staging-devopsy.kinsta.cloud

Tag: API

  • Our API Mess is Coming

    Our API Mess is Coming

    APIs are the satisfaction of a long and deep need to be able to create consistent and reliable integrations between disparate systems, operating systems and datasets. As we started to use REST-based APIs, we also realized they fulfilled a previously little-addressed gap in automation. Frankly, they are a force multiplier in DevOps delivery.

    Early efforts at system-independent APIs were not great, but we learned lessons and are steadily getting better. While REST is the default solution for APIs, it probably isn’t the end of the road, as we continue to learn and improve. That’s what we in tech do – get excited about an idea, drive it to broad use and then iteratively improve it.

    And that’s the root of the problem. We have been, and are, saturating our applications with APIs. Internal, external, external-that-rely-on-another-external. The thing that comes to mind every single time I look at an API dependency map for even a medium-sized application is “spaghetti code.” Those lines are going all over, quite frequently returning to themselves.

    And we’re going to have to maintain this creation. Maybe not us, individually, but us as in IT and DevOps teams. The only constant is change, and far too many of us are not only failing to plan for API changes, but failing to even think about them. Because things are generally going swimmingly and most IT managers and DevOps teams have more urgent things to think about.

    But risk management – particularly broad risk – is an essential part of future planning. If you lose access to external API X, what will you do to keep your application running? If you lose an entire internal system that hosts API Y, how can you recover in a timely manner? While most organizations will identify and repair single points of failure, the solution is a pool, and the impact of losing Z servers in the pool needs to be understood, if nothing else – questions like, “At what point would user experience be impacted?” need to be asked, and a plan formulated to keep that from happening.

    We are already seeing issues created by using external APIs. All of them are predictable – an API going offline, or even disappearing completely; an API interface or attendant data format change; a change to the algorithms underlying the API, causing a change in data results, etc.

    API usage is far easier than it has historically been – and companies are working hard to make it easier to access APIs, both yours and others’. This is a good thing, as it means less time spent figuring out and tweaking API calls and request data. But it means our API dependence will only grow, and the number of lines in our dependency graphs – each representing some amount of risk – will only grow.

    “But we’re doing things in the cloud, so we can auto-scale, and failures are no longer and issue for us!” some shout. While that can help some scenarios, it doesn’t help others – if a vendor goes under and takes an API you use with it, or a preferred vendor (or inside component) becomes a security risk, you’ll still need to take steps. Knowing what and where your weaknesses are before they become issues is important, and worth your time.

    Most of you are killing it, cranking out solutions faster, iterating to make them better, using APIs to extend functionality and reduce delivery times. I’m just saying, stay aware of the technical debt you’re creating, because debts always come due, and risks eventually become issues. So know what’s a risk in your infrastructure, and have a plan to react when it does cause problems.

  • Languages and DevOps: API-First

    Languages and DevOps: API-First

    One of the places development has made significant changes is in the land of APIs. These days, a program almost isn’t considered a proper application unless it offers APIs to do most of what can be done via UI (and even command line). This is a big deal for several reasons; first and foremost, the ability to subdivide the work and then test each subdivision. A proper API does one thing and does it well. Testing can validate that this statement is true for any given API. It also allows for distribution of processing. It is far easier to use a load balancer to scale APIs than it is to scale a monolithic application. And finally, it helps with portability, because with API-first, developer users don’t care what platform it’s running on, just that it can be reached via URL and responds as expected.

    Everything is a trade-off, and there are several places that API-first has issues. Integration testing is objectively harder. Loosely coupled applications, in general, are harder to test, and requiring multiple portions of an application be available to run testing becomes onerous, particularly as the layers of APIs start to build up. Second, security is another issue – particularly if you’re offering a public-facing API, but also for internal usage. Once an attacker has access to the network, internal-only APIs are no different to them than external. Smart IT departments design internal APIs with the same controls that external requires. Third is DevOps in general. API-first grew up in the agile/DevOps world, but because there are cascading dependencies, is actually harder to manage DevOps for than most monolithic applications. Finally, documentation – the bane of API-first’s existence. The number of poorly documented APIs vastly outnumbers the astoundingly well-documented ones. Even well-documented APIs, like Google Places, require some thought and testing to use well. Poorly documented ones – my favorite is, “It’s mysite.com/users – everyone knows what’s in there!” – are next to useless without a learning curve.

    With all of that said, what about languages? That greatly depends on what you’re doing. Users don’t care what language it is implemented in – as noted above, they care that the API can be reached via URI, and that it behaves as expected. So you do have options. Indeed, at this point in the evolution of APIs, the language options are so open that you’ll probably be choosing an API definition language instead of a programming language, and just instruct your definition tool to spit out the language of choice.

    Wait, what?

    For those who haven’t followed API development closely over the last few years, it goes something like this. APIs became standardized, then APIs became a big deal, then smart people started to think in terms of code generation á la ancient 4GLs. What if you could use a specification language to describe what you want to do, then pick your language and tell the tool to generate the API? What if you could do this for the entire API hierarchy? That’s where we are. I’m most familiar with OpenAPI and RAML, but there are a large number of specifications out there, and a larger number of tools to develop in them. Before my API friends freak out, of course this is an over-simplification of what happened. I only have 1000 words, max, and my editor would like less. Relax a bit.

    And then you generate.

    I’ll rely on my own experience here, but there are literally dozens of tools available to get you from idea to API interface (doing the work behind the interface is where you come in). With Swagger CodeGen, the tool I’m most familiar with, you specify the API, then generate server interfaces (they call them stubs), and generate client libraries. Finally, you develop what goes on behind the interface on the server side, and (you or others) use the API on the client side. When I say “generate,” I mean “Tell it your favorite language, and click generate.” It spits out the code you can then use to flesh out your API. It does not spit out the info to get the API into your DevOps toolchain, so you’ll have to do the work for that. Which means the more APIs you can generate in a batch, the less DevOps lifting you’ll have to do (more on that if/when we cover microservices). Since most languages can easily be placed into DevOps via their build tools, I don’t want to make this sound too onerous – security and testing will be a bigger problem, by far.

    API security is slowly getting automated and placed right into generated code, but all the usual bits, like certificate management, still have to be handled. The interesting thing, that hasn’t (yet) gotten a ton of attention, is security of clients. It is far harder to verify that the client library is secure when the purpose of that library is to grant access to your API. I expect we’ll eventually have to get there for some applications, but today, it appears to be largely individual orgs figuring out how best to lock down client libraries, and most of the limiting factors end up server-side.

    I’ve generated Python, Java and Ruby APIs via various toolsets, and considering the problem domain is still relatively small, they work fine. And it saves one piece of the puzzle … but does introduce another piece – coordinating client and server updates via DevOps.

    Keep making the apps that run the world, and use API generators to further standardization, spitting out whatever language you need on the backend, and a portable interface for clients.

    You. Rock.

    Whatever your language of choice, in the end, it will likely be your build tool that links it to DevOps. Your biggest concern for DevOps will probably be portability, at least in the near future. Meanwhile, keep rocking it. You’re doing the work that needs doing, even if it isn’t the part of DevOps everyone is currently buzzing about.

  • HAProxy Updates Data Plane API to Simplify Service Discovery

    HAProxy Updates Data Plane API to Simplify Service Discovery

    HAProxy has updated the open source HAProxy Data Plane application programing interface (API) to include service discovery capabilities alongside native support for the Consul service mesh being developed by HashiCorp.

    Version 2.2 of the HAProxy Data Plane API also adds support for file handling, SSL certificates, map files and configuration files for its Stream Processing Offload Engine (SPOE).

    HAProxy is best known for the open source load balancing software of the same name that is widely deployed by DevOps teams. As an extension of that platform, HAProxy Data Plane makes it possible to dynamically add and configure front ends and back ends and then route application traffic between them.

    Daniel Corbett, director of product and marketing for HAProxy, said the HAProxy load balancing software coupled with the Data Plane allows IT organizations to create a gateway through which they can manage their APIs.

    Corbett said the company is currently rounding out a suite of offerings, dubbed HAProxyOne, that are extensions to its load balancing software and that add a control plane and a service mesh to its portfolio. The company has already added a Kubernetes ingress controller to its portfolio.

    At the same time, however, HAProxy Data Plane will make it simpler for DevOps teams to integrate other offerings, such as Consul, within the control plane being developed by HAProxy, Corbett says.

    Corbett said, collectively, HAProxy will provide a suite of tools through which DevOps teams will be able to more easily manage application delivery across IT environments that span multiple platforms.

    HAProxy is not the only load balancing software provider with similar ambitions. The issue that HAProxy will address is the need for a framework that can apply equally well to both monolithic applications and microservices-based applications, Corbett says.

    In 2021, a major battle is brewing for control over the infrastructure that DevOps teams use to deploy applications at scale. Alongside proxy servers and load balancing software, IT organizations are starting to use API gateways, service meshes and ingress controllers as the number of APIs through which data flows rapidly increases. In effect, these technologies provide the connectivity infrastructure required to route data between the applications that make up a distributed computing environment.

    It’s still early days, though, as advocates for competing open source projects make their cases for the hearts and minds of developers. However, given the profound impact these decisions are likely to have on IT operations, it’s likely many organizations will soon standardize on one suite of offerings to minimize complexity.

    Whatever the result, applications environments are, in fact, becoming more complex. IT organizations will soon find themselves running monolithic applications alongside a raft of microservices-based applications, all of which need to use APIs to integrate with each other. Decisions made today about how best to integrate those APIs will have a profound impact for years to come.

  • Internal Makes On-Premises Edition of No-Code Tool Available

    Internal Makes On-Premises Edition of No-Code Tool Available

    Internal today launched an on-premises edition of its eponymous no-code development platform that promises to simplify application development backlog reduction for enterprise IT organizations.

    Arisa Amano, CEO of Internal, said many professional developers regularly find themselves building simple applications that, if given access to the right tools, could be built instead by end users. Internal was created to provide that capability either through a cloud service or, now, through an on-premises edition.

    The on-premises edition makes it simpler for developers to work with databases, application programming interfaces (APIs) and applications that reside behind a corporate firewall, Amano said.

    As organizations accelerate digital business transformation initiatives, professional developers are pressed to build and deploy more complex applications faster. Many lower-priority application development projects are sidelined simply because developers don’t have time to take them on. Internal, and other no-code and low-code application development tool providers, are making a strong case for enabling end users to build their own applications using graphical tools, or encouraging professional developers to use these tools to build less complex applications more quickly.

    A survey published by Internal finds 62% of respondents report their developers spend more than 25% of their time fulfilling requests to build internal tools. More than three-quarters of respondents (79%) said engineers use any extra time they can find to develop customer-facing features or other applications.

    Most of the projects filling application backlogs tend to involve workflows that span multiple applications running in a data center. The challenge organizations face is low-code and no-code tools that reside in the cloud can’t easily access that data. As such, Amano said there’s a clear need for a no-code tool that can be deployed in an on-premises IT environment.

    Of course, regardless of who builds the application, it still must be deployed. Challenges arise as developers build more applications, which must be concurrently deployed in production environments. The addition of no-code and low-code tools, and the applications created with those tools, will require organizations to rethink and refine their DevOps processes.

    Historically, professional developers have tended to view no-code and low-code tools as the application development equivalent of a black box preventing them from building reliable, robust applications that can scale. Many low-code and no-code applications that organizations build, however, are meant to used internally by a finite number of end users. At the same time, no-code and low-code tools become more robust with each major update.

    Whatever bias may exist, the current application development backlog in most organizations is only going to increase. Professional developers may prefer to write code at a lower level than a no-code or low-code tool allows; however, most business executives are more interested in application performance than in how it was constructed.

  • Low-Code Opens API Integration Potential for Citizen Developers

    Low-Code Opens API Integration Potential for Citizen Developers

    APIs have an accessibility problem. Could a low-code API approach make them more inclusive and accessible for citizen developers?

    Most software today integrates with third-party SaaS. TomTom for geolocation. Stripe for payment. Twilio for SMS. Kairos for facial recognition. AWS for NLP … the list goes on. Whatever function you’re looking for, there’s probably an API for that.

    These specialized utilities help software teams avoid reinventing the wheel for things outside of their core competency. They’re standardized, too — most services are programmable following REST or GraphQL guidelines. Serving integrations over HTTP aids in decoupling the business logic from the client interface. Furthermore, OpenAPI Specification, now part of the Linux Foundation, has risen as a standard description format, making APIs more defined and interoperable.

    However, for all this progress, there’s still friction in the integration process, even for seasoned developers. Each API is a bit unique. It takes time to read through the documentation to authenticate and discover methods, parameters and accepted fields. It requires effort to test requests within Postman, decipher error messages and set up calls within your unique environment. On top of all this, downtime and breaking changes are legitimate concerns for over-the-wire dependencies.

    APIs require effort and maintenance, and just integrating third-party APIs alone won’t build you a fully functional application. Yet, organizations find themselves scrambling to find and hire engineers, and increasingly the onus is on non-tech folks to create new digital workflows. Naturally, companies are currently searching for even more low-code solutions.

    So, what’s the glue holding together API products? How can we better connect APIs to the software applications we’re already building and allow citizen developers to reap the benefits?

    My theory: If low-code platforms decrease the API integration hurdle, more problem-solvers can stitch together business utilities to create more advanced internal applications. Making third-party APIs more inclusive for citizen developers could boost the API economy at large and enable creative solutions to flourish throughout enterprises.

    The Need for Low-Code API Integration

    Perhaps the best way to demonstrate the need for low-code is to scare you with some code. Let’s say you want your application to initiate a direct deposit into a bank account.

    You don’t want to build a financial service from scratch, so you turn to Stripe API. You set up an account, grab a key, and dig in to the ‘Getting Started’ materials. You peruse their comprehensive documentation, and find out how to create a payout.

    The docs demonstrate a sample HTTP curl request for creating a payout. Other parameters to pass include destination, methodor source_type.

    curl https://api.stripe.com/v1/payouts \
 -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
 -d amount=1100 \
 -d currency=usd

    If there are no errors with the request, the response will be a payout object, formatted in structured JSON. It details the balance, data created, currency type and other fields. It looks something like this:

    {
 "id": "po_1HmNKQ2eZvKYlo2CuRNBRWUW",
 "object": "payout",
 "amount": 1100,
 "arrival_date": 1605116514,
 "automatic": true,
 "balance_transaction": "txn_1032HU2eZvKYlo2CEPtcnUvl",
 "created": 1605116514,
 "currency": "usd",
 "description": "STRIPE PAYOUT",
 "destination": "ba_1HmNKQ2eZvKYlo2CzlT6m5gk",
 "failure_balance_transaction": null,
 "failure_code": null,
 "failure_message": null,
 "livemode": false,
 "metadata": {},
 "method": "standard",
 "original_payout": null,
 "reversed_by": null,
 "source_type": "card",
 "statement_descriptor": null,
 "status": "in_transit",
 "type": "bank_account"
}

    Now, you can parse the response into data your application can ingest. You can also send other requests to update, return and cancel this payout object.

    For specialists familiar with HTTP calls and JSON, this is pretty straightforward. Stripe docs typically generate sample requests for many languages, like PHP, Ruby, Python and others to help within your own environment.

    But what if your users are more comfortable with UIs? Integrations are pretty opaque to the non-technical. Even for seasoned developers, APIs can quickly become too dense, and not all have a great developer experience.

    Hurdles In API Integration

    There are many issues preventing non-tech-savvy users and novice programmers from inserting API-driven business logic into their applications:

    • Onboarding time: Setting up an account, acquiring authentication, reading docs and testing calls to familiarize yourself all takes time.
    • Data Types: Data passed in headers must conform to explicit data types, or else requests may fail.
    • Parsing: Responses are often too verbose; developers must parse the JSON object responses to retrieve the data they need.
    • Programming language: Each language handles network communication differently, meaning requests look different in every language.
    • Breaking Change: Third-party services change often, so if you’re not actively maintaining an integration, you may face breaking change.
    • Differing formats: There are various API standards. REST, SOAP, GraphQL all are designed a bit differently. Plus, webhooks, event-driven and publish-subscribe patterns have nuanced communication models.
    • Security and legal: If APIs aren’t appropriately secured, hackers could easily steal user data or cause outages.

    As you can see, a simple programming task like “make a bank deposit” becomes extremely complex the more closely we look at the logistics of actual implementation.

    Furthermore, just because you’ve made an integration work, that doesn’t mean you have a fully-functional application, securely deployed in the cloud, with a friendly UI  that’s universally accessible. A single application may have numerous integrations, meaning the integration process must be duplicated for every external service you connect.

    API Categories

    Overcoming these challenges is a lot to ask of non-professional developers, who may not be comfortable consuming business applications from the command line. This issue is further compounded as integration extends beyond public APIs, too.

    In a recent interview with Divyesh Kharade, co-founder/CEO at DronaHQ, he identified three main API categories:

    • Ready-made SaaS APIs: Microsoft Teams, Slack, Sendgrid, Twilio, Google Sheets, etc. These are the connections to third-party operational business utilities and applications.
    • Connecting to databases: Many companies require secure connections with custom internal databases or legacy internal MySQL databases.
    • Internal custom software: These are private or partner APIs. They may be new API-first microservices or custom-built legacy internal apps. Such middleware may expose many APIs.

    If a low-code application wants to become a holistic software development add-on, it must anticipate all potential environments. This includes integration realities for various scenarios.

    Citizen Developer vs. “Non-Tech”

    So who, exactly, benefits from more accessible integrations? The term citizen developer is often used to refer to any non-professional software developer with minimal programming training. “When most people speak of citizen developers, they mean functional guys that can build tech,” says Kharade.

    As I’ve said previously, fostering a culture of citizen developers is becoming a bigger priority amid dwindling developer talent and a combination of other forces impacting software development requirements.

    Yet, citizen developer tends to means something a bit different to everyone involved in the low-code space. Kharade seems to prefer the term “non-tech.” Non-tech users may include designers who may not have a software background but the relevant UX/UI knowledge to assemble workflows. “Design is a big part of any application,” said Kharade.

    Masking the Complexity of APIs

    If low-code tools mask complexity and make APIs more accessible for these non-engineers, it could inspire many creative solutions. With low-code, “creativity is limitless, and integrations really help with that,” said Kharade. “Once they see that power, they aren’t going back again.”

    For example, if an internal application requires chat functionality, Slack could be a great addition. Low-code layers could hide complexity in the backend, to the point that it “doesn’t feel like a request in background,” noted Kharade.

    The solution works both ways. Utilizing low-code platforms to manage data and expose APIs could enhance visibility and extensibility. Kharade described an industrial manufacturing use case, in which shift workers manually entered machine operations data on paper. That was before COVID; once lockdowns were mandated, the company started digitizing this record-keeping process.

    Access to real-time data, and the ability to filter and organize historical data, is already beneficial.  “Six months down the line, they may consider how to make this data accessible using APIs,” said Kharade. “The journey has just started — it will only get smarter and better.”

    Trigger-Based Workarounds Were A Start

    Of course, tools like IFTTT and Zapier have already been making APIs accessible to laymen for years. These platforms use friendly, UI-based front ends over the foundation of complex integrations, allowing users to set up trigger-based automation. If this particular service does this, this other service should do that.

    These are cool tools for stitching together service-service communications. However, these workflows are a bit isolated from the larger application development picture. For example, an application for hiring employees may require more than connecting a form API’s input to a Google Sheets field. This could require a custom database, a way to upload documents and the ability to deploy an application to the cloud.

    Since low-code already has standard components for drag-and-drop, adding a common method to insert third-party API integrations feels like a natural feature enhancement for these platforms.

    Making Integration More Accessible

    ProgrammableWeb records 23,000 APIs as of this moment. Clearly, many innovative, programmable SaaS offerings are ripe for the picking. Some paid, some open, all with slightly different integrations.

    “It’s difficult for a non-tech user to register and interact with an API,” said Kharade. Figuring out authentication, OAuth, headers, payloads and responses requires expertise. Though a professional developer may be required to “register them once,” says Kharade, once third party integrations are set up, they “should be drag-and-drop and click-to-use.”

    Solving integration problems is just one of the many arenas where low-code platforms could shine. If platforms can successfully establish (and maintain) connectivity, consumers could more easily access the core logic they are searching for. Democratizing robust solutions for traditionally non-tech companies is especially important amid the COVID-19 crisis.

    To be clear, I don’t believe it’s the API provider’s job to cater to end-users. Instead, they should focus on creating excellent developer experiences. Naturally, the type of business application you are trying to construct and your access to software engineering talent will, to an extent, determine a project’s technical stack. The JAMstack style, for example, may be a better fit for your team’s makeup and project objectives.

    That said, there seems to be room in the market for additional low-code layers to open up integration. In the same way APIs avoid reinventing the wheel for specialized utilities, low-code layers could standardize the pains of integration itself.

  • Strengthen API Security With These Tips and Patterns

    Strengthen API Security With These Tips and Patterns

    If you haven’t noticed, digital organizations are building more and more APIs. ProgrammableWeb tracks more than 23,000 public web APIs at the time of writing, and the API market is estimated to be worth $5.1 billion by 2023. Building with APIs increases internal interoperability, reduces development time and can extend product functionality tremendously. In short, the value of APIs is rising. However, opening up with APIs brings security caveats that, if not addressed, could result in serious breaches that negate these benefits.

    I recently spoke with Roey Eliyahu of Salt Security to get his take on the top security concerns for API-enabled companies, and how organizations can safeguard their services. In short, he recommends thoroughly cataloging and understanding your API inventory, designing for legal challenges and mitigating OWASP’s top 10 API security vulnerabilities.

    Challenges for API Security

    APIs have been called “the next frontier in cybercrime.” Rightly so, as API breaches continue to pop up nearly every day. Take the recent API vulnerabilities at Cisco Systems, Shopify, Facebook, U.S. presidential campaign apps, and GCP as evidence. The most infamous was likely the Equifax breach—not enforcing formats on incoming API calls resulted in a massive data breach, which cost the company a $700 million lawsuit.

    So, why have API breaches increased recently? Well, a few years ago, there were simply fewer APIs. And, these older services exposed less sensitive data. Nowadays, companies typically have many more APIs. Banking applications, for example, might expose hundreds of highly valuable endpoints that are very appealing to hackers.

    Companies also build APIs for different audiences: A 2020 study by SmartBear found 72% of companies develop both internal and external-facing APIs. These APIs could easily hold hundreds of parameters, and with every new method added, a company’s potential attack surface grows considerably.

    3 Steps To Secure APIs

    To mitigate the above API security issues, Eliyahu suggested three main steps: Understand your API inventory, audit against privacy policies and data privacy regulations, and track the most pressing API vulnerabilities like a hawk.

    Know Your API Inventory

    “You need a great inventory of APIs to know your attack surface,” said Eliyahu. Many API providers document their APIs with OpenAPI definitions. But, these files don’t always match the production realities—they may be manually generated or leave out certain methods. “In every organization, there is up to a 40% gap between APIs that are actually deployed versus what has been inventoried,” he said. By maintaining a clearer API inventory, organizations can avoid unprotected shadow IT.

    Include Legal Auditing

    A change to an API could result in a change in how it collects user information. Updates involving sensitive data could break privacy policies or regulations such as GDPR. “After you have an inventory, you need to map all parameters manually,” said Eliyahu. He recommended performing a legal audit against your API inventory and establishing a repeatable testing process that plugs into a continuous delivery pipeline.

    As API teams release new updates, they must consider security and legal implications earlier on in the development flow. “Now, with CI/CD, these changes have to happen very frequently,” he noted.

    Watch the OWASP API Security Top 10

    There are many known API security gaps. For example, there may not be proper authorization in place, keys could be exposed, unexpired tokens could be hijacked and so on. Thankfully, OWASP has compiled a resource of top API security vulnerabilities, which Eliyahu and many others in the API economy have rallied behind. “The world should move to adopt the OWASP issues,” he said. These vulnerabilities are as follows:

    1. Broken Object Level Authorization
    2. Broken User Authentication
    3. Excessive Data Exposure
    4. Lack of Resources & Rate Limiting
    5. Broken Function Level Authorization
    6. Mass Assignment
    7. Security Misconfiguration
    8. Injection
    9. Improper Assets Management
    10. Insufficient Logging & Monitoring

    Using AI To Go Beyond ‘Whac-A-Mole’

    Plugging these above holes is a good start. However, these are only defensive measures against known patterns; they aren’t helping you keep guard for nuanced attack types. Ideally, security systems should stop cyberattacks before they even get started.

    Yet, Eliyahu recognizes a big gap in how APIs are monitored. Organizations “don’t have proper tools to know who is looking for vulnerabilities,” he said. “Tools are typically not that advanced. They mainly look for injections and known patterns.”

    Imagine a hacker probing an API. They will likely do so by trial and error, testing undocumented endpoints, sending malformed requests and so on. “They need to probe for hours or days before they find something,” said Eliyahu. If an AI could detect these odd attempts in minutes or seconds, IT could significantly reduce risk.

    Security solutions should thus leverage big data and AI to create baselines of typical behavior, then deter malicious activity the millisecond any nefarious probing begins. Eliyahu said such a security AI must consider dozens of behaviors such as, How is the API being accessed? What parameters are being used? What are the relationships between parameters? What is the flow of API calls? What type of data can be exposed?

    Instead of waiting for a breach to occur, an AI-enabled API security monitoring could instantly pause behavior deemed “out of the norm” by referencing a knowledge base that’s constantly refined using machine learning. Because, if you are only blocking malicious actors without knowing how you need to improve API security posture, “you are just playing Whac-A-Mole,” said Eliyahu.

    DevSecOps Requires API Security Forethought

    Mark O’Neil, analysis at Gartner, predicted that by 2021, 25% of organizations with public APIs will have discontinued or rebooted their public API strategy, citing security concerns as a common breaking point. APIs are now at the core of many application developments. Yet, without the proper DevSec proposition, teams risk wasted efforts and costly breaches.

    To summarize, here are our tips on API security:

    • Ensure everything is well-documented: API inventories cannot be audited if undocumented. Shadow APIs increase the likelihood of an attack.
    • Watch the top 10 OWASP vulnerabilities: These points are foundational to API security. Though not a cure-all, they present a starting point.
    • Build a security and legal review process: Ensure APIs are never deployed without the proper security review. Create validation systems that plug into continuous development processes, so new versions are never breaking privacy policies.
    • Each API is unique: Another thing to keep in mind is each API is unique. Therefore, API security monitoring also requires more context into unique business logic.
    • Continuously monitor your APIs: Continuous search for anyone looking for vulnerabilities.
    • Remove deprecated APIs: Remove old API versions. “You need a mechanism to identify and remove old API versions.” Otherwise, “zombie APIs will bite you!”
    • Learn from the attacks you block: Don’t just play Whac-A-Mole. “You need a mechanism to learn from attacks you block.”
  • Cisco Moves ACI Strategy Forward

    Cisco Moves ACI Strategy Forward

    Cisco is advancing its hybrid cloud computing strategy with release of version 5.0 of Cisco Application Centric Infrastructure (ACI), a software-defined networking (SDN) that can be deployed on cloud platforms or networking equipment deployed in an on-premises IT environment.

    The latest version of Cisco ACI adds support for a transit gateway between virtual private cloud (VPC) instances on the Amazon Web Services (AWS) cloud to provide a more efficient means of routing traffic across east-west and north-south cloud computing environments.

    Cisco is also extending the networking services it provides for the Microsoft Azure cloud to include VNET peering within the Azure Virtual Network, shared service deployments, additional service automation functions.

    Via ACI 5.0, IT teams are now provided a centralized view of cloud resource inventory within AWS and Microsoft Azure along with status indicators and a set of wizards to simplify configurations.

    Cisco also has deepened the level of integration between Cisco ACI and Kubernetes as well, to increase the level of scale that can be achieved in IT environments based on platforms such as VMware vSphere.

    Finally, Cisco has added support for role-based access control (RBAC) in multi-tenancy environments along with two-factor authentication (TFA) capabilities enabled by the Cisco DUO platform. Cisco also added a more flexible policy construct, dubbed Endpoint Security Group (ESG), that can be employed to group endpoints based on L3 attributes and apply contracts between ESGs.

    Srinivas Kotamraju, senior director for product management for Cisco ACI, said the latest update to the SDN platform provides IT teams with the equivalent of many of the same networking services that Cisco historically has provided in an on-premises IT environment.

    Cisco has been making the case for a hybrid cloud computing approach to delivering network services that enables IT teams to manage those services via a central console. Rather than having to manage networking services provided by each cloud service provider, Cisco ACI makes it possible to deploy the same SDN across multiple clouds and on-premises IT environments. Cisco contends that approach ultimately serves to reduce the total cost of delivering networking services by both centralizing management and making it easier to automate the management of networking services across multiple IT environments.

    There are two primary ways Cisco exposes those networking services. DevOps teams can take advantage of application programming interfaces (APIs) to programmatically invoke ACI services. Alternatively, Cisco provides network managers access to a graphical user interface (GUI) through which they can access a command-line interface (CLI) to configure a specific router or switch.

    Kotamraju said with the rise of DevOps the number of organizations giving developer programmatic control over networking services continues to rise. It’s too early to say to what degree NetOps and DevOps teams might ultimately converge; however, Kotamraju said finding ways to meld those two groups is being discussed more frequently within many IT organizations as applications become more distributed.

    Of course, the biggest challenge when it comes to achieving that goal will be more cultural than technical. However, as networking services become more accessible via APIs, it’s now more a question of when and how than if.

  • Netlify Makes Build Plugins Available for Jamstack Platform

    Netlify Makes Build Plugins Available for Jamstack Platform

    Netlify, at a Jamstack Conf Virtual event, announced the general availability of open source Netlify Build Plugins that provide IT teams with a set of continuous integration/continuous delivery (CI/CD) workflows for building websites and web applications using JavaScript, application programming interfaces (APIs) and static HTML files without employing web servers, otherwise known as a Jamstack.

    Designed to be launched from within a Netlify platform to enable IT teams to build a website and launch it from a Git repository to a content delivery network (CDN), the Netlify Build Plugins are intended to make it easier to employ DevOps processes using, for example, a plugin to run an end-to-end Cypress test.

    Netlify CEO Mathias Biilmann said DevOps teams also can make use of a Netlify API to create their own plugins. Ultimately, the goal is to foster the development of a wide range of open source Netlify plugins that would be built and vetted by the Jamstack community, he said.

    Other Netlify Build Plugins provide a means to enable Ghost, an open source headless content management system based on Node.js that can be deployed on any framework; launch a Sentry tool to automatically discover, triage and prioritize bugs; or enable a Gatsby tool to create incremental builds on Netlify.

    A survey of more than 3,000 Jamstack developers published today by Netlify finds one-third have worked on sites serving millions of users over the last year. Nearly two-thirds of survey respondents (62%) work outside the tech industry, including advertising and marketing (21%), media and publishing (14%), education (14%), financial services (13%) and business support and logistics (13%). Well over a third (38%) work for a technology company. About a quarter (26%) have more than 10 years of experience.

    Top Jamstack benefits cited by all types of developers are performance, uptime, speed of development, security, compliance and avoiding vendor lock-in, the survey finds.

    Biilmann said Jamstack as a philosophy enables IT teams to separate the development of front- and backend websites and applications in a way that allows organizations to embrace best DevOps processes. That approach is agnostic as far as any specific JavaScript framework is concerned, but it does define a way to automate the build process for a web application or website.

    In the wake of the economic downturn brought on by the COVID-19 pandemic, more organizations are investing time and resources in modernizing their websites and applications. With more employees and end users depending on those websites to drive digital business processes, the quality of the application experience has become a much bigger concern. At the same time, end users have now come to expect those application experiences to be updated regularly, which in many cases will necessitate the adoption of best DevOps processes.

    In fact, it’s arguable that organizations that don’t embrace best DevOps practices to build and deliver web applications at scale will soon discover they have bitten off more than they can chew.

  • Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

    Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

    As we close out 2019, we at staging-devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the fifth in our weeklong series of the Best of 2019.

    As a result of a broadening threat landscape and the ever-increasing usage of APIs, the OWASP API Security Top 10 Project was launched. From the start, the project was designed to help organizations, developers and application security teams become more aware of the risks associated with APIs. This past September, the OWASP API Security Top 10 release candidate (RC) was finalized and published on OWASP.

    In my previous article, I provided a high-level view of the interaction between API endpoints, modern apps and backend servers, in addition to how they’re different from their traditional browser-based counterparts. I also discussed why this project was so important to the contributors and the industry overall.

    In this article, I aim to clarify the first five risks by highlighting some of the possible attack scenarios to help organizations and end-users understand the dangers associated with deficient API implementations. The following discussion follows the same order as found in the OWASP API Security Top 10.

    API1:2019 – Broken Object Level Authorization

    Attackers can exploit API endpoints vulnerable to broken object level authorization by manipulating the ID of an object sent within the client request. What this means is the client can request information from an API endpoint that they are not supposed to have access to. This attack normally leads to unauthorized information disclosure, modification or destruction of data.

    Example attack scenario:

    Say for instance there is an e-commerce platform that provides financial and hosted services to a group of different online stores (shops). The platform provides an API used to gain access to revenue charts for each of their hosted stores, and each store should only have access to their own revenue charts. However, while inspecting the client request from a single store who wants to gain access to their own revenue charts, an attacker can identify (find) the API endpoint for those revenue charts and identify the URL in use, for example /shops/{shop name}/revenue_data.json. Using the names of other stores being hosted on the e-commerce platform, an attacker can create a simple script to modify the {shop name} ID object in subsequent requests, and gain access to the revenue charts of every other store.

    API2:2019 – Broken Authentication

    Being different than authorization discussed above, authentication on the other hand is a complex and confusing mechanism concerning APIs. Since authentication endpoints are exposed to anyone by design, the endpoints responsible for user-authentication must be treated differently from regular API endpoints and implement extra layers of protection for credential stuffing attempts, in addition to brute force password and token guessing attacks.

    Example attack scenario:

    Suppose an attacker obtained a list of leaked username/password combinations as the result of a data breach from another organizations. If the API endpoint handling authentication does not implement brute force or credential stuffing protections such as CAPTCHA, rate-limiting, account lockout, etc., an attacker can repeatedly attempt to gain access using the list of username/password combinations to determine what combination(s) work.

    API3:2019 – Excessive Data Exposure

    By design, API endpoints often expose sensitive data since they frequently rely on the client app to perform data filtering. Attackers exploit this issue by sniffing the traffic to analyze the responses, looking for sensitive data that should not be exposed. This data is supposed to be filtered on the client app, before being presented to the user.

    Example attack scenario:

    Image an IoT-based camera surveillance system allows administrators to add a newly-hired security guard as a system user, and the administrator wants to ensure the new user should only have access to certain cameras. These cameras are accessible via a mobile app the security guard uses while at work. The newly hired security guard’s mobile app makes an API request to an endpoint in order to receive data about the cameras and relies on the mobile app to filter which cameras the guard has access to. Although the mobile app only shows the cameras the guard can access, the actual API response contains a full list of all the cameras. Using the sniffed traffic, an attacker can manipulate the API request to show all cameras, bypassing the filtering on the mobile app.

    API4:2019 – Lack of Resources and Rate Limiting

    It is common to find API endpoints that do not implement any sort of rate limiting on the number of API requests, or they do not limit the type of requests that can consume considerable network, CPU, memory and storage resources. The amount of resources required to satisfy a request greatly depends on the user input and endpoint business logic. Attackers exploit these issues causing denial-of-service attacks and associated endpoint outages.

    Example attack scenario:

    Let’s say an attacker wants to cause a denial-of-service outage to a certain API that contains a very large list of users. The users’ list can be queried, but the application limits the number of users that can be displayed to 100 users. A normal request to the application would look like this: /api/users?page=1&size=100. In this case, the request would return with the first page and the first 100 users. If the attacker changed the size parameter from 100 to 200,000, it could cause a performance issue on the backend database, since the size parameter in use is so large. As a result, the API becomes unresponsive and is unable to handle further requests from this or any other client.

    API5:2019 – Broken Function Level Authorization

    Although different than API1 above, exploitation of this issue requires the attacker to send API requests to endpoints that they should not have access to, yet are exposed to anonymous users or regular, non-privileged users. These types of flaws are often easy to find and can allow attackers to access unauthorized functionality. For example, administrative functions are prime targets for this type of attack.

    Example attack scenario:

    To illustrate this further, imagine during the registration process to a certain application that only allows invited users to join, the mobile app triggers an API request to GET /api/invites/{invite_guid}. GET is a standard HTTP method used to request information from a particular resource. In this case, the response to the GET contains details about the invite, including the user’s role and email address.

    Now, say an attacker duplicated the request and manipulated the HTTP method by changing GET to POST. POST is an HTTP method used to send information to create or update a resource. The URL would look like this: POST /api/invites/new/{“email”:”hugo@malicious.com”,”role”:”admin”}. In this case, the attacker easily exploits this issue and sends himself an email invite to create an admin account.

    In the context of the five risks above, one could easily image many similar attack scenarios. Those provided were just examples of the nearly unlimited possibilities when attacking vulnerable API implementations. As you can see, the risks above are primarily caused by errors or oversights. I believe these risks could easily be managed or nearly eliminated when organizations improve their secure coding practices, especially when it comes to the way they’re utilizing APIs.

    — Erez Yalon

  • Oracle v. Google and Its Impact on APIs and Code

    Oracle v. Google and Its Impact on APIs and Code

    What, if any, impact will the Supreme Court’s rulings on Oracle v. Google have on the industry at large?

    TL/DR: It is unlikely to impact the vast majority of us, vendor and enterprise developer alike.

    Disclaimer: I am a technologist, not a lawyer. I have done some research, and talked to lawyers, but all that I say here is opinion—even my direct quotes of lawyers are their opinion. The Supreme Court can and has caught people off guard and taken cases in different directions than expected. Though I believe this path is the one it will follow, nothing is assured until the Court rules.

    Throughout this blog, references to The Case are actually using Harvard’s case document.

     

    Background

    On Nov. 15, the U.S. Supreme Court agreed to hear a dispute between Oracle and Google. The Oracle v. Google case has been going on for nearly a decade. And its roots go further back than that.

    The origins of the case go back to 2005, when Google and Sun Microsystems discussed the licensing of Java for Android. This will likely be important because it shows that Google knew it was someone else’s intellectual property (IP). More on this later.

    The crux of the case falls into three parts:

    1. Eight files that were decompiled and used. In one court case, these were found to never have made it into distribution and only used for testing (which limits liability).
    2. Two source files from Java used directly. These were found to be infringing, but damages were set to zero with consent of both legal teams.
    3. The part at issue: the copying of the entirety of Java’s interface from 37 modules. Most tech writers imply this as 37 APIs. It is not—it is hundreds of classes and thousands of methods. The 37 is the number of Java packages.

    The case has been through four court hearings/rulings so far. To sum up the relevant parts of these cases:

    1. The first case (split between a jury and district court, running at the same time) included patent violations. Those were defeated and Oracle has not appealed that defeat. On copyright infringement, the court found that Google did violate copyright on the 37 packages and the directly copied source code, but not on the eight decompiled files. The jury could not settle the issue of “fair use” of the 37 packages.
    2. The district court decided that the eight decompiled files and the copied source code for one function were violations, but remanded the rest back to the jury trial to settle the issue of fair use.
    3. In the end, the district court determined that APIs are not protected by copyright. Their decision involved two parts:
      1. That there is only one way to write a given API.
      2. The interface code and hierarchy can be replicated by third parties without violating copyright because it is an interface and not content.
    4. On appeal, the finding was reversed, and APIs were deemed copyrightable, sending the case back again.
    5. At this point, Google asked the U.S. Supreme Court to intervene. The court chose not to.
    6. Back at jury trial, the court found that while APIs may be copyrightable, Google’s use was fair use in the legal sense. This is the one topic the jury could not originally agree upon.
    7. Oracle appealed this decision, and the District Court ruled that it indeed was not fair use to copy the structure and exact interfaces.
      1. Where the jury and the district court appear to disagree is in the idea of APIs. Calling a publicly documented API is obviously fair use, and the jury latched onto that idea. The District Court appears to acknowledge the difference between calling an API and copying it.
    8. And after some tumultuous hearings and years in the court, the Supreme Court has agreed to weigh in on the issue.

    The real problem here is that copyright law is not a great fit for software. It never has been. If I rewrite your Java application in Python, using the same exact structure and naming, is that protected? All of the “words” have changed. It is akin to rewriting your Hindi book in English without a direct translation  … except that in software there is the added issue of it behaving exactly the same.

    But we can’t fix the misapplication of copyright law to software—only Congress can. So what to expect?

    A fair number of people, from the open source community and from businesses that see opportunity in a ruling that favors their world  view, are advocating that affording copyright protection to APIs is a horrible idea because it would hurt independent developers. This is not quite true. Independent developers are not generally copying Java. While this case does have bearing on coding practices and what developers can do, it does not have a huge impact on innovation, just on copying others’ work and how that is achieved—much like the “reverse engineering” court cases of the ’80s.

    There are good arguments on both sides of this case, but it seems to me that fair use will lose. They are public APIs in the sense that anyone can call them; indeed, some of these packages are so core to the system that you could argue every app must call them. But calling a function is not the same as re-creating it. In that sense, they are not in the public domain. Anyone can read a book, but that does not make it public domain.

    But no matter how this turns out, it is all good for the industry. This uncertainty has hung over the IT sector for nearly a decade. Just like when a new boss comes along and there is uncertainty in the department until everyone adjusts to the new demands/style of the boss, a solid ruling will allow settling to start. The industry can adapt once it knows what the rules are and we can move along.

    The real long-term answer is to have an equivalent to copyright and patent that is designed around the unique parts of computer code. Clearly, software is not the same beast as a book or magazine. The considerations are half-patent and half-copyright, and trying to bend those two to meet the needs of software has created some serious issues. Oracle quoted a 1995 case to showcase this problem—the company is quoting it to win the appeal, but it should be quoted as the reason the protections need to be designed for this medium that is not a publication and yet consists of words whose sequence matters: “Applying copyright law to computer programs is like assembling a jigsaw puzzle whose pieces do not quite fit.” (attributed by Oracle to the concurring opinion of J. Boudin in Lotus Dev. Corp. v. Borland Int’l, Inc., 1995).

    One interesting bit that the courts thus far have ignored, and I would love to see the Supreme Court take up, is this: If Google entered into licensing negotiations, the company knew it was Sun/Oracle’s IP. To then go and copy parts of it without licensing appears wrong on its face. It would be interesting to understand the process that ended in the copying and hear whether that bears upon the legal standing of the case. I can think of both malicious and innocent routes to get there; it would be interesting to hear testimony about motivations and thought processes.

    But even without delving into the licensing negotiations, this case stands to redefine copyright with regards to software. J. Michael Keyes, a partner at the international law firm Dorsey & Whitney, had this (and more) to say:

    “One of the issues that the court may decide is the scope of copyright protection for software. Google had early success below in convincing the trial judge that the snippets of copied code were not copyrightable. Google may urge to have the court take a look at this issue anew. Another huge implication is the scope of fair use. The jury found that Google’s use was “fair” and that it was not liable for infringement. The appellate court overturned that jury conclusion and found that Google’s use was not fair as a matter of law. The Supreme Court will likely wade into this thicket and provide guidance on the scope of fair use in the context of software code and what role an appellate court plays in reviewing jury decisions in that regard.”

    What we (all of us) do not want to do is confuse “good for the industry” with “legal standard.” It is not the court’s job to do what we think is best for the industry; it is the court’s job to determine applicable law and what that implies needs to happen. Presumably, the Supreme Court will do just that. And it may not please any of us. But if we think the current iteration of the law is flawed, that is something to take up with elected representatives, not ask a court to “fix” on the fly.

    For the vast majority of us, though, it impacts our development not at all. Using APIs or forking open source projects are all still perfectly acceptable, and no ruling by the Supreme Court is going to change that—because copying someone’s APIs is the issue here, not using them. And open source licenses allow for copying and improving projects. So carry on. Get the job done. And read about the Supreme Court’s final decision with a smile, because either way, the industry will adapt to the new normal and continue forward.

    — Don Macvittie