staging-devopsy.kinsta.cloud

Tag: Software Supply Chains

  • npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

    npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: The npm registry suffers spam infestation, and Microsoft makes Google sad. (more…)

  • Microservices Adoption and the Software Supply Chain

    Microservices Adoption and the Software Supply Chain

    Software development is undergoing industrialization, with more and more software rapidly assembled out of components and an emphasis on building automation around software validation and release processes.

    Modern cloud-native software is no longer a monolithic application living in a single repo with the majority of its dependencies self-contained. It is integrated from third-party components provided by vendors, cloud providers and open source software (OSS) components, with as much as 90% of code coming from such dependencies. This allows developers to build applications much faster but makes maintenance much harder as it is no longer entirely under their control. If the third-party vendor makes a change to their API, developers are on the hook to update their applications before they break. 

    Each microservice interacts with hundreds of other microservices and each is built and released independently, making it hard to understand how to coordinate API changes across them. For example, if you want to make an API change to a microservice, what impact will this have on the consumers in the organization? Who will need to coordinate the change? When it was all within the same repository and released all at once, it was much easier to make such changes.

    The Changing Nature of Technical Debt

    What we continue to call technical debt is really the activities that are related to tending to and upgrading our software when third-party components are evolving or have common vulnerabilities and exposures (CVEs) and need to be upgraded.

    These are tedious, repetitive tasks that usually fall to the most experienced engineers as they require technical expertise to do correctly. Such activities can paralyze engineering organizations and are a tremendous burden on engineers; that often leads to burnout. Up to 30% of engineering time is spent on technical debt. The perception that somehow developers were responsible for accruing this technical debt and are doing something wrong that prevents them from keeping up is hugely demoralizing and demotivating. 

    However, if we reframe technical debt as software supply chain management and stop blaming engineering for it, we can make maintenance more predictable and consistent. By taking steps like inventorying third-party components and determining how pervasive they are in the application (frameworks take more effort to maintain than a third-party API that you just call from one part of the application), an organization can arrive at a maintenance estimate. 

    These activities are highly repetitive across organizations, as everyone is integrating some subset of the same third-party components to create business value. This high level of repeatability points us to automation. 

    Specialized Developer Tools for Software Maintenance Don’t Exist

    When developers write code, their hands often lag behind their thoughts. They think much faster than they are able to type the code. You often hear developers finishing a tedious task and complaining, “My fingers hurt.” When their IDE can autocomplete syntax, it helps reduce this lag to minutes.

    When a major framework version is released, developers contemplate the codebase. Maybe they need to change this type to that, reorder some method arguments or change dependencies. The types of changes they need to make are enumerable and must be made almost immediately; again, the ability to implement them lags far behind their recognition of the problem. Often, this type of lag can represent months or years of work when dealing with a large codebase. Accumulate enough of this lag and the codebase can grind to a halt.

    That lag kills the joy in development for most senior developers and leads to burnout. Upgrades/migrations amplify that lag by orders of magnitude.

    However, there are few (if any) tools that focus on helping developers automate these remediations/upgrades/migrations. The IDE’s focus is rightly on helping developers write new code and maintain their own code, but code maintenance needs to be coordinated across multiple places across the codebase, within the same repository or across the repository boundaries. Often, developers have a pattern they want to change in their repository, but the IDE only suggests this as an improvement in their current location (one file). Other times, changes need to be coordinated across repository boundaries if developers want to change APIs and their consumers. That’s why this technology needs to exist outside of the IDE.

    OpenRewrite

    OpenRewrite is an open source project that offers semantic analysis and refactoring of code as standalone operations so that everyone can use and contribute over time; composing more and more refactoring operations that make whole framework migrations possible. It is integrated with build tooling and can be plugged into different workflows, from CI integrations to mass refactoring of multiple microservices/repositories. Similar to an IDE or other developer tools, OpenRewrite manipulates an abstract syntax tree (AST) representation of code, but this AST has special characteristics that allow it to transform this AST and generate code back to the standard text representation. 

    The OpenRewrite AST is produced by guiding the compiler through the first two phases of compilation to generate compiler type-attributed AST. This is then mapped to the OpenRewrite AST, preserving formatting and breaking cycles in the AST. The three unique characteristics of OpenRewrite AST are:

    • Type aware, allowing 100% correct semantic code analysis and transformation.
    • Style-preserving, so transformations produced are idiomatic within the projects they are applied to. So, the same transformation applied to multiple projects will potentially look different.
    • Serializable, allowing to output the AST from the build and operate on it outside of the build en masse.

    OpenRewrite ASTs is full of metadata about code. Visualizing the code itself as a highly connected tree of various vertices and edges, we can start to get an idea of the density of this representation. Syntactic elements alone are blue, type attribution is yellow and formatting information is red. If we were to just represent syntactic elements, the tree might look something like this.

    Adding type attribution and formatting, the tree looks quite a bit denser.

    A spanning tree of the data involved in this AST can be laid out in three-dimensional hyperbolic space and then projected onto the unit sphere similar to the way that two-dimensional space can be projected onto a disk—the famous Poincaré disk model. This three-dimensional approach was pioneered by Tamara Munzner at Stanford.

    Every edge that isn’t part of the spanning tree is rendered as a lighter gray, and, as you can see, it is dense! In fact, to make this layout more tractable, we’ve omitted in this view an additional 600,000 vertices representing type information that make the syntax itself seem tiny.

    OpenRewrite calls a single code search or transformation operation a recipe. It provides a number of building block recipes such as find method, change method, find transitive dependency, upgrade or exclude dependency. These recipes, in turn, can be composed into more complex recipes by grouping them together into a composite recipe. When the building blocks are not enough, a recipe can be written as a program in the same language as the code we want to transform, allowing us to encapsulate complex logic with the full expressiveness of the language already familiar to developers. We don’t need to learn a new DSL or programming language.

    These building blocks abstract away many of the details to ensure that edits that we make to source code obey the original style of the project. This is possible for even complex changes like this automated migration from Spring Boot 1.x @ConditionalOnBean to Spring Boot 2.x AnyNestedCondition looks idiomatically consistent in the context of the project that it is inserted into:

    It is important to note that OpenRewrite is not a replacement for an IDE. With software now composed of so many independent components, we need to start to remediate and manage it using new approaches that go beyond single lines of code to looking across entire code repositories.

    Olga Kundzich, co-founder of Moderne, co-authored this article.

  • DevSecOps Trends to Know For 2021

    DevSecOps Trends to Know For 2021

    For DevSecOps leaders, 2021 will be the year of the open source supply chain attack. It’s already starting, in fact. On January 7, security researchers at Sonatype identified three malicious Java components in the Maven Central repository. The components had identical names to reputable components. Then on January 20, the same research team found three more compromised packages in the node package manager (NPM). In all cases, these malicious components had nearly identical names to common Java and NPM packages.

    The bad packages were downloaded thousands of times before they were removed, compromising the open source software supply chain. Such incidents are increasingly common. The practice of intentionally or unintentionally deploying poorly vetted code — we call it shadow code — is inevitable. Shadow code happens when developers include  third-party software code in applications without approval or sufficient safety validations. Developers do this for the same reason that they use libraries and other components – it is quicker than writing the functionality from scratch. Often, these components offload compute-intensive or specialized features. Some examples are shopping cart software, payment systems or responsive design requirements. Sometimes, too, developers mistakenly include shadow code that is fake, but has a similar name (this is referred to as typosquatting), like the examples we described above.

    Shadow code, plus a growing reliance on open source, magnifies the security risk and attracts more bad actors. Modern applications are composed of anywhere from 50% to 90% open source code, depending on the code audit. Rapid code iterations make it impossible for security reviews to police all code changes and dependencies. With many open source libraries containing hundreds of internal dependencies, even the best-automated code checking tools cannot keep up.

    How Shadow Code and Supply Chain Attacks Play Off Each Other

    The compounding risks of shadow code and supply chain attacks are one of the most pressing issues that will face DevOps and DevSecOps teams in the coming year. Security researchers at Sonatype tracked a 430% increase in supply chain attacks against 24,000 open source software components in 2020. The report blamed the growth of these types of attacks on two factors; first, DevOps teams are increasing code velocity to accelerate time to market. Second, security teams are responding to traditional zero-day vulnerabilities more quickly, shortening the window for exploits. Combined, the two factors encourage more supply chain attacks.

    Attacking the Modern Extended Software Supply Chain

    In modern web applications, most front-end and middleware application code comes from third-parties. Some is proprietary, but most code is open source, due to cost considerations. Open source component source code is a massive attack surface that is often poorly monitored, measured and inventoried. DevSecOps teams should create plans to reduce the impact of these attacks in 2021 and beyond.

    Because supply chain attacks are often used either as launch pads for other attacks or to steal sensitive information, the business and financial risk from these compromises is considerable. In its annual “Cost of Breach” survey, IBM found that the average data breach cost $3.8 million and required 280 days to fully contain.

    Supply chain attacks are likely an even more expensive subset of attack, due to several factors. Because they take advantage of trusted relationships, these attacks may go undetected for months or even years. Supply chain attacks also tend to be sophisticated and target an organization’s most valuable assets.

    Evidence is emerging of coordinated attacks attempting to take advantage of supply chain weaknesses at the same time. Multiple attackers have targeted known users of vulnerable and end-of-life (EOL’d) versions of the online shopping platform Magento (aka Magecart attacks). We suspect that advanced hacking groups are either running the same scanning software packages to identify vulnerable targets, or are buying lists of viable targets via the dark web.

    The Open Source Supply Chain Risk

    The benefits of using open source software in building web and mobile applications is clear. Development teams, DevOps and DevSecOps teams can reuse code or libraries maintained by someone else to ship code faster and reduce the amount of code they must maintain.

    Nowhere is this more of a risk than JavaScript and Node.js. The meteoric rise of Node.js as middleware has dramatically increased the number of packages and libraries that can be used by developers. NPM, the package manager for Node.js, is the most frequently downloaded piece of software in use today.

    The rapid usage growth of Node.js and JavaScript has created a chaotic environment conducive to supply chain attacks. Attackers can inject code into trusted open source packages in a variety of ways. (To be clear, Java, Python, PHP, Golang and Ruby have the same potential supply chain vulnerabilities. But JavaScript is, by far, the largest target and it dominates the front-end and near-front-end, where user data is easiest to skim).

    Some big, corporation-backed libraries, such as React (backed by Facebook), are well maintained. But many other commonly used libraries are maintained by under-resourced groups of developer volunteers. These unpaid maintainers have little incentive to do the “grunt work” of following rigorous security best practices, such as setting up automated code analysis or responding quickly to reports of security problems.

    The cost of running all widely used open source JavaScript libraries through a proper code review would likely be in the tens or hundreds of billions of dollars. Because the maintainers of this code rarely profit from their work, they are not inclined to pay for and set up automated review systems. This is true even when maintainers work at large companies; the sheer amount of work behind maintaining a popular open source project can be backbreaking without major organizational support. Even code reviews and running code analysis tools is no panacea. Many popular open source projects change rapidly due to numerous and frequent code commits. So, a clean code review is no guarantee that the component remains safe a week or a month later.

    For projects run by unpaid developers, commit access to core repositories is often poorly regulated, because maintainers are happy to get any help at all on their project. Reviews of pull requests may be superficial and miss well-obfuscated attacks. Worse, sometimes developers just get tired and stop maintaining their library. They may or may not notify users. Their ghost library may sit in GitHub or NPM, still downloading properly, but with no one paying any attention to bug reports or other suspicious activity.

    The problem of abandoned or outdated code is often underestimated. According to a May 2020 study by Synopsys, 91% of commercial applications contain abandoned or outdated open source components. For these reasons, compromising open source libraries that have either been abandoned, are outdated or are lightly policed is a favorite tactic of malicious hackers looking to mount supply chain attacks – and that threat will only grow in 2021.

    Learning to Live with Shadow Code and Supply Chain Risk

    Getting rid of all shadow code is not an option. There’s just too much of it in use; halting the process entirely would negatively impact development cycles. Restricting developer behaviors in a heavy-handed manner usually ends badly (with more shadow code). Likewise, getting rid of the supply chain risk is impossible. Modern applications are too complex, with dependencies on open source components that themselves contain numerous third-party dependencies.

    That doesn’t mean DevSecOps should do nothing. Developers can and should be educated about the risks of shadow code and supply chain attacks. Taking some simple steps to reduce the risk can go a long way. For example, ask developers to create simple checklists for every piece of third-party code they use to ensure the component meets the most basic risk standards. A checklist might include, “recently updated, responds to emails, has more than X number of downloads.” For open source code, use modern software composition analysis tools like Snyk and Sonatype that check open source libraries and can be dropped into the DevOps pipeline as part of the continuous integration process.

    From the other direction, DevSecOps teams should attack the shadow code/supply chain problem by thinking about what application behaviors might indicate a compromise. To this end, teams should create a list of behaviors that are anomalous and create filters or notifications for those. For example, if a sign-up page starts taking consistently longer for users to complete, that might be a red flag for form jacking. As a final backstop, look at using machine learning systems to spot anomalous application behaviors. These systems can monitor all application behaviors, at scale, in a way that humans never can.

    Combined, these defensive practices should reduce shadow code and supply chain risks for DevOps and DevSecOps teams. While thinking about these problems is not fun, a proactive approach to tackling this malicious duo will keep your development process and applications far more secure.

  • The 2016 State of Software Supply Chain Report is Here

    The 2016 State of Software Supply Chain Report is Here

    Our State of the Software Supply Chain Report has just been released.  Over the past year, we’ve amassed a great deal of data with respect to the staggering volume and variety of open source components flowing through software supply chains into development environments. This year, we assessed behaviors across 3,000 organizations and performed deep analysis on more than 25,000 applications.  

    The results we discovered ranged from staggering to surprising to sobering.  For example, we measured organizations consuming an average of 229,000 components annually.  The good news is, these components help companies accelerate their development and innovation.  At the same time, we saw 6.8% of components used in applications marked with at least one known security vulnerability — adding high levels of security debt.  Not all components are created equal.

     

    The 2016 State of the Software Supply Chain Report: software supply chains by the numbers.
    Software supply chain infographic from 2016 report.  See full image here.

     

    The 2016 State of the Software Supply Chain Report blends public and proprietary data with expert research and analysis to reveal the following:

    • Developers are gorging on an ever expanding supply of open source components.  Billions of open source components were downloaded in the last year.
    • Vast networks of open source component suppliers are growing rapidly.  Over 10,000 new versions of open source components are introduced daily.
    • Massive variety and volume of software components vary widely in terms of quality.  1 in 16 parts include a known security defect.
    • Top performing enterprises, federal regulators and industry associations have embraced the principles of software supply chain automation to improve the safety, quality, and security of software.

    We invite you to read the report and leverage the insights to understand how your organization’s practices compare to others. Then, let us know what you think @sonatype.

    If you would like to join a live discussion on this year’s report, please join us on Wednesday, July 13th. Save your seat here.

  • DevOpsSec: 1 in 16 Chances

    DevOpsSec: 1 in 16 Chances

    The quantitative research summarized below, covering over 7,000 repositories across nearly 100 countries, highlights some of the challenges with quality at modern development velocities, especially important for DevOpsSec practices. By leveraging automation in your repository manager, you can improve application quality and reduce unplanned work while lowering exposure to risk. While this practice supports DevOpsSec initiatives, at its core, it is simply a focus on “building quality in”.

    (Written as a collaboration between me and Mike Hansen, Head of Products at Sonatype; Mike wrote the better parts.)

    Innovation and Risk

    Repository managers like Nexus, Artifactory and Archiva have been serving software components for development teams and their tooling for years now.  We can measure over 70,000 active repository manager installations across the globe today.  Use of a repository manager improves build performance and reliability while also providing a secure, private location to host open source, third party, and proprietary components needed across continuous delivery and DevOps tool chains.

    Industry averages show software development organizations electively downloading over 240,000 components a year from public internet repositories.  While these components accelerate development and increase innovation, the vast majority of components go through little to no vetting process to determine if known security vulnerabilities, risky license types, or outdated/unsupported are being used.

    DevOpsSec: tracking known vulnerabilities requires better hygiene.

    Earlier this year, researchers found that 1 in 16 downloads includes a known security vulnerability. That accounts for over 15,000 elective downloads of components with known security vulnerabilities per organization, per year.

    Why the Focus on Repository Managers?

    Repository managers like Archiva, Artifactory, and Nexus have become a critical piece of the DevOps toolchains and are commonly used across the entire application lifecycle. It is effectively a binary parts warehouse for your applications. Given how integral this warehouse has become, software supply chain intelligence and automation can be used to significantly reduce unplanned work and eliminate avoidable risk.

    To understand this opportunity, let’s take a quantitative look at the world’s software development ecosystem.

    Global Visibility

    Not long ago Sonatype introduced a free feature in its Nexus Repository Manager that is providing new data on the health of software supply chains.  This features, called Nexus RHC (Repository Health Check) – offers a simple way to get basic visibility into what OSS is flowing into development via your repository manager.  RHC provides a summary of components in the repository that have known security vulnerabilities and risky license types. Usage of this service is now significant, with over 30 million components across over 15,000 Nexus instances being analyzed every day across the globe.DevOpsSec: identifying known vulnerabilities.

    There are also around a million developers using those repositories. With the law of big numbers working for us, we can gain a statistically valid view into the behaviors of the modern software development ecosystem. The findings are quite telling.

    The Data

    We analyzed over 7,000 repositories containing 500 or more software components using the RHC service during the past 90 days. These repositories are representative of those used by development teams in medium to large organizations. The question was straightforward: To what extent are bad things flowing in, causing downstream rework and creating avoidable risk scenarios?

    In three months time, the average number of new vulnerabilities that flowed into the repositories analyzed was 69. That is an average of 23 vulnerabilities per month, which is a little more than one every weekday. If you only include higher risk vulnerabilities – those with a CVSS score of 5 or greater – the average number was 48, or about 16 per month. That is a little less than one every weekday. For a sense of the kind of vulnerabilities in this higher risk category, the Heartbleed bug had a score of 5, the lowest risk in this grouping.

    DevOpsSec: tracking known vulnerabilities.

    These are not isolated instances or outliers. In the set of repositories analyzed, 98.3% consumed at least 1 vulnerable component and 97.7% consumed at least 1 higher risk (CVSS >= 5) component. In a period of just three months, nearly every repository consumed something the organization running it would rather not have, and that is only with respect to known vulnerabilities. There are plenty of other attributes that are worthy of consideration given availability of the corresponding data.

    Whether or not security is of particular concern, the real point is that a lack of visibility and control combined with an abundance of supply has led to inferior quality parts routinely being used by the world’s development organizations, adding avoidable risk and slowing us down.

    Image title

    Going (Way) Beyond Security

     

    The consumption of vulnerable components is a quantifiable problem that we used to illustrate the current challenges that every organization has with open source component selection. However, security vulnerabilities are only one of the many dimensions.

    DevOpsSec: Surveys show the need for improved OSS governance.Others include architectural aspects such as rules covering component age and popularity, intellectual property coverage with rules related to open source licensing and technical debt controls associated with technology stack selection.  Introducing more formalized vetting processes coupled with software supply chain intelligence and automation will yield significant overall quality and efficiency benefits for an organization, allowing them to operate with a lower risk profile.

    Yes, A Firewall for Repository Managers

    In November, Sonatype became the first company to introduce a “firewall” for use with repository managers.  Called Nexus Firewall, the solution offers perimeter quality control for software development. Similar to a network firewall, it leverages rules you define that automatically shield an organization from unacceptable software components entering and another set for stopping them from exiting your application development. The basic concept looks like this:

    Nexus Firewall: shifting DevOpsSec further left than ever before.

    In order to build quality in from the beginning, this solution represents a “shift (far) left” innovation.  The Nexus Firewall improves speed and reduce risk through the quarantine of components with known vulnerabilities. With a repository firewall in place, organizations can shield your application development from waste and risk by automatically blocking unacceptable software components inbound and preventing release of applications containing such components outbound.  1 in 16 downloads with known security vulnerabilities could be reduced to zero in 16.

    The firewall also goes beyond blocking, providing organizations with the visibility and data needed to make ideal decisions for open source component selection early, significantly reducing waste related to rework and eliminating avoidable risk.

    For DevOpsSec, Shift (Far) Left to Build Quality In

    The consumption of these unacceptable or, at a minimum, inferior components is unnecessary.  For the vast majority of these vulnerable components, there are non-vulnerable alternatives ready to be used in their place. The challenge is that it has been historically difficult for developers to understand these risks and even harder to avoid them.  Few developers have countless hours to commit to researching vulnerabilities in the components they use and fewer more want to rely on colleagues in security, legal, or audit teams to spend hours or weeks to return with “go” or “no go” vetting decisions.

    The volumes and velocity of component consumption we see today validates that any form of manual review process or automated workflows outside of the development teams will be quickly outpaced.  DevOps and Continuous Delivery processes need to rely on automatic intelligence to help developers select the highest quality components and avoid those with known risks.  When known security vulnerabilities and other risks are identified, software supply chain intelligence will be required to guide developers through the identification and selection of safer, higher quality components.

    An example of the kind of information made available is shown below.

    DevOpsSec: Identify the best components that meet policy guidelines from the beginning.

    The information is divided into two areas. On the left side is component data, which includes details related to the component itself. To the right, there’s a graphical display of any security or license issues, as well as popularity data for each version of the component displayed. Selecting different versions updates this information accordingly, including whether or not a particular version passes the established criteria.

    A Must for DevOpsSec: Quality at Velocity

    Simply by enabling a repository firewall, organizations can immediately improve quality and reduce waste and exposure to risk. None of this comes at the expense of development speed.

    The repository firewall automatically quarantines components that do not pass your rules preventing quality issues from entering the software being developed, immediately reducing risk and avoiding wasteful rework at some later point. Quality is improved, efficiencies are gained and risk is reduced, all through automation and all at the speed of modern development.

    One of my favorite reference books is Continuous Delivery by Jez Humble and Dave Farley.  Let me share a few of their words to help remind us of the importance of innovations and practices that allow us to bring pain forward.  Jez and Dave remark:

    “The techniques that we describe in this book, such as continuous integration, comprehensive automated testing, and automated deployment, are designed to catch defects as early in the delivery process as possible (an application of the principle “Bring Pain Forward”).  The next step is to fix them  A fire alarm is useless if everyone ignores it.  Delivery teams must be disciplined about fixing defects are soon as they are found.”

    Well said Dave and Jez.  Well said.

     

  • DevOps Leadership Series: Software Supply Chains

    DevOps Leadership Series: Software Supply Chains

    We kicked off this series on Monday with Gene Kim (@RealGeneKim) sharing his views on the big theme for DevOps in 2015: proving that DevOps is applicable for large organizations.

    Another theme that arose often during our recent “DevOps: Wine-ing, Not Whining” event was the importance of software supply chains.  Every software development organization has a software supply chain, and DevOps leaders are now applying principles from lean manufacturing, Deming, and Toyota supply chain management to improve their operations.

    They are applying software supply chain management principles help:

    • choose higher quality parts (e.g., software binaries, containers, microservices)
    • select higher quality suppliers (e.g., fewer defects, faster MTTR records),  and
    • improve traceability (e.g.., identifying defects or known vulnerabilities to expedite break/fix)

    Today, we will hear from Jez Humble (Chef), Julie Tsai (Retail Giant) and Nick Galbreath (Signal Sciences) as they discuss on importance of software supply chains and DevOps.  Then after watching the videos, you can learn more about the sea change in software supply chain automation in these two blogs from Mike Hansen (Software Supply Chain Automation) and Ericka Chickowski (Security, DevOps and the shift to a software supply chain).

    Here are today’s short videos from our 2015 DevOps Leadership Series:

     

    [youtube http://www.youtube.com/watch?v=4L4nrNpdTM4]

     

    [youtube http://www.youtube.com/watch?v=UEkVYJ1T2NY]
    [youtube http://www.youtube.com/watch?v=nTv8Yx0CpfY]

    NOTE: If you have missed any of the other videos from this series, you can find them here.  (We’re up to 15 so far).