staging-devopsy.kinsta.cloud

Category: Application Performance Management/Monitoring

  • Opvizor Extends Monitoring Reach to the Cloud

    Opvizor Extends Monitoring Reach to the Cloud

    Opvizor, a unit of Codenotary, has extended its monitoring capabilities to cloud computing environments starting with support for Amazon Web Services (AWS) and Microsoft Azure.

    Codenotary CTO Dennis Zimmer said Opvizor aggregates data to provide a more efficient approach to applying analytics to surface performance and risk issues.

    Previously, Opvizor was optimized for instances of VMware vSphere running in an on-premises IT environment. Now the unit of Codenotary, a provider of a service for generating software bills of materials (SBOMs), is extending those monitoring capabilities to the cloud.

    That transition comes as many organizations are looking to strike a balance between monitoring pre-defined metrics and observability platforms that make it possible to surface unknown potential issues that could adversely affect IT environments.
    It’s still early days as far as observability is concerned, so Opvizor is focused on meeting most IT teams where they are today as they further embrace cloud computing, said Zimmer.

    Observability, in general, promises to query not just logs and metrics but also distributed traces that are applied to specific processes. In that way, observability allows organizations to detect patterns that have not been predefined. In effect, an observability platform makes it simpler to debug application environments where the number of dependencies that exist between services are becoming too complex to track manually, much less decipher. Without the aid of an observability platform it can take weeks to discover the exact cause of an application’s performance degradation.

    Of course, those applications will need to be instrumented using agent software capable of generating the data observability platforms require. Fortunately, open source projects such as OpenTelemetry are helping to drive down the total cost of instrumentation.

    However, in the absence of observability, most IT teams will rely on existing monitoring tools. Opvizor is making a case for centralizing monitoring across multiple platforms to reduce costs, noted Zimmer.

    Heading into 2023, each IT team will need to decide how and when to rely on traditional monitoring tools versus observability platforms. Many observability platform providers are justifying the acquisition of a new platform based on cost and the ability to eliminate the need for multiple monitoring tools. However, it remains to be seen whether IT organizations will prefer to employ both monitoring tools to track predefined metrics alongside observability platforms.

    Regardless of approach, the reality is that IT environments are becoming more complex to manage. In addition to running workloads in multiple cloud platforms and on-premises IT environments, organizations are deploying emerging cloud-native applications based on containers alongside existing monolithic applications deployed on legacy virtual machines. As workloads are deployed on multiple platforms, the total cost of IT starts to rise. One way to contain those costs is to standardize on a set of tools that can be applied to multiple platforms.

    The pressure to rationalize toolsets only increases during any period of economic uncertainty. The challenge, of course, is convincing one team to give up their preferred tool in favor of another to reduce costs.

  • DevOps 2022: The Year That Was and Wasn’t

    DevOps 2022: The Year That Was and Wasn’t

    We will be wrapping up our DevOps coverage for the year by the end of this week. It’s hard to believe that another year has gone by. The gravitational lensing of COVID-19 makes it hard to see the progression of the last nine years here at staging-devopsy.kinsta.cloud. Things seem to get a bit hazy looking back. However, 2022 was certainly one for the record books in ways both good and bad. 2022 saw DevOps, our community our countries and our world pushed and pulled in directions we would not have imagined just a few short years ago. To understand what happened in DevOps in 2022, we need to account for several macro factors that played outsized roles this year. They are:

    1. COVID-19: It’s still here; still killing hundreds of people every day. While some in the U.S. and Western Europe have “moved beyond” it, COVID-19 is still draining our resources, sucking the air out of the room and impacting not just the economy but how we interact with each other individually. In the next few months, I am afraid we will see the 1.4 billion people of China experience a backlash as they move away from zero-COVID policies. Not to mention the other viruses and diseases that are circulating.
    2. War: When I was a young boy, I remember a popular poster/sticker that said something like, “War is not healthy for children and other living things.” That’s still true; it’s not healthy for the economy, either (unless you sell ammunition). The human toll of the war in Ukraine is heartbreaking in the present moment, and the toll that this war will take on the once-healthy tech sectors in Ukraine, Russia and the rest of Eastern Europe also will be felt for a very long time.
    3. Inflation: Again, I am transported back to my youth as inflation rears its ugly head and puts its foot on the throat of our economy. Sometimes I wonder if the Fed and other central banks won’t stop raising interest rates until the economy is good and dead, but their fear of inflation is almost genetic. While recent signs point to a potential peak, we still have a long way to go and the possibility of more interest rate hikes ahead of us. These will continue to make it harder to raise money and put a crimp in companies’ budgets. That means layoffs and budget cuts; projects cut or pushed out. We haven’t seen disruption like this in tech for more than 10 years. But for those of my vintage, we have seen cycles like this before. This too shall pass.
    4. Hate: Is it me or has the whole world gone crazy with tribalism, nationalism, racism, antisemitism and a bunch of other “-isms”? We seem to be ripping ourselves apart from the inside out. I don’t have a clue how to combat this on a macro level, but I pray every day that things get better.

    With these four horsemen of the apocalypse riding among us, you may think that I am not very bullish on DevOps now and in the future. But you’d be wrong. Like the little boy in a room full of horse manure, I remain very excited because I know there must be a pony around here somewhere.

    DevOps has continued to evolve, continued to mature and has become an indispensable part of today’s modern software factories. Like their own version of Marvel’s Avengers or DC’s Justice League, DevOps, along with cloud-native, SRE, Agile, CI/CD, DevSecOps and dare I say it, platform engineering, are wielded by IT teams around the globe to continue to push forward as we digitally transform our businesses and the very fabric of our lives. That is what gets me up and out of bed every day and makes me excited to come to work.

    While some marketers are busy running around printing stickers and shirts proclaiming that DevOps is dead, nothing could be further from the truth. Look around—as far as the eye can see, DevOps is everywhere. As we explored in our DevOps Experience virtual event in October (the sessions are still there for you to watch on demand), DevOps is everywhere. It is at the edge; it is in the “connected things”; it is inherent in technologies like WebAssembly (Wasm) and DevOps is distributed.

    In 2023, I think DevOps at the edge, the DevOps of things and Wasm will be huge areas of growth. I realized that some of these folks claiming that DevOps is dead are trying to say platform engineers will replace DevOps engineers. But like Darth Vader trying to strike down Obi-Wan, all they are doing is making DevOps stronger than ever. You see, DevOps was never about DevOps engineers; it’s not about one role or one individual. In fact, some of my best friends helped kick off the DevOps movement and they’d react quite loudly when they heard the term DevOps engineers. DevOps is about building, maintaining and managing our software and applications. It’s a way of thinking about how we can go faster, be smarter and achieve better quality. It’s not about a particular tool you use or the role you play on the team, it’s about the fact that you have a team.

    So, what does 2023 have in store for DevOps? More of the same, I think. DevOps, along with all of the associated frameworks and technologies, will continue to allow us to do more and do it faster and better. As I think about the coming year, I am very excited about the potential for ML and AI and I think they are going to have an outsized role in allowing us to be more successful. I think the novelty of these technologies has worn off and we will get to the productivity phase. I remember interviewing Luke Kanies years ago; he told me that one day software would write software. I asked him when he thought this might happen, and he said maybe 2035 or so. I think we are on track. Much like ChatGPT and Jasper’s impact on copywriting, AI won’t take the place of developers. But it will free them of some of the mundane, repetitive chores they do now.

    I think we’ll also see an emphasis on feedback loops and iteration. I think automation, ML/AI will contribute to this in a big way. We have put so much of our DevOps emphasis on the development, integration and deployment of software. But observability allows us to see patterns and use cases as never before. We need to incorporate those findings into the next revision of our software. This is going to be a big story in 2023. That old infinity loop model for DevOps will rise to the top once again.

    Thank you for all of the support for staging-devopsy.kinsta.cloud this year and over the past few years. I hope this holiday season finds you and yours healthy and happy. Merry, merry. Happy, happy. See you in 2023.

  • 5 Best Practices for Implementing Cloud-Native DevOps

    5 Best Practices for Implementing Cloud-Native DevOps

    As the world transitions to cloud-native offerings as an industry norm, DevOps is gaining traction for its critical role in supporting more efficient IT infrastructure. DevOps is designed to boost collaboration and communication by streamlining the automation process to expedite the creation and deployment of applications. Implementing cloud-native DevOps requires organizations to make a massive cultural shift. Without this change, businesses are unlikely to be as competitive or fully utilize the innovations the cloud has to offer. 

    There’s always a risk in adopting new technology. Following are five best practices to accelerate your company’s implementation process and address potential challenges head-on.

    1. Think cloud, adopt the cloud.

    Optimize technology with a variety of tools tailored for the cloud. Understanding how technology works in the open source environment that cloud-native DevOps relies on is as essential to successfully managing the components as it is to selecting the right tool for you. Don’t get locked into using one tool. Instead, select the right tool for each task while remaining as cloud-agnostic as possible.  

    2. Implement a dynamic security strategy.

    Not considering security is one of the most common pitfalls organizations encounter when using emerging technology. Security must be built into cloud-native development workflows and thought processes and diligently monitored. If funds are available, appoint a chief security officer responsible for scrutinizing DevOps security in the cloud and creating playbooks for each alert triggered. Cybersecurity is becoming more complex due to the unpredictable nature of adversaries and the sheer volume of technologies involved in each solution. One way to start is with a bug bounty program that can bring hackers to your side of the court.

    3. Commit to ongoing training.

    Perform a self-audit and/or penetration testing to reveal weaknesses in company technology, procedures and people. Invest in training to effectively address any gaps in cloud skills. Providing development opportunities will also improve employee engagement and retention, which is a challenge for many employers. Make sure your employees give ongoing feedback to the training’s content, its maintenance and evolution.

    While it may be tempting only to emphasize new-hire training, the rapidly changing technology landscape requires training efforts be geared toward ongoing learning for the entire team. This investment will pay dividends, with DevOps and the cloud netting major long-term savings. To make training more efficient, it can be conducted remotely. Integrate and target training based on employees’ job functions,  but don’t forget about non-IT employees—they must also have a basic understanding of cloud and cloud-native.

    4. Once, twice, automate!

    Automation is critical to minimize software development life cycle friction and eliminate dependence on manual labor. Provisioning networks, infrastructure, application deployment and setup can be automated and versioned with infrastructure-as-code (IaC) tools (such as Terraform). There are also endless automation opportunities for containers, cloud computing, security, testing, monitoring and more. For instance, containers allow developers to compartmentalize applications to work on components without worrying about their potential impact in other areas–making it easier to construct consistent, tangle-free deployments. However, it may be challenging to manage multiple versions, requiring infrastructure and testing investment.

    5. Performance-test your tests.

    Automated testing will save developers time and reduce human errors. It’ll also allow developers to take advantage of doubling up on testing, making sure every change is stable and benchmarked. Test during the development stage and lay the groundwork for an effective CI process, especially if you roll out patches regularly. Automated testing is easier in a cloud-based system and isn’t dependent on sufficient hardware to make it run properly, allowing cloud hosts to scale up or down easily.

    Final Takeaways

    Cloud-native DevOps is revolutionizing the way applications are being built and managed, and it starts with how we perceive our challenges. Embracing DevOps culture enables organizations to leverage cloud and cloud-native solutions to create reliable services that are easily scalable. While it may look resource-intensive, cloud-native DevOps is becoming a business necessity.

  • How Enterprise Software Architects can Design for Sustainability

    How Enterprise Software Architects can Design for Sustainability

    Organizations of all sizes are feeling the pressure to introduce more aggressive sustainability measures to combat climate change. For example, most cloud service providers have announced significant climate pledges to either reduce or offset their carbon emissions. But these initiatives can only go far to move the needle—it’s up to individual companies and departments to participate in curbing their climate impact, and DevOps certainly has a role to play in enacting positive change.

    One way to design for sustainability is to optimize information technology systems to reduce waste. Enterprise IT typically supports a lot of unnecessary technical debt and perpetuates toilsome tasks that contribute to energy inefficiencies. Removing unused instances and features or idle workloads that aren’t contributing value could be an easy way to reduce waste. Yet other optimizations will require more effort.

    I recently chatted with André Christ, co-founder and CEO of LeanIX, to discover what techniques enterprise-level software architects can use to get the most out of their technology modernization efforts while leaving the least carbon footprint. According to Christ, optimizing existing processes, using cloud-native technologies and leveraging value stream management are some tactics architects can take to “do more with less.”

    1. Match Technology to Sustainability Targets

    First, technology leaders must understand the capabilities of their business that could help contribute to meeting sustainability targets. “This requires understanding and systematic mapping,” said Christ. “It’s really about understanding what you can do as a company to contribute to sustainability in a more holistic way.”

    For example, a routing and logistics provider might optimize operations by redesigning its floor plan to maximize efficiency and reduce energy use. Similarly, Christ recommended software professionals take stock of their existing capabilities to determine what should change. This could equate to discovering internal microservices that aid one department and extending them across the entire organization.

    Optimizing data transfers is another way to create a leaner IT system. For example, teams could reduce constant polling by eliminating overfetching and underfetching or embracing a new API standard, like GraphQL, said Christ. Cutting out these superfluous calls reduces unneeded energy consumption.

    2. Ensure Cloud-Architected, Not Lift-and-Shifted

    Cloud migrations can reap significant benefits in terms of scalability and flexibility. But not every cloud migration delivers tangible sustainability benefits. AWS identified six unique strategies for migrating applications into the cloud: Rehosting (known as lift-and-shift), replatforming, repurchasing, refactoring, retiring and retaining. While early gains may be met simply by rehosting applications, replatforming or refactoring them completely with cloud-native components will likely deliver the best optimizations.

    “If you only lift-and-shift applications into the cloud, they will keep running just as inefficiently as a virtual machine,” said Christ. Therefore, architects should design for cloud-native whenever possible rather than simply porting legacy applications to the cloud. On the other hand, it may be prudent to replace specific legacy components with SaaS or retire them altogether. As such, it’s crucial to analyze your existing landscape before making a cloud migration to ensure changes are both cost-effective and advancing sustainability goals.

    3. Practice Value Stream Management

    Practicing value stream management is another potential strategy to cut bloat by ensuring product teams always create more value for the end customer. Planview defined value stream management as such:

    “Value Stream Management is a management technique or practice that focuses on increasing the flow of business value from customer request to customer delivery.”

    This frame of mind could help engineers make more streamlined decisions about what they’re creating. But to achieve value stream management, organizations will require enhanced visibility into how individual services perform, their dependencies under the hood and their approximate emissions.

    The core question you should be asking product teams, said Christ, is how carbon-efficiently are you operating in relation to the number of users you are serving? The answers will highlight which aspects of the business require the most optimization.

    4. Tie Sustainability to the Business Bottom Line

    Many sustainability pledges are founded on carbon offsetting, in which the company purchases “offsets” of its emissions, usually by planting trees or preserving forests. But, the practice of carbon offsetting has received criticism as being ineffective at stemming rising global temperature levels. So what does an honest climate pledge look like that’s more than greenwash? Well, perhaps one more closely associated with the bottom line.

    Taking the idea of value stream management one step further, Christ argued that a real climate pledge would involve reducing carbon footprint by a certain percentage for each new dollar generated. As a business scales and its revenue increases, it should constantly reduce emissions in parallel with this growth. This would equate to assigning a dollar value for every CO2 ton and putting it into the balance sheet, said Christ.

    Doing More With Less

    Nations are striving to reduce carbon emissions through sweeping legislation. Yet, national goals will be insufficient to meet the target decline of 43% percent by 2030 and net zero by 2050, the United Nations reported. Thus, it will take effort from the private sector, as well.

    We’ve identified some steps above to do more with less, yet many current, unsustainable practices are culturally ingrained, making them challenging to change. It will be the work of enterprise software architects to break it down and apply smarter designs to cloud-native applications. This will likely require centralizing views of internal services to enhance visibility and reporting. “You need good visibility into what you’re building and what it’s composed of to reduce the footprint,” said Christ.

  • EU Data Privacy: Try Again | SkyPilot: Choose Cheapest Cloud | WFH: Better Meetings

    EU Data Privacy: Try Again | SkyPilot: Choose Cheapest Cloud | WFH: Better Meetings

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: EU-U.S. Data Privacy Framework reemerges, Berkeley brings cloud selection box, and proof that remote work is good work. (more…)

  • New Relic Extends Observability Reach to More DevOps Platforms

    New Relic Extends Observability Reach to More DevOps Platforms

    New Relic today expanded its ecosystem by adding integrations between its observability platform and a wide variety of DevOps and cybersecurity tools. The New Relic observability platform now integrates with offerings from Atlassian, Amazon Web Services (AWS), CircleCI, Confluent, GitHub, JFrog, Lacework and Snyk and the open source Jenkins continuous integration/continuous delivery (CI/CD) platform.

    Peter Pezaris, senior vice president for strategy and experience at New Relic, said the goal is to make it simple to share analytics based on telemetry data collected from IT operations environments with the tools organizations rely on to build and deploy applications.

    In total, there are now more than 500 integrations with the New Relic observability platform, he noted.

    Only 27% of respondents to a recent New Relic survey reported that they have achieved full-stack observability and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints from end users. More than half (52%) of respondents said they experienced high-business-impact outages once per week or more and 29% said they take more than an hour to resolve those outages.

    On the plus side, nearly three-quarters (72%) of respondents expected to maintain or increase their observability budgets next year, with more than half (52%) expecting observability budgets to increase over the next year.

    Observability, in one form or another, has always been a tenet of any DevOps practice. For the most part, the best that could be achieved until recently was continuous monitoring of a set of pre-defined metrics. With the arrival of observability platforms, it’s becoming much simpler to not only monitor metrics but also launch queries that surface issues long before an IT service can be disrupted.

    An observability platform achieves that goal by making it possible to query logs, metrics and distributed traces in a way that surfaces patterns that have not been pre-defined. In effect, an observability platform makes it simpler to debug application environments where the number of dependencies that exist between services are becoming too complex to track or decipher manually.

    Of course, those applications will need to be instrumented using agent software capable of generating the data observability platforms require. Fortunately, open source projects such as OpenTelemetry are helping to drive down the total cost of instrumentation by making agent software more accessible to a wider range of application developers.

    It’s not clear how quickly IT teams will be embracing observability platforms, but there is now a clear opportunity to narrow the divide between IT operations teams and application developers, noted Pezaris. As each DevOps platform makes it easier to collect data via application programming interfaces (APIs), richer telemetry data can be collected, he added.

    In the meantime, as application environments continue to become more complex in the cloud-native era, most DevOps teams will need additional tools. Existing legacy monitoring tools don’t make it easy to discover, for example, which microservice might be adversely impacting the overall performance of an application because calls are being rerouted. The frustrating thing, as always, is that it can take days or weeks to pinpoint an issue that might take only a few minutes to fix.

  • Shift Left Testing in Microservices Environments

    Shift Left Testing in Microservices Environments

    By now, it’s common knowledge that the later a bug is detected in the software development life cycle (SDLC), the longer it takes and the more expensive it is to fix that bug.

    In 2017, the Ponemon Institute found that it cost around $80 on average to fix a defect detected early in the SDLC and around $7,600 to fix the same defect if it was detected in the production stage.

    Relative Cost of Fixing a Defect Based on Development Stage

    Design

    Implementation

    Testing

    Maintenance

    Cost of Fixing Defects

    1x

    6.5x

    15x

    100x

    While these numbers can be debated, it’s clear that early detection of defects can save organizations significant time and money while enabling those organizations to deliver higher quality software more rapidly. For these reasons, many organizations are implementing shift left testing practices, including in microservices environments.

    Shift left is a set of practices intended to find and prevent defects early in the software delivery process instead of during the testing and maintenance stages. The idea behind shift left testing is to improve software quality by making actionable test signals and feedback available as early in the SDLC as possible.

    Here, we’ll highlight how shift left testing, in combination with test automation, can better enable both innovation and fast delivery of high-quality software. We’ll also address some of the challenges involved in implementing shift left testing in microservices environments, as well as introduce a valuable strategy for successful microservices implementation.

    Shift Left Testing Benefits

    The waterfall model of development involved the explicit passing of responsibilities between highly specialized design, development, QA, and release teams. It also involved lengthy feedback loops.

    Scrum and agile methodologies made the entire SDLC more flexible and nimble by introducing sprints and allowing more frequent iterative development and delivery. Further, DevOps and DevSecOps focus on removing the silos between development, operations, and security through tooling and automation. As a result, the time to market and quality have improved dramatically. Adding shift left testing into the mix better positions teams to handle the broad range of responsibilities from the design stage through the maintenance stage as effectively as possible.

    Shift left testing focuses on prevention rather than detection. Shift left benefits include the following:

    • Increase efficiency by eliminating bugs earlier in the SDLC.
    • Reduce human errors and associated costs.
    • Increase delivery speed and reduce the time between releases.
    • Improve the quality of software.
    • Gain a competitive advantage.

    It may be helpful to think of shift left testing like this: Test early and in parallel with development, instead of the traditional waterfall model of development followed by QA testing. 

    Shift Left Testing Challenges in a Microservices Environment

    By structuring an application as a collection of loosely coupled, independently deployable services, a microservices architecture enables frequent delivery of large, complex applications.

    While shift left testing practices bring many benefits, shift left testing also comes with three challenges that organizations must address when implementing it across a decentralized, microservices-based stack:

    1. Building and managing each microservice separately reduces development time. However, implementing shift left testing at the microservices level means shift left practices must be applied to each individual microservice.
    2. Microservices developers can choose the programming language and service dependencies best suited to their development needs. However, as a result, shift left testing practices need to be implemented and managed across different programming languages and dependencies.
    3. To ensure that microservices function as expected, teams usually run integration tests, end-to-end tests, and performance tests that assess functionality in the context of the larger stack. Shift left testing is especially challenging in the microservices context because of the complexity of testing against the stack.

    Microservices, each with their own SDLC pipeline

    Pipeline Scaling Considerations

    Several classes of automated shift left tests can be integrated into the DevOps pipeline, including unit tests, integration tests, performance tests, and security tests. Different microservices will have different testing needs.

    The key is to apply shift left testing practices to each microservice while maintaining the flexibility and decentralization of those microservices. The ideal workflow system and DevOps pipeline implementation must simplify pipeline creation while enabling development teams to manage the pipeline and requiring minimal input from DevOps and platform teams. In other words, shift left practices must be tailored to the specific needs of the microservice in question to make the testing process scalable and easy for development teams to manage on a continuous basis.

    As just one example, GitHub Actions and similar CI/CD pipeline tools enable development teams to use GitOps-based practices to configure, manage, and administer the entire pipeline for their microservice while leaving infrastructure testing and testing that relies on or affects other parts of the system to DevOps teams.

    Test Environment Considerations

    Test environments are environments where unit tests, integration tests, performance tests and security tests can be run on an isolated collection of microservices and configurations. The nature and quality of these test environments directly correlate with the effectiveness of shift left testing.

    A cloned production or staging environment can be used for testing when the number of microservices is low and overall complexity is relatively low. However, once an organization reaches a threshold number of microservices or a threshold level of complexity, runaway infrastructure costs and operational burdens make such environments untenable.

    In response, some organizations rely upon smoke testing (minimal viable stability testing) and mock-based integration testing early in the SDLC. While such tests can prove useful in the near term, deferring essential testing to a later stage in the SDLC often introduces test signal and validity problems in the staging environment and can lead to delivery delays and serious issues with software quality.

    So how can shift left testing be used successfully in a microservices environment? The solution involves the use of scalable, ephemeral test environments.

    Ephemeral Shift Left Testing Environments

    The availability of high-quality test environments is a common bottleneck for organizations looking to scale shift left testing practices. Using a few high-fidelity staging environments for signal testing can slow the release process. More to the point, it tends to slow down the entire SDLC and increase stack instability over time.

    Ephemeral test environments offer a fundamentally different approach. These sandbox environments use isolated environments that enable users to test independently without impacting other users. In particular, they enable isolation while sharing resources (e.g., external cloud dependencies and hosted databases) using application-level multitenancy instead of the resource duplication approach in traditional environments.

    The idea is that microservice test environments can be used to test changes against a much larger stack of services. Because multiple tests can be conducted at the same time, test coverage and resource efficiency vastly increase.

    High-quality sandbox environments can spin up in seconds and scale to tens of thousands of microservices. In fact, each pull request, branch, or commit can have its own test environment without significantly increasing infrastructure and operational costs. Any change, no matter how small, can be tested in isolation to ensure performance and stability against the entire stack.

    Shift Left Testing Solutions for Microservices

    A microservices-based organization looking to scale shift left testing across the organization must first address several challenges. Considerations include the need to apply testing practices to separate microservices, the need to accommodate different programming languages and service dependencies and the need to test against the stack.

    The right tooling and automation are needed before shift left testing practices are scaled across the organization. In addition, a cultural shift may be required in order to help developers understand the ways in which shift left testing can expedite and improve the core functionality that they are developing.

    To apply valuable shift left testing practices in a microservices environment, organizations should consider using ephemeral sandbox environments. Using the right set of tools in a sandbox environment can enable development teams to test microservices throughout the SDLC. In doing so, organizations can ensure those services follow security best practices and perform exactly as intended, both independently and against the stack–all while reducing costs, speeding time to market and improving software quality and user satisfaction.

  • Massive Number of Transitive Dependencies Traced to Open Source Code

    Massive Number of Transitive Dependencies Traced to Open Source Code

    An analysis of nearly 2,000 software packages published by Endor Labs found 95% of all application vulnerabilities can be traced back to a transitive dependency created when a developer used an open source component.

    The study, conducted by the Station 9 research arm of Endor Labs, a provider of a platform for identifying software dependencies, concluded there’s still a 32% chance those software packages would have known vulnerabilities even after components are upgraded to the latest version.

    Endor Labs CEO Varun Badhwar said vulnerabilities often remain because the last update made to one of those software components may have been made several years ago even though there are existing known vulnerabilities that affect it. Development teams can’t assume the latest version of a software component is secure, he added.

    In general, functional dependencies are created whenever developers download a third-party component, he noted. The most important thing for any software development team to determine when assessing actual risk levels created by those dependences is determining how accessible any given vulnerability is to an attacker regardless of its severity score, noted Badhwar.

    In the wake of a series of high-profile breaches, there has been an increased focus on software supply chains. The challenge is most developers don’t have a lot of cybersecurity expertise. They often create patches for applications based on a list of potential vulnerabilities surfaced by cybersecurity teams. The issue, of course, is that just because there is a vulnerability it does not immediately follow that the affected software component can be externally accessed.

    Endor Labs is making a case for a graph-based platform that identifies dependencies between the software components that make up an application environment. That approach makes it simpler to prioritize remediation efforts based on the impact a vulnerability might actually have and narrows the current divide between cybersecurity teams and application developers. By the time cybersecurity teams alert developers to a vulnerability they often have moved on to other projects. As a result, developers often lack context because it may have been months since they last looked at the impacted code.

    Of course, the inclusion of DevSecOps best practices should improve the overall state of application security as more responsibility for it shifts left toward developers. However, achieving that goal will require more than just giving developers access to software composition analysis (SCA) tools that only surface vulnerabilities in source code as it is being developed, noted Badhwar.

    In the meantime, the efforts to remediate all existing application vulnerabilities will be measured in years, so organizations need to determine where to focus their efforts. After all, every minute spent developing and deploying patches is less time that can be allocated to building and deploying new code. The trouble is that as more new code is written, the number of vulnerabilities that could be exploited increases and makes a bad situation worse.

  • Survey Surfaces Agile Development Challenges

    Survey Surfaces Agile Development Challenges

    A survey of 3,220 software development professionals published this week found that among the 80% of respondents that embraced Agile practices for building software, just under half are tracking on-time delivery (47%) and business objectives achieved (44%).

    The survey, conducted by Regina Corso Consulting on behalf of Digital.ai, a provider of a value stream management (VSM) platform. The survey also found a total of 19% of respondents were not sure what metrics were used to track software development progress.

    At the same time, 40% of respondents noted they are dissatisfied with Agile at their organization; a clash with company culture was listed as the leading cause (41%) of unsuccessful delivery. Half of respondents use a combination of Agile, waterfall and/or iterative processes, but 27% noted this combination of frameworks was a struggle for them.

    The survey also noted that just over a third of product and application owners (36%) that participated in the survey have adopted Agile.

    Derek Holt, general manager of Agile and DevOps at Digital.ai, said that percentage suggests that Agile principles are now being adopted beyond the engineering teams that construct software. As that appreciation for Agile principles continues to grow, the organizations these individuals work for also tend to start tracking value stream metrics, he added. That transition typically begins with a few pragmatic metrics before the organization matures enough to justify the adoption of a VSM platform, Holt noted.

    In time, however, that VSM platform not only helps organizations identify bottlenecks but also identifies potential quality issues and risks to the business, Holt added.

    Overall, the top three top Agile development priorities identified by respondents are to accelerate time-to-market (52%), delivery predictability (44%) and lower risk (31%). However, from a business perspective, the top three priorities are company goals (54%) and end customer satisfaction (43%) followed by time-to-deliver (35%).

    More than half (51%) of respondents, however, also noted that DevOps teams are not incorporating business feedback regularly. Well over half (58%) said the most valuable improvements to DevOps teams would be end-to-end visibility and traceability from business initiative through development, test and deployment to end users, while 55% said the ability to measure cycle time, wait time and bottlenecks would be an important improvement.

    Regarding Agile tools used, Scrum led the pack at 87%, followed by Kanban at 56% and Scrumban at 27%. The Scaled Agile Framework (SAFe) framework has been adopted by more than half of respondents (53%), while Scrum@Scale/Scrum of Scrums was adopted by 28% of respondents followed by Lean Management at 8%, according to survey respondents.

    The top benefit of Agile is increased collaboration (69%) followed by better alignment with the business (54%) and better work environment and increased visibility in the application development life cycle at 39% each.

    The debate over the merits of Agile frameworks, of course, remains fierce and no two organizations may ever have the same level of adoption. However, as the pressure to deliver quality software faster in the age of digital business transformation continues to mount, the more crucial it will become to align Agile and DevOps best practices.

  • Report: Impact of Bad Software on US Economy Reaches $2.41 Trillion

    Report: Impact of Bad Software on US Economy Reaches $2.41 Trillion

    A report published this week by Synopsys in collaboration with the Consortium for Information & Software Quality (CISQ) estimated that software quality issues might adversely impact the U.S. economy to the tune of $2.41 trillion in 2022.

    Cybercrime losses due to a rising number of software vulnerabilities alone are on track to increase 42% in 2022 after growing 64% in 2021, according to the report.

    More troubling still, the number of failures due to weaknesses in open source software components accelerated by an alarming 650% from 2020 to 2021.

    Finally, the report estimated that overall technical debt would increase to $1.52 trillion in 2022.

    Dr. Anita D’Amico, vice president of cross-portfolio solutions and strategy for Synopsys Software Integrity Group, said that despite these issues, there is cause for optimism. The silver lining is that more organizations are aware of software supply chain issues in the wake of a series of high-profile breaches.

    As more responsibility for application security also is shifted further left toward developers, the number of vulnerabilities being introduced into applications should also begin to decline steadily, she added. In addition, more organizations will soon be following the lead of the federal government by requiring software bills of materials (SBOMs) that will make it simpler to remediate vulnerabilities once they are discovered, noted D’Amico.

    Of course, there are still many cultural and technical issues that need to be resolved before DevSecOps best practices are consistently employed across most organizations. However, the expectation is that cybersecurity teams will be able to concentrate more on crafting cybersecurity policies as application development and IT operations teams assume more responsibility for implementing those policies, said D’Amico.

    But while giving individual developers the tools they need to write more secure code is a major step in the right direction, there is also a need to make sure the builds to which they contribute code also remain secure. As such, DevOps teams need to implement guardrails across every stage of a DevOps workflow to make sure vulnerabilities are not created as code is merged. In addition, DevOps teams now need to be aware of any malware added to an open source software component downloaded from a compromised repository. Even once patched, there’s a good chance another developer will have downloaded the same vulnerable component to construct a different application.

    No one knows for certain how many vulnerabilities may have already been introduced into production environments, but it’s unlikely all those issues will ever be completely addressed. Organizations with limited technical resources will need to prioritize the patching of the most severe vulnerabilities in their application environment based on how accessible code may be to cybercriminals.

    Cybercriminals certainly are getting more adept at exploiting vulnerable code. Developers may soon find themselves being required to update software components deployed by someone else many years earlier. In the meantime, more accountability for application security should lead to better quality code that, over time, will reduce the current unsustainable level of technical debt.