staging-devopsy.kinsta.cloud

Category: DevOps Toolbox

  • GitLab Gets an Overhaul

    GitLab Gets an Overhaul

    GitLab launched its next major iteration, GitLab 15, starting with its first release version, 15.0, which the company said pulls together new DevOps and data science capabilities into the platform. With GitLab 15, GitLab says it provides (or soon will provide) continuous security and compliance, enterprise Agile planning, visibility and observability, workflow automation and increased support for data science workloads.

    In an interview with staging-devopsy.kinsta.cloud, Hillary Benson, director of product management, detailed how GitLab 15 provides several security benefits, including the ability to automatically create an exportable software bill of materials (SBOM) and signed attestation for build artifacts, centralized security policy management including a two-step exception approval process, as well as providing compliance managers the ability to see project history changes and better enforce policy. “Supply chain security, especially, has become a major area of focus for us, both because it’s become a high priority in the industry and because GitLab is uniquely positioned to support customers well in this area,” Benson said.

    GitLab 15 also provides improved static application security testing (SAST) with stronger and more flexible rules and finding management, Benson said. Finally, users can pool together GitLab project data to create custom automations.

    According to Benson, GitLab 15 will also introduce more capabilities that interlink data across every stage of the DevOps life cycle from initial planning, implementation, deployment and monitoring. “Some of the things that will happen in this area are additional options for the types of work items that can be managed,” said Benson. That includes supporting a more diverse range of workflows and frameworks—including enterprise Agile frameworks. The platform will continue to support issues, epics and requirements and soon enable tasks, features and objectives.

    GitLab 15 provides new capabilities the company hopes will increase visibility and enable an end-to-end view of value delivery and application health, the company said. The unified collaborative context will help remove DevOps silos within organizations and decrease incident rates as well as provide comprehensive insights into issues that hinder performance in addition to real-time triaging of incidents. These new capabilities can help shorten lead times from code to production, reduce error frequency and severity, help development teams deploy more frequently and reduce mean-time-to-recover (MTTR) after an incident.

    Upcoming features include improved product analytics and experimentation, such as increasing the practicality of surfaced data by enabling testing and validating new ideas and measuring the adoption of DevOps best practices across teams and projects, Benson said. Expanded support for all DevOps research and assessment (DORA) metrics including improvements to value stream analytics for projects and groups is also addressed. “This is about making it easier to understand what value your product is bringing,” said Benson. “We’re introducing product analytics and experimentation to increase the practicality of the data that you’re using and the data you’re collecting to actually move your product forward,” says Benson.

    Finally, GitLab 15 will increase support for data science teams and their workloads. “This is about streamlining the development and deployment of machine learning models to train and test those and then get them into production. We’ll also be working on providing monitoring for those workloads so that it can be understood if those models are being used, how they’re being used and where there may be performance issues,” said Benson.

  • DevSecOps Deluge: Choosing the Right Tools

    DevSecOps Deluge: Choosing the Right Tools

    In the last few years, DevSecOps has become the security process of choice for many forward-thinking enterprises. 

    These organizations have come to understand that fixing bugs in the latter stages of product and application development offers no favors to anyone but cybercriminals. So, they have overhauled traditional processes and united development, operations and security teams to form DevSecOps where the teams now work in tandem, cooperating to embed security into the entire process of application and software development.

    The benefits of this are clear; finding and fixing bugs early closes doors on cybercriminals while bringing peace of mind, cost savings and better products to customers. However, one of the challenges many organizations face when transitioning to DevSecOps is making the process seamless, ensuring it doesn’t slow down the software development life cycle, make security checks more cumbersome or cause frustration among teams. 

    Security teams need to ensure their DevSecOps program is delivering value to the culture of the organization while speeding up time-to-fix, capturing the right metrics, and categorizing and prioritizing issues based on their risk so that nothing goes live that could cause major damage.   

    Additionally, they also need to select the correct DevSecOps toolkit to carry out application security testing while ensuring the tools easily integrate into software development and can be used across multiple projects. When identifying tools for testing, DevSecOps teams are often overwhelmed by the huge range and volume available, which makes choosing which ones to use and learning how to use them a minefield, even for those with the right training and know-how. 

    A DevSecOps automation program needs lots of technical tools and cultural aspects to come together to make it work. Security teams need static analysis tools to check the code, third-party library analysis to check the dependencies, separate analysis to check the infrastructure-as-code (IaC) configuration, scanners to check containers for issues, tools to test the running system, cloud security checks and infrastructure testing for patches and ports. They also need to match these tools up to the right technology each team is using and keep up with the constant changes.

    Given all of these complexities, how can DevSecOps teams overcome these challenges and build out an effective DevSecOps program with the correct toolset? 

    Here are some top tips to help them along the way.

    Keep Security Processes Flexible 

    Your teams are going to be using different technology stacks, different languages, frameworks, and so on. If you tie your process too tightly to a few tools, it’ll be harder to slot in new checks when things change. Remember that the aim is a consistent, repeatable security process with the right visibility—the technical tools feed into that, but they are not the whole process. What security checks will you need in 12 months that you don’t have now? What about 36 months? These are questions that must be asked continuously. 

    Automation is Your Friend

    If development pipelines are running smoothly and automatically, then any manual security steps are not going to fit in with the process. Automating security tools together—from orchestrating their running to aggregating their responses and managing issues—will save a lot of time and give you the results you need in line with your releases.

    Keep an Eye On Your ROI

    It’s very common for big-ticket commercial tools to be underused; you have the licenses for repeated testing and you have frequent code drops, but the processes or the integrations mean testing isn’t applied as often as needed or the results take time to collect and process. Explore ways to bring commercial tools easily into the existing processes.

    There’s No Such Thing as a Free Lunch

    The open source community provides great security tools, but remember: There is a cost associated with the time it takes your teams to use them and to manage the outputs. From developers learning how to run them to the time it takes to actually run them or extract the results or manage false positives—these aren’t free. A ‘free’ tool that costs two hours of work every release might not be worth it.

    DevSecOps is All About Collaboration

    Weaving security checks into development processes, CI/CD pipelines, ticketing systems and sprint meetings help security to be part of the development process, but not every developer is—or wants to be—a security expert. Think about how the aims of development and security can meet in the middle. How can developers include security without much effort or thought? How can security help supply the tooling and help triage issues in developer pipelines? How can the process conform to auditors’ tools and with correct profiles, not just turned down to zero to avoid reporting issues? How can security add value at scale to help developers fix issues faster regardless of the tooling being used?

  • New Relic Expands Scope of Observability Reach

    New Relic Expands Scope of Observability Reach

    At its Futurestack conference, New Relic announced it expanded the integrations and tools it provides for its observability platform and added its first cybersecurity tool.

    The company now provides more than 470 integrations with cloud services, open source tools and other enterprise technologies, with support for offerings from Akamai, Atlassian, CircleCI, Cloudflare, Netlify, PagerDuty and Postman now being added to its Instant Observability platform. The company has also added the ability to now collect logs alongside metrics and traces within its Application Performance Monitoring (APM) offering.

    Finally, New Relic has deepened its cloud ties with Microsoft, added a guided user onboarding interface, support for a wider range of instrumentation methods, tighter integration with Kubernetes and, for the first time, is branching out into the realm of cybersecurity via a New Relic Vulnerability Management service that is being added to its software-as-a-service (SaaS) portfolio.

    Ishan Mukherjee, group vice president for product go-to-market at New Relic, said New Relic Vulnerability Management is the start of a larger effort to enable IT teams to leverage the observability data they already collect via the New Relic platform to drive the adoption of DevSecOps best practices. Legacy approaches to vulnerability management create yet another silo of data that only further aggravates cybersecurity blind spots, he noted. New Relic Vulnerability Management aggregates data collected via its agent software and integrations with a wide range of platforms to provide a unified view of the overall security posture of an IT environment, Mukherjee added.

    It’s still early days as far as adoption of observability platforms is concerned, but it’s apparent that platforms that unify the collection of metrics, logs and traces across both applications and the IT infrastructure they run on are transforming how IT is managed. Most DevOps teams today are able to continuously monitor IT environments using tools that track a set of pre-defined metrics; observability platforms make it possible to aggregate data so that DevOps teams can launch queries that help them uncover the root cause of an IT issue. That capability is increasingly becoming critical as IT environments—thanks, in part, to the rise of microservices and cloud computing environments—become too complex for IT teams to manage using legacy tools.

    The challenge is that most IT teams are not quite sure what questions they should ask to get the most value from investment in an observability platform. Observability platforms that accurately surface issues without requiring much intervention from IT teams are likely to gain the most traction, as the competition among providers of these platforms intensifies.

    In the meantime, organizations might want to start reevaluating roles within their IT teams as it becomes simpler to collaborate around a single source of data. The days when IT teams had to triage issues using disparate tools that surfaced conflicting data may finally be coming to an end. As such, it should become possible for a team of IT professionals to manage a wider range of processes across an extended enterprise IT environment.

    Naturally, it may be a while before observability platforms drive wholesale reorganizations of IT teams, but it’s now more a question of when rather than if that day will come.

  • Is Your Future in SaaS? Yes, Except …

    Is Your Future in SaaS? Yes, Except …

    The growth of SaaS—including IaaS—was phenomenal when it came and it has become somewhat pervasive. Today, there are SaaS tools in use at just about every business, large or small. From sales to marketing to customer management, there is a SaaS tool that does it affordably and well. For now. The same is true of cloud vendors. Cloud prices are kept low by two mechanisms—volume and profit. Volume; much like the theory that mass transit is cheaper, by putting many customers on a single server reduces cloud vendors’ operating expenses and thus reduces what the cloud vendor must charge to make a profit.

    But as costs rise and growth slows in the current environment, it is important that your IT department, as a customer, pays attention to the cost/benefit equation of all SaaS offerings. Some will remain beneficial because of the work they offload, even as prices increase for your organization. Others, you’ll see, will quickly outgrow the value they bring as costs rise and there is a third class that just isn’t worth it—maybe it never was and you haven’t bothered to pay attention.

    Look for those items. So that this is less vague, here’s an example: A company I am involved with has been paying for a SaaS service that simply emulates an on-site functionality. The organization could (and previously did) drop a couple of boxes in and achieve largely the same result. Recently, we looked more closely at items like this and found that, in this particular case, (though not in most SaaS cases the org is using) the cost of dropping two boxes in and letting them do the job was cheaper (assuming those boxes were upgraded/replaced every other year) than continuing with the SaaS solution at the current monthly price. So much cheaper, in fact, that a contractor could be hired to deal with any issues that came up with the physical boxes and the only increased work for IT would be maintenance updates to keep the system secure. If, instead of appliances, the solution consisted of two Windows or Linux boxes, even that maintenance cost would be minimized.

    Service will be the same or better—the devices are right there if a change is needed—and there will be one less link in the chain of tools/vendors for attackers (the physical solutions would be on a private subnet, SaaS cannot be). Assuming the solution is a quality product, there really is no downside here. And yet, for several years this company has used the SaaS solution. These are the low-hanging fruit that people will point at and call “retrenchment” or whatever. Ignore them. Do what’s best for your org—be it SaaS or local—and make that decision on a per-tool/per-project basis.

    And keep kicking it. You’re building apps and infrastructure that keep the company running. Don’t hesitate to take pride in what your team(s) have built and don’t hesitate to cut costs on things that don’t matter so you have more funds for things that do. And once again, thank you—from all the employees in your organization that don’t realize they should say thanks.

  • Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?

    Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Apple lets devs increase subscriptions “without consent,” Twitter has more than 19% spam accounts, and we dig into the claim that Microsoft is about to double salaries. (more…)

  • Increasing Use of SLOs to Enable Observability

    Increasing Use of SLOs to Enable Observability

    Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

    SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

    Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

    State of Observability and SRE

    In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

    These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

    With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

    One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

    Hybrid Environments Complicate Visibility

    Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

    Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

    A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

    Benefits of Tracking SLOs

    The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

    As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:

    • Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
    • Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
    • Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
    • Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
    • Reduce expenses: 90% indicated SLOs saved their company money.

    Final Thoughts

    More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

    Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

    The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.

  • Why Data Lineage Matters and Why it’s so Challenging

    Why Data Lineage Matters and Why it’s so Challenging

    Change introduces risk. It’s one of those foundational principles of software development that most of us learned very early in our careers. Nevertheless, it always seems to keep cropping up in spite of those repeated life lessons. Our inability to foresee the impact of changes, even small ones, often leads to negative outcomes.

    As the complexity of interconnected IT systems has increased, this problem has grown worse. It’s harder and harder to predict how even a small modification might impact systems upstream or downstream of the change. Data lineage accomplishes that, but it’s extraordinarily difficult to do data lineage well.

    Why Data Lineage Matters

    In the old world, data resided in silos with relatively few connection points between them. Today, data flows multi-directionally among multiple systems. Many of those systems share data with external organizations, which means that information subsequently flows to internal systems there, as well. Consequently, it’s easier than ever to overlook the impact of a single change on multiple other systems.

    Imagine, for example, that your company provides a mobile app that enables customers to quickly and easily request a new service call. In the past, your customers could specify which product variant they owned. Your design team has determined that since you already have that information in your CRM database, you no longer need to collect it from users of the mobile app.

    You’ve been tasked with simplifying the app, so you remove that input field and replace it with a query that pulls the customers’ product information from your CRM. Theoretically, the resulting work tickets should include all of the information that your field service personnel need to get the job done.

    Unfortunately, there’s a problem; some of the customers’ product records in your CRM system contain null values. Your mobile app is querying that information, but it’s delivering invalid results to your ticketing system. Your field service personnel no longer know how to prepare for their next service call.

    That might sound like a relatively simple problem of data quality—one that might have been foreseen if someone had been more thorough about vetting the CRM data. Even that might not have solved the problem, though, if null values in the CRM database were not introduced until later. If customer data from a newly acquired company is merged into your CRM system, for example, you might not have product information tied to those new records.

    Sometimes these issues are more difficult to predict, and they may not be apparent right away. Imagine what happens when your marketing team decides to redefine its customer segmentation criteria. A new data field is added to designate the customer category; then it’s populated with values for all customers and the old field is deprecated.

    Unbeknownst to the marketing department, though, the company’s top executives are still looking at the old data. They’re using dashboards that rely on data from the deprecated field. New customers are being added every day, but no one is entering any values in the deprecated field. As a result, the executive team is looking at distorted data. Suddenly, the C-suite is tracking very different KPIs than the rest of the organization. What’s worse, no one even notices that there’s an issue until weeks or months after the fact.

    Next, imagine a similar scenario, but instead of populating an executive dashboard, the source system is feeding data to the AI algorithms that drive customer recommendations on your e-commerce site. When AI is trained on flawed data—or if the data changes materially in some way—it can dramatically impact the effectiveness of that technology. In this case, it distorts your customer recommendation engine.

    Tackling the Challenge With Data Lineage

    Data lineage tools provide a systematic means of understanding the impact of changes by presenting a complete roadmap of potential implications, both upstream and downstream.

    This amounts to risk reduction and cost control. If we can predict how changes might impact other systems, we can avoid creating problems like the ones we described earlier. That leads to less wasted effort, faster time-to-market and lower costs. We’re all familiar with the principle that says it’s faster and cheaper to solve a problem when you catch it early in the development process; data lineage provides a critical missing link, potentially ferreting out problems before they even reach QA.

    There are benefits to data governance and regulatory compliance as well. When the European Union passed the General Data Protection Regulation (GDPR) back in 2016, it set off a chain of new requirements for systems that house personally identifiable information (PII). If a customer asks you to delete their data, you’re legally obligated to comply with that request. But what happens upstream and downstream of that change? Will it break anything? Or will some of the customer’s PII remain elsewhere in your systems, rendering you non-compliant with the law?

    Data lineage addresses these questions by automating the process of detecting upstream and downstream effects. It provides clear auditability that can help determine where changes originated and how the data arrived at its current state.

    Column-level data lineage is especially difficult because it requires that SQL queries be parsed to discern exactly what changed and how. This means tapping into database logs, but because there are so many database vendors—each with their own unique SQL dialect—parsing that information for various databases can be particularly challenging. As each of those SQL dialects evolves over time, data lineage tools must constantly be refined to accommodate those changes.

    An example column-level lineage view that shows all the tables and columns that are involved in computing the sales table used for reporting in an imaginary e-commerce business selling craft beers.

    As interconnected systems extend their reach even further and the flow of information accelerates, column-level data lineage will become more important. Today, it’s an extraordinarily useful tool that saves time, effort and money. Soon, data lineage will be indispensable.

  • Observe, Inc. Dives Deeper Into Observability

    Observe, Inc. Dives Deeper Into Observability

    Fresh from picking up an additional $70 million in funding, Observe, Inc. announced this week it has added multiple tools, dubbed Data Universe Maps, to its observability platform to visualize all datasets and surface underlying event data.

    The company also added pre-built applications for Kubernetes, Amazon Web Services (AWS) and Jenkins environments with editions for Google Cloud Platform (GCP) and Microsoft Azure planned for later this year. Observe Apps are intended to make it easy for IT teams to ingest data that is surfaced via pre-built dashboards that will generate alerts when issues arise.

    Finally, Observe, Inc. has added dashboards that provide transparency into its usage-based pricing model. IT teams can now more easily determine how many credits are consumed by both users and specific datasets to help contain costs.

    Observe, Inc. CEO Jeremy Burton said those tools are crucial because in the absence of tools to control data costs, most organizations will soon discover that observability is unsustainable.

    Of course, it’s not clear yet how many organizations are moving beyond monitoring metrics to embrace observability and launch queries to discover the root cause of performance issues. Many organizations don’t have the level of expertise required to structure such queries. However, as more organizations struggle to investigate and remediate issues impacting customer experience, interest in observability platforms is rising, noted Burton.

    Burton said thus far, as a startup, Observe, Inc. has expanded its customer base by a factor of three to now include 50 organizations, which has resulted in a five-fold increase in the number of monthly active users of the platform. The platform is now ingesting more than 40TB of data each day, with more than 25 million queries each day launched across nearly one trillion rows of more than 10 petabytes of data. Capital One is included among the investors in Observe, Inc.

    At the core of the Observe platform is a graph engine that makes it simpler to uncover the relationships between machine data that is ingested into the platform.

    Going forward, Burton said Observe intends to add support for distributed tracing using the open source agent software created by the OpenTelemetry project that operates under the auspices of the Cloud Native Computing Foundation (CNCF).

    Change within enterprise IT organizations happens slowly. Many will be using legacy monitoring tools alongside observability platforms for years to come. However, there is an opportunity to rationalize monitoring tools as the capabilities of observability platforms continue to expand.

    In the meantime, IT environments will continue to become more complex as more modern applications based on microservices and constructed using containers are deployed. These applications, over time, are generally more resilient than monolithic applications; given all the dependencies between microservices, determining the root cause of a performance issue can be a major challenge. As a result, it may be just a matter of time before this new class of applications ultimately forces the observability issue in the enterprise.

  • Top 3 Requirements for Next-Gen ML Tools

    Top 3 Requirements for Next-Gen ML Tools

    As the machine learning market matures, new tools are evolving that better match data science and machine learning teams’ needs. Vendors, both open source and private, have been quick to introduce new products that better meet requirements making it easier and faster to develop models and enable collaboration.

    These new offerings range from cloud-based platforms that make it easy to build and deploy models to specialized software that can speed up the training process. As an example, the existing field of ML experiment tracking is a multi-billion dollar market with a crowded amount of vendors: Comet ML, Weights & Biases, Neptune AI, Databricks’ MLflow and more. These tools will need to evolve to move beyond meeting only the basic requirements of ML teams like experiment tracking, visualizations and model training to address collaboration and technical requirements so these teams can get models to market faster.

    The types of teams that are driving modern requirements for data science are the bet-the-farm types, where it’s all about finding new technology and solutions to drive efficiencies and get models to market faster for their organizations to maintain a competitive edge. Going forward, we’re seeing the following as requirements for ML teams as they evaluate tools to add to their MLOps stack.

     An Open, Modular Approach

    Data scientists and ML engineers use multiple tools to do their jobs, from Jupyter notebooks for model coding to DVC for experiment and data versioning, to automation tools like AirFlow to model visualization tools. Solutions will need to be open to integration with other tools instead of a walled garden that many ML tools, like those offered by cloud vendors (e.g. Amazon SageMaker, etc.) are now.

    Familiar Developer Experience

    ML developers and engineers are familiar with standard DevOps tools and workflows like GitOps (think GitHub or GitLab) and live in CLI-oriented solutions. ML tools should integrate tightly with these existing tools instead of complicating the developer experience with a separate ecosystem. And as a part of the developer experience, workflow is also important. Data scientists can run hundreds of experiments using their preferred CLI-based solutions but, often, all those experiments are streamed and posted into existing ML tools. A more optimal approach would be to offer data scientists the experience of running experiments locally and letting them commit only the most important experiments for further discussion and collaboration so the team isn’t overwhelmed.

    Strong Connection to Source Code

    This approach makes a stronger foundation for something like tracking ML experiments. As ML teams race to get models to market and continually improve them, being in lockstep with DevOps teams to deploy those ML models into production is critical. An example would be a developer tracking five experiments with changing hyperparameters. For these experiments, they may have a link to a Git repo for a particular version of source code. But what if the source code changes or data set changes are not committed or reported in their existing ML solution? Then, the connection is lost and existing metrics would be based on a previous version of source code or data that’s no longer accurate. That lost connection slows down time-to-market and makes it harder to optimize models.

    As seen with the above examples, ML teams juggle myriad tools and workflows as they develop models for various applications and services. The requirements discussed will help these teams to collaborate better and build models faster in an increasingly heterogeneous MLOps landscape.

  • Container Storage for DevOps: New Options With NVMe

    Container Storage for DevOps: New Options With NVMe

    The DevOps world is ripe for a storage paradigm change, both in how it operates within the enterprise and how it implements applications. 

    NVMe, especially NVMe-over-fabrics (NVMeoF), is both a tool and a host target that can provide such a change—especially in the new world of microservices and containerization. NVMe can drastically improve application performance over classic implementations and, given the right tools, transform the process of writing those implementations to be faster, more robust and more flexible. 

    I/O-Intensive Work

    A quick recap shows why this is so. The DevOps build toolchain, especially for large codebases, has always been heavily I/O-dependent and storage access has classically been the biggest bottleneck for building an image. The ability to cluster small, inexpensive Linux x86 servers connected to SANs twenty years ago was a first step in separating storage hardware from compute, but I/O latency was pretty awful. Build/link operations could actually take longer than the compilation of object files. 

    Virtualization helped make hardware utilization better and cheaper but didn’t really improve throughput. Cloud infrastructures such as OpenStack and CloudStack made those infrastructures simpler and more elastic to manage but, again, the IOPs bottleneck remained. 

    Containerization for DevOps made it even simpler from the applications level—just deploy the app and ramp up container instances as needed—no more dedicated infrastructure needed. And it’s cheaper for incremental functions. 

    However, not much was done to clear the I/O bottlenecks for largescale storage and increasing overall performance. 

    When the new nonvolatile memory technology of NVMe came to market in 2014 for the single server world, it delivered two game-changing capabilities. 

    1. An increase of 5x disk access I/O speed. This was, arguably, the biggest single performance improvement since multicore processors with onboard cache (think Opteron) emerged. With Linux adding support for standard, open source NVMe drivers in 2021, the options increased beyond Windows and Mac users.However, SANs and their protocol stack were still an issue. SATA- and SCSI-based SANS performance was gated to the 1.2GB limitations of those technologies and big iron to head-end the SAN still dictated high-cost infrastructure.
    2. Unlike SATA or iSCSI, NVMe has the native capability for TCP/IP or fibre channel addressing. This allows NVMeoF to be an autonomous point for data storage in a normal network. A user can now keep control of their data but use network-attached compute as a utility with roughly 6Gb speed for I/O—greatly lessening the big penalty for disk I/O.  

    Applying Containers

    The dramatic uptick in container deployments and the debut of container attached storage (CAS)—software that includes microservices-based storage controllers that are orchestrated by Kubernetes—represented another industry milestone. These storage controllers can run anywhere that Kubernetes can run, which means any cloud or even bare metal servers or on top of a traditional shared storage system. Critically, the data itself is also accessed via containers as opposed to being stored in an off-platform, shared scale-out storage system. 

    Since container-attached, cloud-native storage enables the use of small, disaggregated architecture without the need for large-scale SANs or storage controllers dovetails nicely with DevOps work. 

    Container-based applications can benefit enormously from NVMe-based storage. As experts have pointed out, since DevOps applications are all about scalability, the parallelism embedded in the NVMe spec fits nicely to support them. 

    While SANs are good for large-scale contiguous operations, they’re expensive chunks of capacity and not nimble to deploy. They’re also bound to the slower I/O storage protocols, which can negate performance gains possible by adding NVMe to that array (as opposed to discretely).

    Container-based storage with NVMe gives DevOps teams the opportunity to add storage at low cost with deterministic management of that stateful storage—in theory, of course. In practice, the temptation to add like infrastructure to current infrastructure is powerful, even with all its limitations.

    To make that theory a reality, a control plane with the fast data plane is necessary—and the market is beginning to address this need.  

    As one example, OpenEBS recently introduced an upgraded engine, MayaStor (now part of DataCore Software), which enables use of NVMeoF at impressive speed: Benchmarks with Intel SPDK architecture show a 5x read/write speed improvement over SATA.

    OpenEBS debuted a few years back as a containerized storage solution for Kubernetes-based applications. With an optimized storage engine, Kubernetes has the opportunity to perform like other, more specialized hosting and development solutions. 

    Impact for DevOps

    What does this mean for DevOps teams? Firstly, using NVMe in containerized DevOps applications speeds up the time required for large builds, thereby speeding iterations of writing software. It also removes the need for complex and expensive SANs to host build environments. This double whammy of benefits means that development is quicker and cheaper and, in turn, applications can deploy faster while using more flexible tools.

    Second, NVMe greatly mitigates I/O concerns for how application code is designed and how environments are implemented. No more being beholden to the storage trunk—NVMeoF drops the performance penalty for I/O ops and distributes traffic across a normal TCP/IP network. This is now a triple whammy—faster, cheaper and more flexible containers built atop a fast, flexible virtualized infrastructure.

    Backed by innovations in container-attached storage, DevOps teams can achieve the agility and flexibility that are paramount to its mission—and act as a proving ground for the rest of the IT team to consider alternative paths to greater efficiency and performance.

    Join us for KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain (and virtual) from May 16-20—the first in-person European event in three years!