Tag: devops

  • From Noise to Narrative: Rethinking Observability for AI-Augmented DevOps Pipelines 

    From Noise to Narrative: Rethinking Observability for AI-Augmented DevOps Pipelines 

    Because DevOps teams deal with so much data, they may feel like they are drowning, but rarely find the insights they need. There is more observability data now than ever, yet it often stays separate, disorderly, inaccessible or confusing. As AI enters more organizations’ processes, observability moves from just monitoring to gathering early insight through meaningful narratives. It is not only about having new tools but also changing how development is done and organized. Supporting today’s DevOps — where AI is involved — requires developers to switch from seeing observability as a solely technical aspect to thinking of it as a way to explain what happens in the system: One that links unrelated incidents, highlights how each one is related, and warns users ahead of any failure. 

    The Telemetry Tsunami 

    Legacy monitoring was created for stable, predictable systems, including static servers, stable networks and well-known failure points. Nowadays, architectures made of ephemeral containers, dynamic microservices and AI-driven automation result in high-volume and high-velocity telemetry data that older monitoring systems cannot handle. 

    AI models add more factors to consider in analyzing systems – model drift, low data quality, inference delays and unclear ways AI decides make things difficult to observe. Logs and alerts are insufficient to make sense of what is happening in such systems. Observing these complex, AI-driven systems calls for tracing and feedback that can reveal valuable insights with a lot of noise around them. The result? As a result, most teams experience so much data that it’s hard to find the reason behind any performance, regression or security problems. 

    Observability as Narrative 

    Moving forward, DevOps teams need to treat observability as a way to construct coherent stories. DevOps teams should focus on building stories from the signals they gather that help explain what happened and what to do next. This shift involves three fundamental changes: 

    Correlated Context Over Raw Data: Systems should first add metadata like deployment IDs, feature flags, user behavior and model versions to metrics, traces and logs to describe their life cycle independently (refer to figure 1). 

    Declarative Telemetry Pipelines: Bringing AI/ML into the system will allow systems to describe their activities using words humans can easily understand. A pipeline should ensure that the effects of an AI model update on latency can be noted and followed, not simply added later on. 

    Semantic Layering with AI Assistance: Instead of being used only in the system, AI/ML can simplify understanding what is happening. With the help of AI/ML, observability tools are becoming easier for humans to understand. 

     

    Figure 1. The three pillars of observability that are essential in an AI-augmented DevOps pipeline. 

    From Dashboards to Dialogues 

    Most current observability tools rely solely on the user to interpret data. However, more practitioners are starting to wonder: Why can’t the system simply offer explanations? It all boils down to how observability tools combine AI to help sum up what’s happening, pinpoint regressions and provide possible explanations. With the help of AI, systems can now describe issues, indicate regression points and identify probable problems in the data. An AI-augmented observability system could tell you: “Starting at 12:06, service X saw a 30% increase in latency because many cold starts from model Y, which had just been redeployed recently. Service level objectives (SLOS) did not meet expectations by 12:12. Consider raising the pool size of warm data or re-adjusting the model’s time frame for batch processing”. 

    This is no longer science fiction but something that is becoming true. Several open-source and commercial solutions integrate machine learning into observability stacks to help find root causes, group anomalies and make suitable suggestions for resolving issues. 

    Designing for Explainability 

    AI models used in DevOps also require monitoring for issues, as they can perform worse than expected for reasons that might not be obvious in tests. Unlike standard software, changes in AI models can be challenging to detect until they appear in production. Sometimes, a model that looks good during testing can have problems in the real world, caused by small or unusual changes in the input data. 

    Explainability is key. AI observability must track: 

    • Input/output distributions over time 
    • Feature importance changes 
    • Data pipeline lineage 
    • Confidence scores and decision paths 

    When correlated with infrastructure and application data, this information will tell teams if the incident was caused by a system problem, an issue with the pipeline code, or faulty model behavior. 

    Practitioner-Centric Observability 

    For observability to progress, it must be designed for all system users, not just for central teams. This means that not only platform teams but also the mid-level workers responsible for trying to understand user complaints, triaging incidents, or fine-tuning production models need observability tools. 

    Some key principles include: 

    • Narrative-Driven Interfaces: Less reliance on dashboards; more chances to look at timelines, see how things are linked and get simple explanations from AI. 
    • Integrated Feedback Loops: Ability for engineers to fix, add notes to and correct system alerts so the information shown in the future gets better. 
    • Cross-Domain Visibility: Bringing together data about how applications are doing, how the systems are working and how well your AI is doing all together into one place to help you see your systems better. 

    Observability must evolve, so it is smarter than simply collecting more data. It will require the entire culture of the industry to change, not just new technology. People in the industry should start seeing observability as telling a story, where each log, trace and metric adds to how the system can be easily explained. This evolution means that your systems must be able to define themselves well, rather than simply based on how much data you can collect. 

  • Datadog Adds Raft of Additional AI Agents and Tools to Portfolio

    Datadog Adds Raft of Additional AI Agents and Tools to Portfolio

    Datadog at its DASH 2025 conference today previewed a raft of artificial intelligence (AI) agents that automate tasks ranging from assessing infrastructure alerts that normally require the expertise of a site reliability engineer (SRE), to fixing code and triaging cybersecurity issues using the company’s security information event management (SIEM) platform.

    In addition, Datadog is also adding Proactive App Recommendations that analyze telemetry data to suggest the next best action and APM Investigator to identify and trouble shoot bottlenecks impacting application performance.

    At the same time, Datadog is adding data observability tools, along with extensions to its service for monitoring large language models (LLMs), to add support for AI agents and previews of tools to test and validate the impact of prompt changes, model swaps or application changes on the performance of LLM applications. The company has also added an AI Agents Console to make it simpler to discover and invoke its AI agents. Datadog LLM Observability, now generally available, monitors the integrity of AI models and performs toxicity checks that look for harmful behavior across prompts and responses, while Datadog Workload Protection monitors interactions between LLMs and their host platforms. A Datadog Cloud Security and Sensitive Data Scanner (SDS) also enables organizations to meet compliance requirements.

    Datadog is also making generally available Datadog Code Security, a service that leverages AI to detect, prioritize and remediate vulnerabilities in their custom code and open-source libraries based on runtime threat activity and business impact. It has also extended the capabilities of its log management suite to run in an on-premises IT environment—meeting data sovereignty requirements in regulated industries—and to help reduce costs with a lower-cost archive tier of storage.

    Finally, Datadog has added a managed internal developer portal (IDP) that is integrated with the company’s core application performance monitoring platform. It provides a live system of record for telemetry data to surface what software is running, who is responsible for it, and how it is performing. There are also templates powered by Datadog’s App Builder and Workflow Automation to automate tasks such as creating the scaffolding for a new service along with engineering reports and scorecards to track compliance, reliability, security, observability, cost and other metrics.

    In general, Datadog is making use of a range of proprietary and open source LLMs, including one optimized for time-series data that it has developed. Over time, Datadog plans to mix and match LLMs to address specific use cases as additional AI advances are made.

    Michael Whetten, vice president of product for Datadog, said, for example, the latest Bits AI SRE, Bits AI Dev Agent and Bits AI Security Analyst agents not only extend the range of tasks being automated, but also enables them to query data, analyze anomalies or scale infrastructure using shared memory in a way makes it possible for AI agents to reuse functions. Datadog has also developed a Model Context Protocol (MCP) server to make it simpler for AI agents to consume its data and interoperate with each other. The overall goal is to not, for example, replace SREs, but rather augment their expertise in a way that makes it easier to manage complex IT environments, he added.

    It’s still early days so far as adoption of AI technologies across DevOps workflows is concerned, but a recent Futurum Research survey finds 41% of respondents expect generative AI tools and platforms will be used to generate, review and test code. In fact, arguably the challenge now is not so much deciding whether to use AI as much as it is determining where AI will have the biggest impact on improving both software quality and the speed at which applications are deployed.

  • How DevOps Strategy Is Evolving with Jim Cassens

    How DevOps Strategy Is Evolving with Jim Cassens

    Perforce Software CEO Jim Cassens dives into how the company’s overall approach to DevOps is evolving in the wake of a series of acquisitions.

    Cassens talks through the company’s recent spate of acquisitions and how thoughtful integration—not just portfolio expansion—is driving value for customers. He also highlights the growing role of synthetic data, AI-enhanced testing and how automation is shifting the responsibilities of development teams.

    A recurring theme is how generative AI is disrupting—not replacing—traditional developer roles. Cassens sees AI as a force multiplier, taking on repetitive, low-level tasks while freeing developers to focus on higher-order problems. But with that shift comes a challenge: How do early-career engineers gain the hands-on experience they need when entry-level work is being automated? That’s a question Cassens believes employers, not just universities, will need to answer.

    They also delve into the pressure DevOps teams face as pipelines become more brittle and overloaded. Cassens warns that the current approach to automation and scripting isn’t sustainable in a world where AI can flood pipelines with code at unprecedented volume. Organizations will need to rethink how they structure CI/CD, data management, and even pipeline observability.

    Finally, Cassens touches on the human element behind every digital transformation. As leaders weigh new tools and AI-powered workflows, he advises them to consider the fear and uncertainty many team members may be feeling—and to support them through education and open communication. The DevOps landscape may be changing fast, but people, not just platforms, will determine how successful the transition is.

  • How AI Is Transforming the Future of Software Development with Wing To

    How AI Is Transforming the Future of Software Development with Wing To

    Digital.ai CTO Wing To delves into how the way code is written, deployed and managed will fundamentally change in the artificial intelligence (AI) era.

    Cutting through the extremes, AI won’t erase entry-level dev jobs, nor will it fully replace senior developers. Instead, it’s changing what both groups focus on. Entry-level devs will need to learn how to effectively prompt, evaluate and refine AI-generated code, while senior devs may shift toward mentoring, reviewing AI-assisted output, and ensuring software quality. The fundamental knowledge of how systems work still matters—AI doesn’t remove the need for strong foundational skills.

    But AI does change the coding landscape. Developers are writing less and reading more. Reviewing AI-generated code—often verbose, not elegant—has become part of the job. And while some fear that the code is harder to debug, it may just be different: more mechanical, less idiosyncratic.

    From a DevOps perspective, AI introduces new challenges and opportunities. DevOps engineers are facing a surge in code volume and speed—often without a corresponding bump in quality. Automation, golden paths and better processes are essential to handle the influx of AI-assisted code. AI can also help streamline repetitive work: writing scripts, maintaining configurations, generating tests or spotting compliance risks.

    Eventually, DevOps teams may evolve into ecosystems of human engineers and collaborative AI agents. That vision’s still emerging, but orchestration between tools and agents is the next hurdle. The goal? To let developers and engineers focus more on creative, value-adding tasks—and less on toil.

    Still, AI must be used thoughtfully. It’s not a magic fix. It’s a power tool—one that needs skill, discipline and a good understanding of when (and how) to use it.

  • DevEx Got You This Far: What’s Next for True DevSecOps Maturity?

    DevEx Got You This Far: What’s Next for True DevSecOps Maturity?

    The journey toward integrating security into the fast-paced world of DevOps has seen significant strides, largely thanks to a much-needed focus on the developer experience (DevEx). 

    Collectively, the AppSec community and industry have moved mountains in making security tools less intrusive, integrating them into developer workflows, and acknowledging that developers need security as a help, not a hindrance. A focus on DevEx has been crucial, acting as the bridge to bring security considerations earlier into the development lifecycle while simultaneously enhancing the developers’ daily work.  

    The numbers don’t lie: Checkmarx’ recent DevSecOps Evolution 2025 report reveals that a majority of the organizations we surveyed—58% to be exact—are at Stage 2 maturity, a transition stage primarily focused on DevEx: They’ve integrated tools into IDEs, offer formal training, and the relationship between security and development teams is generally positive. While these developments signify important progress, this DevEx-focused stage represents a critical foundation rather than the ultimate destination in achieving mature DevSecOps.

    DevEx Isn’t the Destination

    Organizations haven’t yet achieved true DevSecOps maturity. They haven’t yet embedded security through shared culture, governance and aligned goals.

    And while developer experience is essential, it’s just one brick of the DevSecOps foundation. To achieve the ultimate goal—high-performing, secure code delivered at the pace of business—organizations need to move beyond DevEx to a true culture of shared responsibility and trust.

    Despite better tools and training, developers still report spending a significant chunk of their week—often over 17 hours—on security tasks. Trust in tool results is often hampered by false positives impacting velocity, and a lack of shared goals that align development and AppSec priorities means security can still feel like a separate concern rather than a shared objective. We’ve improved the experience; but haven’t fully embedded security seamlessly and at scale.

    The Scale – and Stakes – Are Growing

    The need to mature beyond DevEx is especially urgent given the accelerating complexity and scale of modern application development. Enterprises are managing sprawling codebases and adopting tools that increase both speed and risk. Checkmarx alone scans over 450 billion lines of code every month, and that figure is only rising. 

    At the same time, the software supply chain is expanding at an unprecedented pace: approximately 700,000 new NPM package versions are introduced monthly, each one a potential point of exposure.

    Add to this the growing adoption of AI-assisted coding tools and agentic AI systems that generate or modify code autonomously, and the attack surface multiplies further. 

    On the other hand, development productivity requirements have never been higher, with organizations tracking DORA metrics and looking for AppSec solutions like Checkmarx’ that deliver on security while optimizing MTTR and other DORA metrics.

    Achieving truly mature DevSecOps—Stage 3, where only 30% of organizations currently reside—requires moving beyond just making security palatable for developers. It demands a deeper, more strategic integration built on shared culture, robust governance, trusted automation, and aligned success metrics.  

    Checkmarx’ Five Pillars of DevSecOps Maturity

    At its heart, DevSecOps is a cultural evolution. It’s about merging the “move fast” ethos of development with the “protect everything” mandate of security to achieve a common goal of delivering secure, high-performance code at the speed the business requires. To progress from Stage 2 (DevEx-focused) to Stage 3 (Mature DevSecOps), organizations need to build on their DevEx foundation with five key elements:

    1. Shared Metrics

    You can’t improve what you don’t measure, yet our research shows a surprising lack of consensus on DevSecOps metrics. While many organizations we surveyed track top vulnerable applications (52.1%) and mean time to detect vulnerabilities (51%), only 28.3% measure mean time to remediate—arguably the most critical metric for balancing security and velocity.

    True DevSecOps maturity requires security and development teams to align shared metrics that measure how effectively they work together, not just how many vulnerabilities they find.

    2. Collaborative Governance

    In mature DevSecOps, security and development teams jointly establish policies, standards and processes. Our data shows this remains rare—when asked to describe the relationship between security and development in one word, respondents most frequently chose “trust,” “policy,” and “risk,” indicating an evolving but not yet fully collaborative relationship.

    For 88% of organizations, the absence of shared metrics, policies and governance suggests security and development teams are aligning but are still struggling to form a fully collaborative governance.

    3. Advanced Security Education

    While essentially all developers have access to training according to our study, the model needs to evolve. Traditional formal training programs still have value, but the premium on developer time makes standalone training challenging.

    Mature DevSecOps organizations are pivoting toward just-in-time training that delivers contextual guidance when and where developers need it. Providing a step further is in-the-moment training with inline feedback from within the IDE, allowing developers to learn while maintaining their workflow.

    4. Trusted Automation

    Our study showed that one in five (20%) of the organizations we surveyed have no AppSec automation in place. Even among the remaining 80%, adoption of key automation capabilities remains low—only 32% have automated security testing during change management, and just 30% auto-populate security tickets with remediation guidance.

    The far-from-optimal adoption rate of automation cannot be attributed to a lack of technological solutions but rather appears as a matter of trust and culture gaps between security and development teams. Without shared governance and metrics, attempts to “automate everything” typically fail because there’s no foundation of trust on which to build automations that both teams will embrace.

    5. Velocity-Security Balance

    The ultimate goal of DevSecOps is to deliver high-performing, secure code quickly. Yet 66% of developers report that false positives impede their velocity, and 28% say their AppSec teams lack the tools or resources to help them fix security issues efficiently.

    Mature DevSecOps organizations recognize that speed isn’t just about how quickly tools scan—it’s about how quickly developers can remediate issues without disrupting their workflow. This requires prioritized, actionable results with clear remediation guidance, not just faster scanning. 

    The Role of Leadership

    Making the transition to DevSecOps takes more than just good tooling—it requires strong leadership. Security leaders must be willing to trade control for collaboration and build systems based on partnership, not policing. Engineering leaders must recognize that security is essential to velocity, not an obstacle.

    This shift requires executive buy-in and strategic commitment. It means moving beyond budget conversations centered on compliance and instead investing in the long-term health of your codebase and your teams.

    The Road Ahead

    As the DevSecOps Evolution 2025 report shows, most organizations are still in the middle of the journey. That’s not a failure. It’s an opportunity. DevEx helped bridge the gap between developers and security, and now it’s time to build the foundation that inspires them to run together. By moving beyond DevEx as the sole focus and embracing these pillars of mature DevSecOps, we can finally embed security seamlessly at scale, achieving the ultimate objective of delivering high-performing, secure code without compromising speed.  

    The transformation won’t happen overnight. It requires sustained investment in building trust between security and development teams. But organizations that successfully make this journey will gain the significant competitive advantage of more secure and advanced applications built with less friction and lower costs.

    As we look to 2026 and beyond, the most successful organizations will be those that move beyond seeing security as just another feature of the development process. Instead, they’ll embrace it as an integral, collaborative aspect of delivering high-quality software in a digital world with ever-rising security risks.

    At Checkmarx, we believe the future of software development is secure by design—and it starts with mature DevSecOps. Let’s keep moving forward.