Tag: devops

  • OpsVerse Adds Backstage Catalogue to Managed DevOps Service

    OpsVerse Adds Backstage Catalogue to Managed DevOps Service

    OpsVerse this week added an open source Backstage catalogue software for building developer portals to its managed DevOps service.

    Originally developed by Spotify, Backstage makes it possible to centralize the management of multiple services that are being built and maintained by multiple developer teams. Backstage is now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

    OpsVerse CEO Arul Francis said the company is now employing Backstage to create a centralized control plane dubbed OpsVerse ONE through which organizations can more easily centralize the management of a DevOps toolchain. Every applications team can see all the services and related resources they own and then via a single click launch a project, he noted.

    The managed OpsVerse platform is based on curated editions of open source software such as Argo continuous delivery (CD) and Prometheus monitoring software that OpsVerse enables organizations to access as a set of software-as-a-service (SaaS) applications. Those applications can then either be deployed in cloud platforms made available by Amazon Web Services (AWS), Microsoft or Google or in an on-premises IT environment.

    The addition of OpsVerse ONE will also make it simpler to extend the OpsVerse toolchain via more than 80 plugins that are currently available for Backstage catalogue software, noted Francis.

    It’s not clear what degree organizations are embracing managed DevOps platforms but at the very least smaller organizations should find it simpler to embrace best DevOps practices by relying on a managed service. There are also plenty of larger organizations that may determine that continuing to manage DevOps toolchains has become problematic after years of integrating various tools and platforms in ways that now are both challenging to maintain and secure. OpsVerse claims to have in the last six months already tripled the number of customers it supports.

    As application development becomes more complex to manage in the age of microservices, many organizations are at a crossroads. They need to decide to what degree it makes sense to continue to devote resources to managing DevOps platforms versus consuming them as a cloud service. Many of the resources that are now allocated to managing DevOps platforms could theoretically be shifted toward building and deploying more applications. In some cases, organizations may decide to continue to rely on legacy DevOps platforms to maintain existing applications while employing a cloud service to build modern cloud-native applications that have many more dependencies than monolithic applications.

    Regardless of the degree to which organizations decide to rely on managed services, platforms such as Backstage as it continues to evolve will make it simpler to extend open source toolchains. Spotify developed Backstage to bring some order to its own internal application development and deployment processes before deciding to contribute Backstage to the open source community. While Backstage is still an incubation-level project within the CNCF, it’s already being used across multiple DevOps production environments to make it easier to integrate additional tools without having to sacrifice their ability to centrally manage a DevOps environment.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    staging-devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it just that much easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Dice.com

    Oscar Technology
    Remote
    DevOps Engineer- Go Language
    $120,000 to $180,000

    LinkedIn

    Proprio
    Seattle, Wash
    Senior ML-DevOps Engineer
    $120,000 to $170,000

    SimplyHired

    DISH
    Littleton, Col.
    Senior DevOps Engineer
    $106,210 to $143,750.

    Indeed.com

    RumbleOn
    Irving, Tex.
    DevOps Engineer
    $115,000 to $135,000

    Monster.com

    PROLIM Corp.
    Washington, D.C.
    AWS DevOps Engineer
    $90,000 to $110,000

  • DevOps Principles, Implementations and Culture

    DevOps Principles, Implementations and Culture

    When organizations begin using DevOps for application development, many only reference DevOps as part of their infrastructure and select a subset of DevOps before understanding the cultural and process changes needed for success. Instead, organizations must first determine which ‘Ops’ approach best promotes a DevOps mindset and integrate their workflows into their culture. In doing so, these organizations will better align their development and operations teams on common goals and protocols.

    There are many variations and subgenres of DevOps. However, it’s important organizations understand the ‘Ops’ choices available and choose the one that’s right for them.

    Principles Vs. Implementation Differences

    General DevOps and its subgenres can be split into two groups. The first and “purest” are the principles that these terms focus on unifying two or more industries with operations.

    The second type is those that focus on an implementation type.

    Principle Types

    DevOps: Improving Technical Processes and Culture

    DevOps is the overarching umbrella term for operations that combine Agile development concepts with IT operations and fast-tracks the development and deployment of applications in cloud environments. While there are different subgenres of DevOps, each subgenre is a different application of DevOps methodology, all geared toward making development more agile, stable and efficient. Because DevOps shortens the development life cycle by a significant margin, this streamlined approach to application development is quite popular.

    DevSecOps: Adding Security Checks

    DevSecOps bakes additional security measures into the workflow. Where DevOps typically has security checks at the end of a cycle, IT pros that leverage DevSecOps integrate security early and often in their DevOps journeys. With DevSecOps, collaboration between developers, security specialists and ops is crucial to maintaining agility and delivery speed, but the extra layers of security can slow processes down. DevSecOps leverages fast testing and robust pipelines to favor better code integrity. This method can be beneficial if an organization needs to elevate security during development.

    NoOps: Reducing In-House Teams for Operations

    NoOps is the lofty vision that DevOps is so automated it doesn’t require in-house expertise, meaning no manual intervention is required in processes. NoOps differs from DevOps because there is less focus on Agile and process management. This process assumes that developers do not need to know how to use the automated tools and uses continuous integration techniques to allow developers to focus only on application development.

    Though NoOps speeds up the deployment processes, if used by itself, NoOps lacks open communication and team management. Successful implementation of NoOps is only achievable if an organization does not have lofty technical goals or if its development team is small but strong and able to handle development and operations.

    GitOps: Using Git to Automate

    GitOps has been gaining traction in the development industry over the past two years because of its use of Git to automate continuous delivery pipelines. GitOps is a robust and manageable form of operations because it uses Git as the single source of truth for operations. In addition, due to developers’ familiarity with Git and pull requests, there is no complicated tool to learn, and changes to existing workflows are often unnecessary.

    The GitOps method can also boast that it is backed by some of the market’s most innovative cloud services, thus automating the process of building an update, testing for errors, reviewing code and pushing the update to production. Because Git is the only source of truth with GitOps, it is vital to effectively fortify the Git repository to avoid needless commits or pull requests.

    CloudOps: Implementing Cloud-Native Features

    CloudOps attempts to take more advantage of cloud-native features offered by modern service providers. Since CloudOps focuses on distributed development and deployment, there is no single point of failure, making the whole cloud environment more reliable. In addition, certain parts of the workflow in this type of operation can go stateless, making it cost-efficient as organizations only pay for the resources they actually use. Though CloudOps can be time and cost-efficient, organizations must get the configuration right for the approach to be beneficial, which could mean added complications to configuring CI/CD pipelines.

    CIOps: Requiring CI Operators for Configuration

    Continuous integration Ops (CIOps) is the latest iteration of DevOps, requiring CI operators to configure the infrastructure needed to support new code before deployment. The manual inputs necessary to ensure CI jobs’ correct configuration and deployment location bring advantages and disadvantages. Control over infrastructure allows two deployments to have different infrastructure configurations. The major disadvantages are the required manual intervention leaves room for error and the need to link an organization’s CI tool to its API can pose security risks.

    CIOps is better-suited for smaller development environments where automation is an inconvenience. While CIOps gives developers flexibility in configuring cloud infrastructures, the approach takes time and effort to run smoothly.

    ITOps: Automating IT Management Pipelines

    IT is loosely defined as the management of end user or employee systems. They are the systems that are required to run a core business. The fundamental difference is that there is little to no development. If your business doesn’t require a software engineering team and can function solely on prebuilt applications, then ITOps is for you.

    ITOps is in charge of delivering and maintaining services, applications and the underlying technologies necessary to run a business, covering jobs like system administrator and network administrator. ITOps focuses on updating laptops and office networks in addition to highly-tested commercial software and solutions for constructing the infrastructure, making updating infrastructure components more complicated. When using ITOps, an organization’s CIO will focus on managing the system via automation. 

    Choosing the Best Op for an Organization

    Choosing between the variations of DevOps is a matter of finding an approach that works best for the kind of apps the organization is developing and the cloud infrastructure they use. Though all of the above Ops strategies are based on unique ideas, they tend to focus on the technical side. Contrarily, DevOps stands out for its ability to comprehensively improve workflows and streamline technical processes while fostering a culture of innovation and collaboration in an organization.

  • Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    The attack surfaces that today’s businesses and public entities must manage have never been more complex and difficult to protect. The introduction of cloud and SaaS offerings over the past decade has caused seemingly insurmountable IT sprawl, with a third of successful attacks now occurring via unmanaged or virtually unknown assets. And while shadow IT is an urgent threat confronting many organizations, the migration of the software development life cycle to the cloud—and the rise of machine identities—is compounding the attack surface in profound ways. Emerging out of the Agile and lean strategies for software development in the 2010s, DevOps predates the mainstreaming of cloud computing.

    But once the value of cloud-based IT resources came to be widely appreciated, development teams began moving their developer environments to the cloud in a big way—to the point where many people today think of DevOps and cloud as inseparable. Recent Gartner research showed that much of enterprise IT spending is now directed toward cloud projects. In fact, for IT categories like application software, infrastructure software, business process services and system infrastructure, more than 51% of IT spending will shift away from traditional solutions to the public cloud by 2025. Nearly two-thirds of spending on application software alone will be directed toward the cloud by 2025.

    The explosion in cloud DevOps has brought with it a huge uptick in the number of privileges, credentials, identities and access rights under management. We’re all familiar with how digital identity and access management (IAM) works in conventional settings through combinations of usernames and passwords. In the DevOps world, where cloud containers can run as distinct microservice processes interacting with other processes (machine-to-machine communication), identities and privileges are usually governed through SSL/TLS certificates. These machine identities have emerged as a critical security priority across multiple industries, not just because they’ve proliferated exponentially, but because of the lack of effective tools and strategies available to IT security teams.

    Machine Identities—Scope of the Issue

    A recent global survey of 1,000 CIOs found that the average organization has 250,000 machine identities under management coming into 2022, with that number expected to double by 2024. The study also found that 83% of organizations suffered a machine identity-related outage in the past year, with 57% of organizations experiencing at least one data breach or security incident related to compromised machine identities over the same time period.

    Because the perimeter defense solutions and strategies that formed the basis of IT security best practices for many years are of little use in cloud environments, security admins have had to rely on existing IAM and privileged access management (PAM) tools to secure data, accounts, and development pipelines. Not surprisingly, identities and access privileges have become the leading attack vector in the cloud, providing attackers with an exponentially higher number of vulnerabilities to target within the enterprise than were available even a few years ago. We’ve seen these points of weakness come into play in recent high-profile breaches and attacks such as SolarWinds, Codecov and PHP.

    Embracing Zero-Trust Principles to Secure the DevOps Pipeline

    Fortunately for today’s DevOps teams, cloud-oriented privilege management technologies have evolved rapidly in recent years, especially in the categories of cloud infrastructure entitlement management (CIEM) and time-based role access management or just-in-time privileging. CIEM solutions target the security weak points and process vulnerabilities that are unique to the cloud environments. These include the lack of deep visibility into user, group, and role privileges, which together make it difficult to conduct oversight and control over the activities of users within cloud infrastructure and applications. Where conventional approaches such as IAM and PAM are designed primarily to secure data, apps, and users within the organization, CIEM is designed to manage permissions risks to prevent cloud breaches and data theft.

    Emerging CIEM solutions are bringing to bear advanced capabilities in discovery, monitoring and auditing to significantly broaden visibility into DevOps processes for IT security professionals. With a CIEM solution, security teams can scan cloud applications for existing users, groups, roles and granted privileges down to the resource level. Each user, group, role, and privilege can be categorized to identify and verify that privileged users have the appropriate levels of access. Access reports can be generated to concisely outline users and groups with the most privilege and the highest risk.

    On the audit front, CIEM solutions provide the capability to track and report on levels of privilege as well as all activity associated with those privileges. In this way, it becomes possible to generate privileged access reports for consumption by internal and external auditors to ensure compliance with regulatory and other requirements such as SOX, GDPR, HIPAA, GLBA, PCI-DSS, ISO 27001, SOC 2, FedRAMP and more. Similarly, with the ability to monitor privileged access across all cloud applications, security teams can apply behavior analysis to identify privilege abuse and abnormal activities from users. As with conventional SIEM tools, CIEM solutions will generate alerts to administrators to suspect activities or behaviors—for instance, if a user attempts to circumvent policies preventing direct privileged access to cloud services.

    Just-In-Time Privileging and Zero Standing Privileges

    Importantly, CIEM solutions are also effectively eliminating standing privileges to cloud resources based on least-privilege access and zero-trust principles. The automated granting and expiring of permissions—just-in-time (JIT) privilege grants—is highly effective at minimizing attack surfaces. Because these JIT/zero standing privilege (ZSP) solutions work on a zero-trust model, no one and nothing is trusted with standing access to cloud accounts and data.

    With JIT permissioning, elevated privileges can extend either for the duration of a session or task, or for a set amount of time. Once the task is complete, those elevated privileges are automatically revoked–all without sys-admin involvement. Where a user previously had standing access privileges potentially extending around the clock for months at a time, converting to JIT granting would compress that attack surface to several hours per month. Further, JIT permissioning largely frees organizations from having to maintain and pay for both privileged and non-privileged accounts. Dynamic secrets generation – where a dynamic secret is generated on demand and is unique to a client, instead of a static secret being defined and shared ahead of time – also provides a better model for securing temporarily deployed services and features.

    DevOps and DevSecOps are still new and fast-evolving concepts within the wider computer science and cybersecurity universe. No doubt, DevOps has been wildly successful in accelerating automation and speeding time to market for innovative applications and business services. To date, however, security solutions providers have struggled with cloud access management. In other words, they have struggled to accelerate privileged access solutions that could secure the devices, data, and resources used by DevOps teams, especially in cross-cloud environments. Dynamic permissioning platforms using JIT privilege grants and employing ZSP principles show great promise in solving these problems.

  • Compliance Synergy – DevOps Unbound EP 25

    Compliance Synergy – DevOps Unbound EP 25

    Achieving compliance might be challenging when embracing DevOps due to the rapid pace of development. However, the agility and automation associated with DevOps might actually simplify compliance. When integrating compliance into the SDLC, DevOps teams may have to use different tools or slightly modify part of their processes to adapt to specific requirements. In this episode,Sonya Lowrence (Tricentis), Anna Murray (Rocket Software), Dan Kirsch (Techstrong Research) and Tim Johnson (Cloudbees) join hosts Alan Shimel and Mitch Ashley to discuss how to effectively integrate compliance as part of your DevOps process and the challenges that come with it. Is it possible to deliver high-quality software at the speed of DevOps whilst ensuring security compliance? The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey everyone. Welcome to another episode of DevOps Unbound. DevOps Unbound is a biweekly video series that explores all aspects of DevOps. We’ve been doing this for over a year now and it amazes me the breadth and depth of DevOps topics that we’ve had a chance to dive into here on DevOps Unbound.

    The show is sponsored by our good friends at Tricentis who have been our partners in bringing DevOps Unbound to you now as I said for more than a year and have really, in every sense of the word, partnered with us to really explore, go where no man or woman has gone before and DevOps and kind of really looked at things and take them apart. So many thanks to them for sponsoring. Welcome to our show.

    We have to sort of formats for DevOps Unbound. The every other show, every other week show is about 40 or 45 minutes of a prerecorded conversation with a panel of experts. And then around once a month we do do a live roundtable of DevOps Unbound in which the stars of the show are you, our audience. And you are invited to come on, participate in the conversation and ask questions, comment, chat with each other and it’s just – when it goes right, it’s a rip-roaring conversation of a thousand different directions. But it’s a fantastic experience. I invite you to, I think the next one might be around March 15 or something. But check it out, we’ll get the dates for you. But I do invite you to come to our next live roundtable as well.

    Anyway, let’s get on with today’s show, though. Today’s show is about DevOps and Compliance Synergy, Reality or Fiction? I have some thoughts on this but before we do, let me introduce you to today’s All-Star panel.

    On my screen, he’s on the top left. I don’t know where he is on your screen. It’s my friend Tim Johnson from CloudBees. Tim, why don’t you introduce yourself.

    Tim Johnson: Thanks, Alan and thanks for having me here. As he said, Tim Johnson. I’m in product marketing at CloudBees. I’ve been in DevOps since about 2016 and a whole lot of years in security prior to that. So happy to be here.

    Alan Shimel: Thank you, Tim. Thanks for being here. Next up I want to introduce you to Anna Murray. Anna, if you wouldn’t mind introducing yourself to the audience.

    Anna Murray: Thank you, Alan. Thanks for having me here. My name is Anna Murray and I work for Rocket Software, a recent acquisition through ASG Technologies, if you see me before. I’ve spent my career in automation. So at this point, I’m a product manager for automation technologies with Rocket Software and I’ve taken teams from Waterfall project management through Agile transformations and Agile has grown to DevOps and I am passionate about automation when it comes to all of that.

    Alan Shimel: Fantastic, Anna, thanks for joining us. Next up I want to introduce you to one of my colleagues, Dan Kirsch. Dan, if you wouldn’t mind introducing yourself and telling people a little bit about you.

    David Kirsch: Thanks, Alan. I’m managing director and one of the principal analysts and cofounder of Techstrong Research. And as a recovering attorney, compliance is certainly an issue that is near and dear to me.

    Alan Shimel: Absolutely. Our next panel member is, we’re very happy to have her, Sonya Lowrence. Sonya, welcome and maybe if you could introduce yourself to the audience.

    Sonya Lowrence: Sure, thank you, Alan. Hi, I’m Sonya Lowrence. I’m the head of compliance at Tricentis and I have been in this role and working closely with DevOps actually in a couple rules now since 2018. And before that, I was a programming manager heavily embedded with the integrations and Cloud operations for about 15 years. So I’m also very passionate about how to bring this to the dev team and streamline DevOps. So looking forward to it.

    Alan Shimel: Excellent. Last but not least, my cohost for DevOps Unbound and business partner, associate for 20 years, Mitch Ashley. Mitchell, if you wouldn’t mind saying a little bit.

    Mitch Ashley: Yeah, Alan. What a great panel. I’m excited to hear everybody chat about this topic. I’m CTO with the Techstrong Group and also work with Dan as part of Techstrong Research. So looking forward to us diving in.

    Alan Shimel: Absolutely. And thank you, Mitchell. All right. So guys, this episode is DevOps and Compliance Synergy, Reality or Fiction? Let me start it off with this.

    My background is in cybersecurity, 20+ years, Mitchell and I have worked together in it. Tim, you’ve been in cyber a long time, too. I will be really honest with you. One of my most hated subjects or aspects of security is compliance. I think compliance has done as much to hurt security as it’s done as much to help security. It’s introduced this least common denominator, lowest possible bar for security where instead of doing what we should be doing, we are doing what the minimal needs are, checking it off, check that box and we move on. And then we act surprised when stuff hits the fan.

    Now, I started staging-devopsy.kinsta.cloud because I thought DevOps was a godsend for security. Finally, a chance to get it right. Not to be bolted on, to be built in. And where security goes, compliance follows, right? Keeping that low bar here. And the promise of DevOps, especially with compliance but even more so than security, was my God, we could automate this. If we could just automate the compliance, we don’t have to worry about it anymore and we can focus on really making secure software. Because compliance becomes sort of a byproduct of security automation in DevOps.

    Now, as we sit here today, we know that that hasn’t necessarily been the outcome. Let’s leave it at that. Maybe it has been to some of you. I’m interested to hear. Don’t let me bully you. Has DevOps been a good friend to compliance or are we barking up the wrong tree? Are we chasing our tail? I don’t know how many dog analogies I can come up with here but what is the deal with compliance and DevOps? Who wants to start us off?

    Tim Johnson: I’ll jump in, Alan. You crack me up. We’ve been hearing that for forever and the dichotomies that we have to deal with, can you be compliant and fast? Can you be projectable and flexible? It’s like yeah, come on. Knock it off. It isn’t like that.

    The way we’re seeing it is DevOps is kind of going through this evolution of can we just automate the pipeline or can we get the CI part to work? And then can we automate the testing part? So it’s kind of evolutionary. And security has kind of come along, I think about 2017, John Wallace, we did a webinar with him where he’s talking about dev sec ops and saying that the sec should be silent because if you’re doing it right, the security is there. And now we’re coming to the next evolution and the next source of friction, if you will, is compliance. Because okay, we’re going faster, we’re going secure. Somebody says can you prove that to the regulators? And the way we see it is we have a bank customer of ours has 100 people devoted full-time on a 90-day rotating basis to doing compliance work.

    Alan Shimel: Sounds automated to me.

    Tim Johnson: Companies are doing. So that’s kind of a state of where we are. So from where I am, if you can’t do an audit easily, you ain’t running DevOps. You ain’t doing DevOps right in the first place. So if you are compliant and you can prove it to the regulators, you’re doing DevOps really well. That’s my opening salvo. Back to you.

    Alan Shimel: All right. I guess someone else wanted to say something.

    Sonya Lowrence: Yeah. I’m very passionate about this definitely at Tricentis but honestly just anywhere. I feel like it’s a two-part answer right now. I think I’m finding more compliance people in my space who understand, again, I come from a heavily background so for me, it’s very native to say compliance is an afterthought. If you look at it the right way, even though it may be a low bar, it’s an input. It’s a set of basic requirements. If you can accommodate that upfront, there are very few surprises. 

    And I would even extend on that and say in the world that we are moving to in DevOps and better automated tools where your CICD pipeline takes all of this into account, both your code and your Cloud environment, I really feel like it’s going to start to deliver inherently secure code and infrastructure. It’s just that that vision and the tools that are bit lagging, right? I don’t see that there are really any surprises. Couple of example frameworks SOC 2 type 2, ISA 2701. The objectives are clear. If you wrap that in at the beginning as a requirement, then there are very few people that one, don’t get trained as they go and two, can’t deliver because it actually is not that hard or complex to deliver technically.

    So I think that part of what we are seeing, and this is in part to me having been a program manager for years, Agile took and pared it down to a very lean perspective of we’re looking at features first, customer needs and typically things like encryption and administration and users was kind of an afterthought and it’s not built-in to the traditional Agile tools. I think that along with what Tricentis is delivering for quality, many companies are starting to respond with respect to compliance and security and putting it back at the front where I think it belongs. Again, this is not technically difficult. It’s the fact that it’s basically not built in to the current CICD tools. It makes it challenging, I think, for developing teams and infrastructure teams.

    David Kirsch: Yeah, I think what Sonya said made a lot of sense. It’s also, now when we are talking about DevOps, we talk a lot about culture change and changes in process as much as technology. So if the dev team is just go, go, go, go, what new features have you added today? They are never going to think about compliance. But if the development team feels that they have a working relationship with the compliance team and they are working together and they understand what the compliance team needs and why they need it, not just oh, it’s a checkbox but it’s important and it’s important for more than just our auditor to look at. That sort of merging together of collaboration between DevOps and compliance is an important aspect.

    Sonya Lowrence: Absolutely.

    Tim Johnson: I think part of that is some of the evolution that we are seeing and thinking, at least from the dev side is a maturing from a bug is a bug but it’s matured into a security vulnerability is a bug and noncompliance is now starting to be thought of as a bug. And there really aren’t too many developers out there who aren’t passionate about writing bugless code. But the challenge is, what is compliant? And how many standards do you have to adhere to? And how does that translate into all the different environments, the runtime environments, the binaries, the code, the access to the data that you’re dealing with. How do you translate all of those mumbo-jumbo from those regulatory standards down into being compliant with this piece of code that your writing? That’s a big challenge.

    Sonya Lowrence: No, I agree. And especially to that point, when I am evaluating alongside our dev teams and our product teams tools that help augment and fill these gaps, I’m definitely very aware and making sure that we’re starting from the rate baselines. There are baselines that if you work from it is kind of that magic decoder ring so all the different frameworks you need to apply going forward from a compliance standpoint. And once that’s in place, again, it just streamlines all of this for all of the teams. So it’s a really important consideration in tool selection.

    Anna Murray: And I’m going to finally jump into the fray here. I think that’s also been part of the challenge that we’ve seen among people we are talking to is that there are so many tools in place. How do you get that report so you can see what’s happened and you can reply to your auditor? And so we’ve really been talking to people a lot more about making sure that you have a plan that surrounds all of this because not only do you have a lot of tools, you might be replacing one or two of them every year because there’s better technology. And so you need good, planned workflows that are going to work for you no matter what you put in play into that workflow. And it’s not an easy answer to get from here to there and as somebody said, it is a journey for every company. And so it really does behoove the leadership to make that culture change as Dan talked about. But it isn’t just about adopting DevOps. It’s about having an end to end plan or making sure that you are delivering that value on a consistent basis as things change around us all the time.

    Alan Shimel: Agreed. To me, I’m going to be just the devil’s advocate today, I’m sorry. But I almost hear us saying hey, you know what’s great about us this DevOps thing? We are going to make compliance the developer’s problem, not the security person’s problem. Right? Developers, they’re well-paid, what the heck. But they have enough on their plate. I don’t need my developer necessarily to be my tester as I think the folks at Tricentis will clearly tell you. I don’t need my developer to be my security person. I need them to be security conscious, security aware. But I don’t need them necessarily to be a compliance expert. So therefore, if we can’t develop the tools that can automate compliance as almost a byproduct of the process and instead throw it onto the shoulders, put another straw on the camel’s back of the developer, I don’t know if it’s successful. If it can be successful.

    Tim Johnson: Agreed, agreed. The shift security left thing just really bugs the bejeebers out of me on a couple of levels. 

    [Laughter]

    Tim Johnson: One, like you talked about Alan, just the burden that it puts on the developers to know more stuff. But there’s also kind of the implication if we do all the shift to all the left, all the rest of the stuff to the right of that magically takes care of itself. Environments can be changed. Credentials can be mucked about with and stuff once the process has started. So there’s two levels here. Your pipeline, defined pipeline should have that level of governance and guardrails and gates and everything else that goes along. But you still beyond that have to look at all of the other moving parts around there and attest that everybody did what they are supposed to do. Because at the end of the day, it’s folks like Sonya or maybe Sonya’s boss who have to sign the paper that said this meets our risk profile and we can go forward. That still has to be a separate thing. But you need that visibility outside of the pipeline itself to be able to see it and control that.

    Sonya Lowrence: Yeah. And to speak to that, I feel like again I’m going to come back to people aspect of this challenge. One of my biggest partners in any company that I work at, especially when you start looking at the program and platform level, future state, making this inherent in the design are architects or I know that that’s not a role that’s very frequently filled at smaller companies, the group or team of leaders in the engineering org that fill that role of architect. If I can work with the architects upfront, again, it becomes inherent in the design. I agree, Alan, I do think it’s exorbitantly way too much data or knowledge for one engineer to consume. But if you can plan your builds for containers or images and get things like that in place with the right team leads, a lot of this comes together as an affect where they are working within a very specific, still flexible. You can still achieve the future and growth with customers that you need. But at least the infrastructure around them is really secure. 

    And a quick data point that I have because I’ve been through I think two or three SOC that I’ve looked at the secure control framework, 70 percent to 75 percent of SOC 2 audits are about the infrastructure controls. It actually doesn’t directly deal with the codes that the developers are producing for the actual app. And I feel like within the industry, if that understanding were more common and the DevOps team really supported and they had the right tools for secured builds and really making sure they could see things before going to production that were insecure at that layer around the perimeter and with user access, I feel like most of these issues around certification and compliance, there are baselines, would actually be met more consistently.

    So I feel like there’s also kind of a challenge right now of a lot of people look at those product engineers as that’s the source. Those are the people with the root cause answer. They are the ones coating the solution. But the truth is, it’s your infrastructure team in the Cloud. And that’s really something that I’ve seen on premise a little bit different. But it’s still the same statement. You’ve still got networks and IP and ports and perimeters to control. So I just feel like the infrastructure piece of this is typically left out and then it becomes, unfortunately for many companies, the tech debt when they start looking. 

    And I’ve coined this phrase recently, I’m calling it, it’s not a moment of realization. It’s becoming a moment of terror because when you do get the right tools and you can see it, everyone steps back and says I had no idea this is what we had built. And it takes a lot of fortitude to walk through that backlog, that tech debt.

    Alan Shimel: Does anybody want to comment on that?

    David Kirsch: Well, yeah. I had the same feeling that you do, Alan, about dev sec ops and we are going to have these amazing – there are a lot of amazing developers, developers who know everything there is about security. Everything there is about operations. Everything there is about compliance. It’s not realistic.

    So instead, part of it is, I think, building guardrails. So I’m sure Sonya is working on that. What sort of data are you going to allow your developers to put into the Cloud, for instance. Because at the end of the day, we’re talking about data often times in terms of compliance. So what kind of data are you going to allow in the Cloud or out of a certain region? When you’re testing applications, are you going to allow the testers to see all of the data? Probably. Probably not, I don’t think you want them to see all of your production data. What sort of role-based controls do you have in place? So setting those foundational guardrails, I think, is really important to keep the whole DevOps process clean and compliant.

    Alan Shimel: Okay. Go ahead.

    Tim Johnson: I would take that a little bit further back to the first principles of the – and this gets into culture, this gets into cooperation, this gets into silos, this gets into doing it all right is starting with a description. Can you describe your software delivery process? If you want to call it a value stream mapping exercise, call it that. But you get everybody in the room to describe how this software moves through the system then agree that’s how it’s done. And then agree on how to measure that. And then the final piece of the puzzle that I see overlooked way too often is that everybody’s compensation plans or bonus structures or measurements have to be aligned on that description of what that is. Because I’ve seen companies where the three different groups, we are all in on DevOps but the engineers are doing things differently and the security folks are doing differently and the DevOps teams are doing differently because they are measured differently. They agree in principle to doing DevOps but because they are compensated this way and they are measured over here by something else and something else, they are never going to be as successful as they are. So if you want to make this whole process as friction-free as possible, get everybody to agree and one of the things we’ve always seen in these value stream mapping exercises is somebody says and this happens and then somebody else immediately jumps in and says actually, no, that’s not how it works. And you have to get those out of the way because that’s going to come back to compliance. Well, it doesn’t work that way. Well, then how can we be compliant and so forth. Anna, I think you wanted to jump in on that.

    Anna Murray: There’s so many things to talk about but when you guys talked, first Dan and then you, it comes back to culture. It comes back to leadership. And I think it reminds me of when we were making our transition from Waterfall to Agile. It’s this go faster, go faster, go faster. And there’s more emphasis on go more faster. Go faster, go faster. Wait, slow down. We actually go slower if we don’t do the right planning. And so there’s absolute proof that if you plan well, you will actually go faster. And part of that planning must include your security and your compliance. The plan is, you can go back to your Agile definition of done. This is done when. And part of the when is your security rules. It will pass these tests. It will go through these scanning tools. Whatever your definition of done is, that’s how you make DevOps help you with compliance is you first start with the right plan. And that only happens if you have cultural leadership. And like Tim said, you get everybody in the room and you agree.

    And so it is a big challenge and I think more than anything it comes back to leadership because the people who built software love building software. They love writing code. They love delivering bug free software. That’s how they get their value for themselves of what I do every day in my life. If I deliver code that gets hacked, that puts my company on the news, I’m not very happy with myself.

    So when you think about it, leadership can put things in place and help teams go where they need to go while leaving the guard rails in. But if the leaders are not on board, you end up with this fracturing that Tim is describing.

    Alan Shimel: So to me what we are describing here, though, is quality. No one wants to produce shoddy work whether it’s secure, insecure compliance, really, we’ve really strove in security to make security synonymous with quality. If you are delivering a quality product, and whether that’s software or a guitar or whatever it is you’re selling, you want to deliver quality.

    For many – and I’m going to put forth this. For many developers and even testers, quite frankly, and folks, other folks in the software development lifecycle, security and certainly compliance are not quite synonymous with quality. Security is other to them and compliance is other’s little brother. But it’s not quality. It’s not quality. And I think that again, that’s one of the promises of DevOps is making that quality.

    I’ll throw just something else out here because it’s at the heart of DevOps and kind of this whole way of looking at things. Infrastructure is code. You know Sonya, you made an excellent point that if you set your infrastructure up right, compliance almost does become automated and easy because so much of your compliance is based on that infrastructure.

    But in the world where infrastructure is code, is code, not as code, is code, that can be changed, that can be duplicated, that can be cloned. When I first started staging-devopsy.kinsta.cloud, puppet and chef ruled, Britannia ruled the waves because you can create that same instance over and over, that same configuration, that gold config that was rigid, it was secure, gave me compliance. That was one of the big benefits here.

    Sonya Lowrence: Absolutely. I feel like that’s why it ties back in now, like at this point in the conversation, naturally to a tenant of a privacy that is becoming more and more standard in many countries. That’s security by design is the expectation. In the world of Cloud, some of that challenges, you’ve got a lot of options and not everyone understands on that front how to build it securely. There is still a lot of base designs out there that don’t necessarily have the configuration controls that maybe they could or should. That’s debatable. But I feel like that’s the challenge here is there’s a huge explosion of growth in Cloud environments and the engineers who understand the secure tenants, architects who know how to build it consistently to those standards are lagging a bit. They are learning, too, as we go. 

    So I feel like again that’s why these tools, and going back to the understanding that if you were to look at that process like you mentioned, Tim, it’s really important to understand the SDLC. And I recently did this just for a dec. A very simple here’s how many times you have to touch code or infrastructure if you put this up at the front in a beautiful DevOps environment and approach, it’s one hand off and then your security team, ideally, is watching for vulnerabilities. But the truth is, I think most companies end up touching code or infrastructure 10 to 12 times before they see the vulnerabilities to deal with them. 

    So I feel like the more we can find the right tools to actually put this up front or working with the architects or leads to really build out that future state environment and build these automation points, we will continue to see this challenge. And it is very, very challenging for any one developer to account for all of that. These are many layers of expertise that we are touching on today.

    Alan Shimel: Absolutely.

    Tim Johnson: One of the things, Alan, I saw a statistic last week and I apologize, I can’t remember the source. But to your comment about infrastructure is code and repeatability and we can get into the concept of immutable objects and the pipeline and all that other stuff. There is a potential fallacy or vulnerability there because yes, you were starting with that. You are using this approved component and its repeatable. But the statistics said that only half of the controls from a regulatory standpoint actually covered infrastructure is code. So there were other things that were happening that your IAC solution or methodology or component wasn’t going to see. So that’s – okay, you started well. Did you continue well and did you end well is kind of the thing that you have to kind of keep along and keep in mind as you go through that.

    Alan Shimel: I don’t disagree. Mitchell, you’ve been letting people talk today. I’m interested in your thoughts.

    Mitch Ashley: Absolutely. You’ve got a great panel. It was just thinking about several things. One thing Anna said was talking about culture and leadership. And I’ve learned the hard way. Automation does not fix culture. One of the toughest things –

    Alan Shimel: You can’t automate culture?

    Mitch Ashley: No. I mean, automation doesn’t solve your culture problem. And one of the hardest things to do is to try to automate across silos. That’s something nearly impossible and you’ll end up with a mess. And so I think we kind of have to shed a little bit of the paradigms of thinking of these as fixed objects. Compliance is not fixed. It’s constantly changing more quickly. Tim and I know this from some of our research together. DevOps is speeding up things, Agile is speeding up things. We are changing our toolchain. So it’s almost like instead of thinking about we are building a highway, it’s like we are trying to do compliance for something that’s constantly in motion, that is flowing by like a river. And so we have to be really good at things like understanding the changes that are happening and understanding the data that might be helpful. Maybe it’s not the panacea that solves all of our problems. The data that we are already throwing off or can capture as part of our software development process. And understanding the tool changes and I think configurations are going to change that whether it’s infrastructure’s code or we plug in a different tool. And that’s kind of the mindset we need to adopt whether it’s compliance, security or DevOps and software.

    Alan Shimel: Yeah. Might very well be. Let me defend the common man developer, common man or woman developer. As a security person, I fancy myself a security person. I am bewildered and befuddled by the amount of compliance regulations that are out there, depending where I am, who I’m talking to and what I’m doing. How can we honestly, earnestly, expect the developer to untangle this mess?

    I’m reminded when politicians hold up those big – these are the regulations we cut this year. And they slam down this big fat book of regulations. The regulations never go away. How is a developer supposed to make heads or tails of this, though? Or maybe they shouldn’t and that’s DevOps job?

    David Kirsch: Yeah, I mean that’s really the job of different tools and platforms because you have to codify those compliance, those huge documents that what are the 1500 actual requirements? What is Regulation 43BC Subsection R? What does that actually mean? And that means you’ve got to check this little thing of code. And so that’s got to be automated into the platforms to make it easier because it’s just unrealistic to expect developers to understand it but they are not paid to care about it. They are paid to put out really high quality software. And that’s what you’re paying them for.

    Tim Johnson: A lot of money.

    David Kirsch: You’re paying people like Sonya to figure out the compliance problems and you want the developers to put out often software.

    Alan Shimel: But you know what? Anna, Tim, that sounds like your job. You guys got to be making products that do that. How do you do it?

    Tim Johnson: I’m very difficultly avoiding a commercial for the products I’m associated with right now.

    [Laughter] 

    Tim Johnson: There’s other aspects to this which you haven’t really talked about is the context of it, of whatever issue that you find. And just what the heck is the programmer or developer supposed to do about it? So there’s any number of securities scanning tools, stass, yast, blast, bast, cast, whatever. And they all come back with this storm of alerts. There’s scary things happening. Developers are what the heck do I do about it? The compliance officer, like my colleague here, Sonya, says what the heck does this mean? Because it’s just as challenging for the compliance officers and security people as it is for the developers. And you use Bob’s software that has some AI magic that says we determined this is a threat level of 563. What does that mean? And some other Joe’s software says oh, that’s a threat level of 12. And things like that. So when you discover this, what does it mean and what does the context of what it is that we are doing? So you look at things like where is this in the lifecycle? Yeah, it’s an open connection to the Internet but this environment is a test environment. It has no connection to the Internet whatsoever. So do we need to panic about this? No. If it’s in production, obviously, yes. But it should never get there in the first place.

    But also, you’ve got this issue, you’ve got to be able to feed it back to the developer in the tools that they want to work with in a way that they can do something with it. So they don’t necessarily have to understand you’re not compliant with this particular control of CIS 12, you can speak Urdu or Phoenician to them. You get better results. Here’s this thing, here’s what’s wrong. Here’s how you fix it. Go do it.

    Alan Shimel: Anna, what about you?

    Anna Murray: I echo what Tim says. It has to be clearly defined back to the developer where is there a hole? We recently were looking at something ourselves that we had found a problem and okay, wait. Does it affect this – doesn’t even matter? First you have to evaluate it. And it does matter, most of the time, but you have to have your architects in play. Your architects, as Sonya said, she doesn’t even meet with some titles called architect. If you don’t have somebody in that level, you don’t have anybody you can look at the whole product and say what the threat level is because of your product and what it’s meant for. And the access that can be to that software for it to cause a problem.

    So that’s where you really have to have those people that are at the architect level, whether that’s their title or not, and they are interfacing with your compliance people and understanding the things that need to be architected in and you have to have standards. You have to have them codified into your tools. And we are scanning and we choose to use this solution from CloudBees or that solution from Rocket. And we’re going to make sure that we meet this threat level that’s defined by that product and then your compliance people review it and decide whether it’s good enough or not. And if not, up your game. But automate it, give the reports back and you better hope those reports, as Tim said, tell the developer what’s going on. Because really, these things change all the time. And developers are good at writing code and making software work and they can make it work the way we wanted to given the requirements. But that comes back to giving them clear requirements. We cannot ask them to do more than that. We want them to be creative, building whatever it is that they are building and we give them requirements to fit. And if we’re not doing that from leadership and architect and compliance, then we are failing, not them.

    Sonya Lowrence: I completely agree. And I would like to add to that. For me, I have seen some organizations where compliance will walk into any design meeting and say here’s my list of 200 requirements and regulations or whatever their source is for their input. And I really would challenge any security or compliance professional, especially in a leadership role, to really think in terms of make it simple. Make it accessible. It is not hard for developers to understand. Data is what we are protecting. It’s kind of like you’re building a fortress around your gold as a dragon. And from there, think about in terms of what you are protecting. It’s the perimeter. It’s who has access to it. It’s principles of least privilege.

    And so I feel like approaching the development team to help them come up to speed and not terrify them with too many details is the right way to go. And it just changes the culture. It shifts naturally because they do understand these principles and they do want to deliver quality code that doesn’t have these types of issues both for security and for compliance. So I feel like there’s definitely a lot of improvement in this space for training, too. Really explaining as you go why it matters and then making sure you’re working at the right level, finding those leaders on the engineering side to partner with.

    Alan Shimel: Training around – I’m sorry, go ahead, Tim.

    Tim Johnson: Well, I was going to jump back in. On that standpoint, we’re talking about the developers, we’re talking about a number of people. But part of this equation is the leadership of the organization. And you talked about, Alan, you talked about quality. And is quality enough of an incentive for upper management to really get behind compliance efforts and security efforts. And I would argue that no, it’s not. Typically, it’s kind of nice to have. And I remember statistically, from a couple years ago where managers have said they’d rather go fast than secure. This was only like three or four years ago. That’s scary. I hope it’s changed.

    But one of the things in talking with some of our customers and prospects in the last six or seven months is the concept of a tax, the compliance tax. One customer actually used that phrase. And we go back to this bank that I mentioned earlier. It’s 100 people year-round devoted to doing that. What is your fully burdened cost of that labor to do zero value add work? Right? Compliance is something you have to have but it doesn’t add any value to the organization whatsoever until you don’t have it in a big way. Then it takes a lot of value on. But that’s kind of funny money because you’re going to pay them anyway.

    But if you also add in the concept of what should they be doing in terms of technical debt or the opportunity cost of them spending this time on zero value at work versus value add, so you take that fully burdened cost, what are the expected returns? So if your fully burdened cost is $100,000, I think the company wants you to generate more than $100,000 in value in a year. Maybe it’s $200,000. Maybe it’s a total revenue divided by total employees. Some figure that’s more than what you are paying them for. And you add that into the equation of these 100 people doing this work. That’s the labor costs. Then here’s what we are missing. Here’s the value to the organization we are missing from that. Then you get a much more comprehensive picture of the true cost to the organization and how to justify the efforts of changing culture, doing those meetings that nobody wants to go to and putting in different metrics and things like that because you have an measurable impact that you can have on the organization. It’s like a snapshot on day one and then six months later, you halve the people doing compliance. You’ve got a defendable metric to take to senior management and say this is what we’re doing.

    Alan Shimel: I mean, being able to actually show compliance is, it’s a little different than being able to actually show security where if you did good, no one knows because that means you didn’t have an incident. But with compliance, you can say hey, look. I’m compliant with SOC. I’m compliant with PCI, whatever it is, GDP or whatever you are compliant with. And you can hold that up.

    Anyway, team, panel. We are over time here. I knew this was going to be one of those sessions where we ran over. And for me, it’s a very volatile subject. Sonya, as the true compliance person, not making a product around compliance or anything like that but really where the rubber meets the road, I was going to give you the last word today on today’s show to leave with our audience in leaving with compliance and DevOps.

    Sonya Lowrence: Yeah, absolutely. I think just to summarize a lot of what I’ve heard here today and what I’m trying to really make sure we are influencing here at Tricentis, you know that true realization that along with leadership, compliance is an absolutely important part of the team. And when done well and when done from the right approach is whatever tools you’re working from. You start with what you have, you identify the challenges and you get better. A lot of this boils down to knowing your requirements, following a pretty classic Deming cycle: plan, do, check, act. And I feel like those tenants really have never changed. I just hope that more organizations really start to add this back in within Agile development environments that we all have. So I think it’s very achievable and the tools are catching up quickly to get us there.

    Alan Shimel: Excellent. Well, Tim, Dan, Anna, Sonya, Mitchell, thanks for being on our panel in this episode of DevOps Unbound. We’ll be back, as I mentioned, every other week so in another week or another two weeks we will be back with a fresh panel and a fresh subject and something DevOps related. But until then, many thanks to Tricentis for your sponsorship. Many thanks to you out there watching this. We appreciate your time that you set aside to watch this. And we are grateful for it.

    Until then, though, this is Alan Shimel for Techstrong TV, DevOps Unbound. Have a great day, everyone.

  • Finding Value in AI-Augmented DevOps

    Finding Value in AI-Augmented DevOps

    While artificial intelligence (AI) and machine learning (ML) are emerging technologies, we know they can help an organization parse large data sets and glean actionable insights. But do AI-infused processes actually make a difference in organizations that employ DevOps? The answer is explored in a recent survey by Tricentis. Working in partnership with Techstrong Group, Tricentis surveyed more than 2,600 DevOps practitioners across the globe about their AI-augmented DevOps processes. The results pointed to a bright future in DevOps.

    In this survey, the majority of respondents cited they’re already getting value from AI-powered DevOps processes. A third of survey respondents found AI-augmented DevOps extremely useful today and nearly half found it very useful. Although the technology is still relatively young, a 2021 report by IT infrastructure automation company Puppet Labs found that 83% of organizations are implementing DevOps processes. This rapid adoption speaks to the promise of the technology.

    Interestingly, nearly two-thirds of respondents cited that testing is the area within DevOps where AI will have the most impact. Testing is a major pain point for DevOps organizations because a scaled DevOps environment calls for complex testing scenarios and large amounts of data, and many organizations struggle to scale their automation to the level DevOps requires. Without automation, manual testing can significantly delay releases. In AI-infused DevOps processes, AI can help accelerate the authoring of tests, help testers understand where the risks are, and repair broken tests.

    AI’s Greatest Potential is in Functional Testing

    We typically think of software testing as either functional, where the program enables users to properly execute tasks, or non-functional, where the application is secure, fast and stable. Respondents were asked which disciplines of testing stood to gain most from AI and 65% of them indicated functional testing. Unit testing and UI testing were the sub-categories most often selected.

    The complexity of the functional testing process is where AI shines. Functional testing is open-ended and many permutations are required to ensure proper test coverage. AI steps in and manages those permutations, parsing large amounts of data and providing insights. Additionally, AI can help wrangle production signals that aren’t discovered in a test environment.

    Throughout the survey, respondents specifically noted AI’s promise in regard to UI testing. UI testing is perhaps the most time-intensive and manual type of testing in most organizations. With the many variations of the user experience paths that exist, AI can help mimic the actions of a real end-user, adjusting for application nuances or variations in experience. AI and ML can also boost your build process, increasing the efficiency of repetitive tasks. The ability to quickly identify and resolve issues can improve the effectiveness of your automation.

    While survey respondents see the most potential in testing, they acknowledge that AI-powered DevOps can address other business challenges. More than 40% claim that these processes can reduce the skills gap and empower junior staff to perform complex tasks, improve the customer experience, cut costs, boost innovation and increase efficiency among developers. Nearly 40% also believe that AI-powered DevOps delivers leadership the insights they need for continuous improvement.

    Automation is the Key to Scaling DevOps

    It’s worth noting that among the DevOps organizations that see benefits from AI and ML, most of them are mature organizations; they employ DevOps workflow pipelines, toolchains, automation and cloud technologies, and they’ve automated more than half of their testing. Only 21% of practitioners who are just dipping their toes into DevOps have achieved this level of automation. And even in mature organizations, testing is still the main bottleneck; only 40% of them have automated more than half of their testing.

    The key to scaling DevOps practices is developing test automation that can keep up with an increasing volume and speed of releases. But the road to DevOps isn’t always easy. Practitioners cited obstacles like a lack of tech skills (44%), insufficient budget (25%) and tool selection (19%). Organizations that want to implement DevOps need to rely on external vendors and internal resources to address these challenges and effectively scale.

    Looking to Adapt More DevOps Practices?

    So where does an organization that wants to implement DevOps start? With the big questions. It’s crucial to identify the specific business problem that you’d like to solve. What’s the gain you’re looking for? Where in the software life cycle do you expect to find a benefit? AI and ML can add business value, but they don’t solve every problem across the life cycle.

    After an organization has identified its DevOps goals, its focus should shift to automation. To reap the benefits of AI-powered testing, an organization should automate repetitive functional testing, especially unit and regression testing. Most organizations start by implementing continuous integration and continuous delivery. Once an organization has taken these steps, it will start to amass a large cache of results and artifacts generated from running the test cases, which AI can mine to increase test stability and identify recurring problems. The more you automate, the more you’ll get out of it and the faster you’ll accelerate your delivery.

    Organizations just starting out with DevOps should also understand the distinction between artificial intelligence and ML. Most people refer to ML as AI, but they’re not interchangeable, and most organizations haven’t advanced to the use of AI. A true AI is broadly powerful, while ML technologies are narrow and solve a specific domain problem that a model was trained for.

    When you’re evaluating potential vendors, there are a number of key questions to ask, including:

    • What specific AI technology is the vendor using? Is it an off-the-shelf library or something they built themselves?

    • How are they using AI? Which algorithms are in play and what approach do they take?

    • How do they train their models? Different types of patterns emerge in data sets depending on the industry, the frequency with which data is updated and the application. The data population that the vendor is training on must look like your production data, or the model could take you in the wrong direction.

    • How do they ensure that bias doesn’t come into play? If they train models in a different domain or vertical, for example, you could be an outlier for that vendor.

    • How do they handle failure? How do they monitor for miscategorization and misapplication of the AI in a way that lets you trust the data?

    • Will your data be used to train and improve your learning models? If so, does that raise any privacy concerns? Expectations around data privacy should be explicit and leave no questions.

    • Are there other customers in your vertical with a similar business profile that are getting the benefits you’re looking for? Can the vendor provide examples?

    What Does AI-Infused DevOps Look Like in Practice?

    Consider a financial services organization that has more than 20 applications to manage. With so many applications, it’s necessary to conduct significant UI testing and the tests must be stable. One of the biggest challenges to UI testing lies in locating various web elements; most people start with manual script-based automation, but it can be challenging and time-consuming to create and maintain. Employing AI and smart locators to identify these elements can save time and drastically cut down on UI testing maintenance.

    Survey respondents were asked how they’re currently using AI to augment their testing processes. Thirty-seven percent say they’re accelerating the creation of their automated test cases, while 44% focus their testing on the areas of highest risk. Almost half are able to reduce test case maintenance with self-healing and 43% use AI to identify the root cause of failed tests. Thirty-one percent claim that AI offers insights into test process improvements and 34% use AI to help identify tests to run based on changes within the application.

    Where is AI-Powered DevOps Going Next?

    Going forward, AI will become more and more embedded in testing processes. Great progress with off-the-shelf vendors is being made, and their products are a lot more accessible than they used to be. As the tooling continues to mature, these products will reduce the need for specialized skills. Internal initiatives like data lakes or data warehouses will persist, but there will be less need to build large data science teams to get the advantages of AI.

  • The Definition of Done: Why it’s Important for Success

    The Definition of Done: Why it’s Important for Success

    Establishing clear communications and expectations across software organizations is essential to the success of any project. A company is less likely to experience unexpected delays or other unpleasant surprises if everyone is on the same page. To ensure all team members are synchronized and to eliminate unnecessary ambiguity, engineering teams should create a “Definition of Done” to guide projects.

    What is the “Definition of Done”?

    In Agile software development, the definition of done is a working agreement between teams that defines a set of standards to identify something as “done.” Because Agile is an iterative approach to software development, sometimes it is unclear when a task is complete. Product managers often report that something is ready for delivery when it’s “Dev done” and not “QA done.” In other words, the code has been written, but no documentation or tests have been created or executed. For other parts of the software organization, code without QA or documentation is not “done.” Intuitively, the definition of done exists to eliminate confusion about what “done” means across an organization.

    How to Create a Definition of Done for Your Software Organization

    For some software organizations, the definition of done is a static set of guidelines and benchmarks established at the start of a project. For other organizations, including here at Allstacks, engineers have the latitude to amend the definition of done as they go. That approach empowers an engineering team to make decisions that will produce the best possible results.

    When establishing a definition of done within an engineering organization, consider these tips:

    ● Communicate criteria in a way that the whole software organization can universally understand.
    ● Ensure that the definition of done addresses both tactical and technical requirements.
    ● Create a living document for alignment that can be amended because of unforeseen circumstances, like losing a team member or a significant market shift.
    ● Consider that the robustness of the document will vary by team maturity, and specific teams might need an addendum for special requirements. For example, bigger enterprise IT teams frequently need more detailed documentation compared to two-person shops.
    ● Establish an understanding among all teams that, until all criteria are met, the user story, task or project is not “done.”

    A definition of done acknowledges the shared responsibility between the frontend, backend and QA. Beyond those roles, other stakeholders might contribute to defining “done,” like Scrum masters, board members or business analysts, depending on the organization and its structure. Further, the product owner may also define “acceptance criteria” with goals targeted toward the end-user experience that the engineering team must also address as part of their work.

    How a Definition of Done Can Help Teams be Successful

    The definition of done helps a software organization maintain alignment on projects and keeps everyone on the same page. Misalignment can have heavy consequences. For example, it can initiate premature conversations with customers before a product is really ready, create friction between internal stakeholders and/or risk other go-to-market initiatives. When the definition of done encompasses defined steps for a project down to the task level, Agile teams can create more cohesion across an organization and eliminate unforeseen delays.

    A definition of done outlines each task in a software engineering project. Noting the completion of each task can be an indicator of whether a project is on track to meet established deadlines. By noting the time it takes to complete each task, an engineering team can measure the velocity of a project, and that can help protect due dates or raise a warning flag about the need to shift timelines.

    The Definition of Done Helps Support Software Team OKRs

    Objectives and key results (OKRs) are a goal-setting framework to help senior leadership focus on what matters to the organization. By defining software engineering OKRs, senior developer leaders can keep their teams focused on aligning engineering goals and activities to business outcomes while also giving them room to experiment.

    Creating engineering OKRs involves taking the organization’s objectives and crafting actionable, generalized engineering objectives while identifying the metrics and milestones that drive progress. Translating the organization’s objectives for the engineering team helps them to understand exactly how their work impacts the company.

    One of the key benefits of the OKR framework is giving the engineering team a shared language and vision for success. However, without an established definition of done, OKRs will have a lot of unintentional gaps (and teams will have a lot of misunderstandings). Start with a definition of done to ensure that each team knows what their part is and when it’s officially ready for handoff to another team. Teams will pass the baton smoothly across the organization and there will be fewer obstacles in the race toward release in the market.

    A Definition of Done Can Help Organizations Grow Smarter

    In a small startup with a handful of software employees, establishing a shared understanding of timelines and goals is easy. As software organizations grow, casual chats at the water cooler aren’t enough to keep an organization in alignment. All team members must understand and have input on goals and objectives and those goals must be in writing and accessible to everyone. The benefits of establishing a definition of done include creating a shared understanding and unified language for software delivery, ensuring that new employees have access to tribal knowledge and process expectations, and detecting and removing false agreements between teams to create real consensus. By creating a definition of done early in an organization’s history, company leaders can ensure alignment even as more people join.

  • DevOps World 2022: DevOps Tail Wags the Dog of Digital Business

    DevOps World 2022: DevOps Tail Wags the Dog of Digital Business

    As a methodology for building and deploying applications faster, DevOps has been around for more than a decade. But it’s only now that the methodologies and concepts pioneered years ago are starting to wag the dog of digital business transformation.

    Since the onset of the COVID-19 pandemic, digital business transformation has been all the rage. Organizations of all sizes now have a much deeper appreciation for digital processes that enable them to respond more adroitly to changing business conditions regardless of whether the global economy is expanding or contracting.

    DevOps Wags the Dog

    Not too long ago, ironically, business executives routinely complained that IT simply wasn’t fast enough to keep pace with the needs of the organization. At the CloudBees DevOps World 2022 conference in Orlando next month, it will be made crystal clear that businesses are now adopting concepts originally defined by DevOps pioneers to not only keep pace with the current rate of application development innovation but in many ways actually manage the business itself.

    As Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group, publisher of both staging-devopsy.kinsta.cloud and Digital CxO, recently noted during a podcast, “It’s all about the software.”

    Of course, the shift in the relationship between IT and the business it supports has been a long time coming. The debate over how to close the divide between IT and the rest of business has been going on for decades. Even with the rise of DevOps, it’s been more than 11 years since the Jenkins continuous integration/continuous delivery (CI/CD) platform was officially launched. Today, advances in IT are now happening at a rate that many businesses are struggling to absorb.

    In fact, what started out as small teams of application development and deployment specialists looking for ways to automate the building and deploying of applications has now evolved into a multi-billion dollar sector of the software industry. And it is driving a level of business transformation that hasn’t been seen since PCs running word processing and spreadsheet applications arrived in the 1980s.

    Software is (Still) Eating the World

    Just how big an impact DevOps is going to have going forward is still to be seen. It’s still early days as far as pervasive adoption of DevOps best practices is concerned but, as Marc Andreessen once noted, software continues to eat the world. Today, the bulk of that software is being constructed with DevOps best practices in mind and using everything from containers to serverless computing frameworks that make it simpler than ever to add new capabilities to applications both inside and outside of the cloud. The pace at which technological innovations are being made is only going to accelerate.

    As such, it’s arguably never been more important for DevOps professionals to gather in person to share knowledge and insights. During the COVID-19 pandemic, DevOps platforms evolved tremendously. The teams that use those platforms exercise more influence than ever within their organizations. The best way to appreciate the value of DevOps has always been to learn firsthand how practitioners are employing those principles to achieve a wide range of strategic business goals.

    No matter how large or small the organization, the need to embrace and then continuously improve DevOps processes and workflows is, as always, a pressing concern. Achieving that goal in isolation may be possible, but there’s no doubt in anyone’s mind there’s nothing quite like spending a few days with a few thousand like-minded souls to make attaining those goals a whole lot easier.


    Want to learn more about this and other related topics? Join CloudBees for DevOps World 2022, held in Orlando, Florida, at the World Marriott Center. Use discount code DW22 when registering and get access to all the fantastic offerings, including keynotes, sessions, training and other interactive activities. We can’t wait to see you there for the DevOps Remix!

  • Study: Real-Time Data Has A Transformative Effect

    Study: Real-Time Data Has A Transformative Effect

    We all know that data is the heart of the modern digital business. And real-time data is arguably the next evolution—having up-to-date data can empower energy savings and optimizations to reduce cost. Real-time data processing also aids fraud detection and improves health care responsiveness.

    Especially in DevOps, access to real-time data is necessary to reduce incident response time and inform ongoing SRE objectives. Leveraging performance data can also empower future decision-making and open new revenue opportunities. In essence, real-time data can have a transformational effect on the operations of a digital business.

    DataStax and ClearPath Technologies recently released the 2022 State of the Data Race Report. The study found that the majority of data leaders are using real-time data to create additional value. The report also suggested that working with cutting-edge real-time data technologies increases developer experience, which is good to maintain amid ongoing talent shortages.

    Below, I’ll review the key findings from the study to infer how organizations can effectively use these technologies and how they can mitigate roadblocks that arise throughout the process.

    The State of the Data Race

    Simply put, customers expect real-time experiences. And it’s not only the end consumers—developers are also increasingly accustomed to building in this modality. This is evidenced by the recent rise of asynchronous, event-driven architectures. The survey depicted how fundamental real-time data is to modern software, finding that 78% of respondents agreed that it is a “must-have,” not just “nice-to-have.”

    The investment appears to be paying off. A full 71% of all respondents said they can directly tie revenue growth to real-time data. Not only does it seem to have a transformative impact on revenue growth, but it enhances productivity, too—66% of real-time data-focused companies agreed that developer productivity has improved. Naturally, the report found that developers are the ones most likely to be working with real-time data.

    Although modern databases and real-time data processing tools are more readily available these days (and open sourced), the maturity of data handling is not homogenous. Based on the classifications of this report, “data leaders” made up only 10% of respondents.

    Roadblocks to Value

    The road to data maturity has a few obstacles that could limit adoption. First is data complexity—33% said this is one of the biggest barriers to leveraging real-time data. This was closely followed by controlling data costs (32%), data accessibility (30%) and available skillsets (28%).

    Most notably, a skills gap continues to be a commonly cited hindrance throughout industry reports and across technology niches. Interestingly, a lack of in-house talent rises slightly higher in organizations with a high real-time data maturity. Those surveyed in this category said that their top barrier is finding the appropriate skills in their business unit.

    Cybersecurity is also a top concern—51% say security challenges are a big concern when it comes to the use of real-time data. As the report surmised, the business-critical nature of real-time data raises its worth as a commodity and area for attack. Furthermore, amid new data sharing regulations, IT departments are now under more pressure than ever to uphold data security and avoid the leakage of sensitive personal information.

    Unlocking Real-Time Data

    This area has a lot of potential; 86% of developers at organizations with a strategic focus on real-time data said that the “technology is more exciting than ever,” the report found. Real-time data is beginning to be applied at the edge, where AI/ML can be trained to give a more responsive advantage to modern applications. Yet, organizations must prioritize its adoption and mitigate the roadblocks mentioned above to unlock the benefits of real-time data.

    Prioritizing real-time data will take more than tools—it will take vision. According to the study, those who created more value with real-time data initiatives were found to:

    • Have clear product owners
    • Have business unit accountability of data
    • And have department representatives working together in a cross-functional team.

    As you can see, these foundational goalposts have less to do with technology and more with lucid leadership and collaboration. That being said, an investment in the appropriate tooling is undoubtedly required to achieve real-time data.

    For example, this tooling might encompass NoSQL databases optimized for real-time or Apache Cassandra for high throughput. The transition to real-time will likely require streaming and messaging technologies as well. Less obvious, however, is an API layer to sit between the applications and the database. This element could greatly help open data accessibility in a standardized fashion.

    The State of the Data Race: Final Thoughts

    Working with real-time data can be a differentiator and have a transformative impact. As a result, most developers and leaders see real-time data as an exciting area. Yet, organizations must reduce data siloes to open up data access to realize its benefits. It will also take collaboration between data scientists and business divisions to construct meaningful outcomes.

    The State of the Data Race report surveyed 500 leaders across various industries. Above, we’ve summarized the key points from the survey. Readers are encouraged to pick up a copy for more nuanced data points and further commentary.

  • Tips for Better Governance – Techstrong TV

    Tips for Better Governance – Techstrong TV

    David Williams, VP of Porduct Strategy at Quali, and Charlene discuss David’s top tips to better governance in DevOps. The video is below followed by a transcript of the conversation.

    Charlene O’Hanlon: Hey, everybody. Welcome back to Techstrong TV. I’m Charlene O’Hanlon, and I’m here now with David Williams, who is the VP of product strategy over at Quali. David, thank you so much for joining me. Good to see you.

    David Williams: It’s good to see you, too, Charlene. It’s good to be here.

    O’Hanlon: Excellent, excellent. So, lots of stuff going on in the world of DevOps these days, but one of the things that we have been hearing a lot more noise about is governance in DevOps. And especially with just the insane amount of automation that’s going on to help kind of speed up the processes, obviously, it can be pretty easy for things to kinda go off the rails. So I wanna kinda get a sense from you as to what you see happening in the space with respect to governance in DevOps, and how we can kind of maybe move away from the herding cats mindset when it comes to governance.

    Williams: Yeah. It’s been a journey on DevOps. DevOps, when it was introduced many, many years ago as a concept immediately aligned itself with Agile, so it became only the ability to get things out as continuously as possible, and also as fast as possible. And so what happened is that at that time, things were very much managed. It was small teams doing lots of work. As enterprises started kicking in, with DevOps become a serious strategic way of actually delivering applications to their consumers, the amount of people got involved grew. 

      The amount of teams that were involved grew. Even though the DevOps philosophy was small teams, fast moving, application componentry going out into the ether as fast as possible, it stopped being that quite a while ago, and started becoming much more mainstream. So unfortunately, DevOps grew up a lot, and one of the things that became an issue was governance. And governance from that perspective wasn’t really just about security – y’know, SOC 2 compliance, and making sure that your secrets are held and managed efficiently across the pipeline – 

      but it was also around, how do you understand what people are doing? And what is the interaction that they’re having? And why are they having that interaction? And basically it’s a fragmentation challenge, more than anything else. DevOps pipelines are fragmented, and so how do you get control without inhibiting innovation? And how do you get visibility, having lots of teams that wanna innovate in their own areas, but not necessarily give you that full end to end? So that’s the challenge that has emerged. 

      And I think it’s going to continue to do that for quite a while to come, because the applications are getting bigger and more contention-based in regards to infrastructure sharing. So governance is gonna be the thing.

    O’Hanlon: It’s interesting, because I think of it a lot like a company itself growing out there. When you have two or three people working in a company, it can be very difficult to get a lot of achieved without everybody kinda going all in. But as a company grows, you’ve got more people who are doing the work, but maybe they’re not necessarily talking to each other, and not everybody has all the information they need. So you kind of have to kinda build a technology, or at least a process, around being able to get the information out and helping everybody be on the same page. 

      And it kinda sounds like the same thing is happening in the DevOps space with respect to just having a greater understanding. Not just an understanding of the process, but just, I guess, a better way of enabling the process from an understanding and a communication perspective – as well as just making sure that all the right things are getting done at the right time. So what do you guys propose? What do you see that you believe could make a real difference?

    Williams: You used two words that always make me smile when I talk DevOps. I used to be an analyst, talking to a lot of DevOps practitioners, and the two words that I used to run for the hills were “process;” y’know, that was one thing. And so they wanted collaboration, but they wanted to be able to collaborate without being governed in any way. So I think that that process word is one, and anything to do with “abstraction.” So abstraction’s not a good thing. It takes you away from the core. 

      So what I’m saying to you is that any solution that enables you to get greater control on governance and understanding has got to be able to do it without impeding people’s ability to do their jobs. And it’s gotta be something that is subsumed within their normal breathing in and out. It can’t be something that’s another job to do. That defeats the whole point of, anything that’s to do with lean practice is about removing stuff. We’re the software industry, so the way that we make things less complex is by adding something, y’know; that’s the normal inclination. 

      I think DevOps, when you actually use it, whatever you add into the mix has got to be subsumed within your normal day-to-day activities. So it can’t be yet another thing to do. For example, security has gotta be there, but not intrusive. Why I came to Quali was because I’d spent a lot of time with companies where infrastructure, the ability to provision into the cloud, do all the things, was becoming a bit of a mystery. People would assume infrastructure. 

      So the governance around what you’re deploying and how you’re provisioning your infrastructure not only became fragmented through your pipelines – Because you plan something, and then you code it, you build it, you release it, and test it, et cetera. Each one of those is a very defined area. It might be one person; it might be many teams. But each one of those is a very specific thing you have to do. But wouldn’t it be cool if you could subsume testing into your code? Wouldn’t it be cool if you could consume compliance while you’re coding? 

      So you’re coding, and it’s automatically looking at configurations and modifications, and logging that appropriately without you having to add another layer. And I think the same with infrastructure. I think that the companies I’ve worked for before really didn’t know what was going on, and I worked for DevOps companies that were trying to work out, “Okay, so the infrastructure was there; now it’s gone,” or “It’s there, and it’s modified.” 

      Every commit that a coder does is, y’know, a point in time. “So what has happened between that commit and the next commit?” And so all these stages of infrastructure, how do you build that in? So at Quali, what we looked at is providing a method with which you can let people use infrastructure as code practices, configuration management tools. Do what you need, but it gets automatically subsumed into a model. And the model is what you rate on. That’s what applies the policies around access and et cetera. 

      But it should allow the coders, the release managers, the testers to do what they need, but also it provides sort of like a control plane on top that enables you to talk and be governed, but without that Big Brother type of thing looking at you. So I think that’s the magic about DevOps, is if you add something in, it’s gotta be something that doesn’t get in the way of speed. It’s gotta not provide another barrier. It’s gotta do these things. It’s a hard problem to solve, but if you approach it in a way that assumes that people are going to do things, they’re gonna break stuff – 

      People do that; that’s what they do. They do their own stuff. They don’t like being told what to do. So if you can provide as subliminal a level of control over infrastructure that says, “I know how you’re doing it. I’m now going to provide that same capabilities that we used in build to help the testers understand what to build.” So now you’re providing some consistency across, using the control plane. So you’re using your models, and you’re moving them across. But it subsumes what they already use, so you can add it in. 

      Your infrastructure stack and the way that you deploy it is implemented in the model that has policy, access control, secrets integration. And so that’s really the philosophy, is to make sure that we can provide the end-to-end capabilities, accelerate the delivery life cycle, but not impede the practitioners from what they do, but provide the business with visibility which they do not have today. So I would just follow up by saying cost isn’t a governance thing, but unless you govern it, and you understand why you’re doing something, then how do you associate a cost of the cloud, if you can’t associate it with a pipeline? 

      At the moment, cost is associated with instances, and you’re having to group that together. What if you could provide the model with a value while you’re doing it, the application that you’re delivering on? And if you could do that, then you’re subtly applying what they do at the practitioner level with an application, with an owner, with a cost. So that is a way that governance actually comes in with the access, the controls, without being intrusive. We can basically provide all the capabilities there, and enable the business to get better visibility control over all sorts of things, including, for the first time, cost on the end to end. 

    O’Hanlon: Yeah, that makes a whole lot of sense. And in my mind – please tell me if I’m wrong – but it also means that the developers don’t really have to think about, how shall I say? They don’t actually have to learn more about something that would – If they’re moving from, say, a regular kind of a developer motion into more of a DevOps motion, it doesn’t require them to kinda add on a level of expertise that they might not need otherwise. This kinda does it for them, and kind of fills in the blanks for them, and also kinda helps other organizations or other teams within the organization understand the flow a little bit better, which makes a whole lot of sense. 

      And I think that may be part of kinda the governance conversation as well, is if we all have a greater level of understanding about what is happening at any particular time, we can better manage it.

    Williams: Yeah. I mean, secrets management, people hear a lot about that. They don’t quite know what it is when you talk to them about secrets management. I mean, something like a password or a username would be used maybe throughout a process, and it’s a shared piece of knowledge that you don’t wanna expose. So you keep that consistent throughout your environment, but keep the access controls consistent throughout whoever uses that access throughout the pipeline. One thing that we’ve learned is that DevOps isn’t a thing; it’s an interpretation of a thing. 

      The objective is very clear for people, but how they implement it can differ, depending on organization, application, design, et cetera. So what we also have to do is make sure that people with lesser skills – And I don’t mean lesser skills in regards to infrastructure; infrastructure can be cloud provisioned. But what if they have to manage all the networking components and the database instances, the things that aren’t inherent within a cloud-provisioning environment? 

      So we’ve gotta make sure that as developers of different skill levels interact with the infrastructure, we can fill the gaps for them, so they can do what they need to do, but if there’s networking, et cetera that needs to be managed, then the model should be something to give them the guardrails with which to implement that in support of the application requirements to that part of the pipeline. So it is something that we’re working on quite heavily, to make sure that we can get those guardrails in place.

    O’Hanlon: That’s great. And I think about all of the things that have kind of expanded DevOps, if you will, beyond the original parameters, including the implementation of DevSecOps, and the adoption –

    Williams: Yeah.

    O’Hanlon: – and DevNetOps, and all of these things that have kinda piled on top of DevOps. 

    Williams: [Chuckling] Yeah.

    O’Hanlon: It can be very, very difficult for the DevSecOps community, or the DevOps community, especially developers, to kinda have to manage all of that knowledge. So having those guardrails in place, I imagine, is going to be very well received by a lot of organizations, simply because we’re also talking about the dearth of talent in the developer community these days.

    Williams: Yeah.

    O’Hanlon: So anything that an organization can do to help ensure that there is a level of governance around the way their DevSecOps tools and practices and culture and everything is run, I think is a huge benefit. David, thanks very much for walking me through what you guys see happening with DevOps and governance. And hopefully this is an issue that is going to get better as we move forward. To your point, we do tend to throw technologies at problems, and hopefully this is one that will stick, if you know what I mean. 

    Williams: Yeah.

    O’Hanlon: So thanks again for your time. I do appreciate it.

    Williams: Right. Thank you, Charlene. It’s been a pleasure. 

    O’Hanlon: All right, everybody. Please stick around. We’ve got lots more Techstrong TV coming up, so stay tuned.