staging-devopsy.kinsta.cloud

Author: Mitch Ashley

  • SAST vs DAST: What’s the Difference?

    SAST vs DAST: What’s the Difference?

    One of the most effective practices for ensuring that your software is secure and safeguarded against security vulnerabilities is using the right secure coding tools — like SAST and DAST.

    The Key Differences Between SAST and DAST

    Both SAST and DAST are used to find software security vulnerabilities in your code. However, these DevOps tools are used at different times during the development process.

    Here are the key differences between SAST and DAST.

    SAST:

    • White Box Security Testing
    • Source code is required.
    • Vulnerabilities found earlier in development and are less expensive to fix.
    • Unable to identify timing- and environment-related issues.
    • Generally, supports all kinds of software.

    DAST:

    • Black Box Security Testing
    • A running application is required.
    • Vulnerabilities found later in development and are more expensive to fix.
    • Can identify run-time and environment-related issues.

    The Main Advantages of Using SAST Tools

    The main advantages of using SAST tools include the following:

    • SAST tools find coding issues by looking for known vulnerability patterns in internationally recognized coding standards for quality, safety, and security — such as CERT, OWSAP, and CWE.
    • SAST tools detect and fixes defects early in development. This leads to lower costs to fix defects.
    • SAST tools often features a Shift-Left approach, which enables analysis to be done anywhere — including on your desktop and in your CI/CD pipelines.
    • SAST tools are easy to automate, able to effectively scale to your project, and automatically provide the highest levels of code coverage.
    • SAST tools provide fast feedback along with the exact location of vulnerabilities and their cause.

    The Main Advantages of Using DAST Tools

    The main advantages of using DAST tools include the following:

    • DAST tools analyze the entire application as it runs within the full system environment.
    • DAST tools are able to “look inside” your application and dynamically analyze execution logic and live data.
    • DAST tools are language and source code independent.
    • DAST tools check for memory consumption and resource use.
    • DAST tools attempt to break encryption algorithms from outside of your program.
    • DAST tools verify permissions to ensure the isolation of privilege levels.
    • DAST tools check for cross-site scripting, SQL injection, and cookie manipulation.
    • DAST tools test for vulnerabilities in third-party interfaces.
    • DAST tools understand arguments and function calls.
    • DAST tools record application execution for post-mortem test failure analysis.
    • DAST tools catch hard application failures.
    • DAST tools perform unattended script-based dynamic analysis. In order for an effective security program, you will need both a SAST and DAST tool.

    To read more, please visit: https://www.perforce.com/blog/kw/sast-vs-dast

  • ITIL 4 in the Age of Agile and DevOps, With Axelos

    ITIL 4 in the Age of Agile and DevOps, With Axelos

    ITIL is well-established as the process, certification and information resource library for service management across many IT organizations globally. With the recent updates to ITIL 4, influences from Agile and DevOps are helping ITIL to expand and adapt to changes in ways to create software and build operations into the delivery process.

    In this DevOps Chat, Akshay Anand, product ambassador with AXELOS Global Best Practice, and Jon Stevens-Hall, principal product manager at BMC Software, join Accelerated Strategies Group CEO Mitch Ashley to discuss the new innovations in ITIL 4. As major contributors to ITIL 4, Akshay and Jon share the approach they took with ITIL 4, incorporating ideas from the world of Agile and DevOps.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone, this is Mitch Ashley with staging-devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. I have the great pleasure of being joined by a couple of really cool gentlemen here with a great topic I’m excited to talk with you about.

    The first is Akshay Anand, who is Product Ambassador to AXELOS. Also, we have with us Jon Stevens-Hall, a Principal Product Manager with BMC Software. Welcome, gentlemen.

    Akshay Anand: Thanks for having us.

    Ashley: Yeah, I’m excited to talk with you about this. We’re gonna jump into ITIL, ITIL 4, and some of the work that you’ve been contributing to that, but first, why don’t we start out—Jon, would you introduce yourself, tell us a little bit about what you do and a little bit about BMC Software? I’m sure many people know that, but just for the folks who don’t?

    Jon Stevens-Hall: Sure. So, I’m the Principal Product Manager for BMC’s enterprise ITSM suite, which, you know, it comes from the Remedy heritage family of products, now titled BMC Helix ITSM, and I’ve been around service management for over 20 years now. I sort of walked out of university into my first job in the late ‘90s, implementing systems for ITIL 2 over at what became Yell Group.

    And so, I’ve been around service management and ITSM for all that time, and obviously, my primary role at BMC is producing and creating software and tools, but also, we always try to be active members of the ITSM community in general, which is what led me on a personal basis to being one of the contributors or ITIL 4.

    Ashley: Mm-hmm. Awesome, excellent. And Akshay, how about you?

    Anand: Oh, so, I joined AXELOS about four years ago. AXELOS, for those who don’t know, is a joint venture between Her Majesty’s Government and a private sector company called Capita. We like to see ourselves as the custodians of the IP, of not only ITIL, but other best practice arrangements like PRINCE2, which is about project management or MSP which is about program management. I’m Product Ambassador, so like the product evangelist for ITSM and ITIL in general. My background is mostly around consulting in the IT Service Management, consulting and advisory. I’ve also been Head of Service Management for Macmillan Publishing for a couple of years. And yeah, like I said, I joined AXELOS four years ago and it’s been a blast.

    Ashley: Excellent, excellent. Well, I’m sure that consulting, and also Jon, your experience with BMC gives you some real-world perspective on ITIL and ITIL 4.

    Well, talk about the publications that you’ve contributed to the IP. I think the Create, Deliver, and Support around ITIL 4 is one of the areas that you both contributed. Do you wanna jump in first, Akshay?

    Anand: Sure. So, Create, Deliver, and Support is one of four books in a learning stream that we call Managing Professional. All the books in that sort of learning stream are oriented toward managers or professionals who are either managing or designing systems, service management systems. Create, Deliver, and Support in particular talks about things like employee engagement and professionalism, a lot of the commonly used tools needed to create, deliver, and support services, things like value streams, value stream mapping, how to apply value streams within the context of ITIL 4, how to manage suppliers.

    So, it’s a very wide-ranging book, but it goes into a lot of detail about very, very specific techniques. Two of those techniques were actually authored by Jon, and one was around tickets and the use of tickets or the misuse of tickets and what to avoid, and the other was actually around swarming, and how swarming can be a good model to use within service management organizations, a technique that can be used to foster collaboration between development and engineering teams and frontline support teams.

    Ashley: Interesting. Jon, love to hear about that. I know, just to share you a bias about tickets, I think everybody’s impression of IT is, you can’t talk to IT unless you complete a ticket. Well, maybe that’s taking it a little bit far, right? [Laughter]

    Stevens-Hall: No, that’s definitely a true impression. I mean, one of the things I’ve spent much of the last six years doing is really, I’m a bit of a method actor, I’ve been very immersed in the DevOps community. And almost, I joke that I went to my first DevOps conference by accident because I signed up for something called Configuration Management Camp thinking, you know, ITSM, CMDB, configuration management. I soon learned very quickly we were talking more Puppet and Chef and Ansible.

    Anyway, and it was fantastic, because it’s opened up a door into a world that has evolved somewhat differently. And there were many aspects of culture that I think IT Service Management was not understanding very well, which was one of the reasons I was very keen to bring my kind of cross-domain perspective to ITIL 4, because I think a lot of us felt ITIL—bear in mind that version 3 originated before DevOps, it needed a big refresh. And that example of tickets absolutely is true. I mean, people sort of see this negative perception that, you know, to get infrastructure, we used to have to raise a ticket and wait three weeks—now, we can log onto a software-defined infrastructure tool and we can create it right away and it’s all delivered on cloud.

    And absolutely, I mean, one of the—you know, we sometimes, I think the heritage of IT has created these impressions not in a vacuum. You know, they’re not false impressions sometimes. But I mean, ultimately, what is it? You know, a ticket is just a data record that represents a piece of work, and a lot of the work we’ve done around developing systems at BMC has been to try and sort of take away this old way of very sort of skeuomorphically presenting a form to somebody who did all the fields on that piece of work, you know, all the things about that.

    And I wrote a blog on this a year or two ago where I likened it to an old taxi booking service I used to use in California if I flew to work, where you had to kind of fill in about three pages—box, box, box, box, box. All those things are important pieces, probably, for the taxi operator, but you know, this is the world where I can open up my smartphone and press one button and get a taxi, you know? I don’t have to care about all that stuff.

    Likewise, I think where we’ve had to move on is to sort of—you know, what we’ve had to learn as a community on the service management side and maybe, you know, have to convey more to the DevOps world is that, you know, we all record units of work, we just don’t have to necessarily present it in such a complicated way. And we can,  you know, we should focus on the actual flows and the people and not this sort of big-ticket data problem. You know, we probably still want to know that you’re deploying infrastructure, but we don’t have to make you fill out a great big form to do it.

    Ashley: Well, and software engineers kinda think in terms of Kanban cards and stories and epics, you know, applying agile kind of things. Have some of those concepts, then, creeped into the idea around tickets within ITIL 4? Is there some intersection of those ideas at all?

    Stevens-Hall: Yeah, I mean, [Cross talk]—go ahead, Akshay.

    Anand: Oh, I was gonna say, in general, I think we definitely had a lot of inspiration across the entire ITIL 4 contributing team, whether architects, authors, reviewers, et cetera. There were definitely a lot of people who pulled on a lot of lessons learned from the world of agile software development, of DevOps, and of lean, and we tried to incorporate a lot of that thinking into ITIL 4.

    It may have been there in ITIL version 3, but it possibly wasn’t at the forefront, it wasn’t embedded at the heart of the framework. But what we did with ITIL 4 was to say, “Look, these are important, you know? We need to be able to reinforce continual improvement, we need to be able to reinforce elimination of wasteful work. We need to reinforce, you know, that we’re working in complex environments.” Especially as we’re working in environments which are, at this level of service management, is a mix of human and non-human interactions, and that is inherently complex and we need to reinforce things like iterative progression and the role of feedback in such a system and so on.

    So, we brought a lot of those lessons in. Whether that was sort of very foundational models that we used in subsequent publications to present and frame certain arguments or whether that was highlighting specific techniques like Kanban or—I’m drawing a black, sorry, I haven’t had enough coffee. But highlighting specific techniques like Kanban and others in other publications.

    So, there’s another publication called High Velocity IT, and coincidentally, Jon was a contributor to that as well. And in that, we talk about how certain techniques can be used to meet certain objectives in high-velocity service organizations like fast development or creating faster feedback loops at the service level. Sure, we have it at a technical level, but we also need to be able to mirror that or replicate that in some shape or fashion at the service level. So, we definitely try to bring in as much of that foundational thinking from lean and agile and other best practices or emerging practices and bake that into ITIL 4.

    But for the specifics on how we did that is, with queueing and swarming and so on, that’s a lot of the stuff that Jon helped author.

    Stevens-Hall: Yeah, and I think one of the big—I mean, taking a step back from just the concept, you know, the rethinking about things like tickets, one of the things I’m very happy, one of the reasons I was happy to get involved with ITIL 4 was that it’s shifted a lot of thinking, and again, learning from DevOps and lean in particular and making us think much more about value than process and control. And, of course, that’s been a challenge for us to position that we’re thinking that way, but ultimately, I want to sort of talk about the positive things.

    You know, I talked to developers in the DevOps world who see huge value in people taking some of the leg work off their workload, you know? So, we all know that those developers are delivering the most value if they’re innovating the developing. If we can bring service management techniques, better align it with DevOps so that any of the support tickets that are coming the developer’s way, you know, the issues that inevitably come up when the—and especially as DevOps in enterprises goes from sort of small little startup exercises to big and normal and widespread, suddenly, those developers often find themselves doing half of their work resolving issues that have come in, you know, come in from the monitoring or from people. And one thing that service management has done extremely well for a long time is help to kind of industrialize and produce high quality support channels.

    So, what we are in a great position to do is actually, you know, with that focus on value, you know, in a lean culture, if it’s not valuable, why the hell are we doing it, by kind of moving on from some of the old concepts, but taking all the things we’ve done very well and working with the DevOps community with a much better understanding of the way they’ve learned, you know, drawing up their lessons, but at the same time, pushing the fact that there is huge value that we can deliver to help them do what they do, then, you know, suddenly, this IT service management and the ITIL framework, I’m in a much better position to help those things. And I’m seeing people who are very skeptical before now saying, “Actually, this is how we unlocked the organization’s confidence and give them that ability to go faster with DevOps.”

    Ashley: You know, it’s really heartening to hear that’s the approach that you both have taken with this, because it seems, you know, there’s sort of the impression of process for process’ sake where, no matter what we’re talking about, and of course, you know, many folks that are in this highly iterative—right, there’s process in the way we create software that way.

    But if the service management and the software creation processes can better mesh together, right, so that it’s not viewed as an impedance when someone gets a ticket, right, and has to conform to an old process—but it flows with how we develop software and release the change that might go through that. I think maybe that’s that continuity that we start to see between ITIL and DevOps and lean and agile to help everybody see how it can benefit what we’re doing.

    Anand: I remember a conversation with a gentleman called Greg Sanka, who is the CIO of Oregon State Administrative Services. This is a conversation maybe last summer, and we were talking about the guiding principles that we introduced in ITIL 4. In particular, we talked about focus on value. As a principle, we need to be able to focus on value—Jon has just referred to that, as well.

    But what we are talking about is, nobody—very few teams are consistently sure on what value means and how value changes from person to person. So, of course, there’s value for customers who are using the software and the applications that we’re deploying. But there’s also value for your risk officer, there’s value for your supplier, there’s value for your CFO.

    Now, a lot of these people aren’t involved in the actual use of the software day to day, they’re not involved in the development of the software, et cetera, but we’ve got to be able to architect the solutions, the service management wraparound, et cetera, et cetera to be able to create that value.

    Now, part of the reason why sometimes IT service management and ticketing that you were mentioning has a bad reputation is, that’s how that assurance was created at a certain point in time.

    Ashley: Mm-hmm.

    Anand: But we now have to go back and say—look, because technology has changed, we’ve removed certain risks that we were trying to mitigate 10 years ago, but we’ve introduced new risks that we didn’t have 10 years ago.

    Ashley: Mm-hmm.

    Anand: So, what does that assurance now need to look like? How do we re-architect our systems to accomplish that same assurance? How do we introduce more automation, remove the toil, et cetera, while providing—in order to provide that assurance?

    Stevens-Hall: Yeah, it’s almost like the ticket was almost the kind of, the manifestation of the internal memo in a digital system, you know, the old paper memo. I mean, I’m old enough to remember those envelopes where you crossed out the old address and put the next one on and it went around the internal mail system.

    But, you know, it is important to remember that these processes actually helped us to move that kind of customer support in the technical world from adhocracy and Post-it notes to a proper, robust, scalable solution. That moved us on to self-service, to knowledge management, to properly established professional ways of handling and resolving issues that are now extremely valuable when DevOps goes big in organizations and it hits the mainstream and it hits a large volume of users and starts having to deal with support issues.

    The more staff we can use these established service management practices—albeit modernized, heavily rethought, you know, focused much more on value than process for process’ sake? The more we can do that, the better—and this is proving itself over and over again with our customers—the better it actually helps us to enable DevOps to go, you know, to accelerate, to deliver that huge value that we know it can create.

    Ashley: Well, it seems like the, some of the process and the auditability, if you want to think about the benefits of why you have processes, right, so consistency and reaction to issues and resolving them—those are some things that can actually benefit people that are working in a philosophy environment, because sometimes that can be a bit chaotic, or it’s just extremely fast. Obviously, things are happening very quickly.

    And then when you need to start interfacing with other parts of the organization, whether it’s, you know, do we do vulnerability scanning for the security group, you know, how are we handling resolution time for production issues? That’s where that intersection can really be helpful, and some of those well-defined processes and ways of measuring value from an operational standpoint, development teams don’t have to do that, because you’ve already done that through, or are in the process, right?

    Stevens-Hall: Yeah, and also one of the good things in ITIL 4 that we’ve been able to do is actually learn very much hands-on from DevOps. So, one of the big subject areas that’s really growing in interest on both sides, as it were—and I kept on using the word “sides,” but you know where I’m coming from—is this idea of using a more swarming technique than kind of the traditional structures.

    And so, this is a really good example because, you know, very traditionally, service management has kind of operated a three-tier support structure where you’ll kind of have a front line at tier one, which we’ll try and probably mostly successfully fix most minor things, you know, then if they can’t, they’ll escalate to a more expert level, tier two. You know, back in the day, that would’ve been the service desk agency who then did their Microsoft certificates, for example, and became a slightly more specialist team.

    And then things that can’t be solved there, you know, 5, 10 percent or whatever would end up with tier three teams who are more likely to be developers, specialists. There’s nothing wrong with having those teams, but what is wrong is a couple of things. You know, the first is that issues can often take a long time to get to that team, because what we’re effectively doing is putting queues in the way, you know, so it goes into the tier two queue. Someone then has X hours or days to pick it up, so it sits in that queue. It then goes to tier three and sits in a queue, and by the time it’s got there, we’ve already got an angry customer, even though everyone might have met their operational agreements, the other thing that happens is the sort of ping pong effect where, you know, if it goes to a tier—these things that get, especially, to the tier-three level, they might need input from several development teams. They might need a database team to have input to it, they might need a network team or whatever.

    And with these kind of traditional silos, although they’re, if the ticket’s not gonna leave that silo, it’s absolutely the right place for it to be if the issue, I should talk issues, but things ping around and you end up with problems like champions emerging who’s the one person everybody knows can fix it every time, and so they pay their mortgage off in overtime, but they always look a bit unhealthy, and then they leave the company and everybody’s in trouble.

    So, swarming has helped us, actually, it really is taking much leaner techniques. It’s about sort of being more dynamic in the way things are brought together as a team, you know, tearing down some of these silo structures, and in particular, removing queues of unfulfilled work in progress.

    Ashley: Describe for us—

    Stevens-Hall: And that way, we’re really learning directly from lean, we’re really learning directly from the DevOps focus on having much better flow and reducing handoffs and, in particular, not letting things build up in queues.

    Ashley: For someone who’s not familiar with swarming, describe how that would work in this context.

    Stevens-Hall: Okay, well, I mean, at BMC, we’ve done this in our own customer service team for a while, and what it effectively means is a couple of things. You know, you’ll always have kinda, the most severe issues have always pretty much been swarmed everywhere. And that’s, you know, everything’s on fire, so we’ll get everybody we need into a room and we’ll sort it out. And so, that’s always kind of been the most basic form of swarming, even though we haven’t really called it that. And that’s not really where the novelty is.

    For us, the novelty is, we’ve collapsed that tier one and tier two, so instead of having that kinda chain where somebody fairly qualified takes a look and then if they can’t fix it, somebody more qualified takes a look, we kind of put those two people together. And instead of waiting for things to trickle through queues, we look at the things as they’re coming in, but the support people will, maybe every hour, just look at what’s coming in that hour and be very reactive—okay, we can fix those three things, we’ll just get it done—will qualify the rest very well. And you get this lovely sort of information sharing between the two levels as well. So, they learn from each other. I think pair programming is an interesting analogy from the modern software development world.

    So, that actually increases the flow very significantly through those first—you know, it gets things fixed quicker and it increases the throughput. Then when things do go to sort of the tier three product specialist groups, they’re not allowed to start assigning it around other tier-three groups. So, if they’ve got one of these more complicated issues, instead of doing that, they’ll pull together a group of people into a swarm and work it that way.

    And it’s all—they’ve got a lot of leeway to be very self-organizing which, again, probably doesn’t sound like the impression that service management have always given. So, it’s not that rigid, you know? They will figure out what works best for their team. So, we have some teams hold three sessions a day at fixed times and people can come in with issues. We have others that spin things up on a more ad hoc basis, but the interesting point is there, again, things are not going from bucket to bucket, you know, queue to queue, they’re being owned by the person who takes command of them and we’ve found there’s really great engagement, people coming together in a Microsoft Team session or on a call or face to face way back when. And again, it looks a lot more like the kind of practices that have evolved in the DevOps community. So, again, we’re learning to sort of focus on delivering the value rather than focus on hearing sort of a pre-defined, rigid process.

    Ashley: Mm-hmm. I’ve been there, actually. I know you were trying to—I think I may have stepped on what you were gonna say, so jump in.

    Anand: Oh, no. I think it works. As Jon was saying, you know, back in the day or probably even still to this day, you know, every time there’s a priority one incident or a severity one incident or one of those really big things, you know, everything’s on fire type of thing, you know, we used to call it a war room back in the day, we used to have conference breaches back in the day.

    You know, so that technique was perhaps limited in its application, but we’re now starting to see how that same technique is being, if you will, scaled-down or right scaled-down to the appropriate size and being used across the support organization and not just limited for use to, you know, the building’s falling down type of issues.

    Stevens-Hall: And one of the great things that’s happened as we’ve sort of distributed this messaging and talked more about it is, actually, it’s got huge engagement in DevOps community. I’ve presented on this at a number of DevOps events, you know, right up to DevOps Enterprise Summit.

    Because actually, what I believe it’s doing is, it’s sort of showing the DevOps community that there is, not only is the support, the world of kind of technical support and service management, you know, that sort of structured technical customer support and service management, we’re looking to try and offer value to the DevOps community, but we’re also doing it in a way that is much better aligned with the way the DevOps community has learned to work. And, you know, so rather than those developers ending up just being treated as a third line support team and having things thrown over the fence to them, it enables us to kind of bring a much more collaborative system together and reduce those kind of pockets of those silos where knowledge doesn’t escape.

    So, the value—again, thinking about the value—the value, again, to those teams working in the DevOps world is that we can actually be embedding support people much closer to what they do and those support people learn how to fix the issues. And then when there’s more of those issues, those support are there as professional support, you know, as support professionals who are good at this kind of thing can be figuring out ways to proactively resolve those things, automate those processes, move towards customer self-service as an ultimate goal, perhaps, or self-healing and get those developers developing.

    Ashley: Yeah, it also gets them more—I’d say more closely aligned to knowledge about what the application is, how it’s structured, who to contact, you know, help them maybe even diagnose more of the issues themselves.

    Stevens-Hall: Yeah, so again, it perhaps creates some parallels with what, again, companies like Google have innovated with site reliability engineering. It’s not quite the same, you know, these are not necessarily developers and I often caution people from kind of just saying you can just do SRE, you can get service management kind of spaces like the service desk and do SRE. You’re not really doing SRE in the strict sense the way it’s known in the DevOps community where it is kind of developers doing proactive things with infrastructure.

    You know, if you look at an SRE job spec, it’s gonna be a bunch of programming languages and related experience, but we can still take, again, those learning principles, you know, the idea, for example, of thinking in terms of ERA budgets, you know, so rather than every failure being a major—you know, every time we lose a bit of our remaining availability budget against that target, you know, that could be, that’s always been seen as negative, I think.

    But what SRE has introduced is the concept of actually using some of that spare availability capacity to be proactive and make things work better. I like—an analogy I do is, “Why don’t we take somebody off the service desk calls for an hour?” It means our call pickup percentage is 98 percent instead of 99 percent, but they could spend that time doing something that really, really drives improvement.

    So, again, these techniques that have evolved in DevOps really sort of cross-pollinate well in the service management space and enable us to better connect to the way DevOps works. And, again, help them develop.

    Ashley: Mm-hmm. Well, excellent. This has been fascinating. I’ve really enjoyed talking, and I’m sure the audience does, too, that’s listening in. It’s great to talk to the contributors, to the authors of this, because you get the benefit of both, you’re writing a knowledge that’s shared in that medium, but also—what was the thinking behind it? What were the influences and how did you get to those conclusions and sorta that context is also super helpful. So, this has been really great. I’ve enjoyed talking with both of you.

    Anand: Same here.

    Ashley: Great. Well, my thanks to Akshay Anand who is Product Ambassador with AXELOS and Jon Stevens-Hall, Principal Product Manager at BMC Software. Thank you, gentlemen.

    Stevens-Hall: Thank you.

    Anand: Thanks, Mitch, and thank you for listening to us today.

  • Mainframe and DevOps in Traditional Enterprise, With ASG Technologies

    Mainframe and DevOps in Traditional Enterprise, With ASG Technologies

    The assumption that large, established enterprises—from insurance companies to government agencies—can’t adopt Agile processes or DevOps is based on the falsehood that legacy technology stacks won’t allow for it; that existing traditional mainframe applications or legacy applications that large enterprises are built on are incapable of adapting to these approaches.

    Accelerated Strategies Group recently released the new research report, “5 Crucial DevOps Strategies For Cloud and Mainframe 2020,” sponsored by ASG Technologies. In this episode of DevOps Chats, Jeff Cherrington and Anna Murray with ASG Technologies joins Mitch Ashley, CEO of Accelerated Strategies Group, to explore how mainframe app teams embrace DevOps, its processes and tools to equip software teams so mainframe apps continue to provide businesses with vital systems and capabilities far into the future.

    The audio file of the conversation is below, followed by a transcript. Enjoy!

    Transcript

    Mitch Ashley: Hi. I’m joined by some special people, some folks that I really enjoy working with from ASG Technologies—different from the ASG that I work with. [Laughter] So, I’m joined today by Jeff Cherrington and Anna Murray. Thank you, both, for being here.

    Jeff Cherrington: Well, thank you very much, Mitch. I’m Jeff Cherrington, I’m Vice President of Product Management for Systems with ASG Technologies. Many of you may know ASG. We’ve been in enterprise infrastructure management software for over 40 years, and certainly hold a prominent position as well in enterprise content management. We’re very keen to have the conversation today around how traditional infrastructure—meaning the mainframe—aligns with new DevOps initiatives in the enterprise.

    Anna, if you would introduce yourself.

    Anna Murray: Sure. Hi, my name is Anna Murray, and I’m a Senior Product Manager with ASG Technologies, and I work with Jeff Cherrington. And I have extensive experience with automation across platforms and data centers, as well as agile transformation for teams.

    Ashley: Excellent. Jeff was saying before we started that he used to work in radio—you can tell. [Laughter]

    Murray: [Laughter]

    Cherrington: [Laughter]

    Ashley: You’re very comfortable behind a microphone—no doubt, no doubt. We were talking about everybody’s on video these days, so that’s awesome.

    Murray: Right. [Laughter]

    Ashley: Well, great. You know, we’ve been doing some work together and we do have a report coming out to kind of pre-announce that. We’ll be talking about that a little bit later. But there’s a lot of myth about DevOps and mainframes. I know even some analysts—I won’t name any names—have said they’re two different paths, you shouldn’t try to do both. I think the jury has started to change their mind on that.

    Why do you think some of those myths have existed of, “No, we can’t do DevOps on mainframe environments?” Do you wanna start, Jeff?

    Cherrington: I think I will, and then I’m sure Anna’s got some thoughts that she’ll want to contribute. You know, it’s certainly a reality within the industry that we’re going through a generational shift around our leadership or IT, whether it’s infrastructure or application development. There are those who are still working daily, like myself, who represent the traditions of IT from the very beginnings back in the 1960s, and then there are others who are certainly much younger and certainly have much different experiences, which may not include any exposure to the mainframe. And, as a consequence, there might be some resistance, there might be a little bit of anxiety about the technology, and it can be formidable to approach if you don’t have the appropriate background.

    The things that we’d love to talk to, though, is that the mainframe as traditional technology still performs incredibly important roles within the enterprise. And at the end of the day, DevOps is much more about technique than technology, and it can apply as easily to mainframe and should as it does to any other part of the IT infrastructure.

    Anna, do you have some thoughts to build on that?

    Murray: Sure. You know, as I think about the generation gap that you’re talking about, I kind of come on the other end, but I’m Gen X. And so, we were the generation that said, “Oh, the mainframe’s dying,” right? [Laughter] So, that—obviously, I’ve learned that’s totally wrong, and the generation coming after us is coming in to learn the mainframe and realize how powerful it is, right?

    And so, I think a lot of the leaders that are coming in are coming from this generation that didn’t even think about mainframe for a while, right? And so, they did all their planning and their agile transformation and their DevOps transformations thinking about the cloud and all of their Windows and Linux and Unix servers and didn’t even think about the mainframe.

    And so, I think that’s the other side of the gap that happens. As the younger leaders are coming up, you know, especially, I would say blame my generation the most, right? [Laughter] Because there’s this gap, and—learn more and make sure you’re talking to both ends of the spectrum and learn about crossing the gap.

    Cherrington: And actually, the way I like to phrase it now is far less to use jargon like “generational gap,” but instead talk about generational bridge, because that’s what I’m seeing.

    Murray: That’s so much better, yeah.

    Cherrington: We’re seeing that the younger IT professionals are identifying that there’s great opportunity to work with mainframe in the coming years. And so, we see Millennials, or is it Gen Y—the generation after yours, Anna—who are stepping forward and saying, “This is really interesting. The mainframe does all of the technologies, it supports all of the technologies that I learned in uni, university. I can do Java, I can do TCP/IP connections, I’ve got great sources for strong cryptography, it supports Unix, I can put Linux onto a mainframe LPAR—this is a good space to be in.”

    And so, more and more, we’re seeing that the people we engage with are a mix of people that are my age with the white beard or the gray beard and a mix of young, fresh faces that are really eager and really excited to get involved, and who come predisposed to new techniques such as DevOps.

    Murray: Mm-hmm.

    Ashley: You know, there’s so much important things that both of you said. One is that in DevOps, there is no technology definition of what it applies to, and it’s very easy to associate it with Cloud Native and maybe newer applications, because that’s where it sort of grew up initially.

    But DevOps is really about how we create software, not what technology we use to create software. And it’s about how teams collaborate, it’s about how you share information, it’s about how you get different functions from the business together in a collaborative team to kinda do things earlier in the process, do automation. All of that applies to the mainframe, of course, and there’s tools to do much of what we’ve talked about. Maybe you can say a little bit about how people approach DevOps in the mainframe environment.

    You wanna go, Anna?

    Murray: Right? Well, you know, when you approach DevOps in the mainframe environment, it’s not so much just about the mainframe environment, but the company. The whole organization has to be behind DevOps, and there has to be leadership saying, “Hey, this is something we need across silos,” right? And then you have to look at the tools available to you. DevOps is a lot about the toolchain itself. There’s hundreds of tools in that toolchain, and so, you have to start looking for solutions that are gonna meet your needs and help you cross those silos.

    Cherrington: You know, it’s certainly true that everybody resists change. I think that’s an innate part of human nature, but both Anna and I have seen that things of this nature can come forward, can be used with teams that are mainframe-centric, perhaps have been doing things the way they’ve been doing them for 10 years or 20 years or 30 years.

    And there’s natural resistance. “Gosh, you know, I’ve done things the way I’ve done them successfully for so long, why should I change?” Certainly, I’ve seen agile adopted by mainframe development teams, sometimes kicking and screaming, but once the resistance curve is overcome, it becomes very productive and becomes very natural.

    And I think we’re at that stage now in terms of adoption of DevOps for the entire enterprise infrastructure, from Windows all the way to the mainframe and onto the cloud and even onto mobile, even as there are initial reactions of going, you know, “How can this apply? This is not something I learned when I went to my classes to learn more about the mainframe at SHARE in 1995 or 1985.” But we have proof points.

    We have large international enterprises, we have small regional enterprises that are coming forward and saying, “I must have mainframe. It plays a critical role for things that I need to do, particularly around payments processing, anything that needs highly available, real-time online absolute reliable performance. And it is only part of my infrastructure, and I need to treat how I develop there the same way I treat developing anywhere else.”

    Ashley: You were talking about, also—so many apps are not just a mobile app, right? There’s a whole back end to it, oftentimes sharing data, even talking transactions between multiple systems, one of those or multiple may be mainframe applications. You know, and if I step back for a minute and think about our current environment and how much things have accelerated in doing sort of these digital transformation, digital experiences for customers, organizations are looking to both strategically move quicker, but also be able to react opportunistically and sometimes based on conditions. And it seems like some of the flexibility with DevOps makes it easier to coordinate some of those activities across multiple applications. So, if you do have to do three different release to make up one new capability, that comes together much easier.

    Cherrington: It does. You know, the thing that I always look to as I engage with anything around business is, to use an old phrase, “why is the juice worth the squeeze?” And while we talk about DevOps a lot because of its elegance, because of its collaboration, because of the ways in which it makes development easier, the real juice of the squeeze is time to market. The whole reason to consider DevOps is how can we get something we need to get in terms of new technology enablement to market quickly, more quickly than we did before, but not sacrifice quality.

    And that’s what we see time and time again is, we have something we need to get to market. I want to present new capabilities to my customers through an app on the phone because that’s where people live their digital lives. But at the back end, it’s dependent upon the mainframe for things such as account balances, customer statuses—an unending list of capabilities that still remain on the platform as the best place for that type of activity to occur.

    Ashley: Mm-hmm. You know, I’m thinking about just the recent experiences with the reactions to COVID and contactless delivery, inventory, product inventories are often something that’s still on the mainframe, has been for a long time, and now, you know, retail stores, as well as a lot of others, have to be able to maintain a really accurate enough inventory and also present that to the customers so they know, what store can I go at to pick up whatever I’m looking at buying.

    I’m curious, too—I know that ASG Technologies has some of your own products, your own technology around this idea of orchestration across platforms because you work both mainframe and non-mainframe environments. Talk to us a little bit about how you approached coming up with that product and maybe a little bit about how it works and what it does.

    Murray: Sure, I’ll jump in for a little bit here. You know, you talk about orchestration—that’s combining your toolchain, right? You’ve got all the little pieces of your DevOps toolchain, and you need to figure out how to connect all the dots and bring it together and, you know, ASG has been doing automation and orchestration for a long time, but we’ve changed our focus into bringing in the development part of that toolchain, right?

    Operations has been kind of a focus of automation for a long time, and now we have to pull in all of the development pieces that happen and the configuration management that happens, and all of those—there’s lots of little tools there, right?

    So, we haven’t tried to come at the market as trying to provide all the tools, but we’re trying to provide some of them, right, and we definitely want to orchestrate as many as possible, right, and cover the entire environment for the customer. On the mainframe, the most popular thing we’ve been working with recently is JCL Management. Every mainframe’s running JCL, right? Nothing happens on the mainframe without it, and so making sure that there’s a DevOps interface to your JCL Management solution is really important. And so, we’ve made an investment in making sure that ours has both an Eclipse plug-in that makes it easier for the developers to interface with that JCL as well as a RESTful web service API so that it can be integrated with building of the products and crossing across the silos to other products.

    Ashley: Mm-hmm.

    Cherrington: And certainly, in the approach that we’ve taken, we look to be a good citizen of the DevOps enterprise community, and the work that Anna and her team have done, we ensure that we support the most popular integrated development environments for doing this sort of work. So, the JCL quality assurance and management offerings that we have for the mainframe are also certified for the Compuware Topaz integrated development environment and for the IBM—what’s the name of it again, Anna? It’s—

    Murray: IDZ.

    Cherrington: IDZ. What was the rational development environment that’s now rebranded?

    Murray: Yes. [Laughter]

    Cherrington: As well as any other Eclipse compatible IDE that a customer may want to use.

    Ashley: You know, it’s interesting, just thinking about all the, a number of the terms that you both used in just those last two comments, you know, mainframe 10 years ago, you wouldn’t have been talking about, I don’t think, RESTful IDE—all the tools that we kinda think of as sort of in cloud applications, much of it is the same toolset, a lot of the same ideas, just working with integrating some other different tools that are specific to the mainframe environment.

    Cherrington: It is, and you know, I certainly want to encourage those who are listening and watching—and particularly, those who may be ASG customers now or are considering becoming ASG customers—to join us at our worldwide customer conference, a virtual conference this year, EVOLVE, that will be occurring on October 6th and 7th, because we will be making some significant announcements about extending support in this area, including opportunities for creating interfaces to mainframe applications that can extend all the way to mobile, if that’s of use to a customer.

    We’ve had a lot of excitement coming into this, we have a lot of partners and customers who are going to be presenting, and there’s a great deal of energy that we’ll be putting forth around our DevOps support and DevOps enablement.

    Murray: And can folks sign up? How do they register to attend the virtual conference? Is that on your website?

    Cherrington: It’s on our website if you come to www.asg.com, I’m sure you’ll find something right on the home page that says EVOLVE 2020 and it’ll give you the opportunities to get, well, involved.

    Ashley: [Laughter] EVOLVE and involved at the same time.

    Cherrington: Exactly. [Laughter]

    Murray: [Laughter]

    Ashley: You know, the fact that you get to work with so many organizations that are adopting DevOps, I’m kinda curious about two things. How does the mainframe teams approach wanting to learn about DevOps and coming up to speed and trying it, and then how do the existing practitioners maybe outside of the mainframe group that are also using DevOps—how do they see what’s happening in the organization and how do they…do they want to participate, do they support it, are they kinda not sure? What do you see happening, there?

    Murray: So, yeah, you know, there’s a mixed feeling. You know, there’s definitely differences between, you know, if it’s led from the top down and people are being told they have to change, right, it depends on how you’ve adjusted the expectations in your team, right? Helping them learn what’s coming and why instead of just telling people what to do makes a huge difference, right?

    Ashley: Mm-hmm.

    Murray: We’re all much more interested in solving problems and learning new things together. So, it’s really important conversation is held properly, right?

    So, we have some that are really excited that they want to learn and they want to jump in from the mainframe side and grow, and then there’s others that are really just struggling and pulling their mainframe teams along, right? So, one customer I think of, you know, is standardized on Compuware Topaz Workbench, but the developers are less interested in switching over. They have some of the older generation who’s very comfortable with the green screen and they think, “Why bother switching to Topaz?” You know? They just don’t even get it.

    Ashley: Mm-hmm.

    Murray: And so, you know, the truth is, we really need to have solutions that meet the needs of all of the people that are part of the DevOps environment, right? So, it isn’t—you know, a very experienced mainframe programmer doesn’t have to use Eclipse, right? They can be quite happy on the green screen and we should let them stay there, right? [Laughter] And we do, our JCL Management solutions have a very robust green screen interface, and then we bridged the gap, as Jeff was talking about earlier, with having those APIs, right?

    And so—yes, so the DevOps toolchain is still brought together with something like our ASG Enterprise Orchestrator pulling in the code and grabbing with the API the JCL and making sure that the standards are applied and then bringing them across for compiles and distribution into another environment, right? Not that you compile JCL, but there’s other parts that are related.

    Ashley: Mm-hmm.

    Murray: So, you know, you asked—it’s kind of a mixed bag. And we see that, too, right? And as the conversation becomes more common, right, I think we’ll have more people who are more comfortable having that conversation and learning together.

    Ashley: Excellent. Well, Jeff has been bitten by the Zoom bug, so he had to call in from his call dropping, so glad you’re still with us, Jeff.

    Cherrington: Mm-hmm.

    Ashley: You know, we were just talking about the adoption process, and Anna, you were talking about how mainframe teams kind of approach this. I’d like the other side of it. DevOps teams that maybe aren’t in the mainframe group that have been practicing this for a while, maybe a few months, a few years—how do they see this happening, and is there collaboration? Do they pitch in, do they tend to work together or not? How does it go?

    Cherrington: Well, some of the things I’ve seen actually goes to either end of the spectrum. For the teams working with the more modern technologies and have to adopt DevOps very frequently, we find evangelists, those who have gotten to the point where they believe in the techniques and the approaches so passionately that they’re out there actually promoting them everywhere they can. It’s much like what we saw with the adoption of agile in the last decade. We’re seeing that same sort of passion surround this approach to development in this decade.

    You know, it’s one of those things, we have to keep in mind—DevOps was not even a term within the industry until 2009. I mean, it’s barely a decade old, and it’s in these last five years that we have seen the acceleration for adoption.

    Now, having said that, you know there certainly are always going to be pockets within any enterprise where, again, there’s resistance. There are those who are not looking for change and those that, perhaps, while they have embraced the DevOps approaches for their distributed systems or cloud development, look at the ideas of what they have to change or adapt to on the mainframe side and really don’t want to go there. The idea that there might be a long compile that has to be done on the mainframe before the next step in the DevOps process can occur will cause some—a little bit of agita.

    And that’s why it’s so important we need to take a look at the strategies that are necessary for DevOps success. I think that’s something that you speak to very eloquently, Mitch. And out of those steps, the one that I think is so critically important is culture shift. Just like with agile adoption, DevOps adoption needs support from the top, and it needs to be constantly vocal support to help people pull through these resistance curves. Resistance curves are natural and the quicker an enterprise gets through them, the quicker they get to a point that they’re delivering new capabilities to market much more quickly.

    Murray: Right.

    Ashley: Thanks for your kind words about the culture, too, and I want to get your thoughts on this, Anna. Culture is a lot of things, but I think we’re all human at the end of the day and none of us want to look dumb, right? We don’t want to tackle something we don’t know if we can learn or maybe not kinda look so good in front of our peers if we’re the ones behind the curve.

    And I think something that can help combat that is, one of the principles of DevOps is sharing, is really—it’s collaborating, but it’s also sharing. Sharing scripts, process, tools, content, methods—all of those kinds of things which, if you truly adopt DevOps, as you said, Jeff, you can become an advocate and then also help with that transition and very quickly, you know, they kinda get over that curve of, “Okay, alright, I’m not—I’m familiar with this,” as you mentioned, Anna, I can still use my green screen or many of the tools that I’m already using and I’m just adding to kinda what I’ve been doing.

    So, your thoughts, Anna?

    Murray: Right, well, I’ve recently been having other conversations with partners about this culture shift and it’s become really clear with other companies that are supporting DevOps and those initiatives that you really have to have not just buy-in from your senior management, you’ve got to have active support. You know, they have to sponsor the change.

    Because we’ve also seen companies who don’t go through the change, they fail. If it’s from the bottom up only and the senior leadership says, “Okay, yeah, sure, that’s a great idea,” but they’re not actually investing in it themselves, the team, when they hit road bumps—because they will. It’s not, you know, switching to DevOps doesn’t happen overnight. I think we all know that. And over time, you’re gonna hit roadblocks and you’re gonna get discouraged. And who is it that’s gonna lift you up?

    Now, hopefully, there’s evangelists that’ll help keep pulling you through and encouraging you, but at the end of the day, if your senior management isn’t coming back in to say, “Hey, guys, we believe in this process, it’s worth fixing, we’re gonna invest in whatever it takes to solve this problem,” then they’re encouraged to keep moving forward, right?

    But, you know, there is the difference happening out there where the leadership isn’t always invested in it. And so, that’s what we’re really encouraging when it comes to that culture shift that has to happen, make sure that you get not just buy-in but investment from your senior leadership.

    Ashley: Mm-hmm. Hugely important and it goes back to the, “Why are we doing this question?” that you talked about also, Jeff. You talked about really responding quickly and getting software out to market or quickly in today’s situation with COVID and uncertain as well as very quickly changing conditions, you know, something could very easily come down from Product Management or whoever to the development teams and say, “Okay, we need to add this capability or change how we’re doing something.” And the team’s ability to react quickly—I mean, they already do today, but even more quickly under these circumstances, I think, demonstrates the value not only of DevOps, but also those teams.

    Cherrington: Absolutely. Absolutely. I mean, it’s always a virtue to be able to get to market quickly. And in the current period of the pandemic, it’s a virtue that actually can impact lives on a broad scale.

    As the different national governments are taking the steps that they need to take to provide stimulus and secure the safety of their populations during all of this disruption, one of the things we’ve seen is that there have to be changes to critical traditional mainframe systems for all of this to happen. Certainly, the unemployment compensation that was distributed here in the U.S. touched legacy traditional systems at the state level that perhaps have not been actively developed for some period of years.

    And certainly, those government enterprises or commercial enterprises that were pre-prepared to apply their DevOps techniques to making those changes to their traditional offerings, traditional applications were better prepared to deal with this and to deal with the ongoing change as our government’s tried to figure out how to stabilize economies and get us through this challenging time.

    Murray: Mm-hmm.

    Ashley: You’re absolutely right, and that’s one example of many. I mean, there’s insurance claims processing to get your request fulfilled quickly, you know, new medicines—all of that kind of thing. There’s so many areas where mainframe technology is involved today.

    Well, I wish we could keep going. Maybe I have to come to the conference to continue the conversation. [Laughter]

    Cherrington: [Laughter]

    Murray: Yeah, we would love that.

    Cherrington: You would always be welcome.

    Ashley: Well, I appreciate that very much. And so, just a reminder for our audience, the EVOLVE conference is on October 6th and 7th. Go to the ASG.com website to find out how to register; I absolutely recommend doing that. And as I mentioned earlier, we have a research report coming out and the working title of that report, I think, was “Five Critical Strategies for DevOps and Mainframe.” So, we’ll see, maybe that title will stick or something close to that. I’m excited to get that out because that’s been a great effort working with you.

    Well, Anna Murray, thank you very much; Jeff Cherrington, also—thank you for joining us today.

    Cherrington: Thank you so much.

    Murray: Thank you, Mitch.

    Cherrington: And folks—stay safe.

    Murray: Yeah.

    Ashley: Alright.

    Murray: Alright, thank you.

    Ashley: You bet.

  • DevOps Chats: Software Delivery Leadership Forum and DevOps World 2020, CloudBees

    DevOps Chats: Software Delivery Leadership Forum and DevOps World 2020, CloudBees

    DevOps World | Jenkins World has long established itself as a leading venue for DevOps practitioners to learn, share their knowledge and build a top-notch community. But, what about leaders, the people in leadership roles at companies who utilize DevOps as a vital part of their business and technology strategies?

    Sam Fell, CloudBees area VP enterprise markets, joins DevOps Chats to discuss the newly announced Software Delivery Leadership Forum. DevOps World 2020 is expanding its focus and content by bringing leaders together with content specifically tailored for business and technology leaders. Accelerated Strategies Group, where Mitch Ashley is CEO, is leading the programming of the DevOps World 2020 Software Delivery Leadership Forum.

    As usual, the streaming audio is immediately below, followed by the transcript of the conversation.

    Transcript

    Mitch Ashley: Hi, everyone, this is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Sam Fell, area vice president, enterprise marketing at CloudBees. And we’re talking about, actually, a subject of interest to both of us. It’s the new Software Delivery Leadership Forum, and we’ll get into that.

    Welcome to DevOps Chat, Sam.

    Sam Fell: Thank you, Mitch. Pleasure to be here, as always—great speaking with you.

    Ashley: Absolutely. I always enjoy talking with you. For folks who don’t know you, I know you’ve been in the industry around DevOps for a while. [Laughter] So, a lot of folks know you, but—

    Fell: Are you calling me old? Is that it?

    Ashley: Actually, I didn’t say that. I’ve got the gray hair; you’re still young, so.

    Fell: [Laughter]

    Ashley: Just to introduce yourself, for folks that may not know you.

    Fell: Alright, wonderful. So, I am Sam Fell. I am area vice president at CloudBees, where I’ve been for the past year. We were acquired in, as part of the Electric Cloud acquisition where I ran marketing at Electric Cloud.

    Ashley: Yeah, a lot of folks know Electric Cloud, so probably know you from that as well as CloudBees, of course.

    Fell: Yeah. That’s right.

    Ashley: How long ago did that acquisition happen?

    Fell: That was literally a year ago in May.

    Ashley: Okay.

    Fell: So, we are just a year, really just coming up on a year. So, all the exciting stuff was happening, all the stuff that made me lose my hair and all that was happening right now. And we were just so thrilled with the outcome to be, you know, the company that Electric Cloud was. We had a pretty nice name in the space. But when you look at CloudBees and you look at the community that they’ve created and you look at the access to the brains that they have, it’s a tremendous opportunity. You know, obviously, I’m really excited to be able to be here. And I’m excited about sharing this journey with the Accelerated Strategies Group, right?

    Ashley: Well, thank you. I’m excited about that, too. Yeah, CloudBees is a great organization, as was Electric Cloud.

    So, let’s get into the—we made an announcement, or actually, you did a press release about the Software Delivery Leadership Forum, and there’s a connection to DevOps World and some other things. Maybe we start with that and then we can talk about Accelerated Strategies’ involvement in it.

    Fell: Lovely. Awesome. Okay, so, you know, we’ve been running probably, people would say, the most successful DevOps conference for years now, DevOps World/Jenkins World. Almost 2,000 people show up to learn more about what’s happening in the space, there’s hands on practitioner based workshops and training for people to really get their hands on and get them dirty about how to implement these changes that have to happen as you’re digitally transforming your organization.

    One of the things that we took a step back and we were thinking about is, what we don’t really have is a place for the leadership to be able to participate. And, you know, we’ve done leadership seminars and workshops before, but we’ve never said that DevOps World needs to be a place where not only the practitioners are coming, but where the leaders are coming to learn not the practitioner level information, but how to lead the practitioners. What are the processes, what are the methods that people are learning and using and trying and failing with and learning from?

    And so, we thought we would add that to the program. And as we were thinking about adding it to the program, we also thought, because of the way that the DevOps World/Jenkins World show has evolved, I as a vendor looked back at DevOps World/Jenkins World and I was always very willing to sponsor that event when I was at Electric Cloud, because it was an open event. There was not a ton of bias towards CloudBees products, necessarily. It was really about the community, and it was about educating people.

    And so, in that same spirit, when we were thinking about how do we make sure that there’s some objective independence in the way that we’re putting out the ideas around, “How do you lead? How do you lead this transformation? How do you make sure that your teams are all on the same page?” We felt that it was really helpful to have a third party help us with that programming. And so, that’s, of course, this is the cue to Accelerated Strategies, this is why we sort of reached out to you. There’s obviously great synergy between the work that we’re doing, and we thought, who better to help us craft this program, the programming for this conference than the fine people over at Accelerated Strategies?

    Ashley: Good.

    Fell: So, that’s pretty much where we’re going, and then you and I have had lots of conversations about what that Software Delivery Leadership Forum should look like at the event, and so maybe I would pitch it back to you and you could talk a little bit about what we’re thinking about for DevOps World Software Delivery Leadership Forum.

    Ashley: Absolutely. And thanks so much, we appreciate you approaching us, and it was exciting to work with you on this. And I agree—you know, DevOps World, yes, it’s CloudBees behind it, Jenkins World, but the community that you talk about really elevates it above any one vendor, to use that word.

    Fell: Yeah.

    Ashley: And just the fact that Electric Cloud, you know, would—and many others would sponsor a conference that’s put on by another vendor.

    Fell: Yep, yep, coopetition. The competitors were there and happy about it because the audience is the right audience. These are the people that are in the weeds making it happen with their fingers, and now what we’re saying is, “Hey, these people over in the weeds that are using their fingers, they need other people to help on the top that can help support and pave the road for them.”

    Ashley: They do.

    Fell: And so, we’ve been trying to do that.

    Ashley: Well, and I would imagine, too, that, you know, one of the drivers behind leadership as a topic is organization scale. You know, trying to do DevOps on more than two or three or four independent projects, that’s when you get into the heavy lifting. That’s when you’ve got to have management support. And now I see—I know you do, too—leaders reaching out and trying to find resources for themselves, educate themselves, lessons learned, pitfalls, community that they can be part of as well as the technical folks be part of their community and that’s in large part from what we’ve talked to why to go after this Software Delivery Leadership Forum.

    Fell: That’s right, exactly. Do you wanna talk a little bit about what the proposed format would be at the—

    Ashley: Yeah.

    Fell: —Software Delivery Leadership Forum? Wonderful, okay.

    Ashley: Yeah, I’m happy to. We have some research that we’re gonna be launching, and talking about around software delivery that we’ll be discussing and presenting some of the findings from.

    But the main point of how we’ve crafted this to be, it’s a parallel track. It’s not a separate venue, separate place, it’s just a set of content that’s specifically directed at leaders, people in SE or VP director software leader kind of role, but it’s also product leaders as well as transformational leaders. And those are the folks that we really would like to attract to attend to this, because that combination is always what needs to come together to achieve some business strategy, disruptive strategy product delivery, some kind of financial effect that we’re trying to have on the organization.

    So, it’s a track of about 22 different sessions, but we’re gonna have a number of things that will be highly interactive. So, we’re gonna have some panels, we’ll have some audience participation, kinda open forum for questions and answers with different experts, and we’ll have people that—yes, there’ll be some folks who are product company, there’ll be folks who are analysts like from Accelerated, but not only Accelerated; we’ll have other analysts there. And then, we’d like to have as many practitioners in a business role technical leadership role as well to get that perspective.

    Fell: Right.

    Ashley: And we’ve planned some workshops and some things like that. So, my goal is that everybody take three things away from this conference. Whatever those three things are, come back to your company, to your team, to your organization and say, “You know? I picked up some really good stuff. This one, I had no idea somebody was facing the same issue and they actually solved it—”

    Fell: And they fixed it, right.

    Ashley: “—a way I was thinking about, so it validated our thinking.” Or,  you know, “Threw it out the window and we’ve got a whole other approach, but I’m sure glad I didn’t march down that Bataan death march.” [Laughter]

    Fell: Waste my time—failing fast is winning, exactly.

    Ashley: Exactly, exactly. So, the call for presentations is open right now.

    Fell: Yep.

    Ashley: You can go to DevOpsWorld.com, and there’s a button there that is a call for presentations, fill out information, we’d love to have folks, and we are reviewing submissions actively right now, so it’s a good time to get this out.

    Fell: Yeah. We’re getting—actually, from that release, we’re getting quite a few submissions, which is really nice. There’s some suggestions around some of the topics that we’re looking to have people talk about—governance, security, those are big topics. So, yeah, definitely interesting to get people’s perspectives and to see what they have to say. We’re excited about that.

    Ashley: It is. So, full disclosure, we both have an interest in this and participating, but it’s good to be on with you to talk about it.

    Fell: Yeah, for sure.

    Ashley: You know, I’m curious, too, because you’re doing this—well, let me ask you, why did you decide to name it the Software Delivery Leadership Forum? Because I know we went around a couple names, a couple of ideas.

    Fell: Yeah. So, well, when you think about the work that the fine people are doing, whether they’re doing it from a DevOps perspective or they’re just in dev or they’re just in ops or they’re part of the executive team who’s not part of development or operations, but they have a vested interest in development and operations getting their stuff done, right?

    Every—this is such a hackneyed expression, it’s very overused, but every business is a software business. And now, in this age we find ourselves in, the digital connection with your customers is more important than ever.

    Ashley: Mm-hmm, mm-hmm.

    Fell: And so, there are other people beyond just dev and ops. And so, originally, it was going to be the DevOps Leadership Forum, but it’s really, we think it’s more about that, delivering that software or delivering those services that are foundationally built on top of software, and you can call it whatever you want, but down at the base layer, somebody’s written code, somebody’s built that code, somebody had to deploy it to a server so that people could get any value from it.

    And so, the software delivery life cycle, the SDLC, is something that we think a lot about in the SDLF, like, the Software Delivery Leadership Forum. Because the SDLC is still alive, it’s still kicking, and application life cycle management matters, even in the age of DevOps. And so, we thought that this would be an appropriate name for what we’re trying to put together with you is a place for people, a community to come together that’s built on the community that we already have of all these practitioners who already understand how a lot of this stuff works, but now, they’re just looking for some guidance and some, “Hey, I don’t wanna go down that road again, because that guy already failed,” right? Failing fast and learning from it is not a failure.

    Ashley: Exactly. There’s so much sharing that happens in the technical community, I’m confident we can build that with the business product and technology leadership.

    Fell: Yep.

    Ashley: It’s interesting. We’re in the midst of, maybe we’re still in the early part of this, I don’t know the whole COVID-19 and being in lockdown and, of course, there’s the health and, in some cases, even losing people that we know in our family, or friends. So, in all due respect to that, you know, I’ve talked—some of the things I’ve talked about on another venue, which is TechStrong TV, is, it’s interesting to see this—

    Fell: Congratulations on that launch, by the way.

    Ashley: Thank you. It’s a lot of fun to do.

    Fell: Yeah, I love it. Yeah, that’s great.

    Ashley: It’s a lot of fun to do. Some great content on there. If you haven’t checked it out, go to TechStrong.tv.

    Fell: Yes.

    Ashley: Two to four hours a day, Monday through Friday, so—keep you company. I call it “the view” for technology people. [Laughter]

    Fell: Nice. Very nice.

    Ashley: So, even in this situation, you know, we’re locked down, right? Everybody’s working from home. Most people are. I’ve noted, you know, restaurants have all gone to delivery. They’ve gone to curbside delivery. Even restaurants around here have now started to come up with new offerings to essentially take home packaged meals, kinda like you would from one of those box delivery services.

    And even places like Best Buy has curbside delivery. I had to build a computer, and the delays of shipping something from a manufacturer were so long, you know, and they had to change their apps. They rebuilt their app or added to their app and their web notifications to have, you know, a button to push to say, “I’m in the parking lot and I’m in a white this car and here I am, and by the way, here’s my QR code or scan code,” or not even have to do that.

    Fell: To validate—yeah, exactly.

    Ashley: And that happened in weeks. That wasn’t—you know, that’s a few weeks that those two examples happened. Now think about—

    Fell: Life finds a way. Commerce finds a way, maybe that’s [Cross talk].

    Ashley: It does, it does. And kudos to those folks and many others who were looking for—maybe we didn’t disrupt it, but disruptive business strategies. [Laughter]

    Fell: Yeah. I mean, they have to respond.

    Ashley: Delivery through software.

    Fell: Yeah. They need to respond. Life finds a way, I love it—DevOps finds a way.

    Ashley: It’ll be interesting, I think, you know, given the time frame, the dates are September 21st through the 24th in Las Vegas. And, you know, we’ll have several months behind us, hopefully not too bad, but several months of experience. I bet there is gonna be a lot of stories, many stories of folks who have reacted and responded and some successes, and—

    Fell: Led through that change, right, exactly. Led through that change.

    Ashley: So, talk about a good test—well, not good, but talk about a test situation for the flexibility and agility of our businesses.

    Fell: Yeah. This is the crucible. That’s really the crucible, right here.

    Ashley: Yeah. It truly is.

    Fell: And so, one of the other things, you know, that we’re doing—and Mitch, you’re part of this as well—is that we’re so excited about the Software Delivery Leadership Forum that we actually don’t wanna wait for the September event to get started.

    And so, what we’re actually doing is, we’re creating an online virtual series of Software Delivery Leadership Forums, with our first one coming up at the end of this month in April. Episode one of this series is gonna be on upskilling. It’s on adapting humans at the speed of DevOps and we’re gonna be joined by our friend, Jayne Groll—our mutual friend Jayne Groll—and Eveline Oehrlich, who is her head of research.

    Ashley: Both are mutual friends, yeah.

    Fell: Right, and Eveline—

    Ashley: Both are analysts with Accelerated, too.

    Fell: She’s with you over at Accelerated Strategies, right. And we’ll be joined even further after they’ve given their little talk about the survey results, and we’re actually gonna get an interesting cut of data for the European, because we’re doing two versions of that, event—one for the U.S. time zone friendly and one for EMEA time zone friendly.

    Ashley: Mm-hmm.

    Fell: And so, for the EMEA one, there’s gonna be a different cut of the data that’s gonna be much more focused on that geography, which we’re really excited about. But we’re having a whole bunch of other folks join us as well to be able to have that conversation, because we’re really wanting to try and create, again, a community where there’s interaction between the attendees and the folks who are on the webinar or on this virtual event. Because so many events, people parade out folks that are saying, “Yeah, here’s how we’re successful” and then maybe you get five or 10 minutes at the end for questions and answers.

    And so, we’re kinda gonna flip that on its head and we’re gonna have 30 minutes up front for the Software Delivery Leadership Forum—30 minutes up front for Jayne and Eveline to go through their findings to talk as subject matter experts about this topic that they certainly know a lot about. And at that point, we’re going to introduce a couple of other panelists to join us on camera and start fielding questions directly from the audience.

    And so, the folks who are going to be joining us this time around, we’ve got Ellen Thorne, who is the head of HR here at CloudBees, so who better than a person who’s in HR to talk about what’s—that’s an interesting perspective. What’s unique about that from a—

    Ashley: Skills and training of their people, yeah.

    Fell: Yeah, training. Robert Reeves, who’s the CTO at Datical, right? So, how is the technical person who used to hire technical people, what is that like? What kind of people are you looking for? What skills do you find deficient? What skills do you find lacking that you’d like to see? Of course, you’re joining us.

    And so, really, really excited about that. And then over in Europe, we’ve got a couple of other people joining as well—Eveline Oehrlich will be on that one as well as Cheryl Razzell, who is over at, I think, PolyCom right now, and she used to be at HSBC. She’s not a customer, I think, any more, but she was a customer ad she definitely has an opinion about this.

    So, we’re really looking for people with different perspectives to have a conversation. I’m gonna be on there, you’re gonna be on there. I’ll be moderating, I’m gonna be looking at the chat window the entire time. All the panelists will be able to respond to the questions that are posed in the chat. There’s gonna be a mechanism for people to vote the different answers up or down, which is pretty cool.

    Ashley: Mm-hmm.

    Fell: And my role is really gonna just be to interject and speak on behalf of the people on the chat and say, “Hey, Bill brings up a good point,” “Hey, Debbie just brought up a really good point, she disagrees with you,” “He thinks that that is wrong,” whatever. I’m gonna try and be a conduit to make sure that the audience feels like they’re part of that conversation.

    So, we’re gonna have 30 minutes of up front, set the table subject matter expert speaking stuff, and then we’re gonna have an hour of live Q&A or as long as that audience has questions, we’re gonna work on trying to get to them.

    Ashley: Wow, an hour. That’s great, that’s fantastic. I mean, you know, community in some ways starts with conversation. That’s what draws people in and say, “What’s happening? What are those folks talking about? What is that? Oh, I have a question, I have a thought.” And I’m sure we’ll get some opinions coming in on those questions. [Laughter]

    Fell: That’s right.

    Ashley: Sometimes, they’re sort of written that way, that, “Hey, don’t you think this is true, instead of the way you said it?”

    Fell: Yeah, exactly.

    Ashley: Which is actually, it’s some of the challenge as well as seeking information that helps advance it for all of us, and—you know, there’s more than one way to skin a cat, if you will.

    Fell: I was on a crowd chat with DevOps Institute, I think it was about a week ago, and we had folks from all over the world chiming in and giving their, some prompts that we were asked. It’s fantastic to see people, like you said, to challenge the status quo, to give their opinion of what’s worked for them to understand. Because that—you know, you get an understanding of how some people’s situations are just different. And you can’t—there is no best practice. I think Manuel Pais and Matthew Skelton, they talk about this all the time. There’s no best practice, there’s just lots of good practices—

    Ashley: Good practices.

    Fell: —that you can apply to various different, apply to different areas, but you can’t just say that there is one best practice.

    Ashley: So, kind of a variation on the best practice theme, I’ll just ask you your—neither one of us can predict the future that accurately. [Laughter] But what are your thoughts about this virtual event, virtual conferences? Is that something you think will stay with us in a meaningful form post COVID and whatever, how we’re working, all that environment?

    Fell: I do. You know, you saw O’Reilly has canceled all of their physical conferences.

    Ashley: Yeah, yeah.

    Fell: And so, regardless of what ends up happening, I think the same way that the businesses like Best Buy and the other folks that you talked about are transitioning and they’re shifting.

    Ashley: Mm-hmm.

    Fell: I mean, I’m seeing—this week, I’m starting to see all the activities that my children were involved in are now, they’re finally acquiescing and saying, “Okay, we’re transitioning, and now, here’s the Zoom link for you to join for the kung fu class and you’re just gonna do it in your living room, but you have to still wear all of your things and you still have to show up and be polite.”

    So, they’re also transitioning, and businesses are transitioning. They’re learning how to work more remote. I’m very lucky to be working at a company that’s, you know, we’re 99, 90% remote already, so it’s not a huge change for us, but it is a huge change for the folks that are working remotely in my company who have, now, children at home.

    Ashley: Mm-hmm.

    Fell: And so, education is gonna be impacted and I’m talking with Charlie Betz, who’s an analyst over at Forrester, he’s a teacher over at the University of St. Paul—or St. Thomas, excuse me. And he’s talking, he and I talk all the time about how is education gonna be impacted by this? What are—as a parent, I’m sitting at home and I’m working, but my children are here, and they should be doing some school things, but it’s very hard for me to keep up with that and keep track of it. And I think it’s hard for a lot of the educators, because they haven’t had the experience in the virtual meeting space that we’ve all been sort of used to for a while.

    And so, I think there’s a lot of learning that needs to come out of this, but I see—and one of the things that Charlie says is, when people have no jobs, they can do a couple different things, but one of the things a lot of different people do is they go back to school, which is a fantastic thing to do.

    Ashley: Mm-hmm. Oh, yeah, absolutely.

    Fell: And if you give people viable options to educate themselves and make themselves—again, all about the upskilling conversation that we were just having. How do we enable these people to be able to make sure that they’re gonna stay relevant, and how do we help them get jobs for the folks that are right now, that they don’t have a job because they’re forced to be in an office that’s closed?

    Ashley: Right.

    Fell: So, I think that there will be quite a bit of focus on how do we educate at a distance, how do we make the collaboration work better. And I think it’s happening already naturally. People are getting more accustomed to being on video calls, right? People who used to never want to be on a video call, now they’re flipping on their video, because it’s like—you know what? We’re all in this together, and one of the things that I really appreciate, before we started scheduling this, I ran around to everyone in my house and I said, “Hey, guys, please don’t scream for the next 30 minutes, because I’m gonna be recording something.”

    Ashley: [Laughter]

    Fell: But the fact is, is that I’m on meetings with people and, you know, my kid or my wife will walk by and we’ll speak. And now, it’s—it used to be where I was embarrassed by that, because it was an intrusion into my workday. And sort of the mind shift that I think is happening is that now, what’s happened is that I’m allowing work into my home.

    Ashley: Yep. And you’re entering into other people’s homes at the same time.

    Fell: And I’m also entering—yeah, absolutely, I’m entering into other people’s homes. But I think there’s a much more understanding and a much more—I’m more comfortable, anyways, with there being life happening outside of me. It’s like the classic BBC video of the girl, the 2-year-old girl who runs in the room. He was—he was, like, appalled that that would happen. And now, if that happened, hopefully, he wouldn’t be, right?

    Ashley: It’s called life.

    Fell: So, I think, as an organization—it’s life, right? And hopefully, as a humanity, we’re getting better about that, and I think we will.

    Ashley: You know, to give you some, a little bit of data to back up part of what you were saying is, we’ve launched—we started an IT, health emergency IT preparedness survey back in February, and things happened so fast that just a traditional survey couldn’t keep up with the changes that are happening with work at home and how this has evolved.

    So, we’ve also launched a flash poll survey, and I’ll be talking about it on TechStrong TV this week. But one of the initial ones before we announced this flash poll, one of the initial ones was, “What is the most critical—you’ve only got one choice—between kinds of services that you use to support your work at home staff? There’s e-mail, collaboration services, meeting, file sharing, phone—all kinds of things.

    Fell: And which ones had degradations, yeah.

    Ashley: Well, yeah. And it’s, surprisingly, collaboration and online meeting services were equal. Those are the top two, within 1% of each other.

    Fell: The top two worst performing?

    Ashley: No, this is of our most critical to my business.

    Fell: Most critical—okay, got it.

    Ashley: Supporting it. We’re gonna do another survey on what have you had issues with that’s actually launched this week.

    Fell: Yeah, I just—that’s why I saw it, because I just saw that one. I was like, “What?”

    Ashley: You just saw it. [Laughter] So, 37 and 36%, I think, online meetings was 37. And so—and email was, like, 20. So, that tells me people have already made this shift into Slack and Microsoft Teams and Hangout and Google and whatever other forms they have for kinda their corporate chat.

    So, I think that skill of collaborating that way is something that will be a natural outcome of the situation we’re in.

    Fell: Yeah, I think so. I mean—yeah, hard times force people to think differently.

    Ashley: Mm-hmm.

    Fell: And we’re a very resilient species. We’ve been around for a while, and we’ll figure this out, but it’s terrible to have to go through it.

    Ashley: It is, it is. We just hope everybody gets through okay.

    Fell: Yeah, exactly.

    Ashley: That’s what we’re hoping for. Well, Sam, it’s a great pleasure, a lot of fun, always, to talk with you. Thanks for being on the podcast and for joining us today, sharing and also sharing with Accelerated Strategies the Software Delivery Leadership Forum. Excited to do all of those events with you and take that dialogue to the next level and really build the community amongst the leadership side.

    Fell: Me, too—wonderful. Thank you, Mitch. I appreciate your invitation and stay safe.

    Ashley: You bet. So, you’ve listened to another DevOps Chat podcast. This is Mitch Ashley thanking everyone for joining us today. Be safe, be careful out there.

  • DevOps Chats: What Is Your Data Really Worth? with Splunk

    DevOps Chats: What Is Your Data Really Worth? with Splunk

    Gaining real insights can catapult companies ahead of the competition, but that’s easier said than done. Splunk recently released its new report What Is Your Data Really Worth? which produced some very compelling results.

    Leading-edge data innovators use data to raise gross profits by 12.5%. Ninety-seven percent of this top tier meet or beat their customer retention targets. Mature organizations are almost 10 times more likely to draw more than 20% of their revenue from new, innovative products and services. A select group of companies, categorized as data innovators, achieved impressive and measurable results.

    Andi Mann, Splunk chief technology advocate, joins Mitch Ashley on DevOps Chats to share some of the key insights in the report and discuss how companies are utilizing their data to achieve higher revenue growth, improved customer experiences and gain cost savings.

    As usual, the streaming audio is immediately below, followed by the transcript of the conversation.

    Transcript

    Mitch Ashley: Hi, everyone, this is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Andi Mann, chief technology advocate with Splunk, and our topic is a report that just recently released called, “What Is Your Data Really Worth?” The ultimate question, you know? It’s kind of, from, do you remember—what’s the, 42? Is that the answer? Okay, anyway, another topic. [Laughter]

    So, welcome, Andi—good to have you on DevOps Chat.

    Andi Mann: Hi, Mitch. It’s so good to be with you again.

    Ashley: Absolutely. Andi and I are friends from way back, so we’ve known each other for quite some time now. [Laughter]

    Mann: Yeah, this could get a bit squirrelly, mate. Let’s see how we go. [Laughter]

    Ashley: I made sure I didn’t have too much caffeine before we recorded, so hopefully that will kinda keep me in check, anyway, we’ll see. [Laughter]

    Mann: [Laughter]

    Ashley: So, for folks who don’t know you, tell us a little bit about what you do and what you do at Splunk and a little bit of your background.

    Mann: Sure. So, Mitch, I’m chief technology advocate at Splunk for our IT Markets Group. What that means, as an advocate, I spend my time explaining stuff to other people and advocating for technology mostly. So, I talk a lot. A lot of people will see me doing podcasts like this, they’ll see me at conferences and keynotes, writing, and other things. A lot of outbound talking about Splunk and our data delivery platform and all the great solutions we’ve got to bring data to everything.

    But a lot of people don’t see the other side of it, which is me advocating for my customers. It’s one of Splunk’s core values that we have two ears and one mouth, and we should try and use them at least in that proportion. So, I try to listen a lot to experts like yourself, like other analysts and pundits, customers, market makers, and try and help Splunk create the best possible products and solutions to make our customers successful.

    Ashley: Sounds like when I was a kid, the saying was, “God gave you two ears for a reason, Mitch.” [Laughter]

    Mann: [Laughter] Exactly, right? Oh, you weren’t the only one who got told that as a kid, huh?

    Ashley: No, I’m guilty as charged. [Laughter] So, let’s just jump to the report. Tell us a little bit about, obviously, you’re in the data aggregation analysis collection, all kinds of interesting parts of the data world and lots of different sources. How did you decide to try to answer the question of what’s your data really worth? What was the genesis of this report?

    Mann: Well, the first thing we did was, we decided to find an independent expert to help us. So, we went to Enterprise Strategy Group. Now, you know them, they’re an analyst firm out of Boston.

    Ashley: Yeah. Good folks, good folks, yeah.

    Mann: Really good folks. Their key areas where they focus on is data and security. And so, for the data part of this survey, that was a very obvious choice. So, they hope to figure out a bunch of questions to ask to see if companies were using their data in advanced ways or not. So, we were looking at things like how much data they use, where do they get their data sources from, which business departments use data in their decision making and a whole bunch of other outcomes.

    And we were able to figure out from that with working with ESG, within Enterprise Strategy Group, we were able to figure out a certain percentage, around about 10% of the survey respondents out of 1,300 respondents were what we call data innovators. So, they were taking more data, they were using it in more deliberate ways, they were using it in more business departments and more business decisions.

    And so, we were able to figure out, well, if they’re using data in better ways, what are the good things that happen when you do that, and comparatively, for the companies that are using data in sophisticated ways, what are the down sides for them?

    So, it’s a really interesting set of questions and answers that we managed to find some really interesting data on.

    Ashley: Great question to ask. So, tell us, what was the number one thing that jumped out to you? What was the learning you didn’t expect to get from this study?

    Mann: So, I think the, one of the learnings that I think we did expect to get was that using data better helps you in your business, in all sorts of ways. So, the one big area I think I was a little bit surprised at was that using data better is not just gonna help you save money, it’s gonna help you make money. So, for these data innovators, on average, they had a profitability of around 12% gross, across different sizes of businesses as well. That meant about $38 million average total gross profit for these innovative organizations who are collecting, managing, and analyzing data to improve their business.

    I think the second thing that surprised me was that this wasn’t even split between top line and bottom line. You know, in IT especially, we often look at ourselves as a cost center. We’re often told to do more with less. We’re often told to find ways to save. What we don’t hear a lot about IT is how important it is to making money, to adding revenue, but that’s what we’ve found from this research. Data innovators added on average over 5% to their annual revenue because they were using data better, and that added to reduction of cost of around 5% as well.

    So that’s how we get up to that 10 to 12% on the bottom line, but it’s a combo of making more money and saving money, which is—you know, there’s not a lot of technologies which you can point to for that.

    Ashley: Definitely both sides of the line, now. There was 5.32% over 12 months as a result of their data use. Talk about—how do you define a data innovator?

    Mann: So, some of the things we looked at for the data innovators—are they more or less sophisticated in their strategy? So, do they have a data strategy, do they have a chief data officer? Do they have specific plans over a 12 to 24 month period of how they’re gonna get more data in and use that data? Do they have analytics programs in place? Do they have a data science program in place? You know, these are some of the signals that we looked at to see what they were able to do with data.

    And then in terms of the outcomes, we looked at things like revenue growth, we looked at operational cost reduction, the ability to innovate. How long does it take to get new products or new ideas to market? We looked at outcomes as well, like customer satisfaction, customer retention, ability to make faster decisions.

    So, all of this sort of gave us this picture of what does a data innovator look like?

    Ashley: Now, just to give folks an idea here, this isn’t like baseball, everybody gets an award and gets to be called a data innovator. This is 11% of global organizations was your measurement. So, it is kinda top 11% kinda cream of the crop of folks.

    I found another interesting stat that I read in the report that said one in five data innovators generated more than 20% of their annual revenue from products and services developed in the past 24 months, compared to just 2% of data innovators. So, if you have really invested in analyzing, assessing, using, and applying data, that number is backing up what you said about—yes, it does add to topline growth.

    Mann: Yeah. Because, I mean, when you’ve got data to go on, you can make these rapid decisions. If you’re talking about innovation in life, you obviously have written about innovation quite a lot and we’ve talked about it directly a couple of times and, you know, to innovate, you’ve got to be able to make fast decisions. You don’t try things out.

    You know, typically, 95% of innovation will fail. That’s okay, as long as you fail fast, fail small, fail cheap, and fail forward. But if you don’t have the right data to make decisions, then all of a sudden, you’re in a data paralysis. You’re in a decision paralysis. You can’t make those rapid decisions. You end up having to get more information. You end up going with the loudest voice in the room, or you end up going with the loudest person in the room which, by the way, tends to work against some of the greatness of diversity and inclusion about getting different opinions, about getting different, diverse opinions and viewpoints.

    Ashley: Mm-hmm.

    Mann: So, if you—but if you have data, the data speaks for itself, and you can set gates for innovation. There’s a classic innovation theory—you try stuff out, you do it small, you set gates, you pass the gate.

    One example, for example, and directly for this audience as well, is thinking about what is a high quality release? Is it a release that has no bugs? Well, that might be too high a bar to get over, but to understand what level testing has it gone through, what is the pass/fail rate? What is the code quality? What is the compliance quality in your code? These sorts of things, they’re data points that you can then make decisions on.

    Even more so, you can automate decisions based on data points. If I’ve passed 99.92% of my tests and I’ve run 100% of the tests that I expect, that’s a really good mark and I’m probably gonna go straight into production with it.

    Ashley: Mm-hmm.

    Mann: So, I can iterate faster, I can do new things in new ways, because I have surety that the decisions I’m making are real and based on substantive information that will matter when I get to prod.

    So, innovation is absolutely a strong outcome that we see in this research as well, coming from these data innovators.

    Ashley: Now, I kind of threw the trick question at you first. What didn’t you expect to learn from this? I mean, let me toss that one out there again, since we talked about some things you did expect to learn. Were there any small, medium, large surprises that you walked away from some of the answers that came out of the research?

    Mann: Yeah, look, I think some of the surprising stuff was just sort of around the vertical side in the industries that were good and industries that were bad. I would’ve expected some other, some industries who are lower on the ability to get data insights, I would’ve expected them to be higher. Specifically, higher ed and public sector. They have a lot of access to data, they don’t necessarily have the same issues with data analytics and aggregation that private sector do. You know, they’ve got things in place around privacy and protection of data.

    So, there’s some really positive things. Higher education especially, I would’ve thought, well, they have investigative units. Research is something they do. I was a little surprised by that. What we saw was that technology organizations do really well by using data better. I sort of got that, that makes sense. They’re involved in machine learning programs and AI programs and things like that. They’re on the cutting edge. Somewhere around two-thirds of financial organizations had really good results in terms of higher revenue through better utilization of data assets.

    Ashley: Mm-hmm.

    Mann: And again, financial organizations are often on the cutting edge of technology and so forth, so that made sense. But for me? Higher education and public sector were only down around 50% at this ability to use data in operational ways. And honestly, that surprised me. I think they could do a lot better. I think they’ve got the fundamentals, the people, the technology, the inquisitiveness and the opportunity, certainly, to be able to use data in better ways. So, yeah, I’d love to see those numbers come out better.

    Ashley: Now, I wonder if, do you think there’s a correlation or a connection to this idea that those folks that were data innovators, their culture is, what you termed in this report, quote-unquote data obsessed. It was just data driven company, everything is—you know, decisions are driven by data, you know, collecting and using, it’s not just the loudest voices in your room. It seems like that’s a pretty high correlation there to the folks that are really in a place and are leveraging data in a very successful way.

    Mann: Yeah, the cultural aspect is really interesting. You know, we’ve talked about culture in the DevOps community for a decade or more, and how important the cultural change is. And, you know, we know from DevOps you can throw all the tools at the problem. If you don’t have a culture of collaboration and sharing, then you won’t have a collaborative environment to work in regardless of what tools you throw out there.

    Data is the same. The data’s there. The big difference between the data innovators and companies that aren’t necessarily as innovative with data, you know, the companies we call the data detractors and so on. One big difference is if they’re inquisitive about the data that exists and go looking for it and look for ways to use it. It’s not necessarily that they have more data, it’s not necessarily that they have different data, it’s not even necessarily that they have data that people don’t understand or understand better or worse. It’s that they have a culture that values data driven decisions.

    And so, when the decision comes to the—you know, the meeting comes to a decision factor, they have people in that room who deliberately put their hand up and go, “What is the data saying?” Rather than having the people in that room go, “Okay, I think we’ve got everything we need. What do we all think?”

    Ashley: Mm-hmm.

    Mann: Right? And that’s a cultural change, Mitch. That’s a cultural difference. Having that data obsession as we’ve termed it in the report means that your culture is looking for data, actively looking to make those data driven decisions. Actively looking to get data from everywhere and bring it to every decision. Not just the important or less important or whatever, every decision. And that absolutely is a cultural difference.

    Ashley: So, don’t take this—this is not a trick question at all. I’m interested or curious about your thoughts on data, data, data—extremely important, this report showed the value and the impact that can have. I’m thinking in an analysis standpoint, sometimes you can get into analysis paralysis or maybe the insights aren’t always just in the data, but from other factors and things.

    How do you blend both the tools, the experience, the knowledge, the capabilities of the organization and infuse that with data in a really healthy way? What are your thoughts about that?

    Mann: Yeah, look, that’s a super question, Mitch, because you know, there are lies, damned lies, and statistics, right?

    Ashley: Wow. Yeah, we—

    Mann: You can make data say whatever you want. [Laughter]

    Ashley: Yes, we can make it say whatever we want—true.

    Mann: So, you’ve gotta be careful about stuff, right? You’ve gotta be careful about bias, you know? I talked about diversity and inclusion a little bit. If all your algorithms are written by people that look and sound exactly like you, then they’re certainly gonna reflect who you are and what [Cross talk].

    Ashley: Mm-hmm.

    Mann: So, having a variety of data, but having a variety of algorithms created by a variety of people from different backgrounds—so, you’ve gotta have diversity in your teams. Again, it’s a cultural thing, right?

    Ashley: Mm-hmm.

    Mann: Your data will tell you what you want it to if you ask it. So, you’ve gotta have the ability to get more data, you’ve gotta have the ability to ask data, ask continual questions of your data. So, it’s not just the first answer. Typically, when you’re doing data analytics and inquisition, the first answer just pops up more questions for you. So, you’ve gotta be able to go through that iterative cycle of asking more questions. That’s a very fundamental and practical thing to do with how do you structure your data, what tools do you use to inquire after your data. You’ve also gotta have the understanding that some things are not necessarily a data decision. Some things actually don’t have data, and you do need to have personal experience.

    I’m a big believer, Mitch, in letting the machines make the right call on stuff they’re good at. Complex data, time series data, high cardinality data, long periodic data—you know, humans are awful at things like pattern matching, we’re awful at looking at long term data about patterns. Machines are really good at that stuff. So, let machines—

    Ashley: Also really good at large volumes of data where we, as humans, love a data point of one. You know, my kids—

    Mann: Exactly. [Laughter]

    Ashley: Those millennials … I have two data points at home, so I’m super set. [Laughter]

    Mann: Oh, yeah. anecdata, yeah, it’s the bane of our existence, I think. But it’s important to understand that machines can’t make intuitive leaps, either.

    Ashley: Mm-hmm.

    Mann: Machines have no imagination whatsoever. Have you ever seen some of the Harry Potter scripts that have been written by machine learning engines?

    Ashley: I have not.

    Mann: Oh, my goodness—it’s so awful, Mitch. It’s so bad. [Laughter] Because machines have no imagination. So, there is a cutoff point, and it’s a fair question to ask, and I don’t think there’s any definitive answer of where that cut point is. But at some point, you need to have a human to interpret and bring imagination, bring intuition, bring experience to that data driven decision.

    But I would certainly posit that you bring that human experience to the data, you don’t just go with a human gut feeling.

    Ashley: Mm-hmm.

    Mann: The data will tell you what to do.

    Ashley: I don’t recall this as looked at or at least talked about in the report—I would imagine those data innovators have figured out where that balance is, it’s not just about having the most data or the data, the answer is always in the data, right? It’s that balance of, you know, it’s the right sources, it’s the negative validation as well as the positive validation, the correlation, the analytics, how statistically valid is the information. So, all kinds of things that data scientists know how to do that help you be really good about how to use that data, and of course, that’s probably a maturity curve that you work up to.

    Mann: Yeah, exactly right. I mean, we see—and it’s not necessarily specifically in the data, but we absolutely looked at that maturity curve and what it means to be a data innovator versus a data adopter versus a data deliberator, someone who’s in the early phase, for example. We deliberately looked at what it was like as a company, what patterns. And again, in the DevOps community, they’re very familiar with this concept of patterns and anti-patterns.

    Ashley: Mm-hmm.

    Mann: And the patterns that the data innovators took gave us a sense for what is a mature business. You know, we’re not necessarily gonna be able to tell you exactly what those gates are that define good innovation or define a good test outcome, or define a good marketing campaign or whatever it is, a product that will be successful.

    But what we can do is help you understand what the data told us about data maturity. And so, that’s why we’re actually working on, we actually loaded up onto the Splunk.com website a data maturity calculator. So, it’s a free—it’s a web based assessment tool, it’s free, obviously, so you can actually compare yourself against some of these data innovators.

    You know, a really easy way to assess what’s your data use, what tools you need to get the most out of your data, what data you’re missing and how do you compare on this data maturity curve to be able to make those decisions between smart, experienced individuals and definitive or maybe not so definitive data and data driven decisions.

    Ashley: Mm-hmm. Interesting. Great. Well, we’ve used our time pretty well here. Let’s certainly find out how do folks get this information? You talked about this assessment tool of kinda where are you on the data innovator curve—how about getting the report, information about it?

    Mann: Yeah, absolutely. So, it’s all available up on Splunk.com. In fact, if you go there on the homepage right now, you will see the data-to-everything platform, and you’ll see links there to be able to get that video. You’ll be able to get, also, stories from some of our customers who have used data and turned data into delivery. And, you know, household names like Domino’s and others are really using data to create an impact, to be a data driven organization.

    So, yeah, jump onto Splunk.com, you’ll be able to see the report there, you’ll be able to jump on, read the report. You can also take that assessment for yourself. It’s gonna be really fascinating to understand how you line up with those data innovators and where you can go to try and get a slice of that extra money that those innovators are getting.

    Ashley: Great. Splunk.com, great place to go for lots of other information as well as this report. Say, by the way, before we end things, I hope you come back. I’d love to have a conversation with you about data in the DevOps world. One of the challenges—data has such a different nature, right, than—

    Mann: Yeah.

    Ashley: What we can do, you know, more flexibly with software and configuration automation and things around software, oftentimes developers sort of struggle with this amorphous, large piece of data or collections of data and how that’s evolving along with things like DevOps. So, I’d love to have a conversation about that.

    Mann: Yeah, that would be a great conversation. Yeah, I would love that, Mitch. That’d be cool.

    Ashley: Awesome! Well, hey, thanks a lot, Andi, for joining us today.

    Mann: Thank you, Mitch. It’s a pleasure. Always great to talk to you, mate.

    Ashley: Always great to talk with you. And I won’t talk about where Australia fell on the list of data innovators, but that’s another topic, so.

    Mann: [Laughter] I know! Don’t—please, don’t! Thank you!

    Ashley: Okay, alright. [Laughter] Well, you’ve listened to another DevOps Chat podcast. I’d like to thank my friend, colleague, someone I’ve known for quite a while, Andi Mann, chief technology advocate with Splunk for joining us. And thank you, of course—you, our listeners. This is Mitch Ashley with staging-devopsy.kinsta.cloud. You’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: Security Graph SDK Powers Better Policies, with vArmour

    DevOps Chat: Security Graph SDK Powers Better Policies, with vArmour

    The increasing breadth and complexity of the environments we manage are nothing short of breathtaking. Understanding the security risks and applying policies across the cloud, multi-cloud, private clouds and a plethora of software technologies is a challenge faced by every enterprise.

    vArmour recently announced version 5 of the vArmour Application Controller, which opens more significant access to vast amounts of information the Security Graph represents and organizes through its Security Graph and SDK. By streaming cloud, network and agent information into the Security Graph, enterprises now have a centralized understanding of application relationships across their multi-cloud infrastructure, enabling centralized risk and policy management that is simple, accurate and secure.

    In this DevOps Chat, Marc Woolward, vArmour CTO and CISO, and I discuss how establishing and assessing policies becomes more manageable through expanded access to information about software infrastructure, systems, environments across the organization.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.


    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Marc Woolward, who is CTO and CSO at vArmour. We’re talking about policy and risk of cloud applications, how to manage it. Marc, welcome to DevOps Chat.

    Marc Woolward: Thanks, Mitch. It’s great to be here.

    Ashley: Well, thanks for flying over from across the pond, too. [Laughter] You’re in the U.S., with your U.K. accent, so thanks for being on.

    Woolward: There’s quite a few of us out here in Silicon Valley. To be honest, I think it’s half British.

    Ashley: There are, there are quite a few of you here. [Laughter] Would you do us a favor and just introduce yourself, tell us a little bit about you, what you do at vArmour, and just a brief on what vArmour does.

    Woolward: Yeah, of course. Alright, well, hi, everybody. As Mitch said, I’m Marc Woolward, I’m chief technology officer and also, in my spare time, the CSO at vArmour. My background is, I’ve been at vArmour for five years or so and I look at kind of the strategic direction, I speak to the senior technologists and our customers to understand their challenges with cloud security, and I lead some functions such as, you know, the data science team, and handle things like CCPA and GDPR in my role as CSO.

    So, my background is, before I joined vArmour, I was a technology fellow, and I was the CTO for telecommunications and networking at Goldman Sachs. I think I was there for 18 years. Did lots of things there. One of the interesting parts of the role of a tech fellow at Goldman Sachs is, you kinda get involved in all sorts of things. So, I was a member of the private cloud architecture team, you know, almost a decade ago, we built one of the first private cloud environments. And also, I had an interest in application resilience and application design patterns, despite my background keynote infrastructure. So, that kinda set me up really well for a job in a product company in Silicon Valley.

    Ashley: Yeah, very much so, especially the proliferation of cloud and all of that. So, let’s jump into it. I think we’re gonna talk about an announcement that was made back in October—what was it, version 5 of the application controller from vArmour?

    Woolward: Yes.

    Ashley: Can you tell us a little bit about what that announcement was, and then we can get into some of the effects that you’ve seen in the market from it?

    Woolward: Yeah, of course. Well, I mean, first of all, I think I probably should tell everyone a little bit about what vArmour does.

    Ashley: Oh, yes. Please.

    Woolward: So—yeah, no problem. I didn’t answer that in your last question. So, really, we focus on helping our customers to understand how their applications behave within cloud and multi-cloud environments. So, we have a risk and a policy engine that kind of builds a picture of your application behavior, its dependencies, and suggests the ways in which the application should be secured from a policy standpoint, and we help to make it—we make it simple for our customers to implement safe, secure, simple policies within cloud environments.

    So, what version five was all about, underpinning the application controller function itself is a security graph, and affecting that is a graph that describes all of the nodes, you know, the compute instances, the Kubernetes pods, the servers within a multi-cloud environment and the relationships between them. And what we’ve been doing is enriching that graph with all sorts of information about the application, compliance requirements, information about what the application does, is it there, what ___, what realm does it run in, what region does it run in?

    And really, what version 5, release five was about was opening that up and exposing more of the information within the graph so that customers can benefit from it more. So, what that means is the ability to plug more and more information into the graph to gain more value out of it. Things like information that reside in CMDBs, information from systems that handle vulnerability management, information from control planes, things like Kubernetes so that that information within the graph becomes richer and richer and more valuable. And it can be used for all sorts of use cases, and it also allows companies to build the policies based upon their view of risk and their view of their applications, whether that’s compliance driven or risk driven or business unit driven.

    Ashley: Now, can you kinda dig down and open up the hood a little bit more for us? Talk about how all this information gets into vArmour addition the application controller. Are you taking it from logs, configurations, active applications? What are some of the sources for this?

    Woolward: Yeah, and—great question. And what I do, I’ll start out by answering that by saying, in the cloud, we do not have a deficiency of data. So, if you’re running a reasonable sized cloud environment, you have tremendous volumes of logs—you know, cloud trail logs, flow logs, all sorts of event based information. And that’s—well, event based data. And that’s data.

    The problem with that is data—it’s unstructured, and it’s in huge volumes and it’s very difficult to kind of glean too much value from it. And what the application controller does is, it takes the data and it transforms it into knowledge. So, it provides you with insight into your application, which makes it a lot easier to make good security decisions.

    Ashley: Mm-hmm.

    Woolward: So, I’ll kinda come back to that in a minute. But the way that we do it is, we have a distributed telemetry pipeline where we use edge compute functions distributed out to the cloud environments that consume the logs and also consume events off of event buses and APIs. So, when a compute instance is deployed, we pull all of the metadata associated with it.

    The edge compute function we call message bus summarizes that data, enriches it, and then distributes it into the centralized graph where we effectively are transforming that data from being the individual flow log information into adding information into an edge within a graph, which represents the relationship. So, all the time, we’re enriching and summarizing, and that enrichment provides context, which makes it a lot easier to understand what your application is all about.

    Ashley: Well, and given the dynamic nature of our applications—cloud native, all of that, whether it’s Kubernetes clusters instances, microservices, just knowing what you have is half the battle, more or less what’s this information telling you? So, that’s got to be a value, too.

    Woolward: For sure. And, you know, Kubernetes is a fantastic example. Because if you’re capturing that information at the wrong level, if you’re looking at the IP address level or even the pod level, there’s a tremendous amount of noise in there and it’s very temporal and it doesn’t make any sense. If you can consume that at the abstraction, that makes sense to the application, so that would be at the deployment or at the service level and you ingest that into the graph you can build a picture of your application over time.

    And then when the, kind of the instantiation that’s just been deployed now varies from that model, you can see it very quickly. So, what we’re trying to do is summarize the information and represent it as—I say represent it as knowledge while continuing to process kind of the very dynamic and very high volume data and look for the anomalies and look for places where it varies.

    Ashley: That sounds, too, like a perfect application for a graphing technology to be able to visualize, explore, dig in deeper, back out, look at it at different levels.

    Woolward: Yeah. For sure, and you know, like I said, my background of working at Goldman and working in financial services, graph technology is fantastic for describing relationships, and that relates to risk. So, if you look at financial risk modeling, that has been almost the archetype application for graph technology for over a decade. I really would take that same approach and that same principle to cyber risk and cloud security risk.

    Ashley: Very good. Now, I know also that you’ve introduced an SDK with this. What are some of the applications where customers might want to use the SDK?

    Woolward: The whole point of the SDK—so, we integrate with all of the common cloud environments and we integrate with SDNs and the other product is designed to integrate into these different environments, just as part of the product out of the box.

    Ashley: Mm-hmm.

    Woolward: But what we’ve been finding with our customers is, they have different repositories for this context that would enrich an understanding of risk. So, if you have a vulnerability scanner, you have a tremendous wealth of information about the posture of the individual systems running in your environment. By plugging that into the graph, and by using the SDK, it’s very easy to do that for whatever system you’re using, you can begin to ask questions like, “Well, for my application, do I have any dependencies on systems with, let’s say, a CB severity of seven or above.” So, the SDK is a way of pulling in the vulnerability information, pulling the specific fields that are of value to you from your CMDB from, say, your service now, effectively plugging in that context that kind of illuminates your whole kind of understanding of your application and your risks.

    So, really, it’s about allowing our customers to integrate the data sources that matter to them into this graph that plugs into their cloud.

    Ashley: Mm-hmm. So, both in kind of an in and out application—bring in more data, also be able to pull it out for other uses reporting.

    Woolward: Yeah. That’s a superb point. I missed that. Another application that customers use the graph for is, embedded within the app controller, we have classification engines. So, what we do is, we will classify systems based upon our understanding of their behavior, based upon the relationships they have and the systems they have relationships with, we’ll classify them.

    So, one great use of the SDK is to compare the results of our systems classification with the information within a CMDB. So, you know, does your intent, which is in the CMDB match what’s actually happening in the cloud environment?

    Ashley: Mm-hmm.

    Woolward: And by doing that, what you can do is, you can validate that your inventory information is correct, is accurate, continues to be so—which, again, improves your security posture because, you know, one of the fundamental pillars of security is understanding what you have, and having an accurate understanding of the systems that you want to secure, so yes, for sure.

    Ashley: Absolutely.

    Woolward: But, you know, it’s a two-way thing, the SDK.

    Ashley: Well, and you’ve got a little bit of time under your belt with GDPR and we have the California Consumer Privacy Act coming up here. How does this kinda technology help satisfy, meet, report compliance—what are the various ways that you could leverage vArmour and the application controller to help you with those challenges to meeting regulatory requirements?

    Woolward: For sure. Yes, so, I mean, there were different regulatory frameworks that are more or less specific about the security controls required. So, if you take something like PCI, it’s very specific and it’s very directed in terms of the way in which you structure your controls. So, the app controller can allow you to scope your in scope infrastructure around your card holder data environment and build the layers of controls around the tiers of systems, according to the PCI DSS. So, it’s very good at doing that.

    If you look at something more abstract, which talks more about the outcome rather than the means of getting to the outcome—so, if you say it’s something like GDPR where you’re looking at protecting personal information relating to data subjects in Europe or CCPA where it’s more around protecting the personal information around California based consumers, that really comes down to managing risk and ensuring that the control—suitable sets of controls are placed around the systems that either store or process this information. And the first step to do that is to understand the scope of the problem and the scope of the relationships of the systems that are processing this personal information.

    So, the app controller, first and foremost, can allow you to understand, you know, the scope of these environments and begin to put controls and reporting around their behavior and access to those systems. So, it all starts from scoping and understanding your environments and ensuring that you can put the controls around the—you know, place the controls around the edge of them, and you continue to monitor that those controls are effective and that, effectively, in the cloud is what the app controller does.

    Ashley: Mm-hmm. Very good. Well, I certainly don’t profess to be an expert on either of those two regulatory frameworks. [Laughter] I’m really curious, we’re about a month, month and a half in since you’ve released version 5. As a former product creator myself, it’s always interesting to see what people do with new technology, new versions. But also, what you didn’t expect them to do with it. Have you seen kinda instances of both, you know—yep, we thought we’d help customers this way, it’s worked out that, and then we’ve also seen some interesting ways they’re using it they didn’t think they would.

    Woolward: Yeah. So, I’ll start with kind of what we expected and kind of the, if you like, the continued illustration of the value that we designed the product to offer. One of the things that we never—well, I never fail to be amazed by is, you deploy our system within a cloud environment and it begins consuming the telemetry. And within an hour or so, the graph is already beginning to hydrate and you’re finding hundreds or thousands of workloads.

    And all of a sudden, the extent of the interrelationship and often the external dependencies, the relationships outside of the VPC or outside of the VNET kind of are visualized. And almost every time, as we sit there for the first few hours with the customer, we’ll get into the same discussion about, “What’s that relationship? Oh, I have a dependency on something in GCP.”

    So, generally, that kind of uncovering the unknown and unsuspected dependencies opens up a whole kind of, you know, can of worms for understanding what’s going on here and ensuring that the correct policies are in place and ensuring that everybody  understands the extent of the attack surface and the blast radius. So, that, we always see, and that’s just accelerated with version 5.

    In terms of what was less expected, that’s the beauty of the SDK. So, the SDK allows the customer to plug in the information and the metadata that’s important and pertinent to them. An area where I didn’t really expect it was actually the application not just to your traditional cyber risk and your attack surface and your blast radius, but asking some interesting operational questions.

    So, this use case came up about two weeks ago where a customer was taking kind of an operational term for their applications and plugging it into the graph, and asking the question around, for my critical tier applications, my systemically critical applications, if one of their dependencies, one of the applications that they depend upon for, I don’t know, reference data or for other types of communication, like communications gateways failed, what would be my recovery time?

    And all of a sudden, what you can do is, you can draw within the graph the relationship between your application and the dependencies and plug in that recovery time objective, and all of a sudden, I understand that if application A, B or C failed, I might be waiting eight hours for recovery, whereas my system has a recovery time requirement of one hour.

    Ashley: Mm-hmm.

    Woolward: So, they’re beginning to ask these questions that you would not expect from, necessarily, a cyber tool around operational risk. So, I think that’s the beauty and the power of the graph—it allows the customer to think about, what does risk mean to them, and they can plug the information or plug the data in that they require and then they can ask these questions that lead them to knowledge.

    Ashley: Excellent, yeah. It sounds like both information, operational information, but also behaviors of what’s happening between applications and greater insight into it.

    Woolward: Absolutely, yeah. Yeah, for sure. And I think the other thing, I mean, the final thing I’d say is, if you look at most of these data breaches over the last year or two, it kinda comes down to, if you look at cloud and you understand the way that security works, it’s this shared responsibility model. So, the cloud service providers provide a tremendous array of tools—you know, micro-segmentation, identity access management policy enforcement. There’s a tremendous suite of security capability there.

    The problem is always that the lack of knowledge about the application means that the policies that are written and the permissions that are provided by the customer, because that’s their side of the, kinda the shared responsibility model have been deficient. And what the application controller allows you to do is to build the policy in a simple fashion that’s secure and meets your business requirements, and then that’s allowing our customers to meet their part of the shared responsibility model for cloud security.

    Ashley: Mm-hmm. Wow. Well, I feel like we’ve barely scratched the surface and we’re already out of time. Hopefully, we can get you back and we can explore this some more as we get closer to the CCPA coming into effect in January.

    Woolward: Yeah, and as the CSO of a California based company, I’m obviously spending a bit of time on that as well, so.

    Ashley: I’m sure you are.

    Woolward: That’d be fun.

    Ashley: Yeah, exactly. Well, Marc, thank you so much for being on DevOps Chat with us.

    Woolward: Yeah, it’s been a real pleasure, Mitch. Thank you—thank you for inviting me.

    Ashley: Absolutely. The pleasure has been ours. I’d like to thank my guest, Marc Woolward, CTO and CSO with vArmour and of course, thank you, our listeners, for taking your time to join us and listen to this podcast episode. This is Mitch Ashley with staging-devopsy.kinsta.cloud. Have a great day, and be careful out there.

    — Mitchell Ashley

  • DevOps Chat: Is the Software We Create More Secure? Veracode’s 10th Report

    DevOps Chat: Is the Software We Create More Secure? Veracode’s 10th Report

    Application security is top of mind now more than ever. For more than a decade, Veracode examined increasing amounts of code as it passes through their source code vulnerability scanning service. During this period, automation is increasingly prevalent, making it easier to run scans more frequently and regularly. But has automation helped? Is the software we create more secure? We gain key insights about this in Veracode’s The State of Software Security Report X (10th edition).

    Chris Eng, chief research officer at Veracode, joined us on DevOps Chat. We talked about many insights uncovered in the latest report, such as 50% of applications are accruing security debt over time, the regularity of scanning correlates to vulnerability fix times and that scanning frequency directly impacts security debt.

    There is a wealth of information in the report, and you can get a jump on the key findings on this podcast episode with Chris.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Chris Eng, who is chief research officer at Veracode. And we are talking about their latest report, the Veracode State of Software Security. This is volume 10. Chris, welcome to DevOps Chat.

    Chris Eng: Thanks, Mitch.

    Ashley: Great to have you on. Why don’t we start out by having you tell us a little bit about yourself, what you do at Veracode, and for those that don’t know Veracode, just a little bit about Veracode?

    Eng: So, as mentioned, I’m chief research officer here at Veracode. I’ve been here for 13 years now, and my teams are responsible, essentially, for building the knowledge that goes into our products.

    So, we have a number of different products that help our customers identify vulnerabilities in their software. And my team essentially identify—well, what are the patterns that we’re looking for? What are the common mistakes that developers are going to make? What are the important things that we should scan for in each language? And specify to our engineering teams how we do that.

    I also own product security for Veracode, so making sure that we deliver secure products to our customers. Our customers are basically anybody that builds, buys or deploys software. And so, we help them build programs around securing that software at scale, using a number of different technologies, including static analysis, dynamic analysis, software composition and so on.

    So, we work with anybody that deals with software, which is pretty much everybody.

    Ashley: [Laughter] Just about everybody these days with digital transformation, no doubt. Thank you very much for that background on yourself as well as Veracode.

    Well, let’s start out with, you’ve been doing this report for 10 years, and your longevity with the company, you’ve been able to oversee and be part of that over that decade process. What kind of things have you learned since you started out? You know, kinda the size of how many applications and things that you’re scanning to what’s happening today and any trends that you’ve noticed from that?

    Eng: Yeah, it’s been a great process to do this every year, primarily because we’re in this unique position where we actually can see what’s going out there in the industry and when we started doing this in volume one, we only had about 1,500 applications in the data set, and then fast forward to this year, we had 85,000 unique applications.

    Ashley: Wow.

    Eng: So, 50 times more than we had before. And I’m not aware of any other study that has quantitative information on software security that’s anywhere close to as big as this one.

    So, in addition to the size of the data set getting bigger, we looked at fix times, and we found that those have roughly stayed the same, which is sort of a little bit depressing—but that being said, there’s a lot more software now than there was before. So, there’s a lot more to scan, there’s a lot more to secure, and so, you only have so much bandwidth to do this stuff.

    So, we saw fix times stay roughly the same. We did see that, over that 10 year period, there are fewer apps that have no flaws. So, that’s more of a factor of our capabilities than anything else. We can detect more than we could before. But we did also see that there are fewer apps these days with no high severity flaws. So, customers are getting better at identifying and remediating high severity flaws.

    We looked at a number of different angles, including compliance trends and things like that, but for the most part, I think that the fix times is kinda the interesting part, and we dug a lot more into that and explored some of the factors that lead into fix times.

    Ashley: Well, if you just think about what’s changed over a decade, I mean, 10 years ago, we were thinking about SQL injection, cross site scripting—you know, those are the kind of app security flaws, at least a lot of what the focus was, things have changed drastically since then. [Laughter]

    Eng: Yeah, we are still seeing the vast majority of the same flaw categories that we saw 10 years ago. And I don’t think anybody who does app sec on a daily basis would be too surprised by that.

    Ashley: Mm-hmm.

    Eng: These things are really easy to fix technically, but when there’s so much of it and it spans across such a huge application inventory, it does take a lot of effort to close these. So, we still see SQL injection at roughly the same rate that we saw it 10 years ago. Other categories, like cross site scripting, have actually gotten more prevalent.

    And we’ve seen some go down. We’ve seen, like, buffer overflows and numeric errors reduce in prevalence. And part of that, I think, is due to the change in programming languages as well, right? There’s a lot of that that’s stuff that’s prevalent in native code like C++ and we’re seeing less, fewer applications being written in those languages today. So, I think that’s contributing more to the prevalence differences that we’re seeing.

    Ashley: I wonder, too, if service oriented architectures, microservices, things that promote reuse of code maybe help some of those things. Maybe it won’t—we’ll see. [Laughter]

    Eng: Yeah, it certainly code. I mean, any time there’s reuse, you know, you inherit the functionality, but you also inherit the risk, and that’s sort of the trends as we’ve seen people start using a lot more open source than before. Again, you get the functionality, but you also get the risk, and so you introduce new vulnerabilities to your applications that way.

    So, certainly, with code reuse, like you mentioned, you could get that effect as well.

    Ashley: Mm-hmm, potentially even infrastructure as code. There were some really great things that came out of the report. I’d love to have you talk about some of the applications incurring debt, how many folks are driving that down—you know, what kind of effects that had.

    Eng: Yeah, we took a look at security debt for the first time in this report. And, you know, most people have a concept of technical debt—just, you know, things that get old and crusty in your software over time, maybe architecturally, or things that you meant to go back and fix but you never did, and security flaws kinda have the same tendency to build up over time.

    If you think about it like financial debt, right, if you charge something on your credit card and then you only pay the minimum amount every month, you’re gonna be paying a lot, for a long time. And when you add that up, it’s gonna be a lot more than it would’ve just been to pay off your balance right when you accrued it, right?

    And so, security debt, when we look at security debt from that angle, we look at, are applications kind of accumulating new security debt over time, or are they driving it down? And when we look at the number of flaws that are kinda left unfixed in an application, we kinda view that as the security debt, right?

    And, you know, there’s only a certain amount of capacity, right? It’s very difficult for an application that, let’s say, has been accumulating security issues over many years to just say, like, “hey, we’re gonna pay that all down today,” right? You’d have to dedicated a lot of engineers to doing that, you’d have to really focus all your efforts to do that. And so, it’s not typically very practical to do that all at once, but it is important to kinda be measuring whether or not you’re fixing more than you find or finding more than you fix, because that’s kinda directional, right?

    And so, we found that nearly half of the applications are finding more than they fix. And so, essentially, they’re accruing that debt, so they’re never gonna climb out from underneath that unless they start to reverse course there, right? Put more effort into fixing issues so that they, over time, reduce that debt and get to a point where—okay, now there’s nothing kinda outstanding, and you can then make a rule that says, you know, “we’re not gonna allow this application to be promoted into production or whatever if we accrue new findings.”

    Ashley: Mm-hmm.

    Eng: But that’s rare, right? So, barely anybody is there yet. And so, we looked at how different DevOps tendencies or methodologies, practices affect security debt. So, I think those are some of the findings that I’d like to talk about a little bit.

    Ashley: Sure, let’s go to that, because I’m very curious about, as we shift left, right, we’re trying to get security in earlier—

    Eng: Exactly.

    Ashley: [Cross talk] earlier, all of those things, hopefully you fix it before it ever goes into a QA function or an automated testing.

    Eng: Right, there’s all those studies to kind of show that it costs a lot less, right, when you fix it earlier. And that makes sense—if you’re fixing a flaw as a developer is writing the code, if you can sit there and tell them what they did wrong before they even check that code into the repository, it’s gonna be a lot cheaper than if you find it in a penetration test, let’s say, after the code has been deployed and then you’ve gotta go figure out the root cause and then you’ve gotta find a developer that understands that code and get it in their backlog. So, that’s kind of an accepted understanding that it costs more, the later you do it.

    Ashley: That’s after you get done pointing fingers on whose code is it in or is it in the network or the servers or the—you know?

    Eng: Exactly, exactly—then it’s kinda tracing it down. And so, we wanted to really say, like, does DevOps make a difference in how quickly we can fix things and how much security debt we accrue?

    And so, sitting from where we are, remember customers submit their applications to us and DevOps is not just automation, it’s also culture and process and a lot of things that kind of feed into whether you’re doing DevOps or not. But from our vantage point, we can’t see culture, we can’t see process, we can’t see automation. We can’t see to what extent automation has been incorporated into an applications testing cycle, for example.

    We use scan frequency as a proxy for whether an organization is using DevOps. So, by that, I mean, how many scans do they run per year on an application? And so, you have a lot of applications that are scanning literally once per year, right—36% once a year.

    Ashley: Wow.

    Eng: And if you think about how quickly software is changing and how many features are getting added every week or every month, once a year is—that’s not very good. On the other end of the spectrum, you have about 0.3% of apps that are scanning basically every day or more. So, you know that that’s probably being done by automation. You don’t have a person sitting there submitting the application, hopefully.

    And so, if you look at—there’s some charts in the report that kind of show what that distribution is, but essentially, if you scan once per month compared to if you scan every day, your median time to remediation is significantly faster. So, 19 days median time to remediation for a flaw if you’re scanning daily, versus 68 days if you’re scanning monthly or less.

    And so, we then kind of broke it down into different, even more buckets, right? So, we did kind of what does one to three scans per year look like, four to six, seven to 12 and so on. And it’s kinda hard to describe this just via audio, [Laughter] but if you pull out the report, you’ll see all these kind of like iceberg charts. And so, you’ll see, like, pink and blue, and there’s—the pink represents the debt. It’s kind of below the line and that kind of shows the number of findings per app that are outstanding.

    And what you’ll see is that, for the apps that are scanned more frequently, less debt accumulates. Essentially, the teams were able to get after that debt faster, and while it doesn’t go away completely—again, because this is aggregating all of the apps we have—it accumulates less. And so, we see a direct correlation between that scan frequency and both the fixed time and the security debt accumulated.

    Ashley: It certainly makes sense automating that. That’s gonna show up in your numbers around the 19 days to fix, scanning once a day. Is there a way to tell if it’s happening? Do you see scanning happening more frequently than on a daily basis? Is that helpful, or are there other things that help—again, help this shift left?

    Eng: You know, I think once you get to a daily basis, you’re probably—I think there’s gonna be some diminishing returns after that. I mean, imagine you did a full scan every time somebody checked something in. You’d just be getting a lot of information—you wouldn’t be able to act on that quicker than, you know, I think the span of a day or a few hours. So, scanning every few minutes really wouldn’t buy you anything.

    Ashley: Mm-hmm.

    Eng: So, another thing that we looked at that we thought was interesting would be scan cadence. So, not so much how frequently were you scanning, but how regularly are you scanning? And so, you can imagine—and again, there’s this great diagram, it’s my favorite diagram in the report, actually, and it’s just a bunch of dots on a chart. But what it does is, it maps out that every dot represents a scan.

    And so, if an application is scanning on a very steady basis, you would see evenly spaced dots across the course of the year, whereas if you were scanning in kind of a bursty fashion—so, basically, a lot of activity followed by no activity, then you would see a clustering of dots followed by just a bunch of white space.

    And so, we actually calculated that cadence for every single app in the data set, and then we grouped them into buckets, again, so of the ones that were either scanning on a steady basis or a bursty basis or something in the middle which we called irregular—and irregular is basically just a bunch of mini-bursts, right?

    Ashley: Mm-hmm.

    Eng: So, what we wanted to answer was, how does that scan cadence affect security debt? Are you less likely to accumulate debt if you’re scanning steadily or in a bursty fashion, alright? Which one’s gonna be more effective at wiping that out?

    Because you could see it going either way, right? You could set it, you could just—people could get used to seeing results and then they get to the point where they ignore them, maybe. Whereas bursty, you’re like, “okay, well, I’m paying a lot of attention to this. We’re gonna focus all our efforts on this” and then, you know, drive it down.

    So, there was actually a question there. We didn’t know what we were going to see. But when we did break it out, we found out that when you do the bursty scanning, you have all that white space and you just put a flurry of activity over time, like, you’re just—that security debt that accumulates is just massive. You see this huge increase in the amount of pink, which represents the findings that are not addressed, whereas with the steady and even to some extent the irregular scanning, you see the security debt grow a little bit, but then start to decrease. And so, the curve is actually going in the right direction towards the end of the time frame that we’re able to chart out. And so, that was another good finding for us.

    So, what we can do is, we can actually take those conclusions and advise our customers and, really, anybody that’s building software that, if you want to reduce security debt, the data suggests that you should scan frequently and that you should also scan in a steady basis. You shouldn’t ever let up, right? It has to become—and that makes sense, right? Anything that we make a habit of tends to just become part of the way that we do things. And so, it was nice to have some data that actually reflects what we think would be true actually does turn out to be true.

    Ashley: It’s kinda like, brush your teeth daily, right, not the day before you go to the dentist. [Laughter]

    Eng: Yeah, exactly—that’s not gonna do you much good.

    Ashley: [Cross talk] [Laughter]. You know what also occurs to me is, this bursty style is, there’s really no predictability around how long a security flaw is gonna exist in your code, because you may find it and fix it now, but it may be—if it’s another three months or whatever before you scan again, who knows when it gets fixed? It could live out there for a long time.

    Eng: Right. We actually did find—it’s funny that you mention that—that the highest probability for a flaw to get fixed was in the first month or so. And so, the longer you wait, the less likely a given flaw is to get fixed.

    And so, essentially, developers are prioritizing kind of, like, last in, first out. They’re more likely to prioritize something to get fixed if it’s kind of fresh. And that’s not what we wanna see, right? We wanna see developers fix things that are more important. We wanna see them fix more severe items or items that are in applications that are more critical to the business or items that are more exploitable than others.

    But when we measured all of those and we looked for patterns that would suggest that they are prioritizing in that way that’s sensible to a security practitioner, we found that that recency, like, how recently was it found was really the highest correlation in terms of whether something was going to get fixed.

    Ashley: Mm-hmm.

    Eng: And so, we have to do a better job of prioritization, not just understanding what a security person would do, but getting a developer to adopt those same priorities.

    Ashley: You’re obviously not measuring the human psyche element of this, but I have to believe you’ve built up such a mountain of that debt, at some point, it gets too hard to grok, understand, fathom, and there’s probably even an abandonment rate that’s just like, “too big—let’s just deal with what’s on our plate right now, and here’s the scan results” and jump on it.

    Eng: It does, and that’s not—that’s not uncommon to see customers adopt that type of strategy. That, the idea that—alright, well, I’m starting this program now, on day zero. I’m responsible for appsec now. And, you know, anything that happened before I got here? Well, that’s not my problem. Let’s just focus on not introducing new flaws—which is fine. Like, not introducing new flaws is great, but that doesn’t help you with anything in the past.

    Ashley: Right.

    Eng: And ultimately, it’s all risk to the application, right? You can get attacked any number of these ways. But we have seen—we’ve seen big customers try to take that approach and, you know, it’s not advisable. You have to chip away at that debt. Even though you can’t do it all at once, maybe you work in security sprints into your life cycle, you find a way to pay it down over time, and eventually, you get to where you wanna be. But you cannot just ignore it and say, “No new flaws going forward.” It’s not gonna get you where you want as far as the debt.

    And it costs more to fix stuff later, too, right? The longer we wait—you know, imagine you’ve got a library to patch and that library is, you know, one year out of date or five years out of date. It’s gonna cost you a lot more in terms of effort to fix the one that’s five years out of date. Things will inevitably break, so the cost of fixing also increases over time.

    Ashley: There are hundreds of findings, if not more, [Laughter] in this report. Not to pick out just one, but it stood out to me that there was one stat in there of C++ carries three to five times more unresolved flaws than .NET over the same period of time. So, this is to the point of, languages make a big difference, and why is that so?

    Eng: Yeah, you know, we did break it down by language and just kinda took a look at how much security debt accrued into each one. And it’s not really to say that—okay, well, if you’re using C++, you should rewrite that in .NET or some other language. That’s not really practical for most people.

    Ashley: Sure, yeah.

    Eng: But what it does show is that certain languages are more likely to accumulate debt over time. And it’s important not to so much look at the raw number of flaws, because some languages just are inherently more secure against certain classes of flaws, right? It’s a lot harder to shoot yourself in the foot in certain languages than others.

    Ashley: Mm-hmm.

    Eng: But in looking at the shape of the curve, like, does security debt tend to increase for certain languages? That’s kind of—that’s something to look at and at least be aware of, if you have certain parts of your application inventory written in, you know, PHP, for example, you should be aware that those apps are probably gonna be more likely to accumulate debt than, say, the .NET ones.

    Ashley: Mm-hmm.

    Eng: And so, it’s something to take into account as you’re doing your planning.

    Ashley: Okay. Very good. Well, we could talk about this for maybe days, [Laughter] this is so much information, here.

    Eng: [Laughter]

    Ashley: So, we don’t leave everyone with a mountain of, “oh, gosh, there’s so much in here to learn and understand,” are there two or three takeaways if you’re a developer, lead developer, development manager, architect sitting out there listening to this, going “okay, so what do I need to know?” What are the couple of takeaways you would suggest?

    Eng: Yeah. You know, I’ll kinda reiterate some of the things that we talked about, but essentially, security automation, especially if we look at scan frequency, it is definitely lagging the adoption of DevOps in general, right? DevOps has kinda just taken off like a rocket ship and security automation, like I mentioned, only 5% of the apps were being scanned weekly or better. And so, there’s some catching up to do there.

    The good news is that, when you actually do that, and when you actually do that frequent and steady testing, you will probably get to a point where you can start chipping away at security debt and eventually, over time, drive that down so that you can take a strategy of, you know, no new flaws and keep yourself at a clean pace.

    And the last one, I think there’s conversation that needs to happen between security teams and developers in terms of prioritization. You know, I talked about how developers are not prioritizing in, really, a security appropriate manner, though recency is appearing to kind of outweigh every other factor. And so, there’s, I think, some improvement that most teams could make there where, even with the same amount of bandwidth to fix flaws, they could spend their time fixing things that are more important to be fixed as opposed to the ones that are just appearing most recently.

    So, I think those are some of the takeaways, and like you said, there’s a lot in the report. It’s a pretty interesting read, and I would encourage people to go grab a copy and read through it.

    Ashley: And very well done, if I do say so myself—very well put together, there, Chris. Where can folks get the report?

    Eng: It’s on our website, so Veracode.com, and it should be on the front page there, and—yep, there’s about a 50 page PDF behind it.

    Ashley: Okay, perfect. We’ll include a link in the description for this episode, too, so.

    Eng: Excellent.

    Ashley: Well, thank you so much. I appreciate you being on, Chris.

    Eng: Yeah, my pleasure.

    Ashley: It’s been great to have you. Again, thanking my guest today, Chris Eng, who is chief research officer at Veracode, and of course, thanking you, our listeners, for joining us today. We know your time is valuable and it’s a great topic—security is important and having this information, I think, is very valuable to all of us.

    This is Mitch Ashley, with staging-devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chats: 1-Click Vulnerability Scanning on GCP, With Qualys

    DevOps Chats: 1-Click Vulnerability Scanning on GCP, With Qualys

    Sometimes the best way to accomplish something is to choose a path requiring the least friction, or amount of change.

    Qualys customers now have that path available to them in bring vulnerability scanning into Google Cloud Platform. Qualys’ recent announcement means a one-click configuration change enables vulnerability scans in GCP with the results appearing in both Qualys Cloud Center and GCP Security Command Center.

    Join me on this DevOps Chats to explore this announcement with Sumedh Thakar, Qualys President and Chief Product Officer. We discuss Qualys’ and Google’s collaboration and how this benefits DevOps teams.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with staging-devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today I’m joined by Sumedh Thakar who’s President and Chief Product Officer with Qualys. We’re talking about an important announcement that just happened with Qualys and with Google about security natively embedded within Google. Welcome to the podcast, Sumedh.

    Sumedh Thakar: Hey, Mitch. Thank you for having me.

    Ashley: Great to have you. Well, let’s start by just having you introduce yourself and, of course, as President and Chief Product Officer we can imagine what you might do at Qualys, but go ahead and tell us a little bit about what you do.

    Thakar: Yes, I’m not sure you can imagine because I’ve been here for 16 years, so I’ve done a lot.

    Ashley: Okay. [Laughs]

    Thakar: Today, as President and Chief Product Officer, I’m responsible for product strategy and implementation of Qualys product and where we have come from where we were many years, mainly focused on just vulnerability assessment. So today I have the engineering product management support operations team as well as the sales team and that kind of gives a pretty unique perspective because of the engineering and the DevOps teams within Qualys. Also, part of the organization for me helps me to really get a very good understanding of today what are the needs for DevOps and what evolution we’re going through with the DevOps team internally within Qualys, and being a massive SaaS platform ourselves I think that that’s something that I’d really like to focus on when we develop security solutions for DevOps teams.

    Ashley: Well, it does give you a very hands-on perspective, both _____ as well as living with DevOps in your own products internally. It’s great.

    Thakar: Exactly.

    Ashley: It’s good for you. Great role to have. Well, let’s dive into this announcement. So, security that’s natively embedded within Google – Google Cloud I think we’re talking about.

    Thakar: Right.

    Ashley: Tell us a little bit about what it is.

    Thakar: Yeah, we’ve had a fantastic partnership with the Google team and, you know, I think as digital transformation, moving things into cloud container and DevOps have been really getting traction for us to _____ differently than what we have done in the past, which has always been after the fact, adding security solutions. So, today with DevOps and DevSecOps the opportunity to embed security upfront into the infrastructure in an automated way so that you can ensure that you’re not missing – you know, that’s always the biggest issue with security is, oh yeah, I did all of that, but then I missed that one thing because I didn’t know that existed.

    So today, Google Cloud has a fantastic solution that really allows DevOps team, as they’re pushing code continuously into the cloud, to have the ability with this new Qualys integration to automatically have that capability of assessment of the security of that virtual machine embedded directly and done transparently so that the end user does not really have to go about installing agents and consoles and all of that. It’s done; completely embedded in the platform.

    Ashley: Excellent. So interesting, and we’d love to dive into this some more, because you’re talking about not bolting things on after the fact. You know, this is the whole shift left idea within DevOps.

    Thakar: Yes.

    Ashley: So, I imagine with the Qualys cloud agent for Google that’s something that you’re embedding within the software platforms that developers test production environments you’re using.

    Thakar: Yes.

    Ashley: Is it that kind of a lifecycle?

    Thakar: Yes, exactly. So, developers today use that solution to first of all ensure that in their – whether it’s Jenkins or whatever it is that they don’t even push solutions out to production that already have vulnerabilities, right? The unfortunate side effect of doing this bolting on after the fact is that the moment you spin up a new machine it already has so many vulnerabilities in the last two years, you spend all your time trying to patch them. So, this DevOps pipeline does give us the opportunity to ensure that first of all in your build process you already eliminate and patch your images so that you don’t go out with images that are already vulnerable, and _____ customers absolutely use the Qualys agent and scanning to really achieve that.

    But then once it gets pushed out to product you also want to have a monitoring to ensure that somebody doesn’t go to an S3 bucket or to a Google storage unit to pull down a vulnerable version of some sort of a software in their virtual image and create new vulnerabilities, so you also need to have that monitoring. And so that native embedding ensures that once you have done your DevOps process and cleaned the image, as the image is being pushed into the production environment in the cloud, in Google cloud, it is going to have that monitoring capability already embedded transparently without having to do additional effort.

    Ashley: Great. I imagine it’s not a heavy lift for developers to install this in the development test environment, right?

    Thakar: Yeah, and that’s the beauty of it is everything is automated, right? So, once you write the scripting it’s not a process that requires you to run commands every time and do all of that. Once you push that into the CD process then it’s really transparent, because in this case the developers don’t even need to run any command to get the agent onboarded. It’s an option that they check in the Google security center and then every single image that spins up will have that sort of one-click integration and automatically embedded in there.

    Ashley: Great. I imagine then you’re also tying back into integrating into the security consoles both within the Qualys environment and also within Google?

    Thakar: Yes, absolutely. That’s a unique ability here, because you have multiple business units for organizations. Different teams have different accounts in Google and they are, of course, only looking at their own application or their own account. And so the second part of this announcement really is also the fact that those vulnerabilities and those issues that are detected are then pushed back into the security center from Google so that the individual owners of the accounts can see all of the vulnerabilities on all of the issues that they need to fix right there within their console and they can create playbooks and use the automation that is available to quickly fix those issues so that they don’t have to know go to another tool and go to another console and try to do all of that.

    For the IT folks and the DevOps folks who are pushing these applications into the cloud, they just see their own individual view directly in Google, but then of course this information also is available in the Qualys console, which is a wider visibility into the overall risk posture because it will also include the security and vulnerability findings from other types of infrastructure that the customer also has, things like laptops and handheld devices and containers and other things as well. And so the security team will leverage the Qualys platform because then they get a holistic view of the entire outside vector. The individual team in Google will get their view within the Google console so that they can really focus on fixing what they need to fix.

    Ashley: Really kind of empowers that idea of SecOps, right?

    Thakar: Right.

    Ashley: Better right upfront.

    Thakar: Exactly.

    Ashley: Now I imagine that the security engineers – you know, usually there’s some kind of a corporate security team maybe involved in operations, maybe not, but also would be super pleased about having this implemented earlier into the dev cycle and also that it’s pre-integrated already into the Qualys platform, Qualys monitoring as well as Google’s cloud monitoring.

    Thakar: Right. And that’s really a very good point because that’s a big struggle for security teams, right, is that just getting – ensuring that the entire infrastructure is covered by the security solution is a challenge when you do it after the fact, because they may not even know that there’s a new instance that was pushed by the dev team overnight into the Google account, the security may not know. So, with this integration of course it ensures that no matter what gets pushed Qualys is always monitoring every single image that goes out. Now the security team is good for them because they don’t have to spend a lot of time as they have been doing so far trying to get the security solution implemented.

    So instead of focusing on the findings of the security solution a lot of security teams just spend time trying to get that actually installed and making sure it’s up and running and it’s connecting and all of that. So, having the DevOps team actually do all of that work is fantastic because now the security team can only focus on what is important, which is your policy. So, they can be the one that says, “Well, this is a failure if you have a _____,” and they don’t have to worry about how the thing is installed or whatever it is. Qualys is then the third party that ensures that the policy that the security team has defined is actually tested and a pass or a fail is given to both sides of the house and then they each can go ahead and do what they need to do in terms of fixing and reporting those issues.

    Ashley: Now, you described in the announcement sort of one-click integration. What does one-click mean? Can you visualize that for us, how that works?

    Thakar: Yeah. So we say one-click because if you have ever deployed security solutions they are absolutely painful, you know, trying to get the multiple commands –

    Ashley: Or like one month or one quarter. [Laughs]

    Thakar: Yeah, it’s a thousand clicks and multiple consoles and whatever it is. So the one-click is really because when you go and look at that integration, once you kind of go in Google security center and define that Qualys account that you connect to, then after that it’s really just checking one box that says that include the Qualys agent in every single image and then you don’t have to do anything after that, like you don’t even have to run a separate command every time you initiate a new image because that is already taken care of in a very easy manner with the automation that is provided by Google.

    Ashley: Wow. So it truly is a checkbox kind of process.

    Thakar: Right.

    Ashley: Just included.

    Thakar: Exactly.

    Ashley: Are there any other things that maybe you’re already doing or that are part of this announcement in terms of what you’re doing to monitor scan and report vulnerabilities in the Google cloud environment?

    Thakar: Yeah. I mean we have a very strong partnership with them. We’re listed, of course, in the marketplace as well, and with this sort of integration it really provides sort of a security reference architecture where, you know, Google can now provide along with Qualys customers the ability to make it significantly easy by defining the architecture of how you want to secure your workload that you’re putting in Google and then have – not only define the right solutions that should be used but then also work with vendors like Qualys to get those directly implemented with a single click so you also remove the resistance at integration and really simply that overall process. And so that will enable existing Qualys customers as well as new customers who already are going through the digital transformation, they may have Qualys already on their on-prem infrastructure; now it makes it – it removes another hurdle for them to really move into Google cloud because they can get the same kind of reports and the same kind of analysis as they have been used to because their auditors are very comfortable and used to having the Qualys reports so now they can get the same reports and don’t have to go through an entire process to recertify another solution, so there’s a lot of these kind of benefits that go across the board, and just having that embedded I think really helps.

    Ashley: Yeah. It certainly lowers the bar to entry –

    Thakar: Right, right.

    Ashley: – into the Google cloud if it’s already integrated into the Qualys and Google platforms. Let’s circle back to talking about what are some of the benefits to the DevOps teams getting the Qualys agent into their environment earlier? What kind of things do they learn in the dev cycle and the test cycle group normally they would’ve skipped and not found out till way later when they’re ready to go to production or in production?

    Thakar: Yeah. So because the way the platform is structured with APIs, the biggest advantage, there is automation of – and that’s what DevOps is all about, right? How can I script? You know, you talk about infrastructure as a code. I talk about security as a code, right? Can you just code in to say – you know, like you do your infrastructure, can you say if vulnerability is greater than this than failed _____ or, you know, can you write that as a thing?

    And with the way the Qualys platform is and being a cloud-based platform ourselves and with AP as available, that makes it extremely easy for the DevOps folks to really create directly these scripts that they can basically say, “In your pipeline I got a new image, bring that image up, Qualys agent gets already embedded, it does its analysis and the findings show back up in the console and now you can really code that thing with a human, never having to go in and verify and look at the scans or whatever it is, right?” The way the architecture of the Qualys platform and the agent is, is that agent is an extremely lightweight agent that is not focused on doing just vulnerability assessment.

    It does inventory assessment. It does vulnerability, also configuration assessment. And so what happens is that single agent can be used to look at a larger picture of the security of the device and not just if the software is not updated, right? You can do everything from is unauthorized software being put on this build, is end-of-life software being put on this build, is there configuration that is incorrect even though you may have patch or all your vulnerabilities, and so on and so forth. With that sort of an integration, what happens is that once you have done that one task of getting that agent on the virtual machine you can now suddenly with just APIs do a lot more to get a broader view and make sure that that overall workload is heavily locked down, because now you really are ensuring that all the configurations are as per the golden images and all of the CIS benchmarks are followed and all the software is updated and unauthorized software is already eliminated from that whole thing. DevOps is all about how do I eliminate the manual work and how do I use automation and reduce the amount of work that I have to do, and that’s exactly what that architecture enables us to do.

    Ashley: I think it’s an extremely good point; with all the automation, the increase in speed, the decrease in manual intervention or tasks that are being performed. It’s easy for something to slip in.

    Thakar: Yeah.

    Ashley: New opensource code we haven’t used before or something now has got a vulnerability that didn’t two hours ago and you’ll find out right away.

    Thakar: And, you know, one of the things that we have seen is the DevOps push to go quickly into production and sort of was that initial thing of, “Oh, I want to bypass my operations team, I want to bypass my security team,” and the flipside of that is like now your code that goes directly to production and has an issue, that’s on you as a DevOps – like you cannot blame the security person because, you know, they are not really involved anymore. And so that also incentivizes the developers up front to ensure that they’re doing all the right things from a security perspective also so it’s not their microservice that goes out that has their name on it that gets compromised, right? Because then you can basically say, “I’ve coded everything, done everything on my side up front so that what I am pushing out to production using automation is not something that is already vulnerable.”

    Ashley: Excellent. So, let’s talk about how do you get access to this with the release of this capability? Is it already native within Google cloud? You just connect to it and say, “Here – you know, I’m already a Qualys customer,” or do you have to go to Qualys to sign up for something or how do you get access to this?

    Thakar: Yeah. So, we – and as Qualys we provide directly – if you go to Qualys.com there’s – you can sign up for a free trial if you’re not a Qualys customer and, you know, we’re a SaaS service so you immediately get credentials that you need to access the platform, but then it’s completely embedded in the Google cloud console, so when you got there and there’s the configuration settings you will be able to now pick Qualys as a vulnerability management capability that you would like to add and, you know, there’s a one-time configuration where you provide your Qualys agent information and from that point on, once you have that done, every single – you know, do that one-click checkbox, after that that embedding will happen automatically and Google cloud will be the one doing that for you. So within a few minutes you can be up and running with this integration and starting to see the results in a couple hours.

    Ashley: Excellent. Is there any incremental cost for this or is it already included, just an added capability in the Qualys product?

    Thakar: Yeah. So, it’s a cost that the license of vulnerability management that customers already have. They can basically just take that license or use that or they can come and purchase additional licenses to run additional infrastructure in Qualys, but it’s the same license that they would otherwise use that they can use – that they get from Qualys and they can use in Google Cloud as well.

    Ashley: Yeah. Just the licenses they would need to manage a Google could environment anyway.

    Thakar: Exactly.

    Ashley: Okay. Excellent. Well, great. Fantastic news. Anything else we should know before we wrap up?

    Thakar: No, I’m really – you know, as heading the product, I’m always curious to get feedback, and so I encourage as many of your listeners to go and sign up and, you know, send us feedback on how they feel it helps simplify, and if there’s opportunities for us to simplify the DevOps process even more with better integration we’ll be happy to hear that.

    Ashley: Great. Well, let us know how it goes and we encourage our listeners, go check out Qualys. Qualys.com. And especially if you’re an existing customer, it sounds super easy to start to use this, and I know – I’ve been a Qualys customer and also worked with Qualys before as a partner and super easy to get set up and going, so I encourage folks to check that out. Well, thanks for joining us, Sumedh. Congratulations on the announcement.

    Thakar: Thank you very much. Thanks, Mitch. Good talking to you.

    Ashley: Nice to talk with you. I’d like to thank my guest, Sumedh Thakar, President and Chief Product Office of Qualys, and of course, thank you our listeners for joining us today. This is Mitch Ashley with staging-devopsy.kinsta.cloud. Have a great day and be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Software Architecture for Cloud Native, .NET Core and Open Source

    DevOps Chats: Software Architecture for Cloud Native, .NET Core and Open Source

    In this episode of DevOps Chats we talk with Donald Lutz, principal software architect specializing in systems integration and creating large, scalable cloud applications. Occasionally DevOps Chats is fortunate to spotlight DevOps and cloud native developers doing trailblazing work in contemporary software architectures. Donald fits that bill to a T as an entrepreneur and employee at startups such as Faction, BoldTech Systems and his own company Technetronic Solutions, and established companies including Via West.

    Our discussion focuses on creating cloud native applications in startups and large enterprise IT. Donald’s currently working with one of the world’s largest financial institutions to move from legacy applications directly to cloud native apps, bypassing any interim lift-n-shift moves. His work spans many Microsoft technologies, including .NET Core, runtime framework Dapper for RDBMS mapping, Service Fabric and Azure, opensource Kubernetes, Terraform and Puppet (and Enterprise).

    During our discussion, we cover the challenges of architecting and scaling very large cloud native applications, implementing DevOps in less mature software organizations, how established software patterns benefit DevOps and cloud app developers, and the importance of giving back by hosting meetups to share knowledge and mentor others. Donald also gives back as an active leader and mentor of the FIRST Robotics Team 1410 since 2005.

    Join in on our conversation as we explore the complex and sophisticated inter-workings of a cloud native software architect.

    Transcript

    Mitch Ashley: Hi, everyone, this is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by a very special person, a good friend of mine, Donald Lutz. He’s a principal software architect at many companies. We’ll kinda get into that, but he’s focused on microservices cloud, he also is very involved with Microsoft in his background.

    He’s currently involved in several things, but he’s with his own company, Technetronic Systems, and he’s worked with Faction, ViaWest, he actually was one of the original co-founders of BoldTech Systems with me back a while ago.

    So, our topic today is microservices going cloud native. Donald, welcome to DevOps Chat.

    Donald Lutz: Thank you, Mitch. Thanks for having me on. I really appreciate this.

    Ashley: Oh, it’s a great honor of mine, you’re a good friend. And to that point, why don’t you tell us a little bit about yourself, introduce yourself to our audience, tell us about what you do.

    Lutz: Yeah, I’ve been sort of a principal software architect and a director of software engineering for different companies like Faction, my own company TSI and I’ve really in the past five or six years focused on cloud native and taking—you know, getting microservices and applying that to building cloud native applications. That’s really kind of what my focus has been. And there is a lot of DevOps that goes in there. There’s also a lot of architecture and design, so it’s kind of a combination of all of the above.

    Ashley: Mm-hmm, excellent. Well, just full disclosure, I mentioned before, you and I have known each other a long time. We started working together back in the telecommunications industry and I did a startup called BoldTech Systems, which you were a part of founding and getting going and, you know, you’ve hired me, I’ve hired you. [Laughter] We kinda worked with each other—which, by the way, for folks earlier in their career, that’s a great thing to do. Keep your friends close, because you’ll get to work with them again.

    But you’ve had this consistent pattern in your career where I know you were very Microsoft and .NET focused for a while and then kinda moving into cloud and broader technologies. I mean, you’ve worked on Java and other things, too, it’s not like you just did Microsoft, but tell us a little bit about your philosophy on technology. Are you a fanboy Microsoft or how do you kind of view that with everything else that’s going on in the technology world?

    Lutz: I mean, I could be viewed as a fanboy of Microsoft. I mean, I helped—I’m running now .NET Core, an open source Meetup for Microsoft. I really like where they’ve taken .NET Core. I mean, I would claim, six or seven years ago, Microsoft realized that they needed to stop being so proprietary and get more into open source.

    Ashley: Mm-hmm.

    Lutz: So, they really kind of redid all of .NET, and .NET Core is fundamentally, it allows you to build microservices, traditional apps, too. But, you know, they moved to the open source arena because they knew their future was Azure, it wasn’t really Windows in the traditional sense. You know, 75 plus percent of all their things that were under Azure are Linux that run with another product called Service Fabric and, you know, that to me, is a great technology. I have done Java, I’ve done Go, I’ve done a lot of different things, you know? I have a large Microsoft contingent, but they’re all part of the same thing where everybody’s trying to get to cloud native—what does cloud native look like? Whether you’re Microsoft, Pivotal, VMware—everybody’s trying to figure out how to you get there, you know?

    And a lot of people have taken steps in between, like, some people are trying to run VMware on the cloud, but I’m seeing a big push where I’ve been talking to a lot of people, I’ve been working with a really large financial institution right now, and they’re actually punting on the in between and they’re going completely cloud native using both multi-cloud and in their definition, is Azure an AWS?

    Ashley: Interesting. And I’m aware of who you’re working for. I know we’re not talking about the company name specifically, but give us an idea of the scope and magnitude. Because we’re not talking like, you know, a credit union or a large regional bank.

    Lutz: No, they’re one of the largest financial institutions in the world. So, they are looking—they’re moving everything at the top level of their management, you know, CTO level, they decided that they wanted to do microservices, domain driven design. They wanna go all cloud native. It’s either gonna be running in AWS and Azure. They don’t wanna own any hardware any more, they don’t wanna have any of it. For them, it’s an amazing kinda revolution, because they’ve never gone there, and they’re kinda viewing that they wanna run their company in a technological era kind of like a startup, so they’ve really changed their whole view.

    So, I was kinda surprised that they were that interested in that and the fact that they looked at the cloud and they said, “We need to go cloud native, because we don’t really need in between steps because then again, we have another shift and lift then in five years,” and we only wanna have one shift and lift, so I was really surprised they even thought of that.

    Ashley: Well, it is, and I’m very surprised by that, too. It’s certainly—you know, I don’t talk to every big financial institution in the world, but to make that level of commitment of, we are gonna go cloud native, not like we’re gonna lift and shift what we have and just kinda live with legacy, if you will, and build new things in cloud native, they’re actually taking the major part of their software assets, the systems rely on day to day and they’re shifting that just going to cloud native in one big step—well, several one big steps, apparently. [Laughter]

    Lutz: It’s a lot of big steps, but I mean, it was kinda surprising, because you know, they brought in executives who had been more out in Silicon Valley and not traditional financial people. So, they’ve made a different, and they went and found a bunch of us who have been kind of in the cloud native microservices space and brought us on. They kind of viewed it as, they’re trying to build, like a, sort of the Super Bowl winning football team, that if you get the top players, then you can get this to happen.

    Ashley: Mm-hmm.

    Lutz: That doesn’t mean there still aren’t interesting political battles, since this is a company that’s historically, you know, they still have DB2 and green screens, so, you know—

    Ashley: Sure.

    Lutz:—it’s not like it’s completely, “Oh, yeah, this is awesome!”

    Ashley: Right. [Laughter] It’s not like they started in the Linux area, though—era.

    Lutz: Yeah.

    Ashley: So, say a little bit about—so, you have a unique perspective of, you’re not the kind of entrepreneur that’s worked in a from scratch startup in the garage every time, repeat, repeat. You’ve done that, but you’ve also worked with a lot of existing organizations. I know ViaWest had been acquired, I think, by Shaw Communications before you joined and they were going through a transition.

    Lutz: Yeah. I went there because they were—I was looking for a place that wanted to move to microservices and they had been acquired by Shaw. I knew their CTO, we chatted, so they hired me as a Director of Software Development, and the goal—they were all .NET, so I moved into .NET Core and we started building microservices, which is the whole idea of, there’s a lot of stuff that comes out of major design and bounded context.

    So, instead of having, you know, your monolithic application that has customers and data center stuff, we started building services like, “Here’s your customer service.” And it all deals with that. We also brought in some other, DevOps people know about more CQRS and event sourcing. So, the whole idea, each service has its own database that it writes its data to. It then issues commands—so, say I created a customer, I say “create customer.” That causes an event called Customer Created that gets stored in an event store, which almost looks like an audit log. You never delete from it, you only add to it.

    So, if you ever have any problems with all these distributed services failing, you can rerun this audit log and get back to where you were in the beginning.

    Ashley: Mm-hmm.

    Lutz: So, it’s kind of a powerful concept. These concepts have been around for a while, but you’re starting to see them being employed at a larger level. Because, you know, the idea of synchronous communication doesn’t work well in cloud native. In fact, you really just want to expose public APIs, and behind the scenes, you want data streaming, you want eventing. You don’t want any asynchronous communication, because it’s not the right approach. You can’t scale, and you’re gonna be dealing with eventual consistency.

    So, you know, once you start dealing with that, once it’s not in one box, you know, eventual consistency is it.

    Ashley: Mm-hmm. Where you’re going to kinda stateless applications, right? Microservices—it’s interesting that the trend is, obviously, is build much smaller things, reuse them, create services out of them. But there’s gonna be many, many more mass quantities of them. It’s not like you’re going from—you’re going from one monolith with a lot of functions and APIs to really small pieces of software that are containerized microservices.

    But what comes with that is now, all of a sudden, you have the problem of managing that, communicating across and synchronizing when it is asynchronous how you communicate, handling troubleshooting, knowing what all those services are. It sounds like what you’ve done is, you’ve reached back into some maybe techniques and technologies or knowledge of past things and bringing that into this microservices Fabric world. Is that true?

    Lutz: It’s true. I mean, I get a lot of Kubernetes and all that, and you know, even though you start doing Kubernetes and a lot of DevOps things, it’s sort of the thing that I find interesting. It’s kinda like, sorta what you on the Phoenix Project, you know, you need to be able to manage all those things and be able to deliver all the time and monitor it and log it. The problem is that that’s not just answered by creating a bunch of scripts, you know what I mean? Like, even though I’ve used Terraform, Puppet, I think there’s a larger scale of solution.

    Microsoft just released a new framework called Dapper, and Dapper is basically dealing with the microservices run time. So, it’s a run time that can sit on top of whether the microservices can go talk to Kubernetes, because you need to be able to handle that run time stuff as well as the development issues that go on. Those things are not two separate things, and I still think that’s the—one of the large DevOps challenges is how do you get everybody on board, you know?

    And at ViaWest, it was pretty sophisticated, because all our DevOps people were software engineers. So, we just wrote software for most of our problems. We either built it ourselves or used various tools. But then I went to a company called Faction where we had a less mature DevOps practice, and we were building multi-cloud storage, so we had a lot of microservices that allowed us to do Isilon, Dell storage, you know? So, we built a fairly sophisticated microservices platform. We didn’t have the DevOps practice, which resulted in some interesting issues, because the organization was not mature. Everything was manual. So, it’s really hard to roll those out, because you roll them out and then you can’t roll them out as well as you’d like to, because nobody’s catching up.

    Ashley: Mm-hmm.

    Lutz: They’re like, “Oh, well, how do we do that?” I’m like, “Eh, yeah, that’s an issue.” There’s no tests going on, you couldn’t do a Chaos Monkey. So, that was—in that case, in that case, it was pretty challenging, you know? And also the whole—I’m thinking a lot lately about security and the fact that you have all these cloud native accounts, but you have, you know, you have these very, you have credentials that give you access to RDS and Amazon, and then those aren’t really secured real well, and there’s Vault and there’s multiple ways to do it. And the whole security thing is, I don’t think, well cooked in a lot of ways. I would claim it’s not. [Laughter]

    Ashley: Well, application security is now become—becoming, maybe both—you know, one of the top topics. It seems like we’re just sort of, we do things differently, and then we have to go back into and go back and address, like, “Okay, well, how are we doing this now?” Because, you know, it’s not about protecting servers and network devices. We’re exposing so much of the tax service as an application through all these microservices, APIs, et cetera. You gotta kinda rethink it.

    And then you’re automating the development environment to do 15, 20, 100 code deploys a day, potentially. Well, that takes automation—that means all those credentials, security keys, certificates, whatever are built into a process that’s all that secure. I think that’s what you’re talking about is, you have to do it, but you have to also go back and re-look at it to say, “Are we doing this securely?”

    Lutz: Right. You have to look at things, “Do I have—is data at rest encrypted? Do I check any kinda hash on it? Is the data in flight encrypted? What are the levels of security? Do I let my JWT token float to one service, to another service, because that could be an attack, too.” Because I went to service one and now I go to service two and I’m using the same JWT token, but maybe the claims on that second service, it shouldn’t be allowed to do those things, because the claims that it used from the first service lock it down better, but in service number two, it allows you suddenly, “Hey, I could go look at different accounts that Mitch has, and well, that’s interesting.”

    So, then you start thinking about, you know, how—there’s a couple good security books about that whole idea of, you know, JWTs are interesting, but the problem is that if they float, if we haven’t figured out the right way to share those, that’s another thing that causes lots of problems.

    Ashley: Mm-hmm. And for me and for our listeners, can you define what a JWT token is?

    Lutz: Yeah, it’s a JSON web token. It’s just—

    Ashley: JSON, right?

    Lutz: Yeah, yeah. It’s just a token that basically you can use with an API. You know, it’s a little—it’s really for APIs. OAuth, you can do both. OAuth is more effective with users, but those are, you’re seeing the industry starting to adopt JWTs between services. So, now people are saying—but the problem is, the concept behind it is a claim. I wanna make a claim that your name is Mitch and you live at this address and you have this phone number. And in this case—oh, by the way, you can look at your Social Security number, but in this case over here, you shouldn’t be able to look at this other person’s Social Security number.

    So, you start making all these claims, and then you have to start getting really smart about how do we define all these claims for these very distributed systems.

    Ashley: Mm-hmm.

    Lutz: Which, historically, distributed systems were everybody—oh, if you had the root password, you’re golden. We’ll just use that and everything will connect to it.

    Ashley: Yeah, interesting. It’s really interesting, not being that close to it but following what’s happening with it. I think it’s based on, like, RFC 7519, so it’s based on a spec. It’s about how claims essentially get bedded into what we traditionally think of as a data interface specification between services and applications. Anyway, we’re not a JSON JWT podcast, but interesting sideline.

    I wanna get back to talking about microservices in cloud native. So, as you’ve worked with existing applications, and even at Faction with something that maybe didn’t start out in a DevOps world, what do you see as some of the kinda approaches that make sense and maybe things that you or others have tried that turned out not to be the best path or dead ends that you’ve corrected from and now here’s how you do things?

    Lutz: The first problem I see is, you make nanoservices. So, instead of—you know, you make a microservice that tracks time, and it’s so small that it’s not very effective. There’s a software architecture discipline called the main driven design, which is about when you build an application and a service, there’s a domain that it covers, whether it’s a customer or a product, and it needs to be the right size. And if it’s the right size, it’s what’s called bounded context. Because the whole idea is, you want that service just to answer that bounded context, which is based off the idea of a single responsibility principle. If it does more than one thing, you have a problem.

    So, kinda getting people to understand that is complicated at a business level, because you know, business people historically want to throw everything in whatever they’re building. Like, “Oh, can’t we just, why do we need to have such a small bounded context? Can’t we, like, throw all the addresses and products in here?” I’m like—no, because we need to keep those separated, because we’re trying to, each microservice has to have a very distinct separation of concerns, because if you don’t have that separation of concerns, you end up with a monolith, which is what we’re trying to avoid.

    Ashley: Yes, end up back into the spill over again of domains into, this microservice has too many things in it and now one thing’s affecting another, versus let it be a single purpose in our service.

    Lutz: You know, and that’s why we actually saw that this large, large financial institution has adopted domain driven design and they’re actually building domain models for all their applications and everything. So, there will be demand models behind them.

    So, it’s just not the microservice, they actually can say, “Oh, by the way, this is what this financial instrument looks like, and it actually maps to our true domain model and then will map to a true microservice.”

    Ashley: Mm-hmm, okay, so domain driven design, nanoservices, or kind of focused functional microservices that don’t kinda spill over.

    Lutz: There’s another thing, too, you need is what historically—Chris Richardson has a book called Microservice Patterns, and it’s the idea of what’s called a microservice chassis. So, once you start building microservices, you either need to make a framework or use a pre-existing framework, whether it’s Spring Boot or some stuff in .NET Core so that when you write the microservices, all the logging is done the same, all the separation and concerns are handled so you deal with all those non-functional requirements so you don’t spend any time developing it.

    Ashley: Mm-hmm.

    Lutz: But then once you do that, then you have to start saying, “Well, how does this play in the run time world of Kubernetes?” Which gets into something you said earlier—we need to sort of connect the framework with the run time stuff, so that’s kind of a problem you see in Pivotal, Microsoft. They’re all looking at how do we get these really interesting software engineering frameworks that then wrap into run time things that are equivalent, so we end up with, you know, “Oh, they’re the same thing, they’re isomorphic.” They’re not different, they just look different because we view them as different domains.

    Ashley: Mm-hmm. Interesting. Well, we have a couple minutes left. I wanted to just take a little bit of a left turn—not quite a squirrel moment, but a little bit of a left turn. I’ve been doing some podcasts here recently with folks, usually developers, DevOps engineers, Ops folks who are talking at the Spinnaker Summit Conference, speaking on Kubernetes and Spinnaker Open Source.

    So, we’ve had a great opportunity to have other developers, real hands on technical folks on the podcast. And one of the things that I always encourage folks is, even in the open source community, you contribute not just by writing code, it’s by sharing your knowledge, applying these tools and software and sharing the knowledge and networking and building community with others. That’s vital to having, you know, vibrant software is having a vibrant community about it. And I know you’re very active in your own developer circles and communities in creating those. Tell a little bit about what you’re involved in now and why you do it.

    Lutz: I like doing it to share. I mean, I would say about four years ago, there was a gentleman running a group, there’s a group worldwide called ISO, the International Association of Software Architects. And he was trying to get the group growing locally in the Rocky Mountain region—you know, Denver, Wyoming, that kinda area. He hadn’t gotten very far, so I got kind of involved, and we’ve actually grown it over the 4 years that we have 2,000 members and we get about 60 to 100 members coming every meeting, and it’s truly talking about architecture. It could be the Kubernetes architecture, it could be domain driven design. Last month, I talked about how to build microservices that are stateful. Surprisingly, there’s a company called Cloud State. Microsoft and ACCA, they figured out how to build stateful microservices. We won’t get into that.

    Ashley: They didn’t bring pitchforks and, you know—

    Lutz: They didn’t bring pitchforks. It works pretty well.

    Ashley:—torches? [Laughter]

    Lutz: But we get different people. We got the guy who’s the original guy at FedEx who created all their stuff that looked like Kubernetes in the ‘60s come talk. We’ve had people from Slack. We get a lot of people to come talk. I’ve talked at the Carnegie Mellon architecture conference, I can’t remember what it’s called. I talk at a lot of Microsoft groups. I run a Meetup for Microsoft. They have a few .NET groups that are dying, so I pitched to them—why don’t we turn that into a .NET Core and open source group?

    Ashley: Mm-hmm.

    Lutz: And so, I’ve been running that. I’ve grown it—that’s almost to 1,000 members now. It was actually only 200. So, in the three or four months, I’ve actually grown it. I like creating communities of technical people, because when you do that, you actually can solve interesting problems. You also get to hear other people’s experiences.

    I can give you an example—one of the guys who get involved with ISO, I was talking in Colorado Springs at an event, I can’t remember where. And I was outside the event and this guy, Tyler, came up to me and started talking to me and I discovered he was really technical. And he said, “What are you doing?” I said, “Oh, I’m talking about microservices and that.” And he says, “Why don’t I come in?”

    And we actually—he became the person who asked me the most questions during the event and then he became one of the four members to help us run the Denver Software Architects Group.

    Ashley: Mm-hmm. Interesting.

    Lutz: And we became really good friends and it was like, you know, just talking to somebody outside an event I was gonna speak at.

    Ashley: Very interesting. Well, you know, I think it’s also, all of the things you said, I totally agree with. It’s also a way for someone who wants to know about a topic, maybe, and I like how you’re kinda keeping the groups fresh on what’s happening and new in Microsoft and other worlds of software. It’s a place for folks to go and learn about it and hear what other people—even if you’re very engaged in a new technology already, it’s, what are other people doing and what have they learned from it? What can you share, what can they share with you, and vice versa. So, it’s a healthy thing to do.

    Lutz: Yeah, I really enjoy it. I find it more enjoyable—it’s up there with this robotics stuff FIRST I do in terms of raw enjoyment, because it’s so, you’re plugged into so many people where, when you know, when I’m doing consulting or helping build a startup or all of the above, it’s enjoyable. But it has—this has more of a kind of like, not selfless, but it has more of a giving back vibe that I like a lot.

    Ashley: Mm-hmm. Very good. Well, hey, we’ve run out of time. You and I could talk for 12 hours and, you know, up ‘til 3 a.m. in the morning. There’s so much fun things that you’ve done. I really appreciate you being on the podcast.

    Lutz: Well, thank you for having me on. I really appreciate it. I really enjoyed it.

    Ashley: Well, good. Me, too. You’ve listened to another DevOps Chat. I’m very pleased to have my guest, Donald Lutz, on with us today. Donald is a principal software architect, he focuses on microservices, cloud, cloud native, generally Microsoft technologies, but a lot of other things and he’s with Technetronic Systems. He’s also been with Faction, ViaWest, BoldTech Systems, a number of different companies, so he has a lot of software background. And again, our topic today was microservices in cloud native.

    Thank you, also, to your, our listeners, for joining us. This is Mitch Ashley and you’ve listened to another DevOps Chat. Thanks, and be careful out there.

    — Mitchell Ashley

  • DevOps Chats: App Attention Index and The Era of Digital Reflex, with AppDynamics

    DevOps Chats: App Attention Index and The Era of Digital Reflex, with AppDynamics

    Application performance company AppDynamics (acquired by Cisco in 2017) recently released its 2019 edition of the App Attention Index Report. The 2019 report is subtitled The Era of Digital Reflex. Joining me on this episode of DevOps Chats is Steve Long, AppDynamics regional CTO and technology strategy.

    Digital connections are a near-constant. Interacting with our world through digital apps and transactions are omnipresent: making a restaurant reservation using OpenTable, making payments using an app or our smartphone digital wallet and checking our smartphones to plan and move through our day. Thus the name of the report, The Era of Digital Reflex.

    In addition to insights from the report, Steve and I talk about measuring the customer’s experience, tying performance back to the desired business outcoming, strategies for finding problems and reducing the time it takes to fix problems. Join us on this compelling discussion about application performance, customer experience and business outcomes.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi everyone, this is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today I’m joined by Steve Long. He’s regional CTO and technology strategy with AppDynamics. Today we’re going to be talking about the 2019 edition of the App Attention Index Report findings, The Era of Digital Reflex. Whoa, that sounds heavy. Steve, welcome to DevOps Chat.

     Steve Long: Thanks for having me today, Mitch. It’s a pleasure to be here today.

    Ashley: Sorry, I kind of put on my ‘60s accent. “Whoa, man, heavy.” Would you start out be introducing yourself? Tell us a little bit about what you do and of course, in case somebody doesn’t know what AppDynamics does and what AppDynamics is about.

    Long: Yeah, sure, absolutely. So I’ll go back a little bit in time. I was a vice-president of technology operations for a media company and then my last job before joining AppDynamics, I was a chief information officer for a financial tech company, a large banking services company. And I joined AppDynamics because–for two reasons. One was finding problems in the sea of–no longer a needle in the haystack–it’s a needle in a needle stack, at this point. With all of the complexities happening, it changed my career. And when Cisco and AppD joined and that merger took place, I couldn’t think of, you know, two companies that I would love to be part of, that when they joined–you know, after I left being a chief information officer to come and help drive technology strategy for AppDynamics.

    So AppDynamics is a application performance management company that back in the dawn of the day I became one of their first 30 customers, about nine years ago, and really attaching to the run time and basically looking and analyzing your application performance. If the application is the north star of your business and typically how businesses are driving revenue and transactions, we’re at the forefront and the heart of that, to be able to tap into that and give an MRI to what’s happening in your cloud, your data centers and what’s happening with the complex technology running in the business.

    So that’s a little bit about AppDynamics. We’re an application performance company and part of Cisco, and driving businesses to really see into what’s happening in their technology ecosystems.

    Ashley: Excellent. Well you share something with me, and that is both the kind of end consumer of technology like this is a CIO but also a tech person in the product, the tech vendor. So that’s fantastic. I enjoy sharing that with you. Now tell us a little bit about the App Attention Index Report and what that’s about–what it measures, what kind of information that it discloses for us.

    Long: Yeah, so “The Era of Digital Reflex” is the title, but at the core we’ve really entered into really consumers saying they use digital services almost like you would a human, like an extension of your body, like a reflex. And today we’re doing that almost automatically or unconsciously. And that shift in consumers–like if you look past five years ago, we had to proactively do that. Now, with something in the palm of your hand, you don’t realize it, but now you’re relying on digital services like every day and it’s really helping you get your jobs done and your life duties, and you’re taking decisive actions by it being a reflex.

    And it’s kind of a wake-up call for the industry and businesses, because at this point we’re looking at the performance of these applications being application loyalty and it’s taken over to be the new brand loyalty. So if businesses really want to stay competitive, they really have to have a really flawless experience for their users. Otherwise it impacts their brand loyalty and their application loyalty and other things. So that’s the heart of the report at a high level.

    Ashley: Makes total sense. I mean, you’re talking about our digital world and our connection to it. It’s not only true today, but it’s even more so–every day there’s a new function we do on our phones, on our computers, on our TVs, on whatever device is connected to some cloud somewhere. It’s almost all a digital transaction.

    Long: Yeah, absolutely.

    Ashley: So tell us a little bit about this year’s report–what some of the findings were. What would be some of the most interesting things that came out of it?

    Long: I’d like to actually use an example that I have personally, and it’s based on the report. So I actually had a lunch with colleagues, a friend and a customer last week, and it was about six of us. And I said, “Hey, just out of curiosity, how many applications have you guys used today?” And to a one, they looked at their phone and all kind of raised their heads back up and one rattled 13, one 17, one 12, one 19. I was like, “How did you do that?” And they were like, “Well we tend to close out our applications at the end of the day and it’s 2 p.m. and I counted up from the last application used to now.” And I was blown away. And so I did the same thing and went back to the last one that I hadn’t closed and counted and I was at all 11 at that point in time, at 2 p.m. And I thought to myself, “Wow, okay.”

    And then I asked–the secondary question was, “Well, did any of those give you any problems? And I’m relating back to the app index, because if it had a problem, what did you do?” and the first example that came back was, “I had a reservation at a restaurant last night and I had to go close one and go to the other because it didn’t have my reservation, so I ended up having–for every application that I have, I have a competitor.” And if it’s really sticky, maybe like banking or something else, maybe they don’t, but they’ve got a few ride shares, a few travel apps, a few restaurant apps. And it’s really that wake-up call, because the metric that comes out of the report is about 55% of the people can’t go without their mobile device for up to four hours.

    I did that today before this, just to see if I could do it. And right before I got on here I had to give up, because I needed to see if I had a notification or a text or something that I had missed. And so personally it is that reflex, that extension. It’s what we do now unconsciously.

    Ashley: I think I’d rather hold my breath than try to live without my digital–

    Long: No doubt. No doubt.

    Ashley: So when you count the number of applications that people are using, do you count like, for example, on your phone you can look to see, how much battery time did you consume with different applications is another way to measure effectively kind of usage. How do you look at it?

    Long: Well I mean, it’s habitual. So we just looked at ones that were open. But I mean, if I go back and look at my day, my alarm goes up and I pick up my phone. And the first thing I did there was, “Now that I have my phone in my hand, I’m going to look at an application.” My schedule, maybe my e-mail, maybe a text. And we’re basically saying now that the first thing we do is not talk to humans; it’s to look at our apps. And nowadays–you know, if I go back five or six years, my house is now fully connected. So I’m looking at my security camera and any events, I’m looking at my thermostat. Did I leave my air conditioning or heat on? Is my car charged? You know, and do I need to set the thermostat in there? Is it 100 degrees out and do I need to put the AC on before I get in my car? And all of a sudden I’ve got energy usage tools.

    So these applications can be anything from a browser-based service, but they’re there to help you with your life. And then as you get to work it becomes even more–more of a need, with reminders and you know, calendaring and everything going on in your work world and sharing documents and collaboration. And that’s why at 2 o’clock you can hit somebody who’s had 20 and in a day they can be hitting 30 to 50. So it’s pretty amazing what these digital services are doing and how intrinsic they are to our daily lives.

    Ashley: Well it’s amazing, because one of the things–I’m guessing I’m probably not part of your report–but occasionally I’ll look to see how many connected devices or IP addresses have been allocated in my home. And it has gone from a dozen to two dozen to over 70 in a year and a half.

    Long: Wow.

    Ashley: And that’s just as I kind of went to a connected home. And every one of those is not just talking to stuff in the home. It’s talking to the cloud, you know, for the temperature management, for the door locks, for the security system, whatever–it’s all talking to something, if not me or one of my family members through an app on the phone or browser.

    Long: And think about when they don’t work and they can’t talk to that cloud or digital service. And since it’s an extension of your human interactions, now you can have expectations that are super high that you just had a disruption, and it can impact your mood. It can cause frustration. It can do things that you don’t even know are happening. And that’s why we call it–you know, it’s a reflex now. It’s intrinsic, it’s unconscious and it can cause strife in our lives when it doesn’t work.

    Ashley: That’s a really good point, because we always think about networks as end consumers as speed, but the reliability has also gone up significantly, and it’s going to have to increase in reliability moving forward, because that’s–it’s a reflex, it’s a dependency, it’s an expectation, it’s a customer satisfaction measure, right? Of what that experience is like.

    Long: Yeah, and I think the expectation just a few years ago was much lower, and every year that goes by–I think three quarters of the people we queried–that their expectations had gone up. And it’s growing and growing, and the tolerance levels now have also become more important, and these services are just more important because they’re prompting a higher demand and better experiences, so that businesses can help you with your daily lives. And so the  net of it all is we’re hungry for very positive digital experiences, and we’re willing to pay for that and have them be more personalized and know who you are and kind of know what you want, and that’s a huge differentiator between businesses competing with each other.

    Ashley: Interesting. As you looked at this data, were there different ways that you slice and dice, segmented–you know, home versus business or mobile versus browser or any of those kind of ways of looking at the information, to gather some interesting facts?

    Long: Yeah, we did. We have a crosscut of that. Mainly though looking at the consumer base side and how it interacts with the businesses and how the expectations of the consumer are there and need to be met and exceeded. The experiences need to grow. I know in our daily interactions from AppD, we look at e-commerce companies and banking and a number of different online services, and it’s pretty clear that the performance, the loyalties are pretty clean cut when you have a performing app versus a non-performing app. And the impact it has on the consumer. And so those can transcend almost every industry. It’s almost industry-less, where the application performance wins.

    You know, every hundred milliseconds of latency–you know, that causes some level of distress, and when you start to get into situations where something might take 10 seconds, in industry A that might be acceptable because it’s a longer-term type of transaction. But in industry B, which may be more of a real-time field, like I need a ride or I need a stock trade or I need something like that, 10 seconds is unacceptable. And if you’re not able to measure that and have application performance, you can’t dissect what your problem is because ultimately it’s your business transaction which leads to your experience–a journey that you’re going through, your business journey–and at the end of that rainbow is the actual business outcome that you’re looking for. And whether that’s a sale or moving money or buying a ticket or a new shirt, that’s the key, is having that business outcome.

    And the more we tie application performance to that business outcome, we’re better off as a business. At AppDynamics we look at all of the way to the edge, to the browser, to the device, to the data centers, to the cloud, but we also tack into that the ability to look at money or what the business KPI that’s being impacted. Because if you have a seasonal event like an inventory level dropping or a loss of money or you’re making more money because of a promotion, wouldn’t you want to know that your performance or your lack of performance–what the impact is to businesses?

    Ashley: It’s interesting too, because it wasn’t very long ago–I mean, maybe just a few years ago–where we’re almost solely reliant on things like an NPS measurement or call times–things that we do in the call center. And outside of that, you know, one person tells five people when they have a bad experience and you don’t really know those kinds of things. But now, in this age of data for everything, you know, the last time somebody used my Lyft app, “Oh, it was poor performance. We haven’t seen them in three weeks.” Maybe they went to Uber. I may not know that, but I know what their last–there’s data there to mine and to go after, where some of the telemetry information about performance could indicate some symptom that might’ve led to a consumer change in behavior.

    Long: Yeah, and it’s no longer–we talk about expectations and not meeting them, but the downside is that is your reputation risk. I mean, if you get an issue reported to you from internal or you know, somebody inside, that’s one thing, versus it being reported to your call center–that’s another thing–versus somebody switching to a competitor and posting it on a social media site. Now you’re not only worried about stock price and revenue risk; you’re talking about reputation risk. And that reputation risk really does tie back into those expectations not being met, at whatever level it is. So reputation and how, you know, media out there is picking up on that and promoting that when it’s negative–for a reason, because it’s a shared experience.

    Ashley: This may be a bit of an impossible question, so I recognize that up front. Is it possible to measure what customers’ expectations are? Is it too big of a moving target? Is it too many variables? Or is it something we can get a handle on? How do you address that?

    Long: Well, I firmly believe we can. And say that coming from the customer side of AppDynamics, but also–you know, businesses know their business, their KPIs, what outcomes they’re looking for. And if you don’t measure it, you can’t really make change in a positive way. So we see digital transformation–Agile–you know, we’re seeing BizDevOps–those are all things to really move businesses to become more efficient, drive the right decisions around the right business outcomes they’re looking for. And you know, I always say you do have to focus on the application performance. Being a past CIO, I tried to put myself in the customer’s shoes. I mean, if you don’t have a robust way to look at application performance then you don’t have that safeguard or those guardrails that you can put up to make sure that, you know, number one, your mission-critical applications are working, but number two, it’s the user experience. And being able to measure that and seeing that all the way from the edge to your cloud services.

    And the dependencies. I mean, think about it, the number of SaaS providers, cloud providers, third-party data that you’re touching just to execute a login on any application site, it could span hundreds of transactions. And if any one of those little things go wrong, you’re not focusing on application performance or you can’t measure it, and it could have a negative user experience.

    Ashley: Very good point. Because there are very few, maybe if any, closed ecosystems, to throw the “ecosystem” word in there. You really are reliant on a lot of other providers to come together to deliver that overall experience, that service, that functionality.

    Long: Yeah, and that’s how you get your business outcomes. So I do think that it’s achievable and you know, that performance and correlating it to the business performance and the business outcome–that’s really the objective companies are setting, and why they’re moving to APM and companies that are expanding out to give them business metrics on top of APM and consumer metrics on top of APM. Because it’s that trifecta. And if you think about it, there’s lots of ways you can look at logs or network flow data, but at the application, since it’s the central north star of the business, it’s the most granular and contextual data that you can get to see whether or not your business is performing. So that when you have that granular and contextual data down to the line of code maybe not performing, you’re able to look at the two most important metrics to any company, which is finding and fixing problems and reducing the time it takes to get to solutions on either side of that. So the faster you find them and fix them–whether it’s a ten-second issue that you can automate out of your system, or if it turns out to be a ten-hour outage, those are radical, you know, separations that we see in companies by just implementing the right tool set, the right performance measurements and really driving their businesses to reduce those risks in any company.

    Ashley: Okay. Maybe one last thing to kind of touch on–were there any other interesting findings, notable things? Maybe kind of a small little quirk thing that was “Wow, we didn’t expect to see that,” any other things to glean from the report?

    Long: No, I think we’ve covered it. At a high level I always suggest, put yourself in the consumer’s shoes. Really have a robust way to look at performance, and tie that back to the business outcomes. Because ultimately the business teams have a really hard time understanding IT and engineering, but the IT teams actually can make that leap. They can tend to look at the business challenges and situations, take that complex technology of all the ecosystems and translate that into showing the business what’s actually happening under the covers in a meaningful way. And that’s the basis of the report, is that the expectations are rising, the issues are out there. Finding them and fixing them, and really, put yourself in the consumer’s shoes, because it’s only going to grow, the dependency. And the flawless application wins.

    Ashley: Great. How do folks get a hold of this? How do they get to The Era of Digital Reflex report?

    Long: It’s posted on our AppDynamics website. It’s AppDynamics.com, and you can go ahead and get it there, and we’d be happy to share that.

    Ashley: Excellent. For you to download, I assume?

    Long: Yep, sure is.

    Ashley: Nice. Okay, well excellent. Thank you so much, Steve. Appreciate you being on DevOps Chat with us.

    Long: Thanks, Mitch. It was a pleasure, and I appreciate the invite.

    Ashley: Absolutely. I want to thank my guest today, Steve Long, regional CTO with AppDynamics. And of course we want to thank you, our listeners. We appreciate the time that you spend listening to us and ____ some feedback that you give. This is Mitch Ashley with staging-devopsy.kinsta.cloud. Have a great day, and be careful out there.

    — Mitchell Ashley