To read more, please visit: https://www.perforce.com/blog/qac/secure-coding-standards
Author: Mitch Ashley
-
What is CERT? Overview of CERT Secure Coding
As you write code, you must ensure it is secure and safeguarded from potential security threats. That’s why CERT is essential.
Read on to learn more about CERT.
What You Need to Know About CERT Secure Coding
CERT is a secure coding standard that supports commonly used programming languages, like C, C++ and Java. The CERT Secure Coding standard was developed, and is maintained, by the Software Engineering Institute at Carnegie Mellon University. The rules and recommendations outlined in CERT target insecure coding practices and undefined behaviors that lead to security risks. Each rule and recommendation is developed by members of the software development and software security communities.
Why CERT Secure Coding Is Important
CERT helps to ensure that software developed with coding languages — such as C, C++ and Java — is secure and reliable. The aim of secure coding standards like CERT is to not only detect security risks, but to provide guidance on improving the security and quality of that code. By using CERT, you ensure that your software is safeguarded from potential vulnerabilities.
What Is a CERT Risk Assessment?
For each CERT guideline, there is an associated risk assessment to help determine the potential security consequences of violating that specific rule or recommendation. There are three sections of the CERT risk assessment: severity, likelihood and remediation cost. Each risk assessment section is assigned a value between 1 and 3, based upon the results of the assessment. This helps determine the severity of the violation.
Why You Should Use a SAST Tool to Complement CERT Secure Coding
The most effective way to ensure your code is secure and safeguarded from security vulnerabilities is to use a static application security testing (SAST) tool. A SAST tool identifies and eliminates security vulnerabilities and software defects early in the software development process. This helps ensure that your software is secure, reliable and compliant.
To read more, please visit: https://www.perforce.com/blog/kw/what-is-cert
-
What Is AppSec? An Overview
An estimated 84% of security incidents happen at the application layer. And, with the number of cybersecurity threats steadily rising year-over-year, application security (AppSec) has become absolutely essential.
Application security refers to finding, fixing and preventing cybersecurity vulnerabilities throughout the entire development life cycle. By enforcing application security measures, you are able to ensure that weaknesses and vulnerabilities in your software are identified and dealt with as soon as possible in development before they can become serious security breaches. This also helps to keep the overall costs low and enables you to deliver a secure, reliable product on time.
AppSec Best Practices
In order to ensure that your application is secure against security threats and vulnerabilities, you must enforce application security best practices. While you and your team may have application security best practices that are specific to your own process, we recommend that you consider also using these best practices:
- Conduct an application security risk assessment to effectively identify potential security vulnerabilities and weaknesses.
- Eliminate—or, if elimination is not possible, mitigate—the security vulnerabilities and weaknesses that were identified in the risk assessment.
- Examine open source and third-party software for security vulnerabilities. Properly address any weaknesses that your examination may have uncovered.
- Use AppSec tools such as SAST and DAST.
- Provide your team with application security training.
By adopting the above best practices, you can be assured that you and your team will have a strong application security process.
AppSec Tools
As previously mentioned, one of the most effective ways to ensure application security is to use AppSec tools.
The most common AppSec tools are:
- SAST: A static application security testing (SAST) tool analyzes your code as it’s being written in order to detect and report weaknesses that can lead to security vulnerabilities.
- DAST: A dynamic application security testing (DAST) tool enables you to identify security errors, runtime issues and environment-related issues later in the development of your software.
By using these AppSec tools together, you are able to ensure application security throughout development.
Important Secure Coding Standards for AppSec
In order to effectively enforce application security, you should use secure coding standards to efficiently identify, prevent and eliminate software vulnerabilities.
The most important coding standards for application security include:
- CERT
- CWE
- DISA STIG
- OWASP
- ISO/IEC TS 17961
A static code analyzer should be used early in development to effectively enforce coding standards to ensure effective application security.
To read more, please visit: https://www.perforce.com/blog/kw/what-is-appsec
-
Shifting Left and Static Code Analysis with Perforce
Perforce Product Manager Stuart Foster, and Evangelist Steve Howard, join Mitch Ashley to discuss the importance of creating security software from the beginning of the development process. We discuss shift left, SAST, source code scanning and other pursuits towards this goal.
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript
Mitch Ashley: I’m pleased to be joined by Stuart Foster, who’s product manager with Perforce, and also Steve Howard, who is static code analysis evangelist, both with Perforce. Welcome, gentlemen.
Stuart Foster: Hey, Mitch, how are you doing?
Ashley: Very good. How about if we—let’s start off with just a little bit of an overview of Perforce, and then, Steve, why don’t you do that and also tell us a little bit about yourself, and then we’ll toss it over to you, Stuart, to give you introduction.
Steve Howard: Right, yeah. Well, here at Perforce, really, our role as we see it is to help developers write better code more efficiently and more quickly. So, these days, we tend to talk about DevOps and the whole idea of developer operations and how they put that code together, how it’s constructed, how it’s managed, how we control the processes around it as well, of course. There’s more and more regulation, it seems, these days in software development and we’re making that whole process from start to finish, of writing code to delivering it, that whole cycle. It’s hopefully being made easier, more comfortable for our developers that use the products.
So, that’s really where Perforce sits, and we think of that is really enterprise DevOps, and doing it at scale in large scale organizations.
So, my role, really, is specific on the Static Code side or Static Code security testing side, SaaS tools, if you will, trying to make sure that that piece of writing the code, actually developers writing the code itself is done to the highest possible standard from both the security and the quality perspective and also, at the same time, getting maximum productivity from our development teams.
Ashley: Wonderful. Stuart, introduce yourself?
Foster: Yeah, absolutely. I’m Stuart Foster. I’m the Product Manager or Perforce’s Static Analysis products, both Klocwork and Helix QAC. So, I work with Steve, he’s kinda my partner in crime, amongst a number of other sales engineers and developers and everything, and we’re just out to produce a really solid static application security testing tool.
Ashley: Excellent. Well, one of the great things about talking with you both is, since you have products in market, you talk to a lot of customers, especially in an evangelist role, too, Steve. It really helps us kinda get an idea of where the state of security is in applications, what people are doing today. The whole idea of shift left, I think, has evolved a great deal. I think when I first heard it was “move left in the workflow.” And then it became, “Well, move the when the security people get involved.” And you’ve really taken it much farther than that to really thinking about how we start writing secure code.
Maybe one of you could kinda give us an idea of where you think we are as a community on getting to that point of when the code’s written, it’s written securely.
Foster: Yeah, so, I mean, from my perspective, you know, you kind of look historically at people using the term shift left and, you know, from our perspective, that would’ve been, you know, doing some sort of code analysis on a server and then moving that left in terms of the software development life cycle into the developer’s IDEs.
What you see nowadays is that, you know, historically, a lot of those responsibilities would’ve been siloed. You’d have a security team, a QA team, various other teams that, in many ways, because they’re siloed and in different stages of the process, you’d run into kind of bottlenecks. What you’re seeing a lot more with the concept of DevOps, DevSecOps, a lot of development teams themselves are becoming owners completely of the code that they write. And what I mean by owners is that, you know, they are writing unit tests, they’re writing more functional tests, they are ensuring that it’s part of their responsibility that the code is secure.
So, that’s a shift left, but then you have to think how developers work—they’re working in IDEs, they have their own CI/CD pipelines, they may be using cloud services, various containers that are all getting spun up and broken down. So, it’s not just, to me, shift left any more, it’s analysis anywhere. It’s basically having something that can work anywhere in a workflow, right? And in this sense, you know, the Security teams still exist, the Quality Assurance teams still exist, but what it really is, is a culture of security collaboration that are coming together, right?
Steve, I think you may have a little bit of a perspective on this, too.
Ashley: Yeah, I’m interested in your from the field perspective, too.
Howard: [Laughter] Yeah, no, it’s—well, it’s exactly that. I think it’s a really good point that no, we’re not just talking about moving to the desktops, we are talking shifting left throughout the whole cycle. I think that is a key point. And it is really an opportunity using tools like ours in Static Analysis security testing, it’s about being able to enforce the requirements and the needs from that expert security team where there were bottlenecks previously, trying to put that into rules and regulations that we can then push down to the development tasks and processes that are to the left of that kinda final sign-off.
So, yeah, that—in that form, very much, I’d say it’s shift left, and it’s about that optimized process now where we can be doing that checking all of the time. So, it’s making sure we’re continuously compliant to those security requirements or quality requirements, as the case may be, yeah.
Ashley: You know, it’s interesting, we talk about the workflow and we think about the whole tool chain and pipeline, but if you really break it down into much more granular pieces, developers are working on a linear piece of development, right? They’re in multiple places jumping back and forth, solving bugs, adding features, developing new capability—whatever it might be. So, the ability to concentrate on any one aspect of an application, sometimes, is not even possible, right?
So, you need an environment where you’re aided in how you write and create secure code as well as think about writing secure code. You agree with that?
Howard: Yeah, absolutely. I think that’s the point. We’re trying to make sure that all of those pieces that they’re working on at different times and, you know, our colleagues in the Version Control System team know this very well, but obviously, there are tens or hundreds of branches running in parallel, as you’re sort of saying, and developers are jumping between, you know, fixing problems on the current release, maybe, and then carrying back on with their feature work for the next release as well.
And all this time to try and manage what security requirements each one of those had or quality requirements as well would be a really nightmare without sort of tooling and regulation around that, around the whole process, which is a significant part of developer operations as a whole, the idea of automating a lot of that and building it into the system. So, I think it’s just enforcing that with making sure we’re adding all of the extra pieces that we can also include in those processes.
Foster: And also, if you think about those developers, too, they’re experts in their fields or experts on some of the functionality that they’re developing, but they’re not also necessarily experts as a security team might be, right? So, part of the trick, too, is to offer kinda help, knowledge, learning via the tool set so that, you know, the developers can keep the velocity up and develop to those standards more easily.
And I think Steve mentioned a little bit of a buzzword that kinda we like to think about is, it’s continuous compliance, right? It’s having this throughout the process and enforcing it and continuously monitoring, remediating, fixing, and reporting that become important for these processes to run really cleanly.
Ashley: You know, it’s interesting, there was—kinda stepping back in time, and again, I think about the myth of developers don’t wanna, don’t care about writing secure code. Well, of course, they do. They don’t wanna write insecure, vulnerable code. But I think there’s a period where we thought training was—you know, let’s teach them about how to avoid writing code with vulnerabilities in it.
But when you talk about a continuous governance environment, you’re kinda talking about a whole layer of things, right? It’s from the person doing the work to the environment that it’s being checked in and integrated and being tested into the workflow, and then really, the data, the information about that, that’s happening on a continuous, ongoing basis. And that’s really your data set for producing the compliance information, right? We don’t have to go ask you or check your code, it’s already been done, it’s done every step of the way all the time, and here’s the evidence of it, and here’s what we learned from it and here’s where we’re doing well and here’s maybe where we can improve.
That’s how I think of continuous compliance. Yes, it shows up in a report down the road, but it’s continuous, because we have all that data.
Howard: Yeah, I think that’s a really important point from the efficiency as well. I mean, the Security team or the Development Management that want to know when this product or project will shift obviously need to know where they are relative to that compliance requirement. And if we can keep it compliant from day one, we don’t ever have that risk of building up technical debt at the end of the process that we then have to solve, and I think that’s a really important evolution that we’ve seen over the last few years, that we have that knowledge to hand immediately at our fingertips, as you say. So, that’s a really important piece.
And the other point you made there about the developer education, I think, again—yes, you can teach developers and I know myself, in the past, I’ve learned the right way to do things, wrong way to do things, but there’s nothing quite as good as actually showing the examples of when things are potentially risky within the code you are writing, because you really understand, at that point, what you’re writing and how this could then be a problem, and it is the classic on the job experience that you really want for a perfect education, almost.
Ashley: Let’s take—I’d love to hear your kinda definition or how you describe Static Code Analysis SaaS to people about, if you’re embracing that, what does that mean to a development team, to a software project team?
Howard: I think from my perspective, it’s the idea of having an automated code reviewer, almost, looking at the work you’re doing in a fairly pleasant and passive way. [Laughter] Not being critical in any way, shape, or form as well. So, from a cultural point of view, I think the developers perhaps accept it more when it’s from a machine rather than a colleague that perhaps is picking holes in the code they’re writing, which is another good point about having a computer do that initial check.
But yes, it’s like having that guard at the end of the process, and then we often joked historically about the value of the tools and saying, “Well, you know, if you had two developers that were both going for the job, you know, you’re advertising for a new development role, and one of those developers is, you know, the cost of the license is more, but they never make a mistake in the code, they check in clean, compliant code with every commit, whereas the other developer, they both have the same experience, they’re both writing good code, but the other one could make mistakes and could check in, potentially, security vulnerabilities. Which one would you pick as a manager?” That’s quite an interesting way of looking at it, but of course, you would always want to have that assurance that, essentially, it’s clean code from day one.
Foster: Yeah, and also, you know, in terms of the perspective, Steve talked a little bit about the developer perspective or the hiring perspective. From the business perspective, you have to think about the costs of reputation, lives, revenue, anything of a product being in the field and discovering an issue. You know, the costs to fix a bug exponentially gets more expensive as you go further down the software life cycle, right? So, it’s cheapest to fix it then and there, right at the start. And, you know, in terms of kind of pain points that the software is trying to fix is, you know, providing that knowledge, providing that quick remediation as developers are working, ensuring the velocity stays up, ensuring that the continuous compliance enforcement throughout the process means that there’s not a heavy lifting of work later in the process, because you know, you weren’t reaching compliance requirements.
You know, the ease of using the software however a developer wants to work, it’s all about velocity, you know? And in terms of the business having that oversight and understand, you know, what are our risks, and how soon are we gonna be able to ship this product? Because, you know, we know with many products, the faster to market you get, you know, you have that first mover advantage, and that’s what’s needed in this fast-paced software market.
Ashley: Well, that’s very tough to recovery from reputation damage and quality issues or perceived quality issues around your products, too.
You know, one of the ways that I explain it to folks outside of being a developer or within the security world is, it’s kinda like taking your spell check of a document and adding in the grammar checking—so, this is the higher order, more complex, specific things around grammar. In this case, it’s around security and helping you as you’re developing or creating code, right, make sure that you do it the best way, you know? Avoid introducing security issues.
I know that’s a gross oversimplification, so don’t worry, I’m not saying Perforce is like a grammar checker. [Laughter] But that is sort of a non-technical way of describing what we’re doing. I mean, we want it to be that natural, as part of the process.
Howard: Yeah, absolutely. And I think that’s very true. I think, you know, we’ve seen the evolution of Static Analysis tools very much in the same way we’ve seen the evolution of things like the Microsoft Word, you know, spell checker to grammar checker and so on, and it’s kind of evolved to be more and more all of the time, you know, of the systems similarly.
But on top of that, there’s also the opportunity, then, to, what we find quite interesting is to write your own checkers and rules that are specific to your application. So, we can find in some places where there have been security problems in a certain application for some special reason—not necessarily related to the constructs of the code, as most of the coding standards check, but something that’s specific to the project because of the way the project has designed all the code, and that project interacts with outside external interfaces.
And it has been interesting to work with customers in some cases. I worked with very large telecom customer on one occasion and he said, “Well, you know, we have this translation module between sort of the application and the platform, and nothing should ever, you know, go direct between the two unless they’re going through this one piece in the middle.” And it’s kind of the golden rule we’ve taught to all the developers from day one and nobody will ever break this rule.
So, you know, obviously, I was slightly challenged by this and said, “Well, it’d be nice to check the rule. Let’s put that enforcement in just to see if we can, you know, use a custom rule to see if there are any cases and, of course, as you say, it should be clean.” And obviously, you know, a couple of hours later, once the analysis is run, it’s a very big system, we found a couple of cases—only a couple, but there were a couple of cases where exactly what they said should never, ever, ever happen for the security of the entire system, it happened.
And so, we were able to then remediate those problems and keep that rule in for future development, and of course, it will always be enforced. So, that was a really interesting case where there was also that, where you have something specific, it’s not just generic grammar issues, if you will, or spelling issues, but something very specific to that document.
Ashley: Mm-hmm.
Foster: Yeah, you can imagine some of these things coming out of requirements or an internal coding standard or things that a Development team might be following and be able to enforce these. Especially when you start looking at legacy code, giant monoliths, you know, that have been around for 5, 10, 15, 20 years, decades, you know, and you don’t have those original people who are developing, right?
So, you know, it’s—we sometimes think of our tool also as, you know, kind of like a quality assurance on a product or a code base that, you know, it’s legacy and all the new people have no idea how to use it, but you need to expand on it, you need to develop it, because it’s valuable IP, and how do you manage developing new functionality and moving forward when, you know, it could be as brittle as a glass house, right?
So, you know, that’s the other benefit you get out of being able to enforce and write your own standards, I’ll call them, and rules, yeah.
Howard: Another classic case there, Mitch, is—and you’re well aware of this as well from the field is, this system was never intended to be connected to the Internet, you know? This was something that—
Ashley: [Laughter] You’re guaranteed it will be, then.
Howard: Yeah. [Laughter] It was an embedded system that was entirely supposed to run in an embedded environment with no connection to an outside world at all, and all of a sudden, you know, it or something around it that’s connecting to it is now connecting to the outside world and there’s open attack interfaces, and now we have to make sure that they’re not gonna compromise the original legacy code base.
Ashley: Mm-hmm. Yeah, there’s just sort of the rule of unintended uses always occur, right? The thing we don’t want to happen is usually what happens. [Laughter] So, yeah, the demo becomes the production system is another great example.
I’m really interested in your perspective on where you think we are. If I made an analogy to the network security world, the best time to be a network equipment or network security salesperson is the day of or the day after a compromise, and automatically, you’ve got budget and we need to solve this problem. I mean, I think we’re well past that in most cases, but where are we in the software world? Do you see this as a, it’s just a natural thing, “Okay, what’s our SaaS strategy, how are we gonna be doing security within the application, what tools should we use for that?” Or is it something where you have to mature to a certain point or have experienced some issues before you’re really gonna take it seriously?
Where are development—and I’m asking for a generalization I know, but what’s your sense of where people are today?
Foster: I’d say, you know, in general, it seems as though most people don’t take these kinds of things seriously until it bites them. I think, you know, there’s some kind of safety feelings of some types of products that you can use where, you know, you find out about a security vulnerability or intrusion and you’re using a piece of software that basically just tells you that, you know, the barn doors have been opened and all your horses are gone, right? But that’s too late in the process. I think we all need to understand that, right?
So, the point of dealing with it sooner and using tools like security application and security testing tools like Klocwork or QAC is that, you know, you’re trying to build the risk reduction, the known unknowns that you can at least deal with sooner, right? So that, you find out later, in the worst case, that the doors were open, but, you know, nothing happened, you still have your horses, because the stables didn’t open at the same time or whatever, right? Just to make a kind of analogy to it, there.
Howard: Yeah, I think in the—obviously, we do a lot of work in the embedded space, and in that area, having seen it over the last 10, 15 years, I think, you know, there was a time where we were educating people a lot more about what it was, you know, what it was that we could do with Static Analysis to actually find some of these problems. Whereas I think—and there’s a lot more awareness now of the risks in the security field and, obviously, what Static Analysis can do to help. So, most people are starting to think about it if they’ve not already got it built in.
So, I’d say it is becoming more of an expected insurance policy against those kind of things within the tool environments. And obviously, the other kind of driving force there is the regulation. So, we’re starting to see standards themselves creeping into the market like the new ISO 21434 standard for automotive, for example, where they’re actually going to require some level of looking at the security of the system and making sure that we’ve done some of those things correctly and, you know, probably that will include what we expect it to include, some level of Static Analysis and coding standards and so on and so forth.
So, that stuff is really becoming almost regulated into the process just like functional safety regulation has been in that process from the beginning.
Ashley: Definitely.
Foster: Yeah, you’d imagine—
Ashley: Sorry, go ahead.
Foster: I was gonna say, just on that topic there about cyber security for automotive, you think about it, the cars are effectively rolling entertainment systems, right? So, there’s a lot of new kind of vectors for attack, right? An insecure Bluetooth module inside a car and its infotainment system that is also connected to the bus controller that’s driving all the data from every sensor on the car that’s controlling your fly by wire driving and your fly by wire steering and all those things, right? It’s very important—very important.
Ashley: And usually, you have, I believe, you know, both cars and trucks that have starters can start by Bluetooth as one of the mechanisms. So, yeah, you have potentially the same attack vector, you need to be able to isolate that, and now you’re into the function of the engine versus the entertainment system, and there’s a lot of work going in, I know, to the automotive systems. They’re very complex. I mean, just as complex as many, you know, many business applications—maybe even more so because they’re integrated from so many different suppliers, and they are, of course, largely based on, built on embedded systems.
It seems like embedded systems, though, are much more accepting of an update process, of keeping the software current and requiring some way of being able to update software in an embedded system or maybe making it an automatic process so it’s kind of touchless. Is that your experience as well?
Howard: Yeah, I think so. I think that is coming into many systems now. I mean, there are certain manufacturers driving that kind of process—automotive, for example, is a prime case. And it is interesting, because if you go back 10 years, that just was never even considered. You know, once you kind of flashed that system, it was staying as it was until you climbed up the mobile phone master or whatever it was that the embedded system was doing and flashed it again with the latest version. So, yeah, it is, the over the air updates is creeping in, and that will lead us to a more continuous delivery type architecture with the delivery of the software as well, that we’ll be able to go to that.
So, yeah, it’s gonna be an interesting time. And as part of that, I guess now the focus has moved slightly from the cost of fixing an issue out in the wild to just, we don’t want to have the issue because of the potential impacts on having that issue even for a day, you know? [Laughter] The idea that a car might be possible to take control of from an external source, for example.
Foster: Yeah, and I think you’re seeing a lot more emphasis on the security around a lot of these embedded systems because a lot of them are so, you know, requirements driven, compliance driven. They have ISO requirements that are needed to be validated by external bodies to make sure that these things are safe to use, because you know, a lot of our infrastructure is built on the backs of these things, that they’re, in some ways, leading the charge of seriousness for these reasons, right?
Ashley: Mm-hmm. And it seems, also, that industries like, I mean, we were talking about the automotive industry, are putting together their own standards, I think in part of fear out of, “We don’t want government standards to dictate to us, we’d rather us solve the problem, you know, industry solve it” in a way that they’re more accepting and flexible to them. But do you see more regulation coming at a kinda geographical country, government level? Is more of that happening? You talk about ISO standards, of course, being more industry.
Foster: Yeah, I’d say so [Cross talk]—
Ashley: You see it happening? Like, our privacy standards, now we’re talking about software security standards?
Howard: [Laughter] Well, yes, there is that as well. Perhaps in the European level, for example, of something we see on this side, but yeah, mostly, it’s industry standards, from what I can tell that we’re experiencing and seeing come downstream at the moment.
Ashley: Mm-hmm. [Cross talk]
Foster: Yeah, I think it depends on the level of kind of sensitivity of the data. So, you do see requirements as they relate to finance and, like, PCI, DSS, and things like that. You see security in terms of—I’m running a blank, here, but they typically come from when the data is so sensitive, you know, the U.S. government has their STIGs, DISA STIG rules.
Ashley: Mm-hmm.
Foster: So, it tends to come down from the level of sensitivity. I wouldn’t be surprised if we do start seeing more requirements, perhaps, for public data, but right now, we have things like the HIPAA compliance and things like that, as they relate to very, very sensitive data.
Ashley: Yeah, the defense industry is also a good one to start to drive standards, too, and I know they’re doing a lot in security.
So, just to change, shift focus here, we have a few minutes left. We were talking earlier about where people are in their maturity level of adopting SaaS and writing more secure software using those kinds of tools. If you were someone leading a project, maybe an architect or a lead developer or somewhere in the management chain that said, “You know what, we’re almost there, I just need to do a little bit more convincing, or I need something else to help my organization get there to see why we need to do this, and I’d like to do it before we have a major incident”—what’s your best suggestion for kinda doing that? I don’t know if it’s internally selling or if it’s demonstrating the value of it or how do you see people successfully getting the organization to begin to support adopting it?
Howard: Hmm. I think that’s a tough question, just because there’s so many industries and so many different kind of topics of particular interest, it always helps to have experience of a well-publicized kind of issue [Laughter] that’s gone out into the—
Ashley: “We don’t wanna be like those people.”
Howard: Exactly, yeah. That always helps drive things a little bit when you’ve got something tangible to say, “We don’t want to end up there.” I think, again, the regulation piece is where I see a lot of that being driven. You know, the industry bodies are actually saying, “As a whole, the industry doesn’t want to end up there. That’s not somewhere we can afford to be, and so let’s try and make sure we get this right from day one” and that trickles down into individual projects and so on. [Dogs barking]
Ashley: Well, speaking of dogs [Laughter]—okay, hold on.
Howard: Okay. Yeah, so, that’s really where we’re—
Ashley: Okay, just start your answer again.
Howard: Yeah, so, that’s really where we’re seeing it, predominantly, from the—you know, first of all, we don’t want a case where there is an incident, because, you know, if we can use that as a tangible case and say, “We don’t wanna be like those guys, because that didn’t end well for them.” So, that’s, obviously, always a good driving force. But then it’s regulation from the perspective of the industry bodies themselves getting together and saying, “We really can’t afford for this kind of thing to happen, and we want to make sure that, you know, across the industry, we start to follow a best practice that’s gonna do our very best to prevent that,” I guess, is a good way of describing it.
Ashley: It seems this is also an area where security team CISOs can be a real colleague asset for making the case to the rest of the organization, because if you’re _____ on the software team, you have one set of justifications or definitions for why you need that. Another is, there are corporate—you know, the CISO is trying to accomplish these goals, these things with the organization. And it’s always easier to fit your objectives within a larger objective and get that supported and get funding—that kinda thing. So, that’s another suggestion I would throw out there, too, that also helps folks—and plus, it brings security and software teams closer together.
Howard: Yeah. [Cross talk] Sorry.
Foster: Yeah. I think we do see—we do see, we do have, you know, security evangelists that are, as you just said, who are championing it, right? And when we talk to those people and we go over Static Analysis with them and this is the security focus, right? It comes down to, you know, being able to help understand their size of team, what kind of current development practices they have, and what that means and how you can deploy, roll out the tools and, you know, not slow down that velocity, because that’s always the goal. Don’t slow down the velocity, but add more value.
So, that’s what we try to help them with, help them understand. And also, you know, help—sometimes, you know, Steve is sitting with these people and we almost help them mature in their software development process to, you know, move from a couple guys who are just doing nightlies and they have no version control, they’re not using quality gates, they have no type of automated pipelines to setting them up so that they can scale their development groups, they can speed up their testing, their development processes, and that all comes out of engaging with vendors like us.
Ashley: Great. Well, gentlemen, this was a lot of fun. I really enjoy talking security and creating secure applications and, you know, it’s something that I think everybody wants to improve at to different levels and degrees, and hopefully, before those bad things happen. It’s always better to do it before, but if you have to do it after, you do. Maybe sometimes that’s what it takes.
So, it’s been great talking with you. Again, this is Stuart Foster, who is Project Manager with Perforce—thank you, Stuart—and Steve Howard, Static Code Analysis Evangelist with Perforce as well. Take care, gentlemen.
Foster: Great, thank you.
Howard: Thanks, Mitch.
-
Change Management in a Fast-Paced DevOps World
While DevOps speeds up software release cycles, change management often slows them down. So, how do you do change management in a fast-paced DevOps environment? Change control enables organizations to meet compliance and governance while managing changes at the speed of business.
In this interview for TechStrong TV, Greg Sanker, former CIO, IT Operations specialist, author, speaker and contributor to the “ITL4 Change Enablement Guide,” joins us to share his views on the tremendous impact of Agile, DevOps and CI/CD on traditional change control teams. Greg talks about the new role of change control and his contributions to the “ITL4 Change Enablement Guide.”
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript: Change Management in DevOps
Mitch Ashley: Well, I have the pleasure of being joined today by Greg Sanker who is former CIO, operations specialist, he’s a speaker/author, a bit of a futurist in leadership and digital transformation. He’s also a contributor to the “ITIL4 Change Enablement Guide” and a reviewer of a lot of other content—so, long introduction, there. Welcome, Greg, to [Cross talk].
Greg Sanker: Yeah, wow—thank you. That’s kind of awesome. Thanks for having me. I appreciate being on the show. I think we’re gonna have—
Ashley: It’s always—always good to have a fellow musician. Anybody with a Telecaster in their background can be on this any time you want.
Sanker: Yeah.
Ashley: So, tell us, [Laughter] in addition to your wonderful guitar sitting behind you, tell us a little bit about yourself.
Sanker: Well, you mentioned, I am—I specialize in IT Operations. That’s what I’ve done for more decades than I’d maybe like to admit. But service management has been a core part of that, and it’s become really clear to me that my view of service management is a lot more streamlined and agile than a lot of people perceive it as. And so, I’ve begun speaking at conferences and whatnot. I have a book of my own that’s published—I’ll let you guys go find it if you’re interested—about change management.
But most recently, I was asked to work on the ITIL project on a number of ways, but most notably, the change, what we now call “Enablement Guide.” And one of my prerequisites to get engaged with the project was, if you’re gonna ask me to write it, I’m gonna write it the way I see it, which is very different than what we’ve been doing in the past, and we’re gonna be pushing it forward, so. So, that’s a bit about me.
Ashley: That’s wonderful. That’s definitely a consistent theme with ITIL4 and other folks, Mark Smalley and others that we’ve talked with is really recognizing—I think ITIL’s had its sort of process moniker, right?
Sanker: Yeah, pretty much.
Ashley: But there’s a huge emphasis on, you know, how do you fit ITIL and all of the aspects of operations into something that is more of a dynamic DevOps-y agile kind of world, if that’s a way to describe it. And not to say that it didn’t fit in before, but I think it’s making it more prescriptive in that kind of setting is what a lot of the good work that you and others have done.
Sanker: You know, I was an early embracer of DevOps, and part of the reason for it is the mindset that explores possibility that ignores the boundaries in an organization to achieve outcomes. That’s, like, music to my ears—I’m all in. So, it’s a match made in heaven, but I can mention, it’s like, there’s a lot of organizations that have adopted service management, they have used ITIL in ways that are very prescriptive, very heavy weighted, very bureaucratic and slowed down. And it was never intended to be that way.
Ashley: Mm-hmm.
Sanker: It just is, it got adopted that way, right?
Ashley: Well, and I think a lot of the work that’s gone into ITIL4 will help make some of that shift.
Sanker: Yeah.
Ashley: Certainly, it’s positioned that way, and I know from other authors, that was the mindset of how the contributions were made. Now, you specialize around change management and we were chatting when we first got introduced about, you know, how do you do change management in a rapid development, agile, DevOps kinda world? If you were gonna be ask that at a conference by somebody, how would you—what guidance would you give them?
Sanker: Wow. So, I’ve got an hour and a half platform, right, for that?
Ashley: [Laughter] At least an hour and a half compressed into 20 minutes, yes.
Sanker: Twenty minutes, right? Yeah, absolutely. Well, see—well, let me put it this way. I think it really comes down to one fundamental paradigm that we need to change is, traditional change management, which is what I call your CAB-based, you know, Razor request for change, blah blah blah, is based on the idea that we inspect every single change. And as you probably know, Edwards Deming’s said many, many decades ago, “We’ve got to cease reliance on mass inspection to get quality products.” Well, that’s exactly what we’re doing when we say, “Hey, I need to make a change. Well, let’s just take a look at it,” you know, so on Thursday, we’re gonna get together and look at that. But you know full well that, in a DevOps environment or a continuous integration, I mean, we are moving all the time. We don’t slow down and stop.
And so, that mindset, just the mindset of stop and let’s inspect is completely incompatible with a flow-based, whether it’s agile or DevOps or CI/CD or any of the others that are out there that are more flow based—you know, rapid feedback loops, short cycles, et cetera—it’s just incompatible.
And so, what we have to do is elevate out of the inspect mindset into the expectation mindset. And that is—what is it that this organization needs from change management? And some of it’s governance and compliance, some of it’s operational stability and recoverability and some of those kinds of things. But let’s state what that is, we’ll state what those things are for the organization so that those that are working in the pipelines, the value streams can actually optimize how well we achieve those expectations and don’t dictate how they do it, let’s just dictate what the outcomes that are required by the organization. That’s the fundamental shift, and it’s big. I mean, it’s more than a 180-degree shift. It’s very significant.
Ashley: Definitely a different mindset for sure, you know, the inspection versus not. It’s not unfamiliar, though—that’s what security is doing about shifting left and moving ______.
Sanker: Yeah.
Ashley: Iterating the process, DataOps is also—you know, everyone wants to move early in the process, but it’s more than that, because you’re talking about a pretty different mind shift change. So, talk about compliance. Talk about governance from a change management standpoint in this new paradigm.
Sanker: Yeah. So, you jumped right to the jugular. That is the million dollar question.
Ashley: [Laughter] No, but it needs to be done and, you know, we all sort of pooh-pooh it.
Sanker: Right.
Ashley: But you know what? If you don’t do it, it’s really not fun doing it at the end, right?
Sanker: So, maybe you remember, because it certainly rattled my world—Dr. Forsgren said on stage at DevOps Enterprise Summit 2018, “CAB is useless.” And everybody in the audience laughed and cheered and duh duh duh duh duh.
Ashley: You’re not talking about Cabernet, you mean change advisory board.
Sanker: No. [Laughter] I bet you probably would enjoy a good Cabernet, and I’d—
Ashley: CABs, yeah. Those others are still good, but anyway.
Sanker: Merlot, et cetera, they’re all good. But the thing is, she’s not wrong. But she was looking at it from a very specific lens, and that lens was about failed changes—in other words, the success of changes and the recoverability of changes, two things that are really important to the stability of the business. And what she found in the data, she’s a data expert, she has a Ph.D. in this, so there’s no mistake, here.
She’s absolutely right—however, CABs frequently are part of the compliance equation. In other words, it’s useless for those things, but it has a use elsewhere in the organization, and for many organizations, that’s a very scary thought. Because that’s where we’re able to document and demonstrate for internal and external auditors and validators—are we doing the things that we’re required to do? PCI DSS, HIPAA, GDPR—you know, all of those other compliance framework, NIST 800-53, which hopefully we’ll get to in a minute, but they all have requirements around managing changes within your organization. So, if we don’t do it in the CAB, how do we do it? And that’s precisely the point.
So, what I’ve sought to do is help organizations build those compliance points into the pipeline itself. In other words, we can point directly to where that’s being done in the pipeline and don’t require a “let’s slow down and actually check the box, here.”
You know, the governance is—well, can I just say this? There’s a difference between governance and compliance. They’re two very different topics, right?
Ashley: Yeah.
Sanker: So, governance is about how organizations make decisions, and making sure that those decisions that are made throughout the organization align with or help fulfill the purpose of that organization, whereas compliance really has to do with the conformance. Are we conforming to applicable standards or expectations, either that the organization has or we have externally?
So, but when we talk about governance and compliance, we think about performance and conformance. I’m sure you’ve heard that—COBIT and lots of other frameworks have talked about that for years. But performance is about, you know, your strategy and are you utilizing your resources in pursuit, or are you producing value or value creation? And I think that’s fascinating.
I’ll bet you’ve read or heard of Mark Schwartz’ new book, “The Art of Business Value.” It’s a fascinating read.
Ashley: I actually haven’t read that one. I’ll have to check it out.
Sanker: I would put it on your reading list, and I know that sometimes he rubs people the wrong way. I found out that he rubs me exactly the right way, because he asks some really challenging questions, like, “What is business value?” You’d think that’s an easy question to answer, right? It’s like, you know, for a for-profit corporation, it’s like, you know, making your market windows, making your profitability, adding features that attract and retain customers, those kinds of things.
But what if you’re in—as I was as a CIO—what if you’re in a government agency? That’s not a for-profit organization, so what is business value? Delivery of services to citizens, upholding environmental standards? There’s a lot of things that governments do, but they’re not profit motive. In fact, in some cases, it’s not even financial, so what is business value?
And so, when we’re running our backlog, you know, we’re scoring our user stories, like, how do we score that in that kinda context? And then, just when he, when I thought that he had confused me enough, he asked, “Well, what if your business strategy”—and I don’t wanna give away his book. Go buy his book. But—
Ashley: No, just tell me the last chapter, that’s all I need.
Sanker: Exactly. [Laughter] I actually sent him an e-mail. He’s a great guy and he thinks very, very big. But one of the things that he said was, “Well, what if your strategy is, we want to move customers off of this platform and onto one that’s more profitable to us, or one that’s in alignment with our roadmap and our strategy?”
So, if we’re down in the value streams or in the pipelines, scoring our user stories against customer satisfaction, customer value, things that customers really, really demand and want, we’re actually creating negative value—these are my words, not his. We’re creating negative value because we’re making our customers like the platform that we want to get them off of.
Ashley: Mm-hmm.
Sanker: So, business value would actually be—no, do things that make them less likely to like this product. Make them want to migrate. Make it easy for them to say, “You know, this is really getting old. I’m gonna try your news service.” And so—
Ashley: Right, that behavior change.
Sanker: Yeah, exactly. And so, when we really start thinking about, we’re getting, you know, your governance, especially your governance and your compliance, we have to have adaptive, rapidly adaptive systems that allow us to, at any point in time, understand what the current strategy is and how it affects the decision making that we’re doing at any point in time—all without, of course, getting your board of directors involved with every single decision throughout the organization.
Ashley: Well, let me ask you this. I’m really interested, I’m really curious about how you take this kinda shift in mindset and how you’ve woven that into the “Change Enablement Guide“ that you were the author of. So, you know, it isn’t a process discussion.
Sanker: No.
Ashley: You’re talking about viewing things a different way, valuing some things maybe we didn’t value before, maybe some other things less, and maybe delivering a different set of things. Let me just share, briefly. I remember a day, probably not too long ago, where CABs saw their role as saving us from developers and all the bad decisions and lousy code they were gonna create, so that’s what—
Sanker: [Laughter] Oh, yeah! Absolutely.
Ashley: – that’s what they say in those CAB meetings.
Sanker: Yes. Oh, absolutely.
Ashley: But, you know, that’s not what we’re talking about. Tell me how—so, how do you build that into, in a way that’s gonna help people make that shift through ITIL4?
Sanker: So, thank you for saying that, because there is some unspoken or lightly spoken or spoken only in factions kind of frustration, there. It’s like—let’s be honest. I used to be a network engineer, I did network security, I did all things network. So, I used to be a change manager at one point in time, and who the heck are you, Greg, to cast judgment on this piece of code and decide whether that’s gonna work appropriately? Everybody knows that I can’t add value to that, and everybody knows that this external body who’s not familiar with it, certainly not at the pace—you can’t add value, and yet you act like you do. And that just—that doesn’t sit well with anybody.
So, we really need to, if we shift our focus to, “What is it that we’re trying to achieve?”—so, I talk about putting the right measures on the pipeline. How well is this pipeline producing outcomes, those outcome expectations? Is it producing consistently high quality changes? Is it consistently not delivering failed changes—or, for the failed changes that do go through, how quickly and easily can we recover from that with minimizing impact to the customers?
Those are the things that your CAB or whatever your body is should care about, not the individual changes. You’ve got to get away from the individual changes. You can’t add value to that.
And so, I get asked a lot of the times, it’s like, “Well, what do we do with our CAB?” since, you know, they’re very unpopular. And in some cases, the best answer is—now I’m gonna say it publicly, so I’m gonna be quoted on this, but—“Blow it up. Just get rid of it. And don’t ever use the C-A-B word again, because it has such a bad name.”
But it isn’t without value. So, if you have a CAB and the CAB starts saying, “Look, developers, we can’t add value to what you’re doing, but what we really care about is some of these things. So, let’s talk about those things.” And if you’re talking to, particularly DevOps, agile, automated pipeline folks—they were gonna welcome that conversation because they’ve already got it covered. They just do. You know, like you know, I’m the change manager—“Well, have you tested that code before you released it?” Well, dude, that’s the only way it gets in production, if it passed tests. There’s no way for it to make it into production. So, not only is it insulting, but it’s a dumb question, and undermines your credibility.
Ashley: Well, let me ask you this, then. I think where you’re going—and tell me if this is right—is, “We have so much automation in this pipeline process, right? That’s the whole idea of being able to produce software more reliably, more rapidly.”
Sanker: Oh, yeah.
Ashley: There is so much data that’s generated and captured, but not leveraged.
Sanker: Yep.
Ashley: We treat it as data exhaust, right? That’s sort of this fancy term for it.
Sanker: Yep.
Ashley: In that is a lot of compliance, is a lot of governance information—
Sanker: Yeah.
Ashley: – that you can then elevate and connect, and basically save developers time and not having to produce reports at the end. “I’ve already got three-fourths of what I need or what you need, and let me help you put the finishing touches on it so you’ll sail through the next whatever NIST whatever it is.”
Sanker: Yeah. So, there, you get to write a chapter in my next book, because that’s exactly, exactly what we want to focus on is, we want to be—you know, alright, this is potentially controversial, but change management is a feedback loop. It helps us understand the quality of changes that are coming through our pipeline. Sitting there and passing judgment and saying, “Nothing passes until I say so”—that’s not adding value. That’s absolutely antithetical—you know this—to the DevOps continuous flow kinda mindset.
Ashley: So, they’ll go every way around you you can think possible, you know?
Sanker: Yeah, and the other thing, as we well know, is like, develop is most frequently used in the flagship products. “This is the core business. Get out of our way. We have to deliver these changes to maintain our profitability, maintain our market windows—all of those things that are core to the business, and you think I’m gonna stop for three days when I went from, you know, a user story to a value in under 24 hours in some cases? There is no way I’m gonna stop. I’m just not going to. I will take the risk of it failing and you saying, ‘You shouldn’t have done that.’” And that’s—
Ashley: You can talk to the chief product officer about—
Sanker: Exactly.
Ashley: – the stock’s direction. [Laughter]
Sanker: And I helped an organization who was, you know, they had agile coaches and they were doing all that. It’s a large company, but I’d rather not name them. And I was asked to come in and look at their change management.
Well, what they had done was, rather than having one CAB a week, we were having daily CABs. And we were having daily pre-CABs and then the CABs. And one of the things that I tried to get them to understand is, these two are entirely incompatible. If you’re going towards agile DevOps as a development methodology, then you must move away from CAB. CAB has got to go.
And what was interesting is, their flagship products, the ones that really made money for the organization, they could get away with murder. Why? Because it was so critical to massive revenue in the case of this organization.
Ashley: Mm-hmm.
Sanker: And so, we have a big outage, we recover, we move on, that’s okay, it’s an acceptable risk. And change management will still know, we can’t have any defects or whatever in production.
And so, we’ve got to get ourselves wrapped around the idea of managing business risk. You know, you mentioned something about the compliance, and just this last week—like I said, I’m working on a new book, but I was looking at the NIST 800-53. Everybody’s familiar with it, but they finally released version 5. And so, I’m reading through and the author’s made a very fascinating statement, and I encourage everybody to go look at this. Version 5 is phenomenal, because what they said was, “We have eliminated or reduced the specification of who actually does the activity, and we’ve instead focused on the outcomes that need to be achieved.” In other words, some of the compliance requirements or the controls said, “IT will review” or, “The change manager will review.” And so, if you’re gonna comply with NIST 800-53 previous versions, there’s not a lot of wiggle room around that.
But I’ve been saying for over 10 years and others have been saying, it’s like, if we focus on the outcomes, then who’s doing it, how they’re doing it is open for us to do it in the most effective way. And that gets us into the pipeline space. And it’s a major step forward, because we struggle with that. Like you said, we have to meet our compliance, you know? If you’re dealing with sensitive information, you know, pharmacy has HIPAA and credit cards and all that sort of stuff, you have to. It’s just not optional. But you don’t have to do it the way we’ve always done it, and that’s the piece that I like to bring to organizations. It’s like—how are you doing it, and can we demonstrate that we’re doing it? Those are basic audit business controls minimums.
Ashley: I’m curious about, we’re sort of talking also about the whole issue of agility and speed, and on the other end of it of the belief of the best way for something to be stable is don’t change it.
Sanker: Oh, yeah, absolutely.
Ashley: Which is antithetical to the environment we’re in, for sure. And so, it’s kinda like you have to embrace, you just have to say, “We love change, and how do we be super effective at it?” instead of, “How do we not change so we don’t mess it up?”
Sanker: So, there’s two places, two things that I’m gonna mention briefly, but we won’t have time to explore them. There’s the field of study around complex, you know, and Cynefin is one of the most notable frameworks. And the idea that there’s simplex, complex—or complicated, complex, and chaotic, and we treat changes frequently like they’re either simple or complex. Meaning, we need to get some eyeballs on this thing.
But the fact of the matter is, most systems, especially systems that we’re DevOps-ing, are more like a complex adaptive system. There’s a level of complexity that we’ve not seen in days of old, like back in the ‘80s and ‘90s and early 2000s where these systems were, we could actually try to understand them. Modern system—
Ashley: They’re not linear. They’re more, you know, dynamic and self-creating, if you will, in run time.
Sanker: Yeah, exactly. So, that hit me a number of years back. It’s like, this whole mindset, it’s around the idea of, it either is simple or it should be simple, and we can understand it, and we can predict the outcomes, and you can’t. You cannot do that.
And so, we need to build adaptive systems that say, “Hey, look, this is complex.” And you’re not gonna send it to a board of examiners, you know, a CAB who are gonna say, “Let’s take a look at all the risk,” because it would take you years to do that in a complex system. That’s just—that’s not gonna happen.
So, we need to adopt practices that are more adaptive. And one of the things in that complex area within the Cynefin framework says that we need to run some experiments to better understand the mechanism. And how that applies to change management—see, this hit me between the eyes because it was like, “No failures in production. No failures will be allowed”—that’s antithetical to learning how the system works. If you don’t make some changes to see how the thing works in operation, you’re denying yourself learning and you’re probably deluding yourself that you’re actually actively controlling it, right?
The other piece of study that I’ve brought to the table is—and you’ve probably heard of it—this study of anti-fragile. There’s a number of authors whose understanding is astronomically higher than my IQ, you know, Ph.D.s who actively debate such things. But I take complex things and make ‘em simple. Here’s the thing—some things, like a tree, when the wind is blowing, the tree is waving, but there’s more than that going on. It’s the actually fibers, the plant itself becomes more resilient to wind if it’s been exposed to wind.
Here in Oregon, often there’s logging operations that have a lot of controls for the environment—trust me on that. It’s changed a lot since the good, ol’ days. But one of the things that we do hear is, we leave a segment of trees at the edge of a piece of property that’s being logged, and we do that for the animals, for the environment, we protect the riparian areas. But one thing that you’ll notice is, within one year of leaving that stand of trees that used to be next to a large strand of trees—they break. A storm comes, and they get hit. Why? Because they’ve relied on the strength of their neighbors up to this point, and they’re not suited to stand the wind alone, and so those trees break.
Here’s the thing—if we don’t inject some failure into the system, for a system that is truly anti-fragile, we’re actually making our systems more fragile. And so, a mindset of change management that says, “No errors, no problems, no failures ever, ever, ever”—you are making your system more fragile, and you’re treating a complex system like it’s simple. And so, we’ve got to change our mindset.
Now, you know full well—and let me just blast into this—you know full well that, as we take these mammoth changes that we used to make in the old days, you know, it took 6, 8, 12 months to produce, and we’ve turned it into these microscopic things that can go through your pipeline and ours and produce values is your blast radius. What bad thing could happen in production has materially changed. The risk has become very, very small. If that—
Ashley: And your ability to fix it—
Sanker: And your ability to fix it skyrockets.
Ashley: – your responsiveness, it isn’t six months to fix it or three, it’s three minutes.
Sanker: Yeah. So, one of the change outcome expectations that I encourage people to establish is, what are your expectations for recovery from any given change, and can we realistically do that?
So, for instance, if I put something in my flagship product that doesn’t work as desired, if I can recover within two minutes, no harm, no foul, is that okay? In a lot of cases, it is. If it’s a defibrillator keeping somebody alive—probably not, that’s probably not okay. But in a financial transaction or a video streaming service or whatever, you know, modern digital services, it’s probably okay, especially if you’ve engineered the release in such a way that you do it like that. You know, blue/red switches, on/off—all of that kind of stuff that we’re engineering in become, you start seeing why we’re able to do that and what that actually does in production. It’s wonderful.
Sorry, you got me excited about that.
Ashley: Yeah, boy, I could take you a lot of places with that. [Laughter]
Sanker: Yeah.
Ashley: So, let’s take it back to someone who’s gonna pick up a digital version or a printed version of the “Change Enablement Guide” author.
Sanker: Yeah.
Ashley: How is that approachable for folks? Let me be honest. I’m probably not gonna go download that to get my heart pumping to go work out, get a better workout—or maybe I will.
Sanker: [Laughter]
Ashley: Maybe after listening to this, I will—but how do you make that engaging for people?
Sanker: Yeah. Well, so, one of the things—well, I’ll put it this way. It’s like, we’re not selling a product, we’re selling an opportunity to solve some problems. That’s why we call it best practice, right? And there’s a lot of best practices, and I don’t feel like they’re nearly at odds as maybe some of the industry might feel like.
But change enablement is—and I’ve been saying for years that this day has come and it’s now here is, change management is in the crosshairs of modern digital organizations. It’s a pesky remnant of days past. And what the “Change Enablement Guide” is—and there are was more. I mean, for production reasons, I had to kind of limit it. But what I wanted to do was point in this direction of what I’m talking about. It’s like—let’s get people moving towards that. Because like you said, we’re not gonna change overnight. I’m not gonna read the “Change Enablement Guide” and go back to work the next day and say, “Alright, I can save all the world’s problems.”
And so, what I’ve subsequently started doing is writing guidance around, “But what do I do today? You know, I went to a DevOps conference, I’m excited, but what do I do when I go back to work? How do I start moving in that direction?” And blowing up the CAB, while we might all celebrate that, that’s not a good first step. Instead, what we need to do is start moving ourselves in a better direction. It’s not more of same, it’s—what do we actually need to be doing here, and what’s the best way to accomplish that?
And it’s difficult, culturally, because it becomes more of a culture. Like you said, there are CABs galore who despise developers probably as much as developers despise CAB. It’s a match made in—
Ashley: Cats and dogs, living together.
Sanker: – yeah, exactly. [Laughter] Good reference. And so, that group of people who are widely known for having that kind of mindset are not gonna be welcome when they show up to your spring and say, “Hey, we’d really like to be involved earlier in your change management so that we can help you.” It’s like—yeah, sorry, bud. Let’s start somewhere else.
So, if change management starts with, “We’ve gotta get out of your way. We’re gonna start delegating more and more of your changes into your value stream, and instead, we’re gonna be focusing on, we’re gonna partner with you on, let’s look at the results of what you’re doing. Because we believe that you are optimizing your delivery and we believe that you have the right tools and controls in place.” So, we are moving more towards, “Are we achieving all the outcomes that the organization has? Is the organization even clear on what those outcomes really, really are?”
You know, the one that always—I love this one—“All changes must be tested before going into production.” It’s like—well, how many shops have you been in where that’s not true, at least in modern times?
Ashley: Mm-hmm, yeah. I call that the “meh” policy. “We’re gonna test it? Meh.” No, of course we’re gonna test it.
Sanker: Yeah, we call it blow testing, right? You know, it’s like, “Eh, okay, I think we’re alright.” That is true. In modern terms, though? For flagship products, for things that are really, really core to the business, most organizations have some form of automated testing, or that’s just part of the process. So, rather than make everybody say, “Did ya test it?” at a CAB, just, like, “Hey”—
Ashley: “Of course you did.”
Sanker: Yeah, exactly. Is that happening in the value stream? If it is, I’m not—don’t worry about it. We’ll just move on. Is somebody with the appropriate authority approving that? And that’s—I am gonna take a minute to talk about this one, Mitch, is that everybody talks about change approval in terms of who’s gonna approve this idea going into production, this change going into production? Because if something goes bad, we wanna know who to blame, right? In fact, we used to say, “Who has enough—who is sufficiently authority that they’ll get fired if something goes bad?” Which is, of course, the wrong question.
Ashley: Yeah, [Cross talk].
Sanker: Because the right question is, “Who has enough authority that, if it goes bad, they don’t get fired, because they have the authority to put on that level of business risk?” Right?
But if we move—and we could talk for hours, but the idea of product ownership or taking a product view of services or modern systems is, “Isn’t it true that the product owner is accountable to the organization to decide what things we’re going to move into production?” So, to me, that’s the change approval. We’re approving that we want this change to happen.
So, nobody’s adding value by making approval at the end of the chain. We’ve already decided that. We already know that we’re gonna make this change.
So, those are two different approvals. One of them is about a product decision or a governance decision, the other is about a business risk decision of, “Is it actually going to achieve the results that we set out to do?” I believe that most of the latter can be automated. If it’s passed its test, if the testing systems are appropriate, if we’ve done—in some cases, I love peer reviews. It’s like, you know, the code needs to be reviewed by a peer, stamp it, and we ship it out. It happens in a minute or two, you know, not hours or days or whatnot like that.
So, I think as we kind of retool or thinking around it, I think that’s where we can start really solving actual problems that organizations have.
I am reminded, though, I saw this—you’ve seen these before, these overhead views of an Indy car or a Formula One car coming in for a pit stop and all this stuff happens and off they go. I saw one this week that was 1.8 seconds. It was awesome, right? And people like thus always share those on social media of, like, “That’s an agile organization, doing the things” and blah blah blah.
And so, you can’t really argue with 1.8 seconds—four tires, top off the gas, and out you go.
Ashley: Impressive.
Sanker: But the question, to me, is—like, I’m a bit irreverent. Why isn’t it zero? Why can’t we make it zero? If faster is better, let’s make it zero. And the answer is, we won’t make it to the end of the race unless we have fuel. We won’t make it, because [Laughter] when you’re racing at 200 miles an hour around corners, your tires literally don’t last that long. It’s not a matter of preference, it’s a matter of life and death, right? So, we have to change tires.
So, the question isn’t, “Are we balancing speed and agility against risk mitigation?” It’s what it is that we’re trying to accomplish here. And in the case of Formula One, we’re trying to win the race. And to do that, we have to take on certain activities that cause delay—that’s a requirement.
So, the question isn’t, “Do we have to take a pit stop?” We do. We need tires and we need fuel and sometimes minor adjustments. So, the question then becomes, “How quickly can we achieve those outcomes?” And putting four tires on in under two seconds—that is phenomenal. Hats off to those folks that are doing that. And how do you get however many gallons of fuel we put in a Formula One car in two seconds, I don’t know, but that’s amazing, right?
And I feel like that’s the mindset that we need to adopt. We need to quit asking, “Why do we have to do change management?” We have to do change management because part of it is to be compliant. That keeps us in business. That keeps us out of trouble.
Ashley: But it’s a really good example, because it’s not an absolute, it’s not, “Do we need to fuel?” but, “What’s just enough fuel for us to win?” right?
Sanker: Exactly. Oh, and trust me, you’ve watched it. I know, I can see it in your eye, you watch Formula One or Indy and there’s like a thousand computers over there and stopwatches and gauges and dials and they are calculating exactly how much fuel based on the game strategy or the race strategy, the conditions, who they’re up against, what they think their strategy is—that’s how much fuel we need to take on, right? So, it’s—
Ashley: Great model for us to think about of, you know, it’s not absolute quality or absolute whatever.
Sanker: Right.
Ashley: It’s what’s appropriate for the organization. We’re gonna have to leave it at there.
Sanker: Okay.
Ashley: And I want you to come back some time, because the next topic is Stevie Ray Vaughan or Eric Clapton. But I’m gonna—
Sanker: Oh!
Ashley: You can’t answer it. You have to come back and answer that, so we’ll have you here another time. It’s been a real pleasure talking to you, Greg Sanker, who is the author of the “Change Enablement Guide” of ITIL4. So, I think you really gave us some compelling reasons to go check it out. So, thanks a lot—it was great talking with you.
Sanker: Thanks for having me, Mitch.
Ashley: You bet.
-
Data Security in the Cloud with Imperva
Database security startup jSonar, founded by Ron Bennatan, was recently acquired by Imperva, a leading cybersecurity company focused on data and application protection.
In this episode of TechStrong TV, Ron Bennatan, general manager of data security at Imperva, joins us to discuss the plethora of data security challenges across cloud environments. They also talk about the significant increase in data usage and creation as a result of COVID-19.
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript
Mitch Ashley: I have the pleasure of being joined today by Ron Bennatan, who is GM of Imperva Data Security. Welcome, Ron.
Ron Bennatan: Thanks. Glad to be here.
Ashley: Good to be talking with you. I’m excited to chat a little bit about data security. Let’s start out with, just introduce yourself, a little bit of your background. I know you had a recent event with your company, Imperva, and tell us a little bit about what Imperva does.
Bennatan: Okay, yeah. I’ll just start with what Imperva does. So, we’re a cyber security leader, we have over 6,000 global customers and we’re just focused on helping them protect data and all paths to it. And kind of what that means is, we have both application security products and data security products, both of them leading in those spaces.
But kind of the, our view is that, in order to do a good job around data security, you really have to look at this as a whole thing, otherwise it becomes, like, a blanket thing, right? You pull it to one direction and you expose yourself on the other side. Because data is—yeah, it’s very complex. And securing all data and all ways that people access data is a difficult thing.
So, like, for example, I could be—I could be really good at securing access to data from privileged insiders, but then you also get access through the applications. And the way that the data tier looks at things, it doesn’t see enough when it’s accessed through the application, so that’s a problem. And then if you do a good job on the application side, you know, maybe on bots and APIs, but maybe you also have insiders, and maybe insiders are different when they access it directly or they’re admins of the data tier or they’re admins of the application tier.
So, really, our view is, in order to do a good job, you really need to look at the whole access and all the paths to it.
Ashley: Excellent.
Bennatan: How did I get here? So, I’m a 30-year data-slash-security guy, so I’ve been both, kind of working on the data side, whether it’s a DBA or application side, but also on the security side.
So, I started my career in the military. I then worked on Wall Street for a while, areas around Sybase and Sybase security, Oracle and Oracle security. Then I was one of the founders of Guardium back in the early 2000s, built this database security business, got acquired by IBM, was the CTO of data security at IBM for a while. Last, founded jSonar, also in the data security space, and then a month ago got acquired by Imperva, so I now lead the data security setup there.
Ashley: You just keep getting acquired. I need to hang out with you a little bit more.
Bennatan: [Laughter]
Ashley: [Laughter] Going back to Sybase—wow. Yeah, I was a Sybase customer. [Cross talk]
Bennatan: [Laughter] You know, it’s still being used all over the place.
Ashley: Isn’t it amazing? Technology never goes away.
Bennatan: It never goes away.
Ashley: We just don’t talk about it as much as we used to. [Laughter] Well, you know, data security is such a fascinating topic. And I don’t mean this in a pejorative way, but it’s a bit like whack-a-mole, right? It’s everywhere and as soon as you think you’ve kinda gotten control over it, it pops up over here, sorta like that balloon that squeezes between your fingers, [Laughter] you think you’ve got your hands around it. And plus, it’s not a static thing, it’s growing and it’s evolving, it’s being used by different applications in different ways.
It’s a huge challenge. I mean, how do you even begin to tackle that from a data security company?
Bennatan: Yeah, so, I mean, first of all, you’re absolutely right. And, you know, we’re all security professionals, so to us, it looks like a bad thing, but before we go down that route, it’s actually a good thing, right?
I mean, the fact that it is so complex, it is because it’s just hyper efficient, right? The people who are building data architectures are amazing. They’re doing an amazing job. Because if you look at data architectures 20 years ago or even 10 years ago compared to what they are today, it’s just—it’s like, it’s like an immature baby compared to what’s happening now.
Ashley: Mm-hmm.
Bennatan: And it’s a good thing, because these are highly optimized systems, very diverse types of workloads, very diverse technologies, right? A database today is not even close to what it was 20 years ago. A data lake today is an amazing thing, okay? You know, it’s hard to even put your finger on exactly what it is, a data lake, when you look at, say, an AWS data lake or an Azure data lake or a GCP data lake. They’re amazing.
So, these are good things, right? Our applications are getting better, they’re getting faster. We can do, like, crazy things with machine learning algorithms doing analysis, telling us how to improve our lives, okay?
Now, as security people, we’re always playing catch up, okay? We’re always—these people are creating these things, and security is never exactly baked into that stack, although it is getting a little better or we’re trying to get it a little better. So, we are playing catch up, and it does create a lot of challenges on what it means to secure it well.
So, you know, the fundamentals have never really changed, right? There’s authentication, there’s authorization, there’s audit, there’s the notion of finding where the data is, classifying the—that stuff doesn’t change. What does change is the complexity of doing each one of those tasks.
So, if you look at—you know, even the first thing you kind of need to start looking at is, you know, find all your sensitive data and make a catalogue of it and know the lineage and—you know, 40 years, we’re doing the same thing. And it just gets harder and harder, because you look at one of these data lakes, you know, I think 10 years ago, when we were building a data lake or thinking of a data lake, we would think of, “Okay, we’re probably doing some kind of a Hadoop project,” okay?
Ashley: Mm-hmm.
Bennatan: Which, in itself, was complex enough as it is, because you’ve got, like, eight or nine different services running, each one doing something a little different.
Ashley: It wasn’t that long ago. I mean—
Bennatan: It wasn’t that long ago, but it’s dead, right? It’s [Laughter]—that part’s dead. Now, we’re building something which is infinitely more complicated. Because—like, I’ll give you an example. A lot of my customers are building things on AWS.
So, you’ve got this—it’s not like the data lives in one place, okay? You’ve got S3 buckets, and then you’ve got access sometimes from Athena, sometimes through Spectrum, sometimes from both. And then you have some DynamoDB or DocumentDB databases for doing transient things, and then you shove it into Redshift so your Tableau can access it. And then you have AWS Glue moving things around, and maybe you even have a Hadoop stack inside EMR for the landing area.
Ashley: You have data in databases inside containerized—
Bennatan: Yeah, everywhere. And then you say, “Well, how do I secure one of these beasts?” And so, yeah, it’s complicated, it’s risky, and we just need to keep up with these guys who are doing a good job with making very innovative architectures. And, you know, it would be great, by the way, if we could’ve baked this into the stack before they even started, okay?
Ashley: Mm-hmm.
Bennatan: But, like, I’m too old to believe in fairytales.
Ashley: You stopped chasing that rainbow, right? [Laughter]
Bennatan: Yeah. And we just do need to keep up with them. We need to keep up with them, we need—you know, because a lot of my customers are larger, and if they’re larger, then sometimes they’re regulated, somehow, or they need to adhere to something.
Ashley: Mm-hmm.
Bennatan: And certainly, these days, they’re all also troubled by all kinds of privacy compliance issues. So, we need to give them controls that they’re used to getting, you know, on the stuff that they’ve been building for the last 30 years on prem. And it gets to a point that the application team wants—you know, they’re ready. They finished everything, they’re migrating it up, and even simple things, not even complex things like cloud. They have an application using SQL on prem and they want to move it to SQL on Azure—simple enough, right?
Not really, because on prem, they know what their tooling looks like, they know what their controls are, they know what they’re doing with privileged access, they know how to scan, they know how to create the policy for defining access. They know how to manage—they know how to do change management around that policy. Now, they all move it to the cloud. Sometimes, those controls are there; more often, they’re not there.
And, you know, so, for example, one of the things that we view as being fundamentally important for us is to make that transition seamless, okay? Same controls. Different methods, different tools, different policy orchestration, but to them, it should look the same. If it looks the same, then when the app team wants to move it over, security stops being this bad guy that says, “Nope, stop. Okay, tell me what you did, here.”
I mean, we can’t be the bad guys, okay? We have to—
Ashley: Can’t be the cops, can’t be the roadblock.
Bennatan: Yeah, yeah. We need to help. We can’t keep stopping things. It’s just a bad, bad—bad dynamic.
Ashley: So, let me ask you. There’s so many dimensions to this and usages of data. So, is that one, instead of, like, getting ahead of all of it and designing it up front, which we wish we always could and doesn’t happen—is that the right approach?
I’m thinking about how do we do security right—is that creating the, whether it’s the tooling or the framework or the structure of how we secure data, making that consistent so, no matter the environment, we now know how to do it, and we can do it in a reasonably consistent way and know that it’s been done as opposed to, you know, n+1 for every environment that we’re in, it’s all different and it’s all complex and nobody can keep track of any of it, more or less secure it. Is that sort of getting our hands around this problem?
Bennatan: Yes, that is absolutely the one sentence summary is exactly that.
Ashley: A lump sentence, but [Laughter]—
Bennatan: A lump sentence. [Laughter] Yeah, yeah, because we need to—we need to abstract things out and hide the complexity, right?
Ashley: Mm-hmm.
Bennatan: We need—the fact that the data environment is so complex, it doesn’t change that if you go high enough, the problem statement or the what you need to do statement is the same, okay? The problem is that when you translate it to, “Okay, what actually happens?” then it tends to be different here, different here, different here.
But at a high enough level, it’s exactly the same thing. So, you know, what—if the environments are getting more sophisticated and the architectures are getting more complex in a good way, if securing them also gets more complex, then we’ve lost. So, what we need to do is make it maintain a single abstraction layer, and then do the magic that translates it into the individual changes.
So, it’s absolutely what we need to do. Now, some of this is a heterogeneous question. Okay, some of it is a heterogeneous thing where, you know, you have different silos and you know, historically, people have always looked at data as a bunch of silos.
Ashley: Mm-hmm.
Bennatan: Okay, there’s like the database silos, the big data silos, the file silo. That stuff gets blurred (a) because in the cloud, it’s very blurry, okay? Like, you can definitely say that S3 is a file system, okay? It’s an object store, it’s kind of like a file system, but then you slap a FINA on top of it, and that gives you SQL queries on top of that file store. So, is it now a file system, or is it a SQL database? What is that thing? It’s—
Ashley: And DynamoDB and now you’ve got a whole bunch of different ways of accessing and using that data.
Bennatan: Yes. So, that’s one piece of it. The other piece of it, which is almost the same sentence that you said, but semantically, it has a different meaning, okay? If you look at kind of the concept of data governance, okay, from—in the last 20 years, it’s always been, like, at the top, there’s things like policy and things like, you know, what are the business requirements? And at the bottom, there’s the tooling. And always, whenever you talk to anybody in this space, they always say, “Don’t start here, don’t start at the bottom by just implementing tooling, start here and then do this.” Well, I’ve yet to meet a single company that has started here. You always start here.
Ashley: When do you get the luxury, right?
Bennatan: Yeah! Look, part of it is just us people, okay? We’re—us people are problematic, okay? [Laughter] People can understand very concrete things very well, and therefore, we can go do a project that does this and we can do a project that does—and the vendor landscape has never helped, because products are like this and like this, so… And you go to the analysts, and the analysts say, “Oh, you need to pick one of these and one of these and one of these.”
So, you know, for many years, it’s been one of these, “Okay, nobody started here, everybody started here.” And then projects have operationalization problems. It’s not that, you know, tools have problems, it’s the fact that, even if a tool does everything it was supposed to be doing, it gets hard to operationalize things. And then you start looking at it and you say, “Okay, within these silos, there’s actually no difference to operationalize things.” It’s the same thing, okay? You need the same processes, you need the same decisions, you need the same policy push down.
So, why are we doing it six times? Why are we not doing it one time? Well, we’re not doing it one time because we didn’t start here. And so, over time, you know, I kind of gave up on people starting here and assumed people would always start here. The question is, that abstraction—can you now take work that you’ve done within any one of those silos and leverage that in order to create that kind of in the middle, like a single control layer, okay?
Ashley: Mm-hmm.
Bennatan: I mean, a single control layer that then uses those tools, but creates something that is, you know, consistent. And if you start creating something that is consistent, then you’re not, you know, you’re not balled into these silos. And then when you’re—when you get some of these more kind of advanced architectures, you know, you’re not screwed, because you didn’t create something just for this silo and this silo and this silo.
Ashley: And then you know that all the bases are covered, too, you’re not having to do a unique process or solution for all the checkmarks you’ve got to put in place, and who knows if those work reliably in every environment, whatever way you figured out to secure the data or do access control or manage data integrity, whatever that is.
Bennatan: Yeah.
Ashley: Let me ask you. I think, maybe other than you getting acquired by Imperva, most of us want to put a checkbox on 2020, kinda wrap it up and, you know, in cellophane, put it into a bucket of cement and drop it in the river, you know? [Laughter] Kinda, let’s get past this year and move on to 2021.
So, as you think forward in getting security right, we’re in this COVID era, right? And, you know, that light switch isn’t gonna turn off any day soon. We’re gonna be some evolution of this for, you know, whether there’s vaccines or not, right? But that has rapidly, I mean, seriously accelerated digital transformation projects, and changed some things fundamentally about assumptions we make. We’ve kinda, we can go to shed some of that evolution and we’re there, in many cases.
How has that affected your thinking around data security in this era? Because we’re in it, we’re not heading towards being transformed, we’re living it right now.
Bennatan: Yeah. You know, I think this is actually one of the few positive side effects of COVID, you know? I can’t tell you how sick I am of being here. I need to go somewhere. I really need to go somewhere, I’m like, sick, but—
Ashley: I know. What day is it, anyway? I can’t figure it out.
Bennatan: Yeah. I just, I just can’t even tell you. [Laughter] But the acceleration that this has given to cloud projects is one of the side effects that I think is very positive. Because, for years, you know, I’m a technology guy, and you know, if it were up to me, everything would be on cloud, bar none. Nothing, nothing—there would be scorched earth everything else. And, you know, I keep hearing people saying, “Yeah, but you know, it’s more expensive.” You know, it’s more expensive only if you look at it at the actual numbers of the cost you pay the cloud providers. But if you look at how much more effective it is and how much easier, it’s like buying super-duper IT, okay?
Ashley: Mm-hmm.
Bennatan: That’s really what it’s about. It’s not about anything else, it’s about finding people who know exactly what they’re doing and that you never get to a point that you always have when you build your own stuff where you’ve got people finger pointing, you know? “No, it’s a storage problem,” “No, it’s a host problem”—it just works, okay?
So, I think it’s great. I think it does create a lot of challenges or security, just because it’s very new, okay? It’s very new and it’s very, very fast. And new doesn’t mean it’s, like, six months old. But even if it’s six years old, it’s all new. It’s like, somebody built everything that anybody built around data centers from scratch, really, okay? Or it looks different or it’s called different.
And then the problem—again, going back to people, the problem is always people. So—so, a lot of companies, what they’ve done in order to accelerate this very, very quickly is, they’ve built a separate security architecture for the cloud group to parallel the cloud architecture group, which is different from the old guys, okay?
So, now you get people who understand data security pretty well, because they’ve done it for many, many years. You have people who understand data very well. And then you have people who understand cloud very well, but they’re not the same people, okay?
Ashley: Mm-hmm.
Bennatan: So, now, you get a skills issue and almost like a language issue, okay? It’s like Babel again in some respects. It’s like, how do you call this? What is this? Is it the same? You know, what’s a VPC, how many different ways do you have to connect things into a VPC? What does that impact my data architecture, and what are the flows?
And so, I do look at 2021 as a great, great year, because, you know, if people have been going at a certain pace into the cloud, it’s all accelerated. It’s—
Ashley: You know, I think that is a really important point, because when you’ve been disrupted—and we’ve all been disrupted in our personal lives, but when you accelerate something that fast, you know, the saying, “necessity is the mother of invention”? Well, necessity with urgency is, like, the mother of invention right now.
Bennatan: Yeah.
Ashley: And it’s almost like you can’t do things the way you’ve been doing, you can’t go along in parallel paths, because suddenly we’re all now here. We’re all in the cloud together or in a place that’s much different than we were.
Do you think that that is enough of an accelerant to get sort of those silos of people across cloud security and on prem security and those start to work together of thinking how we manage? Because so much more of it now has moved to the cloud—or is that a good assumption? I mean, tell me what your thoughts are.
Bennatan: Eh, you know, it’s—because it’s people—
Ashley: [Laughter]
Bennatan: – [Laughter] I, I don’t know how to answer that, you know? We’re getting into the realms of psychology more than anything. But I can tell you that, you know, it’s gotta happen, and we’re gonna make mistakes, right? As an industry, we’re gonna do this, it’s good that we’re doing it. And we’re gonna make more mistakes, because it’s less known, because we don’t always have the right skills.
Ashley: Mm-hmm.
Bennatan: And I think it’s our job, you know, our job in the industry is to try to reduce those mistakes. Because mistakes lead to things that we don’t want to happen, and we do have the ability to avoid mistakes, okay? We just need to be very clear—we can’t be, like, super techies about it, okay? We have to do things that are practical. Because what you asked is about human behavior. So, really, it’s not enough we create technology, we need to create technology that is accessible to people and that people can consume very easily.
It’s one of the things that, kind of, in security, I think we never—I think we’re doing a much better job now, and my entire focus is really on this, on making things practical and usable very often more important than a certain feature function, okay?
Ashley: Mm-hmm, mm-hmm. I totally agree with that, yeah. Well, you know, what I was thinking about is that, when you have that disruption, think about choosing productivity tools. How many organizations were debating, “We’re gonna go to Teams, we’re gonna go to Slack, let’s study it for a year” and it turns into two years and you final make a decision. All of a sudden, that decision gets made in two weeks or a week or a day—a couple days.
Bennatan: Yes.
Ashley: Like, how could—you just have to. You’re forced to make some decisions and make some changes.
What I’m wondering is, if you accelerate to the cloud and you have, you know, a framework, an architecture, something like an Imperva that—okay, we’ve solved some of those issues, let’s just solve them for other folks and help each other save some time, right? You don’t have to—sure, check me out, validate it, you think this is all the right thing to do. But if you’ve got to do something quick, we’ve got a good bit of that problem solved. It seems like you’re in a good position from that standpoint.
Bennatan: Yeah, and I do believe so. I think that we started very early with the cloud and how do you do this well, and so we are in a good position.
I also really [Laughter]—you know, you said something about this productivity tools making the decision in two weeks, and it’s one of my, you know, one of my pet peeves for my entire life has been something—I don’t see it changing, by the way, yet, but I really wish it would change, okay? Is this concept of a POC, okay? Where you—you know, you need something, like, go back to your two years versus two weeks. What does that mean? It means, somebody picked without doing a POC, right? [Laughter]
Ashley: Make a bet and if it’s wrong, we’ll change it.
Bennatan: Okay, make a bet, and make a bet based on what? You made a bet based on, you know—
Ashley: Best [Cross talk].
Bennatan: – either something that’s more usable or very usable, [Cross talk] or something that everybody else is using.
Ashley: [Cross talk] at the result. I think that’s what it comes down to, right? What is gonna get us the quickest understanding? Nothing’s gonna be perfect, we’re gonna have issues with whatever we choose already, of course.
Bennatan: Yeah, yeah, and this entire industry forever has been, “Okay, I can’t pick anything until I do a POC, but when I do a POC, I’m actually not even checking what’s important, because I can’t reproduce my production environment. I can’t reproduce the load, so I’m just gonna test something, okay? I’ll test, like, really simple things—can you do this, can you do that?”—which has absolutely no bearing on whether you’ll succeed or not.
And what I’m starting to see, especially on the cloud, because it’s much easier to just look at things, much easier to talk to people, much easier to see what your colleagues are doing, what your peers are doing is, I’m hoping that that accelerates things. Because if we need to do something tomorrow, we’re just not gonna wait. We’re not gonna wait nine months.
Ashley: So, here’s what might start to change that behavior. I’m not gonna proclaim that it’s changed that behavior, but through this COVID experience, using that productivity tool example—guess what? Executives, the non-techies of the company, look at us and say, “How come we can make a decision in two days when we were gonna take two years to do it? If we can make that decision in two days, I want us to make the other decisions quickly. Maybe it’s not always two days, but we’ve proven we can make some good decisions under some very difficult conditions and rapidly. Let’s exercise and build up that muscle and do that more often than the thing that we debate about forever.”
Bennatan: Yeah.
Ashley: So, I think there may be some pressure from the business, who’s also, by the way, very disruptive, they’re having to adjust, they’re having to go into defense or offensive mode in the business climate, and they want to be able to experiment very quickly—that’s part of what digital transformation is about, right? Try some things, experiment, and market, go after—you know, learn quickly, much like we do in kinda continuous improvement.
So, that might be the thing that nudges us off of the comfortable stool of, “Let’s take nine months-slash-two years to study it and make a decision or run it—you know, do an RFP for everything.” I hate RFPs, but—
Bennatan: Yeah.
Ashley: – just like POCs for you. [Laughter]
Bennatan: Yeah.
Ashley: But maybe that’s the thing that tilts it a little bit. I mean, I’m not gonna declare success yet, but…
Bennatan: I agree, I agree. I think it’s—you know, I’ve been in startups all of my life, okay? So, one of the things that is really fundamental when you look at a VC that’s investing in a company, they’re investing in the people, not in the company.
Ashley: Exactly.
Bennatan: I mean, this technology, they don’t know yet, okay?
Ashley: They know you don’t have it figured out, otherwise somebody else would’ve done it.
Bennatan: Yeah, yeah.
Ashley: Right? So, your job is to figure it out, but get the right people that can do it.
Bennatan: But they invest a lot of money very quickly, okay? They don’t take nine months, right? So, I believe that, you know, when people look at their partners, who they’re gonna partner with, they need to look at, you know, what are the founding principles of the company? How much is the company gonna care about me? What is it gonna do for me? Are they gonna be there when I need them?
And I’m not saying technology’s not important, because product is everything, but it’s product and the people that stand behind it, and the decisions need to be made, also, from the business level.
Ashley: So, we’re kinda running up against our time, here. I’m gonna point out this irony and hopefully you’ll come back and talk about it. So, you were just talking about investing in people, which is what VCs do, I totally agree with that. And earlier we were talking about—well, the problem we were talking about is human behavior and people, right? So, [Laughter] there are conditions of which will drive different behaviors, and hopefully that’s something we can explore about data and data security some more. We’ll have you back again soon.
Bennatan: Yeah. Okay, I’d love that.
Ashley: Good. Well, Ron, it’s been a pleasure. I wish you all the best. Congratulations on the acquisition and hopefully, you’ll be able to get out of your four brick wall cell there at some point and get out and enjoy the world again. [Laughter]
Bennatan: [Laughter]
Ashley: All of us will, but I wish you the best. It’s been great talking with you.
Bennatan: Hopefully. Thanks, Mitch.
Ashley: Great. Thanks for joining us today, folks. We’ll talk to you soon.
-
Integrating Security in the Development Process With DevSecOps
Occasionally, it’s worthwhile to reflect on how we develop and deliver software during times of rapid change and significant disruption. In those moments of reflection, we learn from the exciting trends and changes that are taking place. Recently I had the opportunity to commiserate with two technical thought leaders at Perforce: Stuart Foster, product manager, and Steve Howard, static code analysis evangelist.
The disruption caused by COVID-19 goes much deeper than just the rapid shift to work-from-home. Development teams are being asked to keep up developer velocity, while the organization as a whole accelerates digital transformation projects, shifting priorities and emphasizing investments needed to successfully navigate difficult and uncertain economic and market conditions.
In our conversation, we note how investments already made in shifting to the cloud, remote and cloud-based development, automated testing and pipeline automation and security tools are helping organizations today. We also explore how current conditions influence product roadmaps, causing software teams to take a holistic look at their development processes and tools, and organizations to rethink how to take their software delivery capability to the next level.
Stuart and Steve share their software best practices on collaboration, automation, security, standardization and more.
Explore our conversation in more depth by viewing the video and transcript below.
Transcript
Mitch Ashley: Well, I’m happy to be joined by Stuart Foster, who’s product manager with Perforce, and Steve Howard, static code analysis evangelist, and I think a lot of other things, too—does a lot with Perforce. So, welcome, gentlemen—good to be talking to you today.
Stuart Foster: Thanks, Mitch.
Steve Howard: Hello.
Ashley: You know, we’re talking about—you know, Perforce does a lot of things, and I think we’re just kind of focusing on what maybe some of the current conditions, current best practices are. There’s a lot happening right now in everybody’s world—a lot of change, a lot of disruption.
But before we get to that, could I have you both introduce yourselves? Maybe Stuart, would you start first?
Foster: Yeah. Hi, everybody, I’m Stuart Foster, I’m Perforce’s product manager for our static code analysis products.
Ashley: Great—Steve?
Howard: Yeah, and I’m—well, I have, as you say, many jobs and roles within the organization, but primarily focusing on the evangelism around using static code analysis for both of our products, Klocwork and QAC, here at Perforce, yeah.
Ashley: If you’re working with developers, you gotta be doing evangelism, right?
Howard: Yeah. [Laughter]
Ashley: They expect you to give them support, and be talking to them, in a different kind of way. Tell us—would one of you tell us a little bit about Perforce and what all you all do?
Howard: Yeah, sure—well, I’ll take that. So, Perforce is an organization, obviously as, you know, originally comes out of the version control software. But really, it’s about developer tools and how we can help make the developer’s life easier in the day to day tasks. So, whether that be trying to prove compliance to standards or just trying to get the stuff out the door in the first place—anything that we can do to help make that process smooth and reliable is kind of where we are. And, you know, these days, that has a bit of a—a buzzword, if you will, around the DevOps tooling ideas, but essentially, it’s developer tools and making their life more comfortable.
Ashley: That’s great, because you all fit into a number of places in the sort of tool chain, pipeline, workflow process across the development life cycle.
Howard: That’s right, yeah.
Ashley: So, that’s awesome. Well, let’s get into this. I’m really curious about what you’re finding in market, and one of the things we like to do when I’m talking with experts like yourself is, let me just try to share some of the things that we’ve experienced within the past, but also kinda recently that it’s happening, because we’re all dealing with economic change, disruption, COVID, work from home, all of those kind of things, but you know, we’ve done some research at my company and there’s a lot—of course, a great, great emphasis on acceleration of digital strategies and, of course, that puts a lot of pressure and emphasis and people are betting money on software being the answer to helping them implement their strategies.
Are you seeing the same kind of thing, that acceleration of digital transformation and software—elevation of software projects?
Foster: Yeah. We’re seeing that across the board, even in our own company here. You know, a lot of us have transitioned to work from home, and then you have to start thinking about the considerations for that, right?
And what we’re finding ourselves and working with our customers is, a lot of people are shifting more technological tools, right, to be able to better produce workflows and code reviews, how to—working from home, security, right? How secure are your remote development work stations? How is security being built into your software and your workflow? So, tools are being used across the board for that. Just being able to rely on a piece of software versus in-office workflow has become more and more important. And amongst all that, then there’s the goal to, you know, keep development velocity up, which could be a challenge when working from home, right? So, we’re seeing a lot more DevOps automation of, you know , a lot of workflow process and things like that.
Ashley: Yeah, there’s no more popping over the edge of the cube or to the table next to you, right? Somebody sitting there at the café, in the office, or down the street. It’s a fully digital world, right? You’re not sitting with people very often.
Foster: Yeah, we’re relying on Slack, we’re relying on Slack calls, Zoom videos, everything to just stay in touch and work and collaborate with one another, yeah.
Ashley: Steve—yeah.
Howard: But I think that’s where—yeah, so, I was gonna say, I think that’s where those centralized processes that we’ve kind of all been working towards, luckily, for the last few years, are really gonna pay off. Because how do we not have that? If you imagine how things were maybe five years ago, it would’ve been a much bigger transition to get to where we are now, whereas it was almost like most of our organization was kind of ready to go. We were already there, just—you could call foresightful, but really, it was a little bit of chance, I think. Yeah, those processes have held things together remarkably well, and we’re very lucky for it.
Ashley: You know, nobody would wish for something like COVID or a pandemic on any of us, but if you look back, you know, I think back when we were virtualizing things and developers were starting to be able to just create a fully self-contained environment to develop on their laptop—I saw people working on planes, right? First time I saw somebody writing code on a plane—that’s awesome.
But so many things, and then of course, adding the cloud and all the services and software and tool chains and pipelines that we can create now, it—I mean, I think we’re lucky, if you were gonna have something like this happen, now is the time where we can actually continue to be productive, and put more emphasis or pressure on ourselves to deliver software. Do you agree?
Howard: Yeah.
Foster: Yeah, absolutely.
Howard: Yeah—sorry, Stuart.
Foster: [Laughter] We agreed at the same time, that was great.
Ashley: [Laughter]
Foster: But you’re right, I think, like, some of it—you know, it’s a little bit accidental, but as you mentioned, setting up the development workflow to be able to do it on a laptop, people are shifting their hardware, they’re removing their hardware and going to the cloud, right? So, it’s kind of been like an expansion and a reduction, all—just naturally, but also, you know, you think about DevOps and there’s also, you know, cost savings to be had by not having to have this footprints, not having to worry about these things.
So, this could be, in some ways, pretty revolutionary or changing for development costs for a lot of companies.
Ashley: Steve, you were gonna jump in there—anything to add?
Howard: Yeah, no, exactly the same. I think the fact is that we—you know, a lot of that infrastructure is being built up and the idea of being able to run our development pipelines kind of in a cloud environment or whatever and having that all ready to go just meant that we didn’t have to pick up ginormous work stations. I remember back in my days of developing in aviation software, we used to literally have, you know, Unix boxes stacked up on the side somewhere that we had to go and log in on each one and run our testing for each single one. It’s just unthinkable. It would never have been possible.
Ashley: That would be insane today. You would never. I mean—
Howard: Yeah.
Ashley: – if you have a choice, you would never do that, right? [Laughter]
Howard: Well, exactly, and the very fact that, you know, 10 years ago, we still had to do that, how could we have transitioned? It wouldn’t have been possible. And so, it really is testament to how far we’ve really come in 10, 15 years to see where we are with our testing processes, the fact that we can do, you know, 99 percent of it, if not all of it can be done completely with no connection to the physical space.
Ashley: Mm-hmm.
Howard: It’s all—and that can even mean, you know, in the cloud completely, so it’s not even within our organization. It’s securely being done off site somewhere in the ether. [Laughter]
Ashley: Yeah.
Howard: And that process is still completely manageable, it’s something we can all see as developers, everyone can get to see where we currently are with that process, you know, what the latest test run gave us across the whole—
Ashley: I’m really curious—yeah, I’m really curious what your experience in market is. Do you see more people who are kinda doubling down and getting even more serious about their DevOps implementation and taking it to the next level, or do you see more people coming to you that maybe are relatively new to DevOps and are looking for tooling and looking for ways of helping them, you know, not just get started and learn all the problems themselves, experience them themselves, but looking from the experience of a software provider like yourself, or is it a little bit of both?
Howard: I’d say a little bit of both. I mean, obviously initially, there was such a lot to do already and everyone kind of had no time to really think—understandably, but probably within, you know, since the end of the summer, maybe, things have started to come back and people are actually going, “Well, hang on a minute. We’ve got a process, but now, how can we actually make this a lot easier for ourselves?” and looking to improve that.
And I think, you know, for a lot of the organizations where, perhaps, doing things off site would never have even been considered. And I think this was to Stuart’s point—that kind of, that’s been forced upon people and it’s almost gonna change the way we develop forever now as a result of that, because we have now, we’ve done it, we’ve experienced that it’s possible, and it’s safe and it’s secure.
How can we now make that better and make our lives easier by not having that massive footprint that Stuart was mentioning? We’ve actually got something that’s very, very lean, it’s in the cloud, we’ve got machines when we need them, but not when we don’t need them, and how can we really make that process as efficient as possible from the results side, you know, making sure we’re doing what we’re supposed to be doing as quickly as possible, but also actually trying to reduce our footprint on process and power, for example? You know, the development time that’s involved in that, how can we make that as efficient as possible?
Foster: Yeah, and I also think what we’re seeing, too, is, you know, our existing customers, they’re doubling down on their tools, you know, because they need to rely on something, right? And that’s also, what it’s done is also, I think, is changing some of the roadmap items that all products are looking at, right? There’s a whole new—not a whole new paradigm, but you know, there’s a little bit more seriousness being taken in terms of some of these features and functionalities of being able to work remote, or the requirements for more cloud usage, right? We’re talking ease of use, we’re talking about things that are, you know, related to workflows to make working in these conditions—not just in these conditions. Assuming you get back to the office, just more efficient as well.
Ashley: Mm-hmm.
Foster: So, we’re seeing that at the same time, too. And then, you know, from the perspective of new customers coming in, they’re almost expecting these things to already be there, right? Because they’ve run up to these problems as COVID has created and they’re trying to find the solutions that are gonna be kinda turnkey out of the box.
And also, I’ve done, I’ve been talking with various analysts related to our product, but it seems as though the analyst community itself is, they’re seeing the requirement for, you know, ease of use in DevOps becoming more important as well at the same time.
Ashley: Mm-hmm. I think that may be also, in part, the expansion of DevOps into the organization. They may have had a core group of teams that were really taking the challenge of DevOps and implementing it and now, how does the rest of the organization do the same thing, but accelerate their adoption of it?
Foster: Yeah, and I think also, a lot of development groups within a single company—you know, you may have a security team, you may have a development team, you may have a LiveOps team—they’re starting to recognize that there are choke points in the development processes.
Ashley: Mm-hmm.
Foster: And so, how do you unblock that? How do you make it more easy? How do you offload some of that choke point onto a broader group, right? A lot of the idea of DevOps, DevSecOps, whatever you wanna call it, right, is that developers are owning the code that they produce in more broad ways. That’s writing unit tests, that’s making sure they do the QA on it, right? That’s when they’re, you know, they have to implement the security requirements then and there, right? They need to ensure the traceability is done, right? They have to—they’re taking that into their hands. It’s becoming not so much that there’s a DevOps group that magically handles all this infrastructure, it’s becoming part of each developer’s responsibility as well, right?
Ashley: Hmm. Interesting. You know, I know, as you scale things out and you adopt new tools, tools do a lot of things for you, but there’s oftentimes, there’s approaches that you need to adapt into your tools, things like what are our coding standards, what are our testing, what level of testing do we do? How do we break it into chunks? When do we do security testing of our code?
And that’s part of that learning curve to is, how do you adapt a tool set to both what your requirements are, but also, you would like it to kinda take you to the next level. What are some of the things that really can help people kind of propel their expertise in the organization’s ability to deliver software more effectively?
Howard: Yeah, that’s a really good point. So, one of the things, you mentioned coding standards there, but obviously, you know, those can come from multiple sources, they can be something that’s actually being required by a standard, for example, if it’s in a safety critical, security critical world. It could be something that’s actually demanded by the organization itself—they have their own rules or there’s certain things in any particular project that, if you do things in a certain way would make, you know, it would be dangerous or insecure or something. So, they have some controls, as well, of that.
And what we do see is, the more that we can get that information out to our developers, for example, the more value that has as well. So, yes, we can help enforce coding standards, for example, with our Static Analysis tools. But that’s only half the story. Enforcement is one thing; actually getting the developers to understand what the problems are as they’re creating them and to have easy access to the help and the quick feedback of those problems makes it much more powerful, because it gives them, (a) you know, all the resources they need to deal with the problem quickly and to not perhaps find it a challenge to have to deal with, but also, they learn.
And so, the next piece of code they write doesn’t generally have the problem. So, there’s a developer learning element there as well, which we can control centrally, and then spread out to all of these remote workers working—you know, without that team collaboration that we used to have, we’re still able to do the same things. So, that’s become something that’s really focused people’s minds on the capability there in terms of what that means. And as I say, that applies to security standards that are well known or requirements for quality that we have to work to for a security or safety standard—
Ashley: Mm-hmm.
Howard: – but also the company’s own rules.
Foster: Yeah, and I think, you know, outside of just simply the tools, development teams are, you know, taking a step back and re-evaluating or thinking more about what their software development process is, holistically, right? You know, Steve mentioned there some of the tools, best practices, but you know, you also have to think what kind of, you know, software development methodologies might you be using. You know, sometimes it doesn’t matter whether it’s agile, waterfall, whether you’re using Scrum or Kanban, you know, I think it’s giving people a moment to think about what’s the most important kind of Dev process that is gonna fit this team or how you have to work now, how it fits the businesses.
And then implementing processes to ensure the organization, communication, and what systems need to be put into place to help prevent any problems that arise during development and kinda identifying and understanding that. Then you’re picking the tools, you’re building your development workflow, whether it’s in environments, it’s setting up DevOps pipelines to ensure that everything’s followed correctly. It’s the tools, but it’s the process, and that’s changed, too.
Ashley: That’s interesting. You know, we’ve done some research about the influence that developers have had on the organization, too. Even outside of development teams, you’re seeing DevOps and agile kind of principles, even though they aren’t calling it that—daily standup meetings, shorter intervals of deliverables, more kinda asynchronous communication tools and collaboration kind of tools, which really makes it easier, I think, for the development teams, the product teams—you’re a product manager yourself, Stuart.
Foster: [Laughter]
Ashley: And beyond—operations, security—everyone is kind of forced to work a little bit differently, but also can benefit from what each other knows about how to do that more effectively. And we seem like we’re at this sort of serendipitous time of being able to leverage that, which fits right into the tooling approaches that we can take.
Foster: Yeah, and you know, when you look at it, like, it always just ends up communication. Whatever the communication is, whether that’s the workflow, whether that’s in-person, whether that’s meeting, whether that’s documentation, it’s all about communication, and that’s true not for just development, but for business, right? Developers working, you know, with other groups—marketing, sales—everything, it’s all communication. It just keeps business running. And that’s kinda the funny kinda perspective that may not necessarily be outright visible by simply saying, “It’s communication.”
Howard: What’s interesting is, that’s also kinda mirrored in the tooling, because a lot of the work we’ve been doing recently is about how to make sure the tools have all got open interfaces to communicate with the next thing in the pipeline or to feed that data back up to some dashboard that’s gonna be shared with the rest of the team.
So, yeah, it’s kind of an interesting point, but that also applies with the tooling that we’re trying to build around that.
Foster: Mm-hmm.
Ashley: Mm-hmm. A really great point. So, to give you fair warning, I’m gonna ask you your three recommendations on best practices in the current climate, so be thinking about that.
I was thinking about your comment, Steve, about being able to better integrate tools together and then taking the data from the tool sets, the pipeline of the process that we’re going through and the pipeline of work—there’s so much value in that data that sometimes ends up as exhausts or on the cutting room floor, right, to use two different metaphors. But that information could be extremely valuable, putting it into dashboards, feeding it into continuous improvement processes, shortening the window for getting feedback for, “Here’s a security vulnerability or a coding standard” or some kind of an error that you can fix in minutes instead of days and weeks and sometimes months, right? The security audit at the end of the release is much easier, because you’ve already gone through so much to it. Those can all be things that can help us.
So, I gave you a little bit of time there to think about your three best practices. I did my best job of stalling that I could. So, who would like to go first? What would you recommend people consider today?
Foster: Yeah, I’ll go. Just, I want to reinforce that message of, you know, figuring out how you’re going to work as a development team, develop that communication, right? And that’s using your tools.
I think in the world of remote work from home, having kind of a common language—again, back to communication—of how, you know, each developer’s system is set up. So, common environments, they should be using as close to the same setups as possible so that, you know, when one of you is running into an issue, it’s easier to get that information across, right? You’re not gonna have to install something obscure because somebody’s using something obscure. You should all be trying to talk and use the same tools so you’re all speaking the same language. That allows you to have that velocity that you could otherwise lose.
And in this world, too, security hasn’t been more important than ever—security is more important than ever. So, think about securing your Dev processes, developing secure software, it’s gonna be very important. And those are kinda my three.
Ashley: Okay, great. All good ones. Steve?
Howard: Yeah, I’d answer—so, my three would be, first of all, to automate; take the process away from the day to day effort of the development teams and organizations around them. So, yeah, that’s the first one.
Report—so, that’s the communication piece, I guess, but get the information to the right people at the right time and hopefully in the right sort of time window.
And then shift left. So, then, we make it more efficient. Then we look to where we can try to remove the fat in the process and take away the time that was wasted and try and move it to the right position in that process. So, yeah, those three, in that order. [Laughter]
Ashley: Great. Excellent, excellent. I’m curious what you both think will be some of the things that will stay with us, the things that’ll last past a forced work at home, forced work remote. Maybe people are returning to the office, maybe they aren’t. What do you think are some of the things that will stick with us? Because it wasn’t like we were doing this for two weeks and then we’d just go back to the office—this is an extended period, we’re developing new habits, we’re changing how we’re working. Why go back, sometimes, right? What are your thoughts?
Foster: I think you’re gonna see a lot of these workflows stay around. I think that the reliance on the numerous tools to keep velocity up and reduce and automate are just gonna stick around. You know, you don’t want to be wasting time on trying to get something to work. It’s easier to just simply automate it and get it out of your hands so that, you know, at least from a development perspective, you can just focus on what’s important, which is, you know, coding, the feature, getting the product out the door, right?
Those are gonna stick around, and I think the tools, over the course of time, are just gonna keep reinforcing and strengthening that kind of automation pipeline.
Ashley: Good.
Howard: Yeah, no, and I think, from my perspective, just to add the one small piece there is, the cloudification, essentially, of the processes. So, I think, you know, people will stop having the premises to host great, big server racks in, and that’s just all gonna move off site, off premise, outside of the organization, for sure, yeah.
Ashley: Yeah. To kinda add what you both have said, which I think supports your thoughts on this is—you know, as we change to whatever we’re gonna go back to, I don’t think it’s gonna be back to what we were doing. But it’s gonna be a hybrid, right? I can’t imagine everyone shifting back to a different way. It’s—some people are still, maybe even more percentage of our teams are working remotely, never going back into an office, but we may have some that do.
We’ve broadened the reach across the globe of people that we’re working with, the product and vendor, tool vendors that we’re working with. Those things aren’t gonna change, right?
Howard: No. No.
Ashley: We’re gonna stay doing that. If we’ve found something that works better, you’re gonna stick with it, right? So, you’re gonna stick with—you’re gonna stay in the cloud, push more in the cloud, right? And leverage some of the automation, the processes that you talked about, Stuart.
Well, this has been really fascinating. Would love to have you both back, have you guys back and we can talk some more about—I’m gonna delve into static code analysis evangelist, your role, Steve, get into some of that, which I know you all have some of your products do and to security and to some other areas within the software pipeline, but this has been really great. I’ve really enjoyed talking with you both.
Howard: Yeah, it’s been a pleasure.
Foster: Great. Thanks, Mitch.
Howard: Bye bye.
Ashley: You bet. We’ll see you soon.
-
DevOps Chat: ITIL 4 and Achieving High Velocity IT
In this DevOps Chat, Mark Smalley of Smalley.IT, and Lead Editor with AXELOS of the ITIL 4 High Velocity IT joins ASG CEO Mitch Ashley to discuss his work creating this new contribution to ITIL 4. Mark shares the 5 value investments important for organizations seeking to operate at high velocity and some of the insights gained from himself and others covered in this ITIL 4 module.
As usual, the streaming audio is immediately below, followed by the transcript of our conversation.
Transcript
Mitch Ashley: Well, I have the pleasure today of being joined by Mark Smalley. Mark is Lead Editor of High Velocity IT, part of the ITIL 4 initiative. Mark, welcome. He’s also with Smalley.IT, I believe a one-person firm. Welcome, Mark. Would you please introduce yourself, tell us a little bit about you and the kinda work that you do?
Mark Smalley: Sure, Mitch—thank you, thank you. Yeah, Smalley.IT, I’ve been on my own for about eight years now; previously, around 30 years at Capgemini and other managed application service providers. So, very much on the application side, very much on the service provision side. That’s really my background. The personal background—born in London, but have lived in the Netherlands, just outside Amsterdam, for about 45 years now, which is not a bad place to be.
Ashley: No, not bad at all.
Smalley: One of the recent—yeah, one of the recent initiatives I was involved with was the AXELOS ITIL 4 update, responsible for the emergence, I say deliberately the emergence of the high velocity IT module, because it was quite an interesting process, pulling in knowledge from various areas in IT service management, but also agile DevOps and out of the IT space completely. For instance, some people may have come across The Toyota Way, the management book about the Toyota manufacturing process. Professor Jeff Liker at Michigan University, he kindly contributed a bit to the book about the Toyota Kata, which is about an improvement method based on scientific thinking. So, drew in people from various fields to try and get this thing off the ground.
Ashley: That’s gotta make it really fun and engaging, too. Why don’t we start out with what do we mean by it? What’s a good definition of high velocity IT? How would you give kind of a working definition of that?
Smalley: I’d first qualify it by saying it’s not actually high velocity IT, but it’s more about higher velocity IT, because high would imply that there’s low and you say, you know, “High is good, low is bad”—well, that’s not the case. We’re on a gradient, on a spectrum from lower to higher velocity. We’re looking at the kind of organizations that you’d say would be more digitally enabled than most.
So, the kind of organizations that use technology to do things significantly differently or even do significantly different things because technology enables them to do that. And that requires a different way of thinking, a different way of working than many people are used to. So, the book’s about different ways of thinking, different ways of working, and primarily the people in the IT service management space who are now confronted with this digital stuff and how do we deal with that? And trying to get them—it’s sort of a guided tour through the kind of things that you come across in these more highly digitally enabled organizations to encourage them to unlearn the old ways of working that aren’t effective any more and integrate new ways of working, new approaches in how they do stuff.
Ashley: I think one of the really important and kind of fascinating parts of your work on this, Mark, is it’s not just about the bubble of IT or the bubble of my part of, my role in IT, it’s that high velocity for the whole organization and how you all work together. And I know it really—there are many places to start, but clearly, product management, there’s some outcome that we’re doing all of this work for, some product or service or offering that we have, and that’s a great place to really engage in kind of a systems thinking about this.
Smalley: Yeah, very much so. It’s taking the end to end perspective. We’ve defined five objectives to give us a bit of guidance in this space, talking firstly about valuable investments—are you doing the right thing in the first place? Are you automating the right kind of business processes? Are you building digital products and services that are appealing to the marketplace? Are you doing the right thing? Fast development—are you doing it quickly? Resilient operations—once it’s up and running, is it resilient? And deliberately using the word resilient rather than reliable, because the kind of systems we’re dealing with are often complex and therefore unpredictable in their nature. They will fail.
Ashley: Mm-hmm.
Smalley: You know, these are fail-safe systems, these are systems that should be safe to fail, so it’s about the resilience—how quickly can you get the systems back up and running? Then the fourth objective is co-created value. Then again, you have that broader chain; that’s when IT starts engaging again, with the user community, in that intricate and even intimate dance that they do together that’s, you know, to help the users derive value from the investment. And finally, the fifth overarching or underlying objective is all of the above—assured conformance, it should be able to assure conformance to governance risk and compliance requirements, whatever they are in your kind of organization.
So, it certainly is, what you said, the big picture, end to end.
Ashley: Mm-hmm, very much makes sense. You know, it’s interesting—tell me if this jives with what you’ve experienced. Highly reliability, the best way to do it is don’t change anything, right? Make it very stable. But we live in a world where business needs to execute quickly, but also dynamically to react to changing market conditions—you know, pandemic being a great example of that.
Smalley: Yeah, yeah, yeah.
Ashley: A tragic one, but very much one that’s outside of our control—but also, you know, offensive strategies to go after markets and that, and the degree that the whole chain, the whole kind of supply chain of the organization working together can effectively do that quickly and reliably, but also resiliently, because you want to be able to experiment and try things. And maybe you need to fail to find the right answer for product mix or whatever.
So, I’m sorry, I don’t wanna go on too long, but it seems like that’s an underlying driver behind all of why you took this approach, at least in part.
Smalley: Yeah. Well, certainly, if you look at the dominant thinking behind most of the guidance there’s something behind it. There’s that concept of co-creation, realizing that you don’t do stuff and chuck it over the wall, you have to engage with other stakeholders—one of the things.
The other main thing is dealing with, working in complex environments. Again, using that word complexity in the sense of complex adaptive systems, organizational systems where you simply can’t predict what’s gonna happen and where you, because of the volatility, the unpredictability of stuff, you can say that’s bad but from a research and development perspective, you want to be able to, you know, 10 initiatives will fail, but it’s that 10th one that’s gonna make you millions. You’re gonna benefit from that unpredictability.
So, it’s really, it’s from the start, through the whole life cycle that you’re dealing with the—with the unpredictability which, for many people in IT service management, is a bit disconcerting, because if you go back—go way back, and you look at where people came from, IT service management people were very much concerned with the stability of systems and putting a wall around them with the best of intentions, with the belief system safe is good, whereas the developers had a different perspective—fast is good. So, you’ve got this fast versus safe. And seemingly, irresolvable, that conflict.
But if you look at it creatively—and this is something that we’ve got a lot to credit the DevOps community for, and we reference extensively stuff that’s gone on in the DevOps world in the High Velocity IT book. The idea that if you—and it goes back to lean as well, lean in particular, agile a little bit—working with small batches of work, if you do work in small batches and more frequently, then you reduce the risk, ________ smaller changes and less risk in bigger changes. And the interesting thing comes along now—if you do changes more frequently, instead of saving them up for each, you know, masses of changes each quarter, no, do them every day, then you develop your capability to change effectively.
So, paradoxically, you’ve turned it around completely. The more often you change—small changes—the better your change capability and therefore the more confident you can be that the changes will go through pretty safely.
Ashley: Mm-hmm.
Smalley: And it’s interesting, the groups that have to interact together, like, if you’ve got a DevOps group or a classic IT service management group, they identify with their own belief system, and often, the belief system is, you know, safe is fast—sorry, safe is good as opposed to fast is good. And how do you—you know, you’ve got to get these people together and talk about this stuff to resolve the apparent conflict. But it’s great stuff trying to build bridges between these communities.
Ashley: Yeah, I’m totally with you and thinking of the sort of lean mindset and agile and iterative, which is very much where software product and organizations have moved to or are moving there. What are the—how do you deal with some of the legacy silos and sort of that self-protection of don’t change things or don’t, you know, let’s focus on keeping things stable or whatever the mantra of those individual groups are? How can this book, for example, this module help people start to expand their thinking around those issues?
Smalley: Well, firstly, if you look at organizations, they will vary in velocity across the various divisions and departments. So, firstly, you’ve got to take a look at which part of the organization are we looking at? What is the appropriate velocity in the business there? What kind of cadence applies to their work? Interesting looking at the pharma industry that’s traditionally been highly regulated with very long periods in which they work. There’s no point speeding things up if you have to wait years before you move to the next stage. It’s like doing the Olympic games—you don’t want the stadium there earlier; there’s no point. So, it’s got to be very much aligned with the business velocity.
So, look at the specific parts of the business and, depending on the kind of speed—and possibly, you’re dealing with legacy systems that don’t need to go that much quicker. And the advice we like to give in the guidance is, the first couple of statements, the guidance is not complete, and the guidance is not intended to be prescriptive. We don’t pretend to be able to dictate, proscribe how people should act. Rather, we’re trying to encourage them to take a look at the 9 concepts and models that we’ve adopted, the 25 techniques to see which techniques do you think would apply to you. So, be selective, pick and mix, take a few things out, experiment with them, see if it works in your environment.
Because this is going back to lean, you mentioned lean, going back to the lean movement, you’ll probably know many of our listeners, audience will know that the Toyota production system was the precursor to the lean movement. And one of the grand figures in the TPS Toyota Production System, was Taiichi Ohno, who came up with some wonderful sayings, one of which is, “You should think for yourself and face your own difficulties. Don’t try to borrow wisdom.” So, in other words, how another organization has done something, that’s not gonna work for you. You have a different kind of organization, so you really do have to dig deep and look at the specifics of your situation and pick the various parts that you think apply to you and just experiment and see whether they work. So, it’s—
Ashley: Really more of an adaptive approach versus prescriptive, right? It has to be done a certain way because one organization succeeded at it that way.
Smalley: Yeah, no, and of course, it’s very tempting, but people are looking for the quick fix. There’s a fabulous cartoon, if anybody wants to look up a great cartoon, if you look up “Simple but Wrong, Complex but Right” it comes up with a great cartoon and you see people following—there’s a sign pointing in two directions of people, the masses are following the sign “simple but wrong,” because they’re looking for a quick fix.
Ashley: Mm-hmm.
Smalley: And what better than looking at how another organization’s done stuff and say, “You know, let’s do that.” It’s not a good idea, usually. [Laughter] So, you do have to be very, very critical, very selective.
Ashley: Well, these are systemic changes, they’re not quick fixes or just change this one thing of how you’re working and suddenly you’ll get different results. And one of those is, we think about specialties or specialization of what we do, but we’re also now thinking, it’s specialty roles, but it’s also how those things work across the life cycle development operations, product teams. So, individuals and teams’ abilities to work cross functionally is a much higher order skill than maybe we valued in the past.
Smalley: Yeah, look at the—I think we can kind of credit the DevOps community for something, but look at the agile community, how the concept of multi-functional, cross-functional product teams have emerged where you’ve got business analyst disciplines, development disciplines, testing disciplines all working together.
It’s interesting to think about the definition of done that they often apply as a concept that you’ll come across in agile and scrum, you know, “When are we done?” It often, in the scrum world, it ends with a potentially deployable increment of software, potentially deployable. The key word is potentially—it hasn’t been deployed yet. So, it’s just asking for an extension of that domain of the team to not stop when it’s deployable but actually deploy it, so start involving the DevOps kind of competencies as well. Then you could extend it further to the service area, because product owners should be aware that the kind of product that they’re building, that needs to be accessed by people, and service is the vehicle that gives users access to your product. You built a great software product, but it’s only through the vehicle of service that people get to it.
And is the product owner, is the product team looking as much at service as they should be? And I think that’s often not the case, and I’d certainly like to, in the role of bridge builder, like to encourage product teams to take a closer look at service and think about the kind of characteristics the service experience as well that they expect their users to have. So, you can see a service substrate going through all of the life cycle, really, alongside the software and infrastructure artifact.
Ashley: I’m curious, you mentioned design earlier and maybe this is what you’re referring to. I’m curious if design thinking or the design approach is something that also influenced your work on this module, the idea that you’re really, the outcome isn’t the car, the outcome is the experience that the customer, the user of that vehicle has and expects, you know, from whatever the product is that you’re creating. Did that influence this work, too?
Smalley: Yeah, two things, there—design thinking, we’ve referenced more at the start of the process, thinking about the kind of specifications requirements that people have. But towards the end, thinking about the service experience, and you’ll be familiar with the concept of service level agreements, which are often unfortunately formulated. We talk in terms of availability and performance and security and stuff like that—all very technical.
But how do the users feel about this? We’re slowly extending that domain service level agreement to include experience level agreements, looking at outcome and how people feel about their engagement with either their technology or the people behind the technology. So, again, that’s that human co-creational dimension that you see in high velocity IT, alongside the technology.
Ashley: Mm-hmm. Yeah, the SRE world, I don’t know if we’ve talked about that, but it has this idea of service level objectives, which is a little less formal and restrictive as what we think of contractual SLAs of, each team really, you know, defining what they’re trying to strive for, for the service that they’re trying to deliver and the experience that people have. So another area, SRE, is a great one.
Smalley: Yeah, no, that’s in there as well. SRE is certainly something that significantly contributes to more resilient operations, just like chaos engineering, another of the techniques that we reference. But speaking about agreements, certainly, you should measure stuff. Whether you—you know, the trouble is, once you put something into an agreement between two parties, people start to game the system. Things—I think it’s Hawthorn’s Law. It’s like as soon as you make something explicit, intrinsic motivation evaporates.
Ashley: Mm-hmm.
Smalley: Yet you do have to agree something with other parties. So, it’s really, it’s a delicate balance, you know, what’s gonna work in this situation. Also looking at the relationship that you have with your customers and other stakeholders—so, assessing what kind of relationship, what kind of agreements and measurements are appropriate for each particular situation.
Ashley: I was thinking about, too, when you mentioned resiliency, and previously, you were talking about sort of the iterative nature, doing work in much smaller increments—that also gives more flexibility for change. So that, you may have a feature turned into a story that might be delivered over multiple iterations in an agile process, and somewhere in that process, you may have part of it released, but you may move on to other things or shift what you’re actually doing. So, it’s a way of dealing with some of the dynamic, changing needs of the business.
Smalley: Yeah, it’s—we reference four characteristics of the approaches that you’ll find in these kind of organizations. We reference lean, agile, resilient, but also continuous, and I think that fits into this category.
You know, things are changing all the time, so it’s interesting when you’ve got these traditional discussions between business people and IT people. You’ve got IT people almost blaming business people for the fact that the world has changed, who it’s the kind of the core values that you have, the belief system you have, if you have the belief system that things are changing all the time and that’s just part of life, that’s a given, then you don’t resist as much. We certainly try to encourage people to embrace and enable change as well. We used to talk about change—management change control, which seems to be a bit defensive. When now, it’s just language, but even then, we like to talk about change enablement—how can you get changes done as quickly and safely as possible?
Ashley: Very interesting. Yeah, words do matter. What’s the state of the community, the ITIL community in the adoption of some of the principles around lean and agile and DevOps and SRE. Are most folks starting to become aware of or are they educating, are they practicing? Where would you say we are in this journey?
Smalley: Yeah, well, I forget who said it. Somebody said, “The future’s already here, it’s not just evenly distributed.”
Ashley: [Laughter] Yeah, that was Stephen Hawking or someone.
Smalley: Yeah, that’s right, yeah. You know, some people are doing it. I’m very pleased to see on LinkedIn or Twitter a response from an organization in the U.K. that says, “This high velocity IT stuff, this reflects the things that we are doing.” This is a guy on the joint DevOps and IT service management come together. I mean, that’s his area of work.
Ashley: Mm-hmm.
Smalley: It’ll depend on the kind of products and services that people, organizations provide are the digitally enabled? How digitally enabled are they? So, it is very diverse. Lots of people for whom this is all pretty new, one of the things we, one of the objectives we have for the book is to familiarize people who know the old way of working, you know, working in silos, that kind of stuff, and get them used to—make them more confident, confident to engage in discussions with their DevOps, with their SRE colleagues about digital to get the conversation going.
Ashley: It seems like that’s very important because, as things evolve, new things come into the picture—DevOps, et cetera, SRE. It’s really easy to be sort of overwhelmed of, “Now, what’s that, and how is that different and why should I care, or does it apply to me?” So, it seems that maybe taking that approach in this module of trying to bring that together from an ITIL perspective, from an operational perspective makes, maybe, those topics a little more approachable, a little easier to consume and understand?
Smalley: Yeah, and one of the things we advise people is to look at the five objectives that I mentioned, from valuable investments to a short conformance, and discover, identify where the weakest link is in the organization or in that part of the organization. Because if you look at the theory of constraints, another body of knowledge that we reference, you should always be working on the weakest link.
So, that helps you to focus on a particular part—don’t be overwhelmed. You can’t do all of this. You wouldn’t want to do all of this at once. Pick your weakest link, pick something that seems feasible that you can think, “Yeah, I can make an improvement, there,” and very much in the sentiment of continual improvement. Of course, one of the DevOps tenets, the third way of DevOps is continuous improvement—learning and improving all the time.
Little steps. I think that’s the way we’d like to encourage people to go—just take it step by step. Because after each step, you refer to the systemic nature of this kind of stuff. After each step, you’ve changed the system, so you’ve got to look around and see what’s changed and then plan the next step. So, moving from what I call confirmatory ways of working in the high velocity IT books, you know, when you’ve got pre-determined plans and deadlines and milestones, because you can more or less predict what’s going to happen, you confirm that—during execution, you confirm that what you predicted in the plan happened.
So, move—and we’re used to that in IT service management, predetermined process. So, moving from confirmatory ways of working to more exploratory ways of working, where you take things step by step—so, evolutionary. As Dave Snowden, who some people will know from the Cynefin framework, says—Dave authored a great piece about ethics for the book, by the way, but his core area is, he’s known from his complexity work—he says, “You’ve got to move to the adjacent possible in an organization.” So, sort of feel where the disposition of the organization is, in which direction are they moving naturally, and try and give them a nudge in that direction, zig-zagging towards, you know, whatever you want to achieve. That more—again, that’s that working in complex environments, reflecting the and respecting the unpredictable nature of the world we work in.
Ashley: You know, we’re kind of borrowing from a lot of disciplines talking about this and one that comes to mind, I believe, it’s quantum mechanics that just observing something changes.
Smalley: That’s it, yeah, the Heisenberg principle.
Ashley: The Heisenberg principle, thank you.
Smalley: Yeah, shine a light on it and it’s moved. You can either detect the velocity or the position, but not both at the same time.
Ashley: Mm-hmm. Very good. Well, this has been really fascinating—yeah, go ahead.
Smalley: I think Schrödinger’s cat came into it as well, didn’t he?
Ashley: [Laughter] Very good. Well, you know, I think what’s fascinating about your work and the contributor’s work on this is, I venture to guess there wasn’t a lot of discussion about tools and technologies, because you’re really thinking about the how of how we do work, how we work together, how this leads to the creation of software toward some co-created outcome.
Smalley: It’s very much the people dimension as well. You see that just talking—I mentioned ethics, for instance. Ethics—people want to do the right thing. That’s an aspiration that people have. They want to trust and be trusted in the workplace—psychological safety. I’m delighted that our industry is talking about this kind of stuff more often. Talking about things like—I remember at a conference, you know, the good old days when we actually went to conferences, a conference in, I think it was New Zealand, in Wellington, where somebody was talking about social anxiety as a rather introvert engineer.
Ashley: Mm-hmm.
Smalley: And it’s great, because you know, people empathize with that, and it’s great to trust and be trusted, fostering an environment where people can feel free to express themselves without fear for their position or reputation. If you build that kind of a foundation in your organization—you know, all the tools fit into place.
By giving people the confidence to—and the privilege, possibly a final thought, people just don’t work in IT. Some people say it, but you know, I’m sure some people don’t just work in IT, they have the privilege of contributing to the well-being and prosperity of digital service consumers, right down at the end of the chain. And I find that inspirational, thinking, you know, not just what happens in my silo, but at the end of the chain, you’re helping change somebody’s world. And that’s the kind of, that’s what I like to say about this book—it’s guidance that matters for people like us, who care.
Ashley: Mm-hmm. It’s definitely inspiring to know that you’re helping people, right, in the work that they do.
So, Mark Smalley, it’s been fascinating talking with you. I applaud you on the approach you took on this, a lot of great concepts that you’ve incorporated in the high velocity IT module, and I hope folks will check it out, because I think there’s some super valuable things. Again, changing the system is just by first observing it and I think just reading the module, checking out parts of it would be a step in that direction. So, appreciate your contribution, and it’s been great talking with you today.
Smalley: My pleasure, Mitch. Thank you very much, indeed.
Ashley: You bet.
-
DevOps Chat: Adopting Agile + DevOps in Large Orgs
In this DevOps Chat, Akshay Anand, ITSM product ambassador at AXELOS, and David Crouch, senior advisor at Beyond20, join Mitch Ashley to share best practices and learnings from introducing and adopting Agile and DevOps in large organizations. Akshay and David discuss overcoming change management challenges and obstacles to increase adoption.
As usual, the streaming audio is immediately below, followed by the transcript of our conversation.
Transcript
Mitch Ashley: Hi, everyone. This is Mitch Ashley with staging-devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. I’m joined by Akshay Anand, who’s product ambassador with AXELOS, and David Crouch, who’s a senior advisor at Beyond20. Welcome. Good to have you on, good to be talking with you.
Akshay Anand: Thanks for having us.
David Crouch: Thank you.
Ashley: Great. Would you first both start out – let’s do a little bit of introduction just to introduce yourself, a little bit of what you do and a little bit about the company that you’re with, if you want to start Akshay.
Anand: Sure. Yeah, my pleasure. Thanks for having us, as I said. My name is Akshay Anand. I’m the product ambassador for ITIL, working for Axelos Global Best Practice. Axelos was a company that was formed about five years ago as a partnership between Her Majesty’s Government and a large private sector company called Capita. We like to think of ourselves as the custodians of best practice bodies of knowledge, not just ITIL. We also manage PRINCE2, which is our project management, MSP which is Managing Successful Programmes, which is our program management, and a few others besides.
Before joining Axelos I was a consultant for most of my career in the IT service management space across both the US and the UK. I spent a couple of years as the head of service management for Macmillan Publishing as well. And that’s me.
Ashley: Great. David?
Crouch: Hi. My name is David Crouch, affectionately called Lower Baltimore at the office because that’s where I live and where I’m coming to you from right now.
Ashley: [Laughs]
Crouch: It doesn’t come with a _____. I spent about 20 years at Johns Hopkins University and Hospital before coming to Beyond20. Finished my time there working in the IT solution center which was our service management center for excellence. At Beyond20 we are a company that focuses on digital transformation. At the core of it we really try to improve the way people and organizations work by looking at their digital transformations and looking at how to improve their IT service management processes. We do consulting work, we do training, and focusing on ITIL but also project management and Agile and DevOps and these other areas as well, and we also have a team of folks who implement IT service management tools.
Ashley: Excellent. Perfect timing for our topic today. So let’s kind of jump into that. We’re talking about Agile and DevOps and ways that that’s introduced into companies’ best practices, learnings from adopting or not [laughs] those different practices. Maybe we can start with how we introduce or you’ve seen companies introduce either Agile or DevOps or both into organizations and kind of thinking probably larger organizations where there’s more change management involved. I imagine that’s a lot of work you get into, David. Is that correct?
Crouch: It is. You know, we see a lot of organizations who know a little bit about DevOps and a little bit about Agile but haven’t really adopted an Agile mindset and don’t really have the full understanding of what organizational structure can really support DevOps, and yet they need these things to really support a digital transformation. So typically we’ll start with doing some sort of assessment to see where they are now to see what pain points they have right now, and then try to figure out, okay, does it make sense to introduce Agile, where does it make sense to introduce DevOps, and how quickly can that be done or how gradually do we need to introduce these concepts?
Ashley: How about you, Akshay? I’m sure you worked with plenty of organizations that have introduced this and gone through that process.
Anand: Yeah. I’ve actually seen broadly two major patterns. The first being where the organization transformation is driven top-down, so it’s usually a board of directors or whatever governance structure says, “Look, you know, this is the vision of what we want the company to be,” and the senior-most leadership team says, “Well, in order to be able to do that we need to be able to iterate faster, we need to be able to have much more of a focus on pushing out digital products and services,” and that change then percolates through the rest of the organization, so very much a top-down thing. The other major pattern that I’ve seen and experienced is where you might have a small team who with the help of let’s just say a more enlightened manager is given the freedom to start to experiment and innovate and change ways of working while sort of at the same time remaining compliant with the rest of the organization’s policies and what have you, and they’re able to demonstrate improvements in either efficiency, productivity, experience, whatever it might be, to the point where the rest of the organization says, “Okay, so what are these guys doing and what can we learn from that and how can we start to pick up some of those changes ourselves?”
So in a way at that point, it’s a branch, that sort of new ways of working or new principles of working start to diffuse horizontally, so there are other sorts of teams, engineering teams, operational teams, et cetera, who start to think and work in those ways. Or it could be that the stories of that change percolate upwards, where the middle management or the senior management say, “This is interesting. Are other organizations doing something like this? Would it benefit us to roll this out a bit wider and give it the sort of support and evangelism and investment that we need in order to create a larger change?” So that’s the sort of branch, but broadly it’s two parts—top-down, bottom-up.
Ashley: You know, I think one of the biggest challenges with introducing any kind of fundamental change like Agile or DevOps is sort of crossing that chasm between saying we’re doing it and kind of really truly embracing the spirit or the concepts or the culture of it. Have you found that also to be true?
Anand: Well, yes there’s a difference. The way I reconcile this difference perhaps is in the following way and David probably has a completely different experience from his vantage point but for me one of the biggest challenges that organizations face or at least let’s say practitioners face, and then I’ll come to organizations—one of the things that I think practitioners are challenged by is in a way thinking that everybody understands things and just gets it. Everyone understands why we have to be Agile. Everyone understands why DevOps is important. And the truth of the matter is not a lot of people understand that and even if they do understand the words they don’t necessarily understand the implications to the rest of the organization.
So there is a little bit of this need to manage upwards, need to manage outwards to be able to champion the need for that change, the need for these new ways of working. The other thing which I think organizations and practitioners are challenged by is understanding where these new ways of working will succeed and where they would not succeed, and this is some of the things that we do actually talk about within the sort of ITIL 4 library of guidance that digital transformation or these new ways of working are really important, yes, but in a certain context they can be a hindrance rather than a boom, and organizations need to understand where these sorts of things can work. If the work that’s being done is highly predictable or has very long cycles of work and feedback then perhaps trying to break it down might only go so far, but if you try to overengineer or impose an Agile way of working you might actually end up doing more harm than good, even though you’re starting out with good intentions.
Certain things, to put it bluntly, don’t care how the work is being done. Once you start reaching the level of governance or supplier management or whatever it might be, in a way yes the CIOs today might say, “DevOps is really important, Agile is really important, et cetera,” but if you press them about what exactly is being done – look, I just say based on my conversations with other leaders I understand this is important, but the exact mechanics of how we’re actually going to execute on this I leave to the team who’s actually doing the work. So you’ve got to understand that change is difficult. The further up you go in an organization the harder it is going to be to try to convince somebody about the nitty-gritties that need to change, and you’ve got to be able to understand your audience, understand what they need to hear, understand how to convince them, and that aspect of communication is one of the critical components of organizational change management. It’s one of the places where I see the failure of organizational change management most commonly occur.
Ashley: Interesting. How about you, David? Share some of your experiences.
Crouch: You know, I think some of the more challenging but also more interesting from the consultant point of view situations are dealing with some companies that have been around for 80, 100, 150 years, and then I think folks pretty much around the globe would know some of these companies and they’re very mature in some respects. They’ve built their brand over time. But then when they start to suddenly – you know, some of them suddenly wake up and realize, “Oh, wait a second. We have gotten out of touch with what our customers really need, our external customers really need and what they want, and there are competitors that we never even dreamed of before that are entering into our space.”
They suddenly wake up and realize not only do we need new products and new services but we also need to work in a different way to produce those because customers change their mind a lot, customers are fickle, and I don’t mean that in a bad way. We want to understand what our customers need. And then the challenge becomes, okay, well if we haven’t been doing this right now it’s necessary but not sufficient just to provide training in these areas. Training goes so far. That’s very useful, but then also where are we going to have an area where doing something like Agile software development really makes sense?
I worked with an organization where any of what are often referred to as systems of record, so these are internal systems that keep the company running, their software team was actually doing a fairly good job because they worked in a relatively Agile way. It wasn’t 100 percent of what you would describe as Agile, but it was pretty darn good. And yet they would often run into roadblocks when they got into the more, for lack of a better term, traditional change management or as we call it these days change engagement processes where they were actually analyzing risk and managing risk all throughout the process because they were doing an Agile approach, and yet sometimes they would have this long delay at the end of the process when they got to their change control board in that company.
On the other hand, that company almost overnight experienced some significant financial losses and significant layoffs and they very quickly needed to develop some, what we call, systems of engagement, systems and tools that their third-party paying customers would use and would purchase, and in that case, the organization bypassed their internal IT entirely and just went to a third party because the third party really was already doing some version of DevOps in that case for years. Then the question becomes does their existing IT shop still have a reason to be? The answer, in that case, was yes, but it becomes very difficult because if you tried to apply that to the entire organization, particularly overnight, I don’t think it would’ve worked very well.
Ashley: Oftentimes in those situations, you also see people _____ project if it is an internal group just to try to isolate it and minimize the overall organizational reaction. Some people call them the corporate antibodies that tend to kick out and respond to change.
Anand: SIt’s interesting you say that, Mitch. What I’ve also seen though is that can also work but it has to be done very carefully because – who was it, which book was it? I think Geoffrey Moore – was it Geoffrey Moore? Yeah, I think Geoffrey Moore wrote an entire book about this phenomenon, a book called “Zone to Win,” and what he was talking about in that book with numerous examples, which I’m sure we’ve all experienced, is that skunkworks, that innovation lab does really well as the innovation lab, but the moment the organization says, “Okay, how do we now scale this new innovative feature, this new innovative product and industrialize it to the point where we can offer it to our thousands, millions of customers,” that’s when it falls down because you’re changing the constraints within which a product exists, if you will. It no longer exists as a small experiment measured in terms of success as an experiment.
It’s now got to be able to stand up to the measures of being like this industrialized product or service. So if you’re moving the yardsticks, is your product capable of actually bridging that—moving across—I don’t want to use the word bridging the chasm—is the product actually capable of moving across into that industrialized state? And what we’ve also seen through some of the research that went into ITIL 4 as well is a lot of organizations failed to make that bridge between that innovation lab and the industrialized operation.
Ashley: I appreciate you sharing that too. It could happen very well to the external group that gets formed as well. How do you reintegrate that, repatriate it back into the organization?
Crouch: And there’s a Harvard professor that famously kind of describes some digital transformation experiments as kind of like attaching a speedboat to an aircraft carrier. You know, they move at different speeds, and although the speed may go really fast, try to get that to translate into the aircraft carrier. It’s probably not going to happen. So it becomes—you know, the challenge is do we do a proof of concept, which sounds like an experiment to me, but if we do that can we really translate that into larger organizational success? And hard to say to come up with a definitive answer, but for larger organizations I think that what I’ve seen is that it’s been very difficult for them to do that successfully, so they have to find other ways of doing it.
Ashley: There’s a lot of things you can look at. You can look at sort of historical what’s worked, what’s failed in the past, you can look at—I call flow—what is the momentum of the organization? Are we a metrics, are we a finance, are we—what are decisions driven by? One of the things that can also help with that is also alignment. It’s why are you introducing Agile? Other than this being an IT fad, which it’s not just that, but it could be just for that reason.
Usually, it’s not a great reason to introduce it. You’ve got to have some real true business drivers like your example, Akshay, of we’ve got to respond to these things in market, right? The customers’ expectations have changed. That creates a sense of urgency but also purpose, like what are we trying to do differently and why and then how do we apply this?
Anand: There’s an additional wrinkle or complication, if you will, for publicly-listed companies as well where they’ve got to report certain metrics, and if you look at any company’s quarterly filings or annual filings where they talk about the year ahead or the quarter ahead, they are using certain types of metrics to communicate their direction and their results to their shareholders, to the market, and you might say, “Look, these measures are no longer relevant for what we’re trying to do or for where we’re trying to go,” but then what does that do to the company’s confidence as a shareholder? The example I cite is when the New York Times tried to move to the digital subscription. You can’t translate print subscribers into digital subscribers or engagement.
How do you measure the success of the digital engagement because you’re not delivering hard copy newspapers anymore? But the investors were used to seeing those sort of subscriber accounts and subscriber churn and all these sorts of things as forward-looking guidance. So there’s an additional complication from a governance and a reporting mechanism, especially if you’re a publicly listed company, and that’s why people like David have so much fun going into these companies trying to help them figure out, well, it’s not just about changing the ways we’re working; how do we educate our peers, how do we educate our managers, how do we educate our suppliers, how do we educate our consumers, how do we educate our shareholders? There’s all these different stakeholders. It’s not just about changing the way your software development is done. It has massive ripple effects across the ecosystem.
Crouch: Well, and what’s really interesting about like New York Times is that they were still successful with their print publication, one of the few, when they began their digital transformation. So you have a tremendous amount of courage to be able to say as the leader, “You know what? We’re successful now but this is not going to last. We can see it in the marketplace. So what we actually need to do is intentionally set up a parallel operating model and say we’re going to move in this new direction.”
And that means that that’s in their case going to siphon away, intentionally siphon away over time the way we do business now and it’s probably going to hurt in the meantime, and we’ll use any of the profits from our existing model to move over to this new model. But gosh, if you’re an investor you have to be pretty nervous about that unless you’re really in it for the long term and you say, “You know what? They’ve seen this and other companies have tried this; how do I know that New York Times is going to be successful when some of their peer publications have not been when they try to do this and for a variety of factors?” It seemed to work for them but it’s pretty daunting.
Ashley: That’s what I would call the disrupt yourself before you’re disrupted model, right? Because usually it’s much more painful when you’re reacting to someone else who’s coming in and taking market share or whatever. That’s a really good example too of in the print media going into digital, it’s not the same product, it’s not the same consumers, it’s not the same behavior. The market really has changed, or maybe you’re just really going after a different market who would never now buy print but they would just use that example.
So a lot of it can be driven by those things that we’ve answered about strategy or understanding of the market or shift’s that are occurring and tie those reasons as part of why we need to create software faster or why we need to create it in waves of some smaller increments to be able to respond to really rapid shifts or experiment in market. That’s another great reason to do that.
Anand: Absolutely.
Ashley: Well, talk a little bit about—you mentioned the speedboat. I can just imagine it tied to the aircraft carrier, kind of sort of like the fly tied to the string, right? How do you start to introduce speed into a larger organization that has a lot of momentum around the way they’ve been doing things that’s worked for them probably very successfully? How do you start introducing something like speed to say, “This is super-critical and here’s how we’ve got to operate now?”
Crouch: I think it starts with trying to understand what they’re primarily trying to do. Where does having more speed make the most sense? So it could be we’re in a consumer marketplace and the customers are changing their minds a lot and there may be a way to develop something more quickly to respond to that need or in the perfect world to actually kind of predict and shape that demand for the great digital masters. On the other hand, some companies may say, “Well, maybe we’re not in a strictly business-to-consumer marketplace but we need to increase operational efficiencies in some areas,” and so maybe speed is more important there. And then for a few companies out there it may be both, right?
It may be if we increase operational efficiencies on the inside of the company we can also drive the outside third party customer, and that’s no easy way to do that, to figure out where speed is going to be appropriate and where doing things quicker might actually get in the way in some cases. Akshay, I don’t know what your experiences have been there.
Anand: I mean, look, there’s a lot of great organizational change, organization design, literature out there for people who want to read up more about this sort of thing. The Satir change model or Kotter’s Seven Steps or whatever else it might be. In my experience, one of the first steps I’ve seen in any successful change is – to use Kotter’s language – creating a sense of urgency. I guess this is something I remember from my time living in the US. Wasn’t there this sort of tagline that the first step to recovery is admitting you have a problem or something along those lines, right? So it’s kind of like that.
The first step towards creating a change is to acknowledge that you need to change, because once everyone acknowledges that you need to change you can start to think about, well, what sort of change do we want to have? And this has to be a highly collaborative effort. It’s not just your CXOs locking themselves in a conference room at a resort for two days and whiteboarding this out. This has to be a truly collaborative exercise across the entire organization. There may need to be some constraints that are in place, and what I’ve typically seen is a board of directors might say, “Look, this is where we want to be five years from now,” not necessarily saying how that needs to be achieved but saying this is where we want to be.
That in combination with creating that sense of urgency says, “Look, if we need to hit this sort of a target we need to be able to develop software faster or close out outages quicker or have a more Agile procurement process or whatever else it might be,” and that’s when you start to identify these sort of strategic and tactical initiatives. Because you’re working collaboratively you’re dispersing that cognitive effort, if you will. So it’s not just somebody from one particular vantage point saying, “As a CFO I think we are giving away too much money so we’re going to be cost-cutting only.” That’s a singular perspective. But once you distribute perspective you can then say, “Right, if we are supposed to meet this we have to change here, we have to change there, we have to modify that,” and so on.
For me that is the essential first two steps; create that sense of urgency and then distribute the cognition of what needs to change. Create that collaborative series of committees, councils, call it what you will, that will help you identify in different areas what needs to change, and then you start investing to execute on that series of tactical initiatives or programs of change.
Ashley: When I first learned about change management—David, maybe you can comment on this—we used the phrase “the burning platform.” What’s the burning platform that we’re on that’s forcing us to change or really compelling us to make a change? It can be as dangerous as are we going to exist but also it could be here’s why a two- or three-year strategy makes sense. Let’s not just communicate this strategy; let’s communicate the why.
What’s changing? What evidence do we have of this? What do we want to get ahead of or prevent ourselves from experiencing? Have you seen that as a successful strategy for helping create that burning platform or at least the sense of it?
Crouch: I think so. I mean I think the top level of support from the leadership is absolutely critical, but when it comes to defining the problem getting down to as specific as you possibly can. It’s not just a matter of we’re not getting new products out to customers quickly enough or it’s not just a matter of we’re not implementing enough infrastructure changes within a certain period of time. It’s a matter of exactly what about that isn’t going well and why would improving that be made better? In some cases it’s obvious. In some cases, for example, it’s not so obvious, and when people think of Agile and DevOps one of the first things that comes to mind is speed, we’re going to get more speed.
That’s an important part of it. It’s not the only reason why you might use some of those techniques. To use a simple example, if I need to be at the theater for a show—not these days—but if I have to be there at 7:00 PM and I only live 15 minutes away, driving faster won’t necessarily improve my results. I get there—it still doesn’t start until 7:00 PM. If I get there at 6:00 PM I guess I get a good parking spot. So speed is just one element, but think about with Agile doing things in smaller iterations, controlling the risk, controlling the scope. All of these are considerations in addition to just speed.
Ashley: What –
Anand: And the other thing that I think the research shows us—sorry, I didn’t mean to –
Ashley: No, no. Go right ahead, Akshay.
Anand: I was going to say one of the other things that our research has shown us as we’ve been working on ITIL 4 and other bits and pieces, even frameworks like managing successful programs and so on, is organizations need to figure out the right duration of their strategy cycle. Now if your company is a two-person company then maybe your strategy cycle is measured in days or weeks, all the way up to if you’re the government of India and you’re trying to manage—not manage, but govern one-point-how-many-ever billion people then your strategy cycle is likely to be measured in years, if not decades. Well, hopefully not decades. So I think the other aspect that we’ve seen, at least in typical organizations, strategy cycles have gone from being sort of this thing that happened every five years to maybe things that are happening every three years because the world’s changing too fast for a five-year cycle to make sense.
We may see a point where three-year cycles are deemed insufficient and we need to move to two-year or one-year cycles. But that’s the other trend that we’re also seeing, that strategy cycles themselves are reducing.
Crouch: I’ve seen that a little with managed service providers, for some of the managed service providers. Even one year seems to be an awfully long time for some of them, you know? So there’s this kind of blend between are we talking about strategy or tactics here, but I agree; the timeline is becoming shorter and shorter it seems.
Ashley: It certainly has, and of course our current COVID situation, I’ve seen some resource. I know some of the sources of this, but some have said, “We’ve accelerated as much as six years in our digital transformation strategies in the last six months.” Probably not everybody has, I imagine. Well, this has been great talking with you both. I know you mentioned, Akshay, Geoffrey Moore’s book “Zone to Win” as a great resource. Any other resources on introducing Agile and DevOps or on change management that you guys can think of might suggest?
Anand: Well, I think I would be very remiss if I didn’t actually mention ITIL 4, which is the product for which I’m an ambassador.
Ashley: That was a setup question.
[Laughter]
Anand: Easy question, though. So look, there are a couple of really good books in ITIL 4 which talk about this. Two books in particular that I’d like to highlight. The first is “High Velocity IT” and the second is “Digital and IT Strategy.” “Digital and IT Strategy” actually does refer back to some of the content in “High Velocity IT.” “High Velocity IT” is more focused on the tactics and operations aspect of introducing these new ways of working and managing the changes and the implications therein to the rest of the organization. “Digital and IT Strategy” is more about the governance and strategy and the interface to tactics, so it’s about how do you determine what’s the appropriate business model, how do you create a change in your business model and from that create a change in your operating model, from that figure out where Agile and DevOps make sense and where it doesn’t make sense and so on.
So it’s very much the sort of top-down but the leader’s perspective of things, whereas “High Velocity IT” is the practitioner’s perspective of things. I think those two would definitely be really good. I’d also plug a couple of books like “Team of Teams” I think is really an interesting read, talking about how especially in an organization like the US Army, which you would typically think is very large and flexible and so on, how they were able to create that kind of a change, not necessarily with software development but around ways of working from across multiple teams, across multiple time zones, so that’s also a really good book to check out.
Ashley: Yeah, I’ve not read that book but I’ve heard it’s good. I think it’s by Stanley McChrystal, right? General McChrystal?
Anand: That’s right.
Ashley: Great author. I think he uses his whole military experiences, examples of that, so I’ve heard him speak about it. How about you, David? Any suggestions?
Crouch: I would just say that, you know, as one of the authors of the ITIL “Digital and IT Strategy” book I’m going to put in a plug for that. I’m just a little bit biased, even though it probably took several years off of my lifespan to help write that—I’m kidding. But no, that’s really talking more about the strategy side of things. I recall reading an article years ago as part of one of his classic books. Michael Porter talked about IT is not strategic, and I struggle with that. I agreed kind of at the time and now I don’t know, and I think if IT is now strategic in this era of digital transformation it really comes down to things like high velocity IT to DevOps to Agile to some of the techniques that we talk about in the “Digital and IT Strategy” book and “High Velocity IT.” You know, I would suggest reading that.
Anand: There’s probably a couple of other plugs to make just before we go, sorry.
Ashley: You bet.
Anand: For those who want to read a little bit deeper into some of the more cutting-edge, I would say, thought leaders, read up on Simon Wardley, Wardley mapping and his whole strategy cycle stuff. He publishes that on his blog. There’s tons and tons of material on his blog, which I think he started condensing to a series of medium articles. But there’s also YouTube talks that he does. Amazing stuff. And the second is Dave Snowden and the Cynefin framework. It’s getting a lot of attention in the Agile space especially, but it’s a decision-making framework. It’s a sense-making and a decision-making framework and it has applications from business strategy to project management and everything in between, and I think that’s another –
Ashley: Excellent.
Anand: – at the moment cutting edge but hopefully a soon mass-market approach.
Ashley: Oh, those are some fantastic resources. And maybe a parting thought for all of us is I don’t know if IT is strategic but I’m pretty sure software is strategic these days that’s driving – really fueling our digital transformation.
Anand: Absolutely.
Ashley: So a topic for another conversation. Well, thank you, gentlemen, Akshay Anand, who’s product ambassador of Axelos, and David Crouch, senior advisor Beyond20. Thank you so much for sharing your experience and your thoughts on this topic. I’m sure it’s been really helpful.
Anand: Thanks for having us, and to the audience, you can find me on Twitter @bloreboy if you care to follow me, and I’d love to hear some feedback either through Mitch or directly, however you’d like. I’d love to know what you thought about what we were talking about today.
Ashley: Wonderful. Happy to do that. And I’ll pass that along if we do. David, do you want to share any contact information?
Crouch: Yeah, same here. I’m on LinkedIn. A slightly younger, bow-tied version of myself is on LinkedIn. Feel free to connect with me. Certainly visit our blog at Beyond20. We’re always writing articles, trying to grapple with some of these issues, changing our mind some of the time, but you find a lot of what I write there.
Ashley: Excellent. Well, we’ll look forward to talking with you both again soon. Take care. Have a great day.
Anand: Thank you.
Ashley: You’ve listened to another DevOps Chat podcast. This is Mitch Ashley thanking everyone for joining us today. Be safe, be careful out there.
-
What Is IEC 62443?
IEC 62443 helps to protect industrial automation and control systems from security breaches. Read on to learn more about the standard.
What You Need to Know About IEC 62443
IEC 62443 is a set of security standards for the secure development of Industrial Automation and Control Systems (IACS) that provides a thorough and systematic set of cybersecurity recommendations. The security standard is used to defend industrial networks against cybersecurity vulnerabilities.
The IEC 62443 Security Levels
A central part of IEC 62443 are security levels (SL), which are used to assess the cybersecurity risks to each system. An additional benefit of security levels is that they help you to understand how to best address the identified cybersecurity risks. There are five security level values, which range from 0 (the minimum level of risk) to 4 (the maximum level of risk).
The 7 Security Level Foundational Requirements
For each security level, there are seven specific foundational requirements that must be met. These requirements help you to ensure that each IACS has the right security and safety safeguards.
The seven foundational requirements for a security level are:
- Identification and Authentication Control
- Use Control
- System Integrity
- Data Confidentiality
- Restricted Data Flow
- Timely Response to Events
- Resource Availability
In addition, each foundational requirement has multiple conditions that must also be met, depending on the safety level.
What You Need to Know to Comply with IEC 62443
An essential part of complying with IEC 62443 is using a static code analysis tool. Static code analysis tools automatically identify vulnerabilities and defects as you code. In addition, IEC 62443 requires that a static analysis tool be used to enforce secure coding standards, such as CWE, CERT, and OWASP. By using secure coding standards, you ensure your software is secure and safeguarded from vulnerabilities.
To read more, please visit: https://www.perforce.com/blog/kw/what-is-iec-62443