Kevin Behr has been in IT game for 26 years and is the author of seven IT management books, including the bestselling The Visible Ops Handbook, which he coauthored with Gene Kim and George Spafford, and The Definitive Guide to IT Management, published by Hewlett Packard. His most recent book, The Phoenix Project, also coauthored with Kim and Spafford, has helped to spark discussion about DevOps globally. Behr is also well-known for his technology and management keynotes for such organizations as the National Academy of Sciences, Hewlett-Packard, the SANS Institute, AFCOM, and the IT Service Management Forum.
Behr is also the founder of the Information Technology Process Institute (ITPI) and chief science officer for the CXO and Board Advisory Practice at Praxisflow (formerly Assemblage Pointe), where Behr consults, mentors, and coaches IT organizations and executives on how to increase their business effectiveness and competitive advantage.
We caught up with Kevin Behr to discuss DevOps, its impact on security, and the challenges of bringing change to the organization.
staging-devopsy.kinsta.cloud: We’re now hearing a lot about continuous deployment, but what does that mean compared to traditional enterprise thinking about deploys? And what does it mean for IT disciplines like security?
Behr: With continuous monitoring, you can catch failure quickly, even if the issue that caused the failure makes it into a production state. With today’s SaaS and cloud models, failures don’t matter as much because you are not shipping software as much as you are managing a service. And one fix works for all of your users instantaneously.
This is the difference between traditional enterprise IT thinking and the thinking around continuous deployment, which really is starting to creep into the enterprise. The new thinking is: how do we do this smaller, faster, and more safe to fail. And organizations aren’t ready to make everything safe to fail, but pushing things in that direction becomes an actual security strategy—versus the notion of, let’s just see what happens.
That’s what we’re getting into with continuous deployment. You’ve actually made the deployment safe to fail. You almost don’t need very much testing because you can actually deploy, then wait to turn on a new feature; and if it doesn’t work, you can turn that feature off. What we’re starting to see is this kind of radio button style of deployment: they’ll deploy software and it’ll be turned on progressively, versus an old-school Big Bang.
That’s really kind of neat when you think about it. Software is being treated more like infrastructure. You hear that phrase a lot: Code is infrastructure, and infrastructure is code. You start to see the different meanings of that everywhere. And I think that’s right. I think the security practices have to be a part of the safe to fail approach. You want to have very good monitoring and detecting controls, because they’re going to help you be more resilient.
staging-devopsy.kinsta.cloud: I think these are concepts that may have challenges when working with some security and IT professionals and others who are used to having control over the process. What are you seeing in regard to the balance between an organization’s need to oversee and protect itself and the benefits of continuous deployment?
Behr: You’re getting at the fear that security practitioners and others may have, that this breaks down controls that they’ve put in place. They’re realizing that it could actually add more resiliency—consider continuous authorization to do certain things, cross-functional teams, continuous monitoring, and all these things that could very possibly increase resiliency.
You’ll see more success if security could embrace DevOps, from the standpoint of, “Oh my: This is amazing and so what are the top ten things we must do to make DevOps work? Not “here’s why it can’t work.” Security teams that roadblock like that are going to get steamrolled right now.
But it’s not good news everywhere. I hate to say that some companies out there are more stupid than others. I’m seeing places use DevOps as a way to excuse all the reduction in force that they’re planning: “We need to get rid of five people because we’re doing DevOps.” What sense does that make? I think mini-backlashes are coming in some organizations. And one of them is going to be where organizations call things DevOps that aren’t, just because it serves their ideas.
For instance, I’ve been into a couple of banks that have created DevOps departments, which is sacrilege to this whole movement, because the whole idea is to get rid of silos, not to create a new one.
Some days I get bothered by the notion of organizations calling it DevOps, and then other days I don’t because it’s just the result of the success of an idea.
staging-devopsy.kinsta.cloud: What steps do you see organizations taking to deal with the change? Are you seeing schisms in organizations where some people are resilient to change?
Behr: I think that the folks who own these systems that they work in have a long way to go. And I think the people who are having a hard time with DevOps are getting steamrolled right now. People in vice president of operations positions, or the director of operations: They may not love the concept of DevOps because it creates cross-functional challenges for them. The first is: “All of these people who don’t report to me and now are in my data center. What are they doing? Why are they on keyboards? How do I control this?” [Story]
Think about it. Their whole working role is judged on this notion of reliability. Their heuristics tell them that having lots of unaccountable folks doing stuff isn’t consistent with reliability in the long term. In the shops where DevOps has been successful, the security folks have been able to get in and say: this is awesome. They love the reduced cycle times. They love that the dev guys are coming down and helping with patching. Getting things done whenever they need it. And that they actually know what this stuff means now; it means that their security feedback loops directly into dev.
I think that’s the other thing that people don’t get. DevOps, when really done right, creates parallel feedback loops, not just serial feedback loops. In the old way, someone asks someone to do something, they do it, and then the provider asks how they liked the service? That’s the feedback loop most operations have today. But when the devs are brought in, all of a sudden the devs are adding another feedback loop. It goes something like, ”Hey, I just watched you do this thing that I don’t do every day. Man, that’s a lot of extra steps. Why are you doing it that way?”
That’s one feedback loop that comes from just teaching somebody something new, and getting the feedback from the outside. And then there’s the feedback loop of the quality of what you delivered. When you’re getting both of those at the same time, parallel feedback loops are how evolution happens. That’s one of the hallmarks of all evolving systems and adapting systems.
If security can become one of those parallel feedback loops, and all the documentation of what you’re doing when it happened, and all that related stuff is in one place, then security becomes part of the refactoring.
I think the other thing is, because the dev and ops people are collaborating more, you’re starting to hear more contextualization happen with the developers. They’ll say things like, “I’ve always wondered why they yell at me whenever I hard-code IP addresses. Now I get it: because you guys have to change these all the time, because we’re now down here changing a bunch of them.” That contextualized understanding goes a long way.
This is a huge thing. Because there’s a ton of infosec ops: patching, log parsing, and different things that happen at an operation level that are really security driven. And if you can get the devs involved in helping out with some of that stuff, they start to understand why you’re doing it. And in many cases, we’ve got technologies in place that we spend a lot of money on, and security, that are just to prevent devs from doing things that we didn’t think we could change.
The organizations that get this—and foster it—are the ones that are going to succeed.
