staging-devopsy.kinsta.cloud

Category: Features

  • Q&A: Bill Burns on Security Within DevOps

    Q&A: Bill Burns on Security Within DevOps

    Last week’s story, Caution: DevOps Ahead, caused a small stir on Twitter and sparked a response, DevOps: Security’s Last Best Hope, from staging-devopsy.kinsta.cloud’s Alan Shimel.

    Coincidently, I recently had a conversation with Bill Burns, former director of IT security and networking at Netflix. Burns is currently an executive-in-residence at ScaleVP, where he provides strategic advice in the security market. He has experience managing security in hybrid cloud/IT environments and traditional IT environments. We thought now would be a perfect time to share his thoughts on how security can potentially be improved in organizations that embrace DevOps.

    Here’s our talk:

    staging-devopsy.kinsta.cloud: What impact do you think DevOps has on its security teams? How should traditional security practitioners look at that movement?

    Bill Burns: DevOps could improve security if it’s done well, and more importantly I think if the business really buys into DevOps. What I noticed, especially at my prior company, was that as product teams were moving to DevOps, everybody was coming to the table with bite-sized building requirements. They weren’t ginormous, “Okay, we’ll do this in the next six months,” but instead: “What do we accomplish in the next two weeks? The next sprint cycle?”

    It forced you to think of the problems early on and know that you would attack them in bite-sized chunks. The security team had to not just throw requirements over the wall, but had to participate in the design phase and in the negotiations.

    With DevOps, you don’t have this ginormous build cycle where many things can go wrong and tests are often done at the very end. In that sense, if it’s done well, incorporating security into DevOps is sort of holy grail for a really robust software development lifecycle with security.

    Everyone says they want to put security upfront within the design requirements and in the architecture. Because you don’t have the upfront cost of a large architecture phase, and you are iterating, it’s an opportunity for the security team to do that and to participate in tighter feedback loops both with developers and with operations.

    staging-devopsy.kinsta.cloud: Well, it sounds like security teams could very well fall on their faces if they tried to be too rigid with DevOps.

    Bill Burns: Yes. It’s the same as if an operations team came in and said, “Well that DevOps stuff is great, but this is the way we do operations.” They would become irrelevant pretty quickly.

    It’s also an opportunity for the security team to become more consultative and advisory in nature. With DevOps, security can say, “These are the most important things to focus on now.” Maybe those things are the session handling or the encryption. For those critical things, it’s best to start on them right away. Whereas the things that are important, but not critical, can be handled later on and over time.

    And security should be agile enough to pivot with the developer team to handle what it needs to do and make the incremental security improvements as the product changes.

    staging-devopsy.kinsta.cloud: What are some of the barriers security teams face when trying to be a part of the conversation in a DevOps environment?

    Bill Burns: What I’ve seen to be effective is, first of all, to approach the product team or the development team as an adviser. Also, speak honestly and say that you may not have all the answers right away, but that you’ll be a part of the process and learn as the project matures. Start with some of the best practices that you know should be a part of development early, and list things that you will want to be added later.

    If you can prioritize the most important security issues along with the rest of the product team, they’re going to go through the process and will have a better understanding – and security is improved.

    I also think it really helps if the security team members are not solely infrastructure-focused. That they’re not just trying to figure out how to get their intrusion detection system to participate, or how to get your security product to snap into the system. I think DevOps is forcing the security teams to become more like developers. I’ve noticed that a security team becomes more of an application security team and less about buying products and bolting them on.

    Ultimately, the security team is not the risk owner. It’s the adviser. And I believe this is true for many organizations even though many don’t realize it. The security team, in my view, should say: Here’s our analysis. Here’s our impact. Here are our threat models.

    And in that process, security can say that it thinks something is really important and a big risk. But, ultimately, the decision whether to accept the risk is on the product owner or business owner – not on the security team.

  • Some changes to staging-devopsy.kinsta.cloud

    Some changes to staging-devopsy.kinsta.cloud

    Well if you are reading this you are one of almost 40,000 unique visitors to staging-devopsy.kinsta.cloud in the last 6 or so weeks. In this relatively short time we have learned a few things. Based upon these we are making a few changes to staging-devopsy.kinsta.cloud. Some you will see right away, some you will see phased in over the coming days and weeks.

    Here are some of the changes we are making and why we are doing it:

    A. Blog Neighborhoods – From the beginning my vision of our many bloggers and blogs was to organize them into “neighborhoods” that would allow people to hone in on what is important to them.  Unfortunately the neighborhood aspect was not well implemented and all of the blogs were lumped together. Going forward you will see our bloggers and blogs grouped into the following areas:

    1. Enterprise DevOps – This is for DevOps in the large enterprise. Organizations doing DevOps by choice.  This neighborhood will feature stories, tips and tricks on adopting DevOps in a large organization and how to make it work. We have some excellent bloggers in this section and will be adding more.

    2. Doin’ DevOps – Formally the DevOps Life section, this is a neighborhood of DevOps practitioners. Not vendors, people who practice DevOps and work and are sharing their experiences. If you are looking for more technical content, this is a good place to find it.

    3. SecOps – This is for DevOps and Security related content.  Again we have some excellent writers in this section and the security tips range from very technical to how to integrate security into the Dev and Ops process.

    4. DevOps in the Cloud – This is primarily startups and companies using the cloud almost exclusively for their infrastructure. Sort of the opposite of the Enterprise DevOps neighborhood.

    5. DevOps Toolbox – This is going to be primarily vendors writing about their DevOps tools here. Yes, you can take what they say with a grain of salt. But to discount them because they are vendors would be a mistake. No one knows their products and use cases better than the vendors themselves. Not to mention that they have experts in the field who will be contributing. There will be much to be learned here!

    6. The Business of DevOps – like it or not, DevOps is on its way to being a genuine commercial business process. There are companies, executives, investors and managers who are exploring the business of DevOps. Sort of the Yin to Doin’ DevOps Yang.

    We hope by calling the neighborhoods out like this it will make it easier for you to zero in on exactly the type of content you enjoy. We recognize that many of our readers are very technical engineers. There are also many others coming to DevOps who are not. We will continue to strive to make staging-devopsy.kinsta.cloud the best it can be, pitching a tent big enough for all.

    B. Newsletter

    Thanks to so many of you who have signed up for our newsletter. Try as we may, we have just not been able to get our RSS generated newsletter to look the way we want it to. Instead if you receive our newsletter you are getting a fire hose of posts. There are too many of them and they are too long. So instead we are going to move to a hand curated newsletter that will only be sent 2 to 3 times a week depending on how much content is on the site.

    C. Building out community features – The business directory is up with some listings.  I encourage vendors to take ownership of their listings and make them better. The pages are actually getting lots of views every day. It wouldn’t hurt. Our jobs board beta is up, but we need more listings. If anyone knows of job listings in the field, this is a good time to put a free listing up there. We will add more community features shortly, so stay tuned.

    We are also adding more content and writers. If you would like to contribute to staging-devopsy.kinsta.cloud, write to me ashimmy@devops.com and we can see if you have what it takes.  Thanks for taking the time to read to the bottom and supporting staging-devopsy.kinsta.cloud

  • Embracing a culture of continuous incident response

    Embracing a culture of continuous incident response

    In a world of continuous development and rapid iteration combined with being connected online 24/7/365, you are bound to encounter threats. In fact, attackers use automated tools so you are under constant siege from continuous threats. The natural response is to implement some form of continuous incident response.

    The standard development model involves different stages of requirements gathering, writing code, and beta testing, resulting eventually in a defined, periodic software release. That model is being replaced by the DevOps culture of continuous development. Security faces a similar transformation from the pervasive legacy model of the “us vs. them” protect-the-perimeter model to a new system of continuous incident response that addresses the current threat landscape in the real world.

    There is no “us”, and there is no “them”. There is no perimeter. Your authorized users are actually more likely to expose sensitive data, or compromise your systems—whether intentionally or inadvertently—than external bad guys. Your authorized users are also just as likely to be “outside” of the mythic perimeter—connecting from personal smartphones or tablets, or logging in from a Starbucks across town. The traditional security model is simply not equipped to deal with today’s technologies or threats.

    So, what is continuous incident response? I spoke with TK Keanini, CTO of Lancope, who explained, “Some people when thinking about process think of it as a linear model where there is a beginning and end; a better way to model adaptive processes is with a loop and continuous incident response is a loop.”

    He elaborated, “Borrowing from John Boyd’s OODA loop, continuous incident response is where there is an observation, orientation, decision and action cycle that outpaces or runs at a faster tempo than the adversaries.”

    Zane Lackey, director of security engineering for Etsy.com, gave a presentation at the Hacktivity conference on the topic of “Attack-driven Defense”. The concepts he described are essentially the sort of logic and pattern that need to be applied for continuous incident response.

    There are a few key elements organizations need to embrace in order to effectively implement continuous incident response. The first is the loop Keanini described. In order to be “continuous”, you have to continue to cycle through the various phases. There is no “end”.

    One of the crucial components of continuous incident response is continuous monitoring. It isn’t good enough to scan the network once a month, or once a week, or even daily. The threat landscape is constantly changing and evolving, and you have to be continuously monitoring for new vulnerabilities, new attack techniques, and anomalous or suspicious activity that might indicate a security incident is occurring.

    Another important factor is to view continuous incident response as a cross-functional business process. Keanini stated, “I think continuous incident response—when implemented well—touches all critical parts of the business, not just IT. Making it an IT thing is just too narrow and people who experiment in this manner are destined to fail.”

    Companies that have a DevOps mindset, or have adopted DevOps principles will find it easier to relate to the concept of continuous incident response. DevOps organizations are not constricted by the departmental barriers of traditional companies. The hierarchy and red tape of an org chart can get in the way of effective continuous incident response, but DevOps businesses have a more open, fluid organizational structure that makes it much easier to adapt to the dynamics of both operating on, and responding to threats from the Internet.

    Adopting continuous incident response may be easier said than done. It is a different approach to security, which may make it more difficult to get buy-in and support from executive management. You may also encounter resistance from the IT admin and existing security personnel because they’re used to doing things the old-fashioned way, and change is uncomfortable.

    Keanini sums it up well, though. “Just step back and ask yourself how important being connected to the Internet is to your business or customers—and then just consider that continuous incident response is the cost of doing business on the Internet.”

  • Security and DevOps showdown

    Security and DevOps showdown

    caution

    George Hulme’s article on proceeding with caution on DevOps implementation offers some advice from a few well known security experts. Article here

     

     

    vs.

    helpmeobiwankenobi Alan Shimel says today’s speed of business isn’t compatible with :go slow”. Security people have to understand that DevOps is our last best hope. But in order to have DevOps help us to security nirvana we need to change how we interact with the rest of our IT teams. Article here

  • DevOps: Caution Ahead

    DevOps: Caution Ahead

    Despite the continued adoption of enterprise DevOps practices, some organizations, especially those in highly-regulated industries remain cautious about moving forward too quickly.

    “There’s no doubt that DevOps brings benefits for some organizations,” says Martin Fisher, director of information security at Atlanta-based WellStar Health System. “However, many pushing for DevOps underestimate the amount of technological and cultural change that is necessary to overcome before moving ahead to boldly, especially among those in security and regulatory compliance roles that are concerned with securing and auditing processes they see as they’re potentially losing control over,” Fisher says. “You can’t audit and secure what you can’t control,” he says.

    Eric Cowperthwaite, former chief information security officer at Providence Health and Services agrees. “There are some who want to move too fast. In many organizations the culture just isn’t yet there, especially where they are used to very rigid quality assurance and audit controls,” says Cowperthwaite, currently a VP at Core Security.

    It’s certainly not uncommon to find chief security officers in heavily regulated industries who are concerned that the move to DevOps is an excuse to cut corners and move more quickly by cutting out necessary oversight. “You’ll find if you have a developer that’s producing a dozen mistakes a day, if you poke at them about it then you’ll find out that they’re tired of the structure, of the bureaucracy,” he says. “DevOps need to be done in a mindful and thoughtful way,” Fisher says.

    Cowperthwaite agrees, and contends that in certain environments DevOps will be – should be – taken slowly. “You have legacy systems and very highly audited and controlled systems where DevOps is just not going to flow for some time,” he says.

    It’s not just the cultural shift, but also the shift in necessary for good governance capabilities to keep up, they contend. “Not all systems are geared to handle 200 commits a day, nor do they have the ability for reasonable tracking and accountability for that kind of speed. In fact, there are many legacy systems were it’s not even possible at all,” Fisher says.

    However, not that all companies that move toward DevOps are attempting to automate everything – at least not at first. But it is higher levels of automation most certainly seek. A survey recently conducted by automated server management software provider JumpCloud found that many companies are turning to DevOps to automate time consuming activity, such as deployment, patching, user management, log file analysis, and forensics activities.

    Some of the trepidation is warranted, to be sure, and some of it may be due to the media and industry focus on the outlier DevOp implementers. “I think, justifiably, a lot of the fear comes from the fact that a lot of the companies that have gotten the most press regarding DevOps are doing things that are radically different than what is going on within the bulk of IT work at traditional enterprises such as banks, healthcare, and organizations like that,” says David Mortman, chief security architect and distinguished engineer at Dell Software Enstratius.

    When it comes to enterprises with trepidations when moving toward DevOps, all of the interviewees we spoke with agreed the key is to move slow. But still move and develop the culture and the toolsets slowly. “Pick one system that is separate and use it as a concept,” Fisher advises. “The DevOps ‘true believers’ are given a hall pass to fully run their systems and group in a DevOps model. As they learn and improve you slowly assimilate into the broader environment,” he says.

    Mortman sees it in a similar way, but notes that it’s not just about technology and absolute speed, but also collaboration, culture, and relative speed. “How fast is fast for your organization? If you’re only making changes to your big medical records database every 18 months, and you switch to every three months – that’s faster. That’s more continuous deployment than you were doing before. No one has to jump right in,” Mortman says.

    “I started working with those sort of DevOps-y concepts back before I was aware of the term “DevOps,” Mortman adds. “I was calling it “Agile InfoSec” and “Agile Ops.” And with that mindset, even the most regulated, paranoid enterprise can embrace DevOps. Start with small, modestly achievable goals and build from there. Collaborate and more tightly integrate the teams.

  • What Cisco’s OpFlex means for DevOps

    What Cisco’s OpFlex means for DevOps

    Some big news in networking land at Interop this year revolved around Cisco’s latest contribution to open source and the Internet, its proposed “OpFlex” control plane protocol.

    The protocol itself is focused on communicating with network elements and specifies encoding in a variety of formats, including JSON, that are developer-friendly. That’s nice, of course, but the big question as someone interested in operations and devOps is why should you care about what appears to be a networking protocol? After all, beauty is only interface deep. What’s in it for you?

    To see where DevOps fits into the picture, consider the description of OpFlex as specified in the IETF draft of the OpFlex protocol:

    The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults. This document defines the OpFlex Control Protocol.  [emphasis mine]

    What’s being offered is not just a protocol for bidirectional communication, but an entire architecture for sharing policies and actionable data. What’s more important to those who for whom “networking” is not their native technical tongue is that policies are designed to be declarative and abstract. That means instead of issuing a set of commands, often specific to the device or service, a policy declares (describes) what’s needed and each component participating in the architecture interprets those needs according to their specific function and capabilities.

    You might be thinking at this point (I know I certainly was) “What is this policy and who is defining it?” because that seems pretty central to the value proposition of this whole endeavor.

    For that, we need to look to the OpenDaylight (ODL) project which has created a new project called the ODL Group Policy plug-in. As explained by Cisco in an OpFlex white paper, “the goal of this project is to provide a policy-based API that can serve, in practice, as a standard information model in OpFlex implementations.”

    Jumping over to the ODL Group Policy Plug In wiki, then, the intention of the Group Policy Plugin is to introduce “the notion of groups of endpoints and policy abstractions that govern communication between these groups. The goal is to support a northbound API that accepts abstract policy based on application requirements while offering numerous concrete southbound implementations for programming network element with configuration/programming stanzas rendered from application policies.”

    Okay, so that’s a whole lot of .. words. What it boils down to is that instead of a policy describing a set of rules governing routing with the appropriate ACLs, one might simply declare that “Web Server A can talk to Database Server A to perform database transactions” while “Administrative Console B can talk to Database Server A about administrative tasks.” If you were to translate that into specifics you would likely have multiple imperative command lines, each describing VLAN participation and routing  as well as entries allowing traffic between specific VLANs or IP addresses on the ports appropriate to the application layer protocols. It would be long and prone to misconfiguration simply due to the number of commands required. Using a declarative (abstract) policy language simplifies not only the policy but also addresses the disparity inherent in competing products across a given market by eliminating it from being problematic.

    The problem with traditional proposals to solve the “different definitions and object models” across a given market space (like ADCs or IPSs or switches) is that they’ve all relied on finding the lowest common model, which inevitably ended up with all offerings needing to dumb down (commoditize) their value in order to ensure consistent support for all “XYZs”. By abstracting to the policy – and not the object model – level, the OpFlex approach effectively eliminates this issue and emphasizes defining what is desired by the application, but not how.

    Now, the actual policy declaration is unlikely to take the form of natural language (that’d be nice, but we’re not quite ready for that… yet, though Watson might disagree with me on that) but the general premise is to provide a far more, shall we say, friendly policy language.

    What this means is the operations folks most likely to be familiar with the application architecture and who understand what it needs and how it needs to work, could be the ones to actually codify that into policies. You’ll note that the intention for OpenDaylight, at least, is to “support a northbound API that accepts abstract policy based on application requirements.” That means someone – hint, hint, that’s you – might be able to integrate into the application deployment lifecycle provisioning of the network services and topological constraints required based on an understanding of the complex web of connections between each piece of an application. Someone who knows from experience that a “three tier architecture” means more than just three components, and can more effectively describe how they should be interacting.

    But that’s just the beginning. You’ll note that in the RFC summary above it specifies “bidirectional communication of policy, events, statistics, and faults.” That means actionable data is being exchanged between participating components. Actionable data enables, well, action. Actions that might address a fault that occurs between end-points. Actions that might address performance issues. Actions that might mitigate attacks in real-time.

    Ultimately this level of capability will drive new solutions – new workflows and processes – that take advantage of event-based systems and policy-driven networking. OpFlex should enable statements like “When X happens, do Y” to be executed upon, even if it requires coordination of three or four or more network components.

    What OpFlex means for the devops practitioner is a way to influence (or, one hopes, specify directly) what services the network provide for the applications you’re tasked with deploying and delivering, and how the network responds to the things that happen on any network that impact the performance, security or stability of those applications.

    As networking continues to recognize and address the need to provide application-driven services, including provisioning and real-time adaptation, it’s going to continue to become more programmable and integrated with the systems driving a more automated, orchestrated data center. And that means devops practitioners are going to have more and more options as to how they manage the entire application (and network) service lifecycle.

  • Defeating the irrational DevOps jobs displacement fear

    Defeating the irrational DevOps jobs displacement fear

    People are generally resistant to change, but when that change comes accompanied with the whiff of potential for job loss down the road you can count on gentle inertia to snowball in to out-and-out opposition.

    Unfortunately, as early misperceptions swirl around the DevOps philosophy, many in the ops world have gotten their backs up in fear that the automation that’s tied to DevOps will endanger their jobs.

    “I think it’s almost impossible to overstate the amount of angst that  DevOps creates in operations,” says Gene Kim, author of The Phoenix Project. “The big fear  in operations is that in order to get to lead times not in months, not to weeks, but to days, you can’t be  manually configuring servers, you can’t be manually reviewing things all of the time.  And so there’s a deeply held fear that these configuration management tools are going to automate everyone’s job away.”

    Now, to the degree that some within IT operations ranks rely solely on large vendors to set the course of their career track, maybe there’s some jeopardy there, Kim says, but competent and capable ops staff needn’t fear the automation reaper.

    “It’s a reasonable fear, but it is ridiculous at  the same time because we all have so much more work that we could get done if we had the time,” agrees David Mortman chief security architect and distinguished engineer for Dell Enstratius. “Doing things manually is negative value for most organizations.”

    And that negative value is having a direct impact on how IT services the business. In the mind of Shridhar Mittal, general manager of application delivery for CA Technologies, one of the big DevOps value propositions is that it helps IT from CIO on down to finally start to meet the pent up demand that the business has been clamoring for from the IT department.

    “I’ve yet to see a CIO tell me that they have more supply than demand. Why do you think the business is heading out to directly work with cloud vendors and doing their own thing? It’s not because the IT said no so that 10 people could sit on the sidelines,” he says. “It’s because these projects are added onto the list of 1,500 other things they got asked to do at once.”

    Mittal says that CIOs who are gung-ho for DevOps should be proactive about potential resistance from the trenches. This starts with open communication.

    “The first thing is start to educate staff that this isn’t about eliminating jobs it is about reducing that gap between demand and supply,” he says.

    And as IT leaders do garner more buy in, both they and operations staff will find that these workers actually become more indispensible rather than expendable through automation. DevOps allows workers that might have seemed lackluster to shine as rock stars because they’re more frequently doing higher value work.

    “Once you get rid of the garbage work or manual work, which is very time consuming and error-prone, you’re going to find that your engineering team is allowed to perform better,” says Nick Galbreath, vice president of engineering at IPONWEB.

  • Merging creation with operations: facilitated collaboration enables continuous delivery

    Merging creation with operations: facilitated collaboration enables continuous delivery

    So much has been written lately about the challenge of improving IT agility in the enterprise. The best sources of insight on why this challenge is so difficult are the CIOs, application owners, ecommerce and release engineering executives, and VPs of I&O, grappling to change their organizations right now.

    At a conference I attended recently, I met two Fortune 100 IT executives from the same company: one the head of development and the other operations. Their story is emblematic of just how hard this is in the real world. As interesting background, both the development and operations leaders were childhood best friends, participated in each others’ weddings, and spend time together socially on an almost weekly basis – but by their own admission, even they couldn’t get effective collaboration and communication to work between their two organizations.

    The lesson learned from this example is that the DevOps collaboration and communication challenge cannot be solved by sheer will, desire or executive fiat. Instead, you must breakdown the barriers that inhibit collaborative behavior and facilitate new ways of communicating and working together. The old standbys of email, instant messaging, SharePoint sites, and conference calls don’t cut it.

    The challenge of two opposing forces: Dev and Ops

    Imagine yourself helping your children put together a new jigsaw puzzle. Each time you turn your attention to a specific piece, the kids reorganize what you have already completed and they add new pieces, but in the wrong places. For sure, three pairs of hands can be better than one, but they can also create chaos, confusion and significantly elongate the completion of the puzzle.

    The collaboration challenge in the DevOps movement is grounded in this analogy. How do you get multiple people working together across teams, locations, and time zones to build and get things deployed faster without chaos, confusion, and delay? How do you get these teams to speak the same language and collaborate together with a singular purpose when their priorities and motivations are so different?

    Faced with this challenge, it’s easy to see why many organizations have stayed in their comfort zone of ‘waterfall’ releases and keep the number of releases per year small. The issue is this method isn’t meeting the demands of the business, the market and competition. As a result, more and more business leaders are going around their IT organizations. Options like public cloud, SaaS, open source tools, skunk works IT and outsourcing are making it easier for them to control IT decisions and implementations within the business unit or department itself. 

    So let’s dive deeper to understand the two forces at the heart of the issue: development (focused on the creation or modification of applications to support a business need) and operations (delivering defined services with stability and quality). It appears these forces are working in opposition, but both groups are focused on doing what leadership asks of them.

    Developers tend to think their job is done once the application is rapidly created, but it’s not because no actual value has been delivered to the business until the application is operational and in production. Operations is severely disciplined when services experience performance and availability issues and they have come to learn that uncontrolled change is the biggest cause of these issues.  As a result, operations teams often believe their number one job is to minimize change to better control impact on performance and availability. This causes operations to be a barrier to the rapid change required to give the business the speed and agility it needs.

    Critical to enabling DevOps is an explicit recognition of this situation and the ability to link discrete phases of the application development and operations lifecycle to enabling ‘fast, continuous flow’ – from defining requirements, to architecting the design, to building the functionality, to testing the application, to deploying the application to both pre-production and production environments and to managing all the underlying infrastructure change required for the application to operate efficiently and effectively in all environments.

    Why current approaches don’t work

    There are several challenges in achieving this ideal.

    1. Developers hate to document (can you blame them?), and, when they do, their communication is in a context they understand, not necessarily empathetic with the language that operations speaks. The view from operations is that the documentation they receive is incomplete, confusing, and/or misleading. With the rapid pace of development, this challenge is getting worse with documentation becoming more and more transient as developers “reconfigure the puzzle” on the fly.

    2. Today’s operations teams typically take responsibility for production environments and their stability. That means there is usually a group wedged in between the two – the quality assurance (QA) team. QA’s job is to validate the application works as expected and often they require multiple environments for each release. This group is typically juggling multiple releases and is, in essence, working on and reconfiguring multiple puzzles at the same time. The challenge of keeping QA environments in sync with both in-process releases and production can be maddening (just talk to any QA leader and they’ll tell you first-hand). The documentation coming from development is inadequate, and the documentation coming from production is often no better, since most operations teams store much of their most current information about configurations in personal files or simply in their brains.

    3. The ad hoc handoffs from development to operations and QA take time away from development’s primary mission: creating applications that advance the business. Some suggest developers should operate and support what they code in order to reduce handoffs and the risk of information distortion or loss. A fundamental risk with this approach is opportunity cost. Does a developer really understand the latest and greatest technology available for infrastructure and how to flex and scale those technologies to fit the organization need? Do you even want them to or would you rather they be coding instead?

    4. Others have suggested that Operations move upstream and own all environments from dev to QA to production, and treat configuration and deployment scripts as code just like a developer would. This may sound like a good option, but it can create a constraint on your operations team and cause valuable intelligence to become hidden in scripts. A particular application deployment could have one or more software packages required and potentially hundreds of different configuration settings. If all that information is embedded in a script, how will other team members know this if they go to make a change to the underlying infrastructure to apply a security patch, upgrade an OS version, or any of the other changes made in IT every day?

    Real DevOps transformation doesn’t mean that you give everyone new jobs, instead, it’s about creating an environment where teams can collaborate together with a common language and where information is immediately available at the point of execution and in a context unique to each team.

    A better way forward

    In The Phoenix Project, written by DevOps thought leaders Kevin Behr, Gene Kim and George Spafford, the authors promote the need for optimizing the entire end-to-end flow of the discrete processes required for application delivery – applying principles as they achieved agility in the discrete manufacturing process.  

    Manufacturing in the 1980s resembled IT operations today, employing rigid silos of people and automation for efficiency and low cost, but this became a huge barrier to the agility, stability and quality the market demanded. They learned if you optimize each workstation in a plant, you don’t optimize for the end-to-end process. They also learned that if quality and compliance processes were ancillary to the manufacturing process, it slowed things down, drove up costs and actually decreased quality and compliance. 

    Successful manufacturers brought a broader view and optimized end-to-end flow rather than operate in a particular silo. They also brought quality and compliance processes inline with the manufacturing process. By addressing quality and compliance early in the cycle and at the moment that an issue occurred, cycle times decreased significantly, costs plummeted and quality and compliance increased dramatically.

    These same principles can be applied to IT resulting in:

    • faster time to market;
    • greater ability to react to competitive pressures;
    • deployments with fewer errors
    • continuous compliance with policies; and
    • improved productivity.

    DevOps can best be realized when IT operates in a social, collaborative environment that ensures all groups are working with a visual model in their context with the necessary information from downstream and upstream teams, as well as in collaborating with relevant experts at the moment when clarifications are needed or issues arise.

    To merge creation with operation, the core idea behind DevOps, requires a cultural change and new methods in which cross-functional teams are in a state of continuous collaboration as they deliver their piece of the puzzle at the right time, in the right way, in-context with the other teams in other silos. Operating something that never existed before requires documentation so that operations teams have the information they need to manage change with stability and quality.

    With more modern collaboration methods, self-documenting capabilities are now possible as development, release and operations do their respective jobs, including visualization of documentation with analytics and with the perspective and context each team needs to effectively do their job downstream. These types of capabilities will transform organizational culture and break down barriers to collaboration that impede agility, stability and quality.

    Is this simply nirvana, unachievable in the real world?  No. Manufacturing achieved the same results by applying these principles; the fundamental point being made in The Phoenix Project.

    The goal is not to write code or to keep infrastructure up and running, or to release new applications or to maintain quality and compliance. Instead, the goal is for IT to take the discrete silos of people, tools, information and automation, and create a continuous delivery environment through facilitated collaboration and communication. This will drive the cultural and operational transformation necessary to enable IT to respond to business needs with agility while ensuring operational stability and quality.

  • Q&A: Speaking DevOps and Threat Modeling with the author of Threat Modeling: Designing for Security

    Q&A: Speaking DevOps and Threat Modeling with the author of Threat Modeling: Designing for Security

    If you want to understand how to threat model systems and applications in most any environment you turn to someone who has done so. That’s why we took 30 minutes to speak with Adam Shostack. Adam is responsible for security development lifecycle threat modeling at Microsoft and he is one of the very few threat modeling experts in the world. We thought, with more and more organizations moving toward DevOps and Continuous Deployment, would be a great time to discuss how this impacts the design of secure software. Earlier this year Adam published his book, Threat Modeling: Designing for Security, available here. The book provides pragmatic and actionable advice on how to make security an integral part of systems, software, or services from the outset.

    This advice applies to systems and development teams both big and small and is especially crucial as application deployment times accelerate.

    staging-devopsy.kinsta.cloud:Why threat modeling? What benefits does it provide the business as well as the application development and operations teams?

    Adam Shostack: For the business, executives, they don’t want to deal with individual problems as they arise, they want to address problems as a category. They want to manage a class of things that can be completed in a consistent, repeatable, or predictable way. When it comes to managing application risks, threat modeling gives you an opportunity to answer questions as to whether you are dealing with certain vulnerabilities across all of the applications – rather than coming at them as one-offs. It’s a way to model and abstract away many of the details and look at things in a more strategic fashion.

    The other value threat modeling provides is the degree of predictability as you’re developing things – instead of forging ahead and building things and then running a pen test at the end only to learn then that you have serious security problems. What ends up happening in that scenario is that once you fix those problems, you then run another pen test and learn of even more or new security problems and you end up in that very unproductive cycle for a very long time.

    So threat modeling helps you think about your security before you start building. This way you can bake security into what you’re doing. My book helps to make both of those value propositions concrete and approachable. You can take these lessons in the book, give them to people and say go do these things and you really get an increased degree of control and predictability through what you’re designing through what you’re delivering.

    Also, when you threat model, you get a picture of where things are likely to go wrong. A lot of the threat modeling practice can be very process heavy. That’s why there’s a lot of very process heavy documentation out there. But actually threat modeling integrates well inside a test driven design approach to software development. Why? Because threat modeling tells you where you need to concentrate your security testing. If you’re looking for bug classes, don’t look at random sets of things or relatively random sets of things and you won’t get nearly as much test coverage.

    staging-devopsy.kinsta.cloud: So, they can operationalize secure development into the processes, which is what security advocates have often spoke of as the optimal state of security’s relationship with application development and operations – rather than be a barrier at the end of the development process.

    Adam Shostack: The whole barrier meme comes from the idea that you do security activity late. A lot of people come into security through hacking. You hear the hacking metaphors all of the time. They are memes like “you can’t secure something if you don’t know how to break it.” Now, knowing how to break things is great. It’s a fun skill. It’s an important skill set. But you notice when you ask about making it operational is that one of the things that security people love to say is you’re never done threat modeling. Now, if you go to a VP of development or a VP of Ops or a VP of DevOps and say, “I want you to do this thing and it’s never done, but it’ll fit into our four week sprints….” You will get a big “huh?” The conversation won’t go anywhere from there.

    In order to work it has to be complete-able. It has to be focused and it has to be prescriptive with checklists. It has to have exit criteria, and it has to have metrics that answer: “Are we doing this well?”

    staging-devopsy.kinsta.cloud: That type of thinking strikes me as fitting well with the continuous improvement objectives of DevOps.

    Adam Shostack: Absolutely. I’ve been threat modeling in various ways for a good 20 years. When I joined Microsoft I encountered people in development and in operations who were passionate about security, who knew that they had to do security and they wanted to do it right. But they also wanted to focus in on reliability, on accessibility, on internationalization, on performance. They wanted to focus on all of these other things each of which is an important property of the system you’re developing. I had to learn to make threat modeling easy. I had to learn to focus in on the highest value deliverables and make sure that those deliverables were accessible and capable. This has to be so even when you didn’t have a security expert groomed because people don’t understand that their decision has a security implication. They’ll make a decision without a security person handy. If you make the security goals accessible and understandable, they’re more likely to understand that what they’re doing has a security implication and they’re more likely to do it well.

    Let me give you an example of this. One of the areas that most often goes wrong in the design of a system is people don’t draw their trust boundaries. They don’t know where different principles are going to interact with one another. And application accounts with different authorization levels interacting are a natural place for security problems to arise.

    So what you can do is draw these trust boundaries on a white board. It doesn’t have to be a fancy diagram. Scribble on a white board and say, this is where things happen in admin. This is where things happen in the database. This is where things happen as the web user, and this is where web sessions are authenticated and you put boundaries between each of them. Developers understand that these things that they’re writing are run at different privilege levels.

    They then say, okay, this code runs on the web user context, this code runs on the database context. That’s great because once you’re doing that you can see the places that your threats are going to arise and look for places to exploit. There are lot of elements in threat modeling that you can work toward incorporating, but the fundamentals are actually easy to teach, easy to learn, and they’re easy to apply.

    staging-devopsy.kinsta.cloud:We started out by talking, from a high level, what threat modeling means to different constituencies in an organization. More concretely, what is the payback of threat modeling for developers? Is it more resilient applications, fewer problems that have to be dealt with after the fact, and generally improved outcomes?

    Adam Shostack: Those are great values. The biggest “what’s in it for me” for a developer is that threat modeling takes all of these details about security that they want to deal with and provides a model, an abstracted way to frame them. So, instead of constantly worrying “Gosh, I might have another security issue” you can get a sense for how much you’ve done to solve the risk, and how much of it might remain. You get a sense of where you want to prioritize that work.

    I think a big misconception is that people believe that only an expert can do threat modeling. One of my aspirations for this book is to show that threat modeling should be like version control. Every developer, every ops person should know a little bit of threat modeling. And it doesn’t have to be a security expert. Perhaps it’s the person who manages your source tree. Or, if you work in a small organization threat modeling will be a side job for someone and that’s okay too. The important thing is to make it a part of your work.

  • DevOps killed the developer star

    DevOps killed the developer star

    The DevOps blogosphere is aflame over Jeff Knupp’s post on “How DevOps is killing the Developer”. While I agree with Knupp’s premise that DevOps, Agile and other technology trends are allowing (is forcing more accurate?) developers to do more outside of pure development, even turning into “full stack” developers, I disagree that this is a bad thing.

    I disagree with Knupp’s assertion that developers are the Orcas of the IT world sitting atop the food chain as the top dogs.  I think Knupp consciously or not, degraded the very real, important and hard job that ops people perform, DBAs do every day, as well as everyone else at an organization. Reading this I was reminded on one time I was sitting in an executive meeting and the VP of engineering actually had the stones to say that his developers were smarter than the rest of the people in the organization and worked harder to boot.  Now that is a real team builder right there.

    One thing I have learned in 30 years of business is that when an organization is achieving its goals it is because just about everyone is working as hard as they can. It is usually not one group, whether it be developers or marketing or operations that is pulling the sled alone.  It takes everyone to really succeed. There is no top of the totem pole or rather who is on top is a constantly changing position as bottlenecks come and go, priorities shift and the market changes.

    In any organization whether it be startup or enterprise, we all have to do what we need to do to accomplish the mission.  There you have it.  Working in the Federal market all of those years taught me something. In the DoD and armed forces it is all about the mission. We have missions as individuals. Those missions can be writing code, deploying or monitoring (heck even support and sales), but ultimately those missions all fold into the greater mission of the organization.  If developers doing full stack work helps the organization better accomplish its mission and goals, than it is good. No man is too big for the army.

    At its core that is what DevOps is too don’t forget. Empowering IT to better serve and meet the organizations goals. Not about letting developers do what they love to do or what floats their boat or what they consider the “top of the totem pole”.  Enterprises are here for one thing, the bottom line. It is something we should all never lose sight of.

    DevOps is changing the mission of developers. It is a change in the environment if you will. Species are faced with changing environments all of the time.  Those that adapt to change succeed and even flourish. Those that fail to adapt fall into the extinction bin.  Generally species that are generalists adapt and thrive much better than species that are too specialized. This is basic evolution theory.

    I think to Mr. Knupp’s point we are going to see a form of DevOps Darwinism play out. Developers who insist they only develop and don’t do full stack or other DevOps type of tasks will find shrinking habitats and less opportunity. They will exist on the margins. Developers who embrace this new environment will find abundant “target rich” environments with rewarding (both monetarily and stimulating) career opportunities.

    I am all for doing something you love for your work. Otherwise I would be practicing law somewhere. I stared working in the technology world because playing with computers was the most fun I was having. I am sure many of you had the same experience. Messing around with code or scripting moved from fun and something you did because you liked it to something you also do to earn a living.

    Earning your livelihood doing something you love is a great blessing by the way. There are untold numbers of people who trudge off to work every day and have to do something they dislike, are not passionate about and do just for a paycheck. I can’t imagine having to live life like that.

    I think that is what Knupp is talking about too. That developers code because they love to, they need to, and it is what they do. Forcing them to do other tasks takes away from their focus and stunts their growth, which in turns comes back to hurt the organization as a whole.

    I don’t think developers will ever stop coding. Mr. Knupp is right, developers code because they want to. That is what they love to do and it is why they became developers.  I think however they will have even more opportunity to code, not less as software continues to gobble up the world. It may be a different kind of development, with coding as infrastructure as software dominates, but developers are still going to code no matter what.

    We may be seeing the rise of the coding generalist versus the pure application developer as the lines between developer, ops, QA and yes even security continue to blur and overlap. But I don’t think it is a bad thing for developers, enterprises or anyone else. It is just part of the constant change in our environment.

    I don’t think this change is bad at all. In fact to paraphrase the great Gordon Gekko, “Change is good, change is what made America great.”