staging-devopsy.kinsta.cloud

Category: Features

  • Q and A with Carmen DeArdo: DevOps at Nationwide

    Q and A with Carmen DeArdo: DevOps at Nationwide

    Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.

    DeArdo is no stranger here at staging-devopsy.kinsta.cloud, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.

    Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.

    staging-devopsy.kinsta.cloud: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?

    DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.

    What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.

    Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.

    These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?

     

    staging-devopsy.kinsta.cloud: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?

    DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.

    But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”

     

    staging-devopsy.kinsta.cloud: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.

    DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.

    We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.

     

    staging-devopsy.kinsta.cloud: What are some of the high-level challenges that you’ve had optimizing that last mile phase?

    DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.

    So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.

    I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.

    One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.

    That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.

    That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.

     

    staging-devopsy.kinsta.cloud: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.

    Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?

    DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.

    DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.

    Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.

    But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.

  • Composable Environments for Continuous Delivery

    Composable Environments for Continuous Delivery

    The Continuous Deployment model discussed below requires some very specific properties for its platform substrate. Currently AWS is the the only environment that supports all the features needed to implement this. So when we start talking about implementation details in this, and future articles, we will be targeting AWS.

    The title of this article is actually reversed from what it should be. This article isn’t as much about using reproducible Composable Environments for Continuous Delivery (CD) as it is about using Continuous Delivery to build reproducible Composible Environments. The Composable Environment is the important piece. Why? To understand that we need to understand what such an environment is. An Environment is a self contained, full software stack that includes every component needed to support the stack along with a version controllable description of that stack. For example, an Environment for a website might be the all of the machines that run the HTTP Servers, other machines that run the RESTful Web Servers, databases, AMQP providers etc. It would also include any infrastructure required for auto scaling, fault recovery etc. This Environment would be completely described in a declarative way by one or more files that are under version control. That description contains every virtual machine image, database schema, DNS server, DNS entry, etc needed to standup a system in a self contained, replicable form that can be put under version control. This description is versioned in a sane way and reproducible through the execution of a single command. A good example of this concept is Cloud Formation Stacks. Stacks declaratively describe a full environment and the Cloud Formation infrastructure takes those stacks and, on request, vivifies them into full running system. Our concept of environment takes that further with ideas around isolation of environments and accessibility via DNS, but these examples provide a good seed for the concept we are describing.

    Reproducible Composable Environments are a simple concept that has profound implications. A few of the most important are:

    • A completely reproducible environment is self contained. Multiple instances of that environment don’t interfere with one another.
    • A declaratively described environment can be changed algorithmically. That is, it can be easily changed via some automated process.
    • An environment can be deployed algorithmically can also be un-deployed algorithmically.

    Once we have created an environment with these properties, we can use it to accomplish things that where not previously possible. It opens up new opportunities through a combination of Testability, Composability and Deployability. You might notice that this sounds like the move towards Structured Programming that happened in the programming community in the 1960s. That transition had a profound and continuing impact on how we create software systems and I think this approach to Devops will have a similar impact on how we create and manage environments.

    Why Environments are Important

    These environments are useful through out the software development process. Start with the individual developer writing and iteratively testing the system. Having such an environment allows him to truly test the changes that he is making by standing up a full environment, including all its participating parts, and testing that system in context. The next big place for using such a system is during Continuous Integration, henceforth CI. At the end of the build cycle the CI system creates an environment, publishes the changes into that environment, stands up the complete system and runs the full test suite for the entire system. The ability to standup the full system in context massively reduces risk in rolling out changes. Being able to roll out, at need, an environment with updated components in an ad-hoc or automated manner improves the testability of our system.

    Automated processes are the correct method to build and modify environments, and are required for Continuous Delivery.

    Continuous Delivery

    I define Continuous Delivery as a idempotent, automated process that is primarily designed to update a reproducible Composable Environment with new, well tested artifacts.

    We can think of Continuous Delivery as a process with a set of well-defined inputs and a set of well-defined outputs. The process itself is composed of a set of stages, each of which have its own inputs and outputs. This allows us to compose stages and even whole CD processes into different processes that serve new purposes.

    For the simplest CD process, the main input is source code and the main outputs are the services that are stood up as part of that system. To be more specific, the outputs are a set of endpoints to which you or your customers connect. In examples I will provide, these will take the form of DNS entries. Knowing the inputs and the outputs, we can now start to think about the process in between.

    The Stages of CD

    The standard CI pipeline is composed of five parts: Develop, Test, Publish, Deploy and Promotion

    Stage: Develop

    The Develop stage is essentially the individual developers development process. Its only input is the output of the CD process that automatically creates and provisions the Development Box as well as the related environment. This is the first example of a CD input being the output of another CD process.

    The reason Development box is necessary is so that the developer can test his work in an environment that exactly matches that of the CI server. It should also exactly match the system that every other developer is working on. This is critical, and often overlooked. Currently, developers rarely write code in development environments that match each others, let alone ones that match the CI server.

    Stage: Test

    The Test stage is where we take the artifacts produced in the Develop stage and test their viability in context. The output of the test phase is usually a more complete set of artifacts published into the environments.

    These Stage usually takes the form of taking developer code, compiling it, turning it into some package-like structure and publishing it to an environment specific known location. The structure is usually an actual package: a LXC/Jail container like a Docker image, a virtual machine image like an EC2 AMI, or some combination of the three.

    During the test phase, the target environment is stood up with the new artifacts, and a series of integration/release tests are run. Test reveal the viability of the changes within the context of the environment as a whole.

    Stage: Publish

    The Publish stage takes the output of the Test phase and publishes those artifacts in a way that makes them available to the Deploy stage. This usually involves putting the packages in a publicly accessible place.

    You may have noticed our heavy reliance on some packaging model. That is intentional. You need turn your system into a manageable unit in order to make it work correctly as part of the process.

    Stage: Deploy

    The final stage is Deploy. This is where the published packages are used as inputs to update the environment. The output of this stage is the canonical environment updated and vivified with the newly published artifacts. The running systems, provisioned databases, accessible URLs, etc are all updated in place.

    Stage: Promotion

    Finally we have the optional step of promotion. This simply involves moving the artifacts outputted from the Deploy state to a new environment. A common example of this is the promotion of a beta environment to production, where some secondary process has been run that has validated the environment making it available to consumers.

    Overcoming Technological Constraints

    I am not going to lie to you. It is non-trivial to get started with this process. Up until a very few years ago, I would have said it is impossible. But, the world has changed, and it is now possible, even approachable. There are some prerequisites that we have to get it working. You must have the ability to programmatically configure your network, configure your DNS, provision hardware etc. I can accomplish this with a mix of new technologies like OpenStack, OpenFlow and customized DNS among other things. Most of us don’t have the ability to implement those technologies in our data center. Fortunately, we have AWS. Between EC2, Route 53 and especially CloudFormation, we have nearly everything needed to make this happen. Over the next few articles, we are going to take the model described here and apply them on a production system. At the series conclusion, you will be able to apply this model in your own AWS-based environments. You should still be able to adapt and apply parts of this model in non-AWS environments, as well.

    And, my team will keep implementing and refining our ideas for composable environments for continuous delivery towards having a more complete solution. Stay tuned!

  • Donnie Berkholz on IoT and the Modern Developer

    Donnie Berkholz on IoT and the Modern Developer

    It’s been nearly four years since Marc Andreessen wrote his famous column Why software is eating the world. It’s proven to be incredibly prescient. While software and the Internet has removed many market frictions from many businesses, software is now just starting to “eat” the physical world as adoption of the so-called Internet of Things accelerates.

    Before his talk, IoT and the Modern Developer, at IBM InterConnect, we caught up with RedMonk senior analyst Donnie Berkholz to discuss how software is in fact now eating the physical world and how the Internet of Things is now more accessible than ever before, and like the description of Berkholz’s talk says, “We live in a world of glue code, where open building blocks are available to Developers to snap together the pieces of the next big thing.”

     

    staging-devopsy.kinsta.cloud: Thanks for taking the time Donnie. Can you tell us a bit about your talk, IoT and the Modern Developer?

    Berkholz: The idea is that IoT has been really hard to work with for a long time because a lot of it has required working with the Arduino tool set, for example, which is a lot of C++ and a lot of developers these days aren’t very interested in working in low level languages anymore. A lot of them want to work in Python or maybe Go or Node, or some form of JavaScript. The challenge for a lot of these embedded devices has been that they’re just really inaccessible to the way people want to write software these days both from a perspective of languages, but also toolsets. For instance, the library support just is not great.

    It all kind of ties together into the same idea of IoT hadn’t taken off for a long time. Not from a developer perspective. Independent developers weren’t doing a lot of playing with, largely because it was a pain in the ass to work with. Whereas more recently we’ve seen stuff coming out like the Raspberry Pi, and the BeagleBone. The Espruino is a really good example and as is the Tessel of this new generation of IoT that is easy to work with and is super affordable, and you’re working with it, in most cases, in Node. Every one of those more current boards supports running Node on the embedded board itself which is a huge change.

    The old point of view used to be having JavaScript on the server and there was a central server that was dealing with all this stuff. But the new perspective is that all these embedded boards are powerful enough that you can actually run JavaScript on every single one. There’s never a point at which you have to interface down to some low level language that you’re not interested in working on.

    That’s basically the point of the talk and it obviously goes into a lot more detail with lots and lots of examples of what’s happening, how the trends have changed over time.

     

    staging-devopsy.kinsta.cloud: That is very interesting, do you know of real-world examples?

    Berkholz: One that I particularly enjoyed is the Internet of toilets. There’s a good company called Hagleitner. They are doing smart restrooms. Big data with washrooms is how they frame it. They’ve Raspberry Pis sitting in these washrooms running, it was at an F50 conference, they’re running SQL Anywhere and integrating with the HANA Cloud platform. And they’ve all the kinds of instrumented paper towel dispensers and that sort of thing so that they can get on-demand maintenance instead of just basically pulling-based maintenance. Obviously it’s the way we want to write software. It’s the same way. It would be great if we could do that in the real world.

     

    staging-devopsy.kinsta.cloud: Where do you think IoT is heading in the next years ahead?

    Berkholz: We’ve run a show called IoT at Scale. We ran one last fall. We’re going to run another one early summer in Europe. The last one was in Palo Alto and the basic premise is that the IoT that’s real today, the IoT that’s easily commercializable today is industrial IoT. With enterprises, all you have to do is show that there’s ROI there. You have to show that if they invest in this instrumentation, they’re going to save money. They’re going to lower risks.

    Whereas with consumer IoT, it’s basically a guessing game. You’re gambling every time you do and sometimes you win, but a lot of times you lose. And so the talk runs through all kinds of examples of what’s going on with industrial IoT and in retail, in construction, in smart buildings, in connected cars, fleets of trucks, all sorts of different things like this. In every one of those cases, it’s a commercial business that’s making money right now and keeping companies alive.

     

    staging-devopsy.kinsta.cloud: That is great to see. Going back 13 years or so at InformationWeek, we actually had a beat called Telematics and it was about covering all of that. It was mostly transportation, but it was so far ahead of the time that we couldn’t find real use cases so we shut it down.

    Berkholz: Exactly. That’s the thing that we’ve been waiting for is all the vendors have been suggesting use cases for years and they think wouldn’t it be cool if somebody did this? That would be awesome, right? But they could never come up with any customers and I think that’s the big change is a lot of these examples are actually customers that have developed their own solutions using vendor products as part of them. GE is a reasonable example. They built their own software stack for IoT and they hit a billion dollars in revenue last year. If that’s not a proof point, I don’t know what is.

     

    staging-devopsy.kinsta.cloud: That’s a great proof point. How do you see this affecting the work load of operations and developer teams? Are they going to have to manage an IT layer of physical devices?

    Berkholz: Yes. It’s definitely going to require extending the DevOps ideas of collaboration to field teams as well, to utility workers and that sort of thing that are out there actually maintaining smart meters. The equivalent of your rackers and stackers in the data center are these field techs who may not be well educated in the technology at all, but still have to understand enough to maintain the hardware on premise and to manage to do software updates. I think that’s one of the critical aspects and certainly one that probably resonates with you from a security perspective is all these embedded devices get sold and then either they’re never updated even though they’re connected or they’re updatable.

     

    staging-devopsy.kinsta.cloud: That’s exactly right. A lot of it feels like it’s 1999 and the Web all over again.

    Berkholz: Yes, exactly. The Internet was not architected with the idea of hackers in mind. We should learn from that lesson as we’re building up the IoT and as we’re creating and experimenting with protocols so that we can make sure we architect in security from the ground up instead of doing the same thing all over again.

     

    staging-devopsy.kinsta.cloud: What are the challenges in working with IoT management teams and building efficient Ops in this environment?

    Berkholz: Yes, I think it’s going to be a very different model. We’re going to have to take a step back in a lot of these cases because on the tech side of things we’re moving forward into more and more technological methods of collaboration, using more and more virtual, using tools like Slack or HipChat or others and relying on that to work together whereas a lot of these people are not at all comfortable with that. In many cases, they may not even use email. I live in Minnesota which is not exactly a software hub, as you, and I’m sure you know many people who may not own computers.

    I think a lot of it is going to have to rely on having field teams out there with much smarter devices themselves that are a step up from the deployed IoT stuff. Like the ruggedized laptops and PDAs that they’re carrying around. So there’s going to have to be a lot of intelligence on there that can cope with what’s happening out in the field and on the back end it’s obviously going to require very scalable and intelligent software to figure out what’s going on out there in the field.

    Techs need to know if alerts require action be taken, or if everything is all good. Because if I can’t figure out if I have to take action until later, that’s going to cost the company money to send somebody back there in person.

     

    staging-devopsy.kinsta.cloud: That brings up a great point. Mistakes here could be potentially expensive, potentially much more expensive than software mistakes because you’re talking about physical devices. Errors in planning or design can be monumental.

    Berkholz: It absolutely increases the risk profile. It’s the same thing that software on things like space shuttles, satellites, airplanes have had to cope with for years and years and years. That’s why in a lot of cases they have to use specialized software development techniques and languages, like Ada is a particularly popular one, to basically prove that they’re going to be reliable, to prove that the architecture is solid, which interestingly is the same kind of approaches that Amazon is taking with building its cloud. I don’t know if you’ve looked into what they’re doing with TLA+?

    They’re proving that their architecture works from a mathematical perspective. I think that’s the same approach that’s going to have to happen with IoT is we’re going to have to do a lot better job of building automated testing and of being able to rapidly roll back and roll forward and of using much more strongly protected languages and testing tools.

     

    staging-devopsy.kinsta.cloud: What things should developers who are looking at IoT for the first time now, as a potential platform to develop upon, be looking at or self-teaching themselves?

    Berkholz: I think for now, for the time being, the answer is the way we start a conversation which is picking up cheap embedded devices like BeagleBoards or Raspberry Pis and putting Node on them and playing with Node.

    That seems, to me, to be the place to be with the Internet of Things right now. Come up with a project that sounds interesting and do it.

  • DevOps & Continuous Change

    DevOps & Continuous Change

    A remark by a colleague while waiting for the coffee machine to complete its cycle started my train of thought. “Should we have multiple minor releases or just do a few major ones in a year?”

    In large organizations, due to many factors, the turnaround time for a single successful release is quite extensive; but if the conceptual change which we are talking about bringing whilst using the #DEVOPS methodology would not only reduce these long cycles but deliver quicker and quality software.

    I have already discussed in my earlier article “DEVOPS: Getting our organizations aligned” on the traditional approach and how the combination of this two (DEV & OPS) has been proven beneficial in various organizations and how many are still hesitant in embarking on that path.

    Continuous Change which encompasses both Continuous Integration and Delivery is paramount to achieving this objective. To attain the objective of delivering rapid and consistent value to the stakeholders regularly, we have to have Continuous Change happening.

    I am sure we have already moved out from the Mainframe era to a more Distributed systems state. But unfortunately we have yet to let go of that culture where we pile up components on a single quasi Distributed Application, instead of creating smaller independent, yet interdependent systems.

    So, back to the same question; shall we go with a few Dinosaur sized, labored releases or make multiple short and swifter runs? In my opinion, we should strive towards the leaner model. Easier said than done you say! Agree. If I have to, then I would go about it this way:

    In an environment having a suite of Distributed Applications, start with Analyzing the components which make up the Major release, List out the changes (fixes and new features) which are planned, asses functional and technological impact, Identify dependencies, upstream/downstream connectivity’s, architectural deficiencies, list out the identified subset of components or group of components by functionality, which can go as an independent release.

    I am resisting the urge to say ‘standalone’ as with the current set of complex intervened collection of applications, it would be an incorrect statement. It would rather be a subset of components which can be independently upgraded. Now the big task is to review and decide if this subset or group would add functional value to the overall suite of applications without the rest of components going in.

    Once identified, the success of this release working as expected would be in the extensive testing (Unit/ Regression/Performance) of these components, and as we gain confidence in this split and deploy procedure, we can start with, prioritizing and scheduling critical must have features or bug fixes which also should be backward compatible with other components lagging behind. We can even come up with scenarios where these components are not just getting updated frequently, but would also have multiple independent tracks. Of course it goes without saying the testing and signoff process is strictly followed albeit in a shorter window.

    Deployments to production have always been a source of heartburn to the OPS teams, we have often seen finger pointing and firefights to contain change fails. In order to maintain stability and maximize availability, stricter controls and reduced deployment windows are put in place. In promoting frequent and smaller component level changes, OPS will also be in control of what’s going in, and smaller component level releases can be rolled back quickly within the Green-Zone window if the change is not working as expected. Though I would categorize this rollback, as a process fail than a deployment fail, and should be reviewed very seriously with right mitigation plan and lessons learnt retrospection, Continuous learning after every successful or failed sprint ensures competent and confident releases.

    Another major factor impacting multiple runs of Production releases is the long Green-Zone windows, by moving to the quicker and leaner deployments cycles, these requests can be substantially reduced, and can eventually aim to reach a stage where changes are deployed online without extensive downtime.

    Similarly, frequent maintenance activities when there are no planned deployments, effectively has the same risk on environment unavailability. Analyzing and moving to a state where these activities are conducted online without a Green-Zone would alleviate the maintenance downtimes and ensure seamless availability, adding stakeholder value.

    I have not touched on the Continuous Integration or the Automation of the deployment process for obvious reasons, we can’t achieve the objective of Continuous Change without having the Integration process in place where incremental changes to code are build into a package swiftly and seamlessly, even a sanity or basic testing might be integrated to ensure the quality of the builds. Same goes for automating the deployments, develop & integrate tools required for rapid deployments. We have seen instances where automating a single activity or revisiting a process has created substantial time savings.

    While it’s easy to propose changes to the Release Process or advocate rapid deployment cycles, we have to ensure adequate checks and balances are in place, compliance adherence and audit trails are created.

    Now to the most important measure of this article, Cost Savings!

    Proponents of the Dinosaur Releases would argue that the common activities which entail a single release like the builds, Testing, Release Management, OPS availability or reduced GZ would cut costs and multiple runs to PROD would logically increase costs.

    But it’s definitely the other way around, we wouldn’t need to feed the Dino anymore, it will be a cost worthy rapid delivery with optimized resources, a leaner process where DEV and OPS are working together to ensure quality value added deliverables are made available to business faster.. This cultural change and collaborative effort also creates new vistas of growth and provides a perfect platform to excel. The cost benefit analysis would eventually be in favor of Continuous Delivery.

    Opportunity cost is another variable which has not been considered; an innovation or a new feature available to customers quickly with reduced time to market cycle will provide the much needed edge in these competitive times. I recall some conversations where a few must have’s were pushed out due to various reasons, frustrating the business.

    DEVOPS was never meant to replace the traditional approach to software development; rather it is an efficient use of the existing resources within a collaborative model to rapidly deliver quality software products. A major cultural shift in our approach is required and to reach that goal we need to embrace the shades of grays, many a times over 50 (sorry, couldn’t resist) when it’s not truly black or white.

    Thank you for the time.

    Visit my blog http://askhurram.com/

    Read On, Live On.

     

  • Five Key Components of Application Release Automation (ARA)

    Five Key Components of Application Release Automation (ARA)

    What are the fundamental building blocks of successful ARA?

    Application Release Automation, or ARA, is the consistent, repeatable and auditable process of packaging and deploying an application or update of an application from development, across various environments, and ultimately to production.

    Successful ARA eliminates the need to build and maintain custom scripts for application deployments, while simultaneously reducing configuration errors and downtime.  By providing a model-based approach to performing critical automation tasks, it effectively increases the speed to market associated with agile development and gives stakeholders the ability to coordinate and automate releases between multiple groups and people.

    In this article, we’re going to take a look at the five fundamental aspects that make up application release automation and talk about how each one serves a critical role in ARA’s success.

    Packaging

    The first tenet of ARA is packaging. Packaging is just what it sounds like – taking a bunch of parts and pieces and bundling them up together in a manageable and functional way. So why is this important and how does it fit within the context of Application Release Automation?

    Consider the following example: You and your company decide to switch offices, from your current building to one just down the street. You hire a couple of movers, get them in, and tell them you want to take all the stuff in your office and move it to the new location. Easy enough, right? So the movers start grabbing chairs, desks, computers – everything – and carrying them  down the street. They’re throwing things out the door, through the windows, everywhere! Next thing you know, a few chairs are broken. A laptop is missing. Your iPhone is gone! The movers say, “Hey, look, those chairs were already broken. I didn’t see any laptop on the table…” In this scenario, there is no bill of materials – just a bunch of confusion and some missing and broken parts. That is where packaging comes in.

    Packaging creates the bill of materials for your deployment. It is your way of knowing exactly what you need to move from point A to point B. You take all the pieces of your release and wrap them up neatly so they are ready to ship. It’s the method by which you can assure that everything that was supposed to make the move, in fact, did. Packaging allows you to confidently say that what you had at the source location is what ended up at the target location – and it is critical to a successful ARA process.

    Dependencies

    Dependencies are another critical cog in the ARA process. When you are working on your release, you need to understand what other applications or components are involved for its success. For example, your web application uses a database hosted on a different server. If the data in that database is changed, or columns are created or deleted, how will that impact your application’s functionality? This is a dependency – the way your application functions is dependent on the data in another location.

    In the case where you are pushing out an update to an existing application, you need to also take a look at how other things might have relied on the old application. Are you doing away with a feature or function that is going to have an impact on something else? This highlights how you need to not only examine what your application relies on to function, but also the mirrored function – what outside processes and/or applications might depend on your existing app.

    As you can imagine, the process of charting dependencies can grow exponentially. If application A depends on databases B and C, and C depends on D, E and F and then F… you get the idea. It can get very big, very quickly. This is why it is helpful to chart dependencies as you move, so you have a dependency map that is up-to-date and adaptable. You know where the dependencies are, how and what your updates or releases will affect them, and what you need to keep an eye on post-deployment. If you start charting dependencies at the beginning of your development, it will make ARA much easier down the road.

    Promotion

    Promotion in the context of Application Release Automation is the process of moving the application or update from one stakeholder to the next so that the release is as close to perfect as possible before it is deployed.

    This is a crucial step in the ARA process because it creates an audit trail that assures that the right people along the way gave this deployment the “OK” to move forward. Getting the input from multiple parties is imperative to a deployment’s success because each party – from the developers to release engineers to the Quality Assurance (QA) team – has a different perspective. For example, developers are looking at an application and saying, “How can I make this the best application possible with the most robust set of features?” While someone on the QA Team says, “How can I break this?” They are running your application through tests, wringing out inefficiencies and errors along the way.

    By passing (or promoting) your release through the chain of stakeholders, you can find the holes in your release before the customer does.

    Deployment

    Deployment is the act of launching your application and from point A to point B. Using the word “launch” is no mistake – it’s just like the commander of a space operation pushing the launch key. It’s pressing one (or maybe a few) buttons and releasing your application or update, configuring its operating settings, and letting it out into the wild. Deployment is making sure the release makes it to its destination safely. At its core, this is the fruit of the labors of ARA – letting your deployment free to do what it was designed to do!  The core objective here is that ARA moves this from a scripted, expert level process to a more model-driven perspective (where less skilled resources can understand and perform the deployments…).  So when applications, components, and dependencies are modeled within an ARA solution, companies now have the means to deploy their critical applications without having to rely on the superhuman resources who designed and built the application.

    Compliance

    Another key aspect of Application Release Automation is compliance. This is literally, “Are you following the rules of the game?” However, the rules of compliance vary wildly within different environments. At a small startup, with a team of three people making a web app – there are probably not too many rules to follow. The compliance necessary for a game launching birds at disgruntled pigs is minimal.

    At the other end of the spectrum are things like banking and government applications. Here, there are laundry lists of rules and regulations that need to be strictly obeyed or you may find your application being tossed – or find yourself in a CIA interrogation room (ok, we haven’t heard about that actually happening…but you can imagine!). With compliance, it can be helpful to start by asking yourself, “What do I really need here?” Once you figure out the requirements that are necessary for your specific application, you can map them out and discover what tools you can use to solve the problem.

    This is an essential key to ARA because if you have four out of the five steps above, but miss on compliance, all your work was a waste because it’s not going to play nice in its environment. Adhering to compliance – and documenting that you did – is crucial to successful ARA.

    With a strong understanding of these five key building blocks, you will have a solid base to start exploring the wonderful world of Application Release Automation. Go forth…and automate!

     

    About the Author/ Wesley Pullen  

    Wesley PWesley Pullen is the general manager and vice president of deployment solutions at Electric Cloud. Pullen leads the Release and Deployment business at Electric Cloud and is responsible for go-to-market strategy to aid in the company’s growth and expansion efforts around DevOps, Continuous Delivery and Deployment. He brings more than 18 years of experience in software development methodologies and design standards, applying this experience to both commercial and private sectors. Linked in: https://www.linkedin.com/in/wgpullen Twitter: @WesleyGPullen

  • eHarmony finds its perfect match with PagerDuty and DevOps

    eHarmony finds its perfect match with PagerDuty and DevOps

    Did you know that 4% of all marriages in the US are the result of people “finding their perfect match” on the eHarmony service? Keeping the systems that are making all of those matches up and doing it 24/7/365 is no easy task. Especially at peak load times like Valentine’s Day for instance. The pioneer in making online relationships relies on many DevOps tools including PagerDuty to minimize downtime and keeps its relationship science systems humming along.

    1672a00I had a chance to speak with Hav Mustamandy, Director of System Operations at eHarmony. Since Hav joined the team there about 7 months ago he has been busy modernizing the eHarmony network using a lot of tools that you would expect. They are a big Chef shop for example. PagerDuty is another tool they lean on a lot. More than tools though Hav has been instilling a sense of team between his Dev, Ops and QA teams.  In a short period of time, Hav has made tremendous progress in making the eHarmony IT business more responsive, reliable and faster.

    Frankly, why shouldn’t eHarmony act more like a Unicorn anyway? After all they are a web based business like many of the companies that we think of as being so successful. But before Hav came it was not necessarily so.

    Upon arriving he had to evaluate everything and everyone and start doing things differently. One of those things was the way they use PagerDuty. More than just an update to the old pager, Hav realized that PagerDuty could help him automate their incident resolution processes. Additionally using PagerDuty’s advanced analytics reporting, Hav was able to pinpoint problem areas in both systems and people. With the problem systems he was able to upgrade or change them. With people he realized where re-training was required.

    As a result Hav has seen some amazing results. The uptime of eHarmony’s systems even during peak load times has improved significantly.  Technical issues are identified and dealt with before they rise to critical levels.

    By leveraging APIs that PagerDuty makes available Hav has integrated several of his other systems into PagerDuty so he can gain insight into their performance as well. To him this is the future. Event management with PagerDuty at the core is what he is aiming for.

    For me PagerDuty and ChatOps applications biggest advantage is the ability to move inter-team communications into hyper-speed. If you are relying on email for communication it is probably not fast enough to keep up with your business.

    We have been using Slack here at staging-devopsy.kinsta.cloud for ChatOps and I can tell you that our communication now moves so much quicker than it did when we communicated strictly through email. However, the discussion I have had with my team is if we are just chatting, what is the advantage of Slack to just google messenger or something like that?

    I think the answer lies in integration. Using APIs we should see more than just chatting going on with ChatOps. Systems will communicate with us in real time as well. This is so much more than chatting on a messenger client.

    The name of the game whether it is the dating game, the online media game or whatever game you play is doing it faster, better and with less downtime. For eHarmony it looks like they have “found their perfect match”.

  • Inside the mind of a true DevOps evangelist

    Inside the mind of a true DevOps evangelist

    IBM InterConnect is imminent. Next week more than 20,000 people will descend on Las Vegas and take over the north end of the strip for IBM’s massive event. One of the primary topics attendees will be focusing on is DevOps. IBM’s Rosalind Radcliffe—a respected evangelist for DevOps—will be there to share her passion and knowledge as well.

    Rosalind Radcliffe is a Distinguished Engineer, in Rational currently responsible for the driving the System z and Power Systems aspects for the Jazz foundation and Jazz foundation based products—or so her LinkedIn profile indicates. Really she’s just a very smart person who loves to help customers solve problems.

    At InterConnect Radcliffe will be talking about DevOps. She is hosting a number of workshops for DevOps for enterprise systems. She will also be engaging in panel discussions with clients to discuss things like how an organization can transform its current environment and the part that DevOps plays.

    I had an opportunity to speak with Rosalind recently, and gain some insight into her perspective on DevOps and what it means for IBM enterprise customers. Two themes seemed to come up repeatedly: breaking down silos, and finding a common ground.

    Radcliffe stressed that top-down support from executive management is crucial to a smooth transition. “At a business level [executives] understand DevOps is necessary. They know the value and they want to do it,” she explained. The challenge is how to adapt and make that transformation without disrupting business.

    Where do you begin? If you’re Rosalind Radcliffe you start by finding common ground. All the way from the business perspective to deployment and across all platforms it’s important for all parties to see the same point of view and work toward a common goal.

    That’s not always easy to do when you’re dealing with different teams using different tools and platforms, and with separate political agendas. The first step is breaking down silos. Rosalind told me, “Everyone says that, but that’s really what has to happen.”

    She explained that customers don’t have to throw out the organizations structure or retool the teams themselves necessarily. It’s about redefining how the teams interact and work together, not about reorganizing the teams.

    Radcliffe is passionate about DevOps, but her real drive is helping customers achieve their goals. DevOps is hot now, but it’s just an evolution of the tools and processes businesses use to find a competitive advantage. DevOps may pass, but the need to streamline processes and accelerate delivery will live on.

    “A few years from now we won’t be using the word DevOps,” said Radcliffe. She told me that DevOps is just the Agile principle taken to the full enterprise. It provides a more comprehensive end-to-end perspective that enables enterprise-scale transformation.

    The reality, according to Radcliffe, is that customers need to come up with an enterprise strategy, but it helps to start small. She works with customers to find those initial baby steps that help them gain momentum that eventually allows them to make the bigger transformation.

    Vegas isn’t really her scene, but she’ll be there next week for IBM InterConnect. She can do without the flashing and dinging of the slot machines or the Vegas shows. But, she’s driven to engage with and help enterprise customers, so she’ll brave the busy Vegas environment to share her passion for DevOps.

  • The DevOps Institute heralds the dawn of enterprise class DevOps training and certification

    The DevOps Institute heralds the dawn of enterprise class DevOps training and certification

    I am very proud to announce that today marks the launch of the DevOps Institute. Myself and the staging-devopsy.kinsta.cloud team are partners in this joint venture with some very successful, experienced IT training professionals, Jayne Groll and Lisa Schwartz. The DevOps Institute (DOI) seeks to codify best practices in DevOps into repeatable training, education and certifications. We are launching with two classes, DevOps Foundation and Certified Agile Service Manager being offered. Several other courses including role and scheme based certifications and classes will be announced shortly. The press release is also available at the end of this article.

    I know there is a segment of the DevOps community who view this as an apocalyptic event. While I obviously don’t agree with that view, I respect their opinion. As I have said all along the DevOps community is big and getting bigger. There is room for a wide spectrum of opinion.  If you don’t think that DevOps needs formal education, training and certification, so be it. I and many folks I have spoken to believe that as DevOps continues to cross the chasm into mainstream enterprise IT, a formal training and certification program will be required. Whether it is from the DOI or some other entity is the only question.

    My co-founders and I believe we are in the best position to bring quality, best practices based DevOps training to market. We have assembled a world-class Board of Regents (BOR) who will help guide the course of the DOI, plan and approve all of our courses and ensure we are training based on best practices as they evolve. You can see and read about the BOR here. We have leading lights of the DevOps community, the IT training world and general IT business leaders as members. Looking at some of the names on this list will show you the kind of people we are involved with and should speak volumes as to what our commitment to this will be.

    We welcome those of you who would like to join us in this mission. Our mandate is to be inclusive. We invite all providers who are offering or planning to offer DevOps training to join with us. We will examine your curriculum, provide you our syllabus to develop your own courses, certify your courses and even allow you to license and teach our courses. You can find out more about this at the DOI website, DevOpsInstitute.com

    If you are interested in DevOps training and would like to be involved with the Institute as a board member or in some other capacity, contact us.

    The launch of the DevOps Institute represents the culmination of over 9 months of work and research. However if we are to succeed this is just the end of the beginning and the start of a great new ad-venture!

    The DevOps Institute Brings Enterprise Class DevOps Training To Enterprise IT Market

    THE RAPID ADOPTION OF DEVOPS NECESSITATES CERTIFIED TRAINING AND EDUCATION FOR ENTERPRISE IT MARKET

    Boca Raton, Fl February, 18, 2015– The DevOps Institute (DOI) launched today with the mission of serving as the premier source for aligning industry standard quality DevOps training and examination services for enterprise IT. The Institute is led by a Board of Regents who will oversee DOI’s offerings in an effort to codify and promote DevOps’ best practices and standards that enable enterprise IT to deliver more value faster to their customers.

    The initial Board of Regents includes Gene Kim, Lori Macvittie, Sanjeev Sharma, Helen Beal, Rohit Antao, Gary Gruver, Lou Hunnebeck, Matthew Selheimer, Donna Knapp and JP Morgenthal. They join co-founders Jayne Groll, Lisa Schwartz and Alan Shimel in leading the DevOps Institute to fulfill its mission.

    “The DevOps movement has matured rapidly over the last few years. We believe that the time is right to have established best practices training and certification”, said Lisa Schwartz, CEO of DevOps Institute. “DevOps training is a logical step in DevOps adoption by the enterprise IT market. The DevOps Institute’s mission is to codify emerging best practices in DevOps and translate them into the highest quality education and certified training courses. Working with leaders of the community, DOI will continue to develop and offer the finest in DevOps and associated programs and certifications.”, continued Schwartz.

    “DevOps is transforming how technology work is done, for Development, Test and IT Operations. I support any effort that helps elevate the state of the practice, and bring consistency and quality to the training offered,” said Gene Kim, DevOps researcher and co-author of The Phoenix Project and the upcoming DevOps Cookbook.

    The DevOps Institute’s initial courses including certification are DevOps Foundation and Certified Agile Service Manager (CASM). Additional certification courses will be announced in the coming weeks including role based schemes and certifications. DOI will offer courses directly and through a network of Registered Education Providers. Courses will be available both in person and virtually utilizing state-of-art virtual classrooms. First courses will begin in early March. More information can be found at DOI’s website: http://www.devopsinstitute.com

    About DevOps Institute
    The DevOps Institute is bringing enterprise class DevOps training and certification to the IT market. Working with thought leaders from the DevOps and IT Service Management communities, as well as the IT training market, the DevOps Institute is the standard in quality, enterprise grade DevOps education.

  • Comparing DevOps to traditional IT: Eight key differences

    Comparing DevOps to traditional IT: Eight key differences

    You can make the argument that in the Software Delivery Life Cycle (SDLC) value is created in two places.  First, when the software is created (development) and second, when the software is delivered to the customer and feedback received.  The rest is transportation.  The faster one can move code from the hands of the developer to the customer and receive feedback, the more value can be realized by both.  Don’t get me wrong, there is benefit to various “transportation” activities (testing, QA, staging etc.).  But those are not value creating activities, they are risk mitigating processes.

    A DevOps organization understands this value equation.  They focus on the processes that add value, organize themselves around those actions and do the best to minimize the risk.  Traditional IT gets it too – at least conceptually.  However they treat all stages in the SDLC with equal importance or worse emphasize the wrong stage(s).

    In general, when comparing DevOps organizations to traditional IT, DevOps will have policies and practices that differ from those of traditional IT along the following eight key dimensions,

    mustafa

    Planning & Organization

    1. Batch Sizes: Go from Big to Mirco

    Traditional IT has a bias for going big and for good reason.  First, most development shops grew out of the waterfall method which by its inherent nature takes a lot of time.  Second, since releases are costly and disruptive, operations allows developers only a few windows a year to release software.  And third, its simply not sexy – doing big is how you get promoted.  As a result, development organizations maximize productivity, by planning big projects, that involve a lot of code, bundled (hastily at times) into a release, and jammed into production.

    A DevOps organization takes the opposite point of view and believe that small is beautiful.  They understand that large batch sizes are inherently complex, risky (since there are so many moving parts), and hard to coordinate.  Small batches sizes, on the other hand, are simple, easy to understand and test rigorously, and less risky.  If things go wrong the impact is minimal and it is much easier and faster to fix.  In other words, by going small organizations are able to perform more frequent releases and become more responsive to the customer. 

    1. Organization: From Skill Centric Silos to Dedicated Cells

    Traditional IT is organized around skill centric silos.  For the most part silos work, they band like skills together, drive greater utilization, and benefit from economies of scale.  However, where these “cost optimized” silos break down is at the hand offs.  In a typical IT environment, a new feature has to go through at least 3 – 4 silos before the customer gets it.  It is not uncommon for an idea / code to spend 80% of its time waiting or ping-ponging back and forth between silos.

    A DevOps organization also operates in a silos but at a different cross section.  Here teams are arranged in “cells”, consisting of dedicated cross functional teams, and focused on only one application.  By creating this self-sufficient cell that consists of developers, testers, business analysts, and operators, an idea can move from one stage to another without hand offs, promotes cross-training / understanding (key to collaboration), focuses the team on the end goal, and encourages the “shift left” thinking.  As for utilization and scale benefits, the rules for response states that for every ¼ reduction in cycle time, productivity will improve 2x, and operating costs by 20%.

    1. Scheduling: Centralize to Decentralize & Continuous

    Efficient scheduling is at the heart of a Traditional IT organization.  Since resources are pooled, projects are usually clamoring for access to SMEs and / or infrastructure.  To get around this, enterprises have invested in sophisticated scheduling planning systems.  And while these systems are quite sensitive, they are inherently inaccurate, take up to much time to manage, and in some cases become the very bottleneck that they are trying to alleviate.

    In a DevOps organization scheduling is pushed to the local cell level.  The combination of smaller batch sizes, dedicated teams, and automated processes makes scheduling more simpler to operate. How? First, forecasting is limited to the very near future (2 – 3 weeks) where the teams have better insight.  Second, there is no fighting for people’s time since it is a dedicated team.  Third, there is no waiting for infrastructure since it has already been defined and automatically provisioned.  And fourth, it eliminates time consuming loop backs to management for escalations and decisions.

    Performance & Culture

    1. Release: Turn a High Risk Event to a Non Event

    In a Traditional IT organization, releasing software into production is a high risk proposition.  It is fraught with issues, escalations, and constant fire-fighting.  The process is tightly managed, governed from the highest levels, and requires participation from all parts of the organization.  It is not uncommon for IT to set up war rooms staffed 24×7 for weeks before and after the release.

    DevOps organizations, on the other hand, make the release of software into a non-event much as possible.  They reduce risk by daily integrating code into the trunk, automating testing, ensuring all environments are in sync, reducing batch sizes (as mentioned above) etc.  In other words, they only promote code from one stage to another after they are confident that it will work in production.  Thus eliminating all the hoopla about release windows and they are able to move new functionality into production at a much faster clip.

    1. Information: Focus is on Dissemination vs. Actionable

    Both types of organizations generate and share a ton of data.  The difference lies in how the teams uses data.  In Traditional IT, information is generated by specialists (e.g. operations team), bundled together with other data into a massive report, goes through management approval, is then shared with other managers, who then send it out their specialist (testers, developers etc.).  In most cases the report goes unread, simply because there is too much data, not timely enough, and / or not the right data.  In other words, information is shared but poorly consumed, and rarely used to take any actions.

    Within a DevOps organization, it’s the team cell that gather and creates the data.  Since the data is to be consumed locally they only collect the data (also automated) that the team deems is necessary.  And since the data is processed within the team, it eliminates the time lag of creating lengthy reports, manager approvals, and queue time (sitting in some ones email box).  As a result, the team is quickly able to read and react to the data – resulting in faster feedback time.

    1. Culture: Do Not Fail vs. Fail Early

    Traditional IT is fundamentally a risk averse organization.  A CIO’s first priority is to do no harm to the business.  It is the reason why IT invests in so much red tape, processes, approvals etc.  All focused on preventing failure.  And yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.

    A DevOps organization is risk averse too but they also understand that failure is inevitable.  So instead of trying to eliminate failure they prefer to choose when and how they fail.  They prefer to fail small, fail early, and recover fast.  And they have built their structure and process around it.  Again the building blocks we have referred to in this article – from test driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this mindset.

    Measure

    1. Metric: Cost & Capacity to Cost, Capacity, & Flow

    Silos work for Traditional IT because they are directly tied to its measurement model – cost and capacity.  IT is measured on how much can it get done (capacity) for the least amount of money (cost).  It is no wonder that cutting cost, while trying to keep capacity constant, has been all the rage in IT over the last two decades.  It’s one of the reasons why outsourcing became so popular.

    And while this model works great for stable legacy applications, the newer “system of engagement” applications require an additional metric – the element of time.  DevOps organization understand this paradigm shift and have added “flow” as an additional metric.  Flow forces an organization to take a look at its end to end cycle time, identify areas of waste, calculate true productive time, quantify quality, and focus on activities that add the most value.

    1. Definition of Done: “I did my job” vs. “it’s ready to deploy”

    Another source of measurement difference is in how the two types of organizations define done.  In Traditional IT, done is defined as the specialist doing “just” their part and handing it off.  In essence, their done is more focused on meeting the hand off deadline vs. making sure what is done is deployable.  This varying definition leads to some bad habits such as sub-par work, quality loss, finger pointing, and an overall sub-optimized process.

    To get way from the finger pointing mentality is another reason why a DevOps organization prefers to create a dedicated cross functional team.  With every member of the various parts of IT represented in this “cell” structure and all of them being held accountable for one and only one thing, they all end up having just one definition of done – bring quality software to market.

    If you would like to learn more about this topic, please join me at IBM InterConnect 2015.  I will be hosting several workshops on DevOps.

    About the Author/Mustafa Kapadia

    Mustafa KapadiaMustafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery.  He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

    Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

    Connect with Mustafa LinkedIn| Twitter

  • GitHub, Bug bounties and DevOps

    GitHub, Bug bounties and DevOps

    A year after starting up its bug bounty program GitHub is showing how complementary bug bounties can be to DevOps practices. Essentially crowdsourcing vulnerability hunting by systematically paying independent researchers prizes for flaws they find, bug bounties are in a sense an extension of the DevOps ethos. They offer a means of continuously delivering application security assurance.

    “The advantage of having a community that is just working 24/7 worldwide is it does tend to be a more continuous approach to vulnerability hunting as opposed to the traditional method of hiring out a consultancy or third party to spot check at some point in time,” explains Shawn Davenport, vice president of security for GitHub.

    And according to GitHub’s security leaders, their program has become one of the most effective and cost-conscious tools they have for extending internal appsec resources. With approximately four application security experts on staff, GitHub has a more than respectable team in place for a company of its size, with an employee total of just over 250. But with so many deploys a day, the team needs air support to ensure they’re thoroughly inspecting the code base.

    “We ship software 100 times a day, we update the site constantly and the security team isn’t looking at every one of those changes,” says Ben Toews, application security engineer at GitHub and the mastermind behind its bug bounty program. “The way we fit security into our software development life cycle is that when someone is getting ready to ship a feature, they’ll reach out to the security team and ask for a review if they think it’s necessary and we’ll get a review in that point in time, but then a lot of smaller changes don’t necessarily get our attention. And then there’s a lot of legacy code we’ve never looked at.”

    Toews says the bug bounty fills the holes a number of ways. He estimates that for any given feature on the site, there’s probably been several hundred researchers who have tested them, “so the odds of someone looking at new code pretty quickly after a change is pretty high.” In its first year, GitHub’s bug bounty program fielded nearly 2,000 reports from bug hunters, 869 of which warranted further review.

    As GitHub rounds the corner into its second year of offering bug bounties, it wants to increase those odds of good coverage with some tweaks to its program. In addition to doubling its max bounty prize to $10,000, it is also experimenting with giving bounty hunters preview access to new features before they ship.

    “We think by doing that we can get even better coverage on features before potentially putting the community at risk,” says Toews, who admits it does complicate rapid release schedules. “We’ve been trying recently to be more methodical about how we ship larger features. If we’re developing a new feature, you will have that feature hidden from the general population and you can develop that and ship it many times a day and iterate on it quickly, but then we do have a little bit of a process around making that feature published and exposing it to the rest of the world.”

    Of course, the efficacy of a bug bounty program depends on organizations finding a good way to marry bounty vulnerability remediation with the deployment process. It’s not good enough to find the vulnerability—you also need to be able to fix it quickly. Fortunately, the continuous delivery model makes this easier than in traditional organizations, says Toews, who reports that independent researchers are often “astounded” that his team can sometimes have problems fixed within hours of receiving a report, considering that they’re used to other organizations taking months to remediate problems.

    Toews says the secret to success is rapid triage of vulnerabilities. If it is simple, he and his team of application security engineers will fix it on the spot. If it is more complex, they’ll loop in the development team. This is where it is important to have buy-in from developers and senior leadership about bug bounties and application security in general, he says.

    “Our developers care a lot about security and they know that our leadership cares a lot about security, so if there’s a problem, everyone is going to jump in and try to fix it,” he says.