staging-devopsy.kinsta.cloud

Category: Enterprise DevOps

  • The ROI of Enterprise DevOps

    The ROI of Enterprise DevOps

    Ever since the 1996 film Jerry Maguire, I can never think about any technology development without hearing Cuba Gooding Jr as professional footballer Rod Tidwell screaming out, “Show me the money!” So I was not surprised when the CIO of a large enterprise recently asked me the enterprise exec version of that famous line:

    Q. Where is the ROI in Enterprise DevOps?

    Fortunately there is plenty of evidence to show enterprise executives exactly where to look when trying to determine the ROI of DevOps.

    “Show me the money!”

    Here on staging-devopsy.kinsta.cloud, Rajat Bhargava has posted on talking to your CFO about DevOps, outlining four key areas of financial impact: generating cash, reducing expenses, enabling growth, long-term stability. I also had some ideas about financial aspects of getting executive support for DevOps in a great article by Erica Chickowski. And site editor Alan Shimel has posted some excellent insight on vendor-sponsored research from IDC on the cost-benefit of downtime avoidance.

    “Show me the money!”

    Not enough? Well, we also have several years of research on ROI factors associated with DevOps adoption in both small businesses and large enterprises. The company I work for, CA Technologies, has commissioned independent third party research into enterprise DevOps adoption since 2013; Puppetlabs has run similar research [PDF] with good data on all sized organizations, but especially mid-sized businesses.

    You can see in the chart below the areas where benefit accrues through the adoption of DevOps (taken from CA Technologies/Vanson Bourne research released in November 2014):

    Image 1

    Still need more? No problem! There is also good data on how much of a return you can expect from DevOps. Again, the following chart is taken from the same CA Technologies/Vanson Bourne research:

    Image 2

    Figure 2 Respondents report seeing double-digit improvements in key technology and business areas. Total varies by benefit. Figure 1 & 2 Source: Vanson Bourne/CA Technologies 2014

     “Show me the money!”

    Okay, I get it – you still want more, and you want a straight answer – a single, simple heuristic to work with. In talking with many enterprise CxOs, faster application delivery is a pressing goal, so just this one factor can act as a key ROI driver. At the Continuous Delivery (CoDe) conference in Copenhagen in October, Jan Bosch (Professor of Software Engineering at Chalmers University of Technology) showed very elegantly how faster delivery acts to reduce (i.e. shorten) the ROI period.

    His theory is based on the premise that for many (most? all?) businesses, time to revenue is dictated substantially by product release cycles. Maximum revenue accrues around product release, then tails off before spiking again for the next new product release. For example, a smartphone manufacturer will get revenue in 6-month peak and trough cycle as they release a new device twice a year; a software vendor will get a revenue peak with a new game release, which tails off until new DLC is released or a new game hits the stores.

    Because DevOps can accelerate business cycles, Jan theorized, business can bring in revenue faster, which in turn makes a DevOps-based business more profitable over the mid- to long-term. This is especially critical in the modern application economy where almost every business is a software business.

    So, for example, if DevOps allows a business to release twice as fast, they can effectively double their annual revenue, as seen below.

    Image 3

    Figure 3 Proportional revenue acceleration from doubling product release frequency with DevOps

    The impact is exaggerated for this illustration, but based on actual data (see above), a 20% increase in annual revenue would be on par with enterprise averages.

     “Show me the … costs!?”

    Of course, any discussion of ROI must realistically reflect costs too. Adopting DevOps in a large enterprise is not free – it requires investment in people, process, and technology. On the people side, even just a small cultural change often requires new activities and other behavioral shaping. Process change requires training and often impacts short-term revenues. Technology investment includes procurement of automation and measurement solutions – key components of a CAMS (Culture, Automation, Measurement, and Sharing) approach to DevOps, especially in a larger enterprise.

    Many of the investment factors can be seen in the following chart, again using CA Technologies/Vanson Bourne research:

    Image 4

    Figure 4 Which of the following is your organization likely to invest in over the next year as part of your implementation of a DevOps methodology? Total = 499 ITDMs who have already or plan to implement DevOps in the next two years. Source: Vanson Bourne/CA Technologies 2014

     “Show me … my own ROI!”

    These ROI factors are of course a combination of averages, research data points, and anecdotes. As with all generalizations, when it comes to a specific enterprise your mileage may vary. However, this is certainly a good starting point when building a business case to prove the expected ROI for your organization.

    You may find other areas for ROI too – upstream in the PMO, DBA, or downstream in NetSec, training, documentation. If you have additional data points on finding the ROI for DevOps, I would love to hear about it. Add your ideas here in the comments, or connect with me and/or staging-devopsy.kinsta.cloud on Twitter – at @AndiMann and @DevOpsDotCom respectively.

     

     

     

  • Q and A with Carmen DeArdo: DevOps at Nationwide

    Q and A with Carmen DeArdo: DevOps at Nationwide

    Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.

    DeArdo is no stranger here at staging-devopsy.kinsta.cloud, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.

    Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.

    staging-devopsy.kinsta.cloud: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?

    DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.

    What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.

    Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.

    These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?

     

    staging-devopsy.kinsta.cloud: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?

    DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.

    But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”

     

    staging-devopsy.kinsta.cloud: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.

    DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.

    We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.

     

    staging-devopsy.kinsta.cloud: What are some of the high-level challenges that you’ve had optimizing that last mile phase?

    DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.

    So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.

    I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.

    One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.

    That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.

    That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.

     

    staging-devopsy.kinsta.cloud: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.

    Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?

    DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.

    DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.

    Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.

    But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.

  • Comparing DevOps to traditional IT: Eight key differences

    Comparing DevOps to traditional IT: Eight key differences

    You can make the argument that in the Software Delivery Life Cycle (SDLC) value is created in two places.  First, when the software is created (development) and second, when the software is delivered to the customer and feedback received.  The rest is transportation.  The faster one can move code from the hands of the developer to the customer and receive feedback, the more value can be realized by both.  Don’t get me wrong, there is benefit to various “transportation” activities (testing, QA, staging etc.).  But those are not value creating activities, they are risk mitigating processes.

    A DevOps organization understands this value equation.  They focus on the processes that add value, organize themselves around those actions and do the best to minimize the risk.  Traditional IT gets it too – at least conceptually.  However they treat all stages in the SDLC with equal importance or worse emphasize the wrong stage(s).

    In general, when comparing DevOps organizations to traditional IT, DevOps will have policies and practices that differ from those of traditional IT along the following eight key dimensions,

    mustafa

    Planning & Organization

    1. Batch Sizes: Go from Big to Mirco

    Traditional IT has a bias for going big and for good reason.  First, most development shops grew out of the waterfall method which by its inherent nature takes a lot of time.  Second, since releases are costly and disruptive, operations allows developers only a few windows a year to release software.  And third, its simply not sexy – doing big is how you get promoted.  As a result, development organizations maximize productivity, by planning big projects, that involve a lot of code, bundled (hastily at times) into a release, and jammed into production.

    A DevOps organization takes the opposite point of view and believe that small is beautiful.  They understand that large batch sizes are inherently complex, risky (since there are so many moving parts), and hard to coordinate.  Small batches sizes, on the other hand, are simple, easy to understand and test rigorously, and less risky.  If things go wrong the impact is minimal and it is much easier and faster to fix.  In other words, by going small organizations are able to perform more frequent releases and become more responsive to the customer. 

    1. Organization: From Skill Centric Silos to Dedicated Cells

    Traditional IT is organized around skill centric silos.  For the most part silos work, they band like skills together, drive greater utilization, and benefit from economies of scale.  However, where these “cost optimized” silos break down is at the hand offs.  In a typical IT environment, a new feature has to go through at least 3 – 4 silos before the customer gets it.  It is not uncommon for an idea / code to spend 80% of its time waiting or ping-ponging back and forth between silos.

    A DevOps organization also operates in a silos but at a different cross section.  Here teams are arranged in “cells”, consisting of dedicated cross functional teams, and focused on only one application.  By creating this self-sufficient cell that consists of developers, testers, business analysts, and operators, an idea can move from one stage to another without hand offs, promotes cross-training / understanding (key to collaboration), focuses the team on the end goal, and encourages the “shift left” thinking.  As for utilization and scale benefits, the rules for response states that for every ¼ reduction in cycle time, productivity will improve 2x, and operating costs by 20%.

    1. Scheduling: Centralize to Decentralize & Continuous

    Efficient scheduling is at the heart of a Traditional IT organization.  Since resources are pooled, projects are usually clamoring for access to SMEs and / or infrastructure.  To get around this, enterprises have invested in sophisticated scheduling planning systems.  And while these systems are quite sensitive, they are inherently inaccurate, take up to much time to manage, and in some cases become the very bottleneck that they are trying to alleviate.

    In a DevOps organization scheduling is pushed to the local cell level.  The combination of smaller batch sizes, dedicated teams, and automated processes makes scheduling more simpler to operate. How? First, forecasting is limited to the very near future (2 – 3 weeks) where the teams have better insight.  Second, there is no fighting for people’s time since it is a dedicated team.  Third, there is no waiting for infrastructure since it has already been defined and automatically provisioned.  And fourth, it eliminates time consuming loop backs to management for escalations and decisions.

    Performance & Culture

    1. Release: Turn a High Risk Event to a Non Event

    In a Traditional IT organization, releasing software into production is a high risk proposition.  It is fraught with issues, escalations, and constant fire-fighting.  The process is tightly managed, governed from the highest levels, and requires participation from all parts of the organization.  It is not uncommon for IT to set up war rooms staffed 24×7 for weeks before and after the release.

    DevOps organizations, on the other hand, make the release of software into a non-event much as possible.  They reduce risk by daily integrating code into the trunk, automating testing, ensuring all environments are in sync, reducing batch sizes (as mentioned above) etc.  In other words, they only promote code from one stage to another after they are confident that it will work in production.  Thus eliminating all the hoopla about release windows and they are able to move new functionality into production at a much faster clip.

    1. Information: Focus is on Dissemination vs. Actionable

    Both types of organizations generate and share a ton of data.  The difference lies in how the teams uses data.  In Traditional IT, information is generated by specialists (e.g. operations team), bundled together with other data into a massive report, goes through management approval, is then shared with other managers, who then send it out their specialist (testers, developers etc.).  In most cases the report goes unread, simply because there is too much data, not timely enough, and / or not the right data.  In other words, information is shared but poorly consumed, and rarely used to take any actions.

    Within a DevOps organization, it’s the team cell that gather and creates the data.  Since the data is to be consumed locally they only collect the data (also automated) that the team deems is necessary.  And since the data is processed within the team, it eliminates the time lag of creating lengthy reports, manager approvals, and queue time (sitting in some ones email box).  As a result, the team is quickly able to read and react to the data – resulting in faster feedback time.

    1. Culture: Do Not Fail vs. Fail Early

    Traditional IT is fundamentally a risk averse organization.  A CIO’s first priority is to do no harm to the business.  It is the reason why IT invests in so much red tape, processes, approvals etc.  All focused on preventing failure.  And yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.

    A DevOps organization is risk averse too but they also understand that failure is inevitable.  So instead of trying to eliminate failure they prefer to choose when and how they fail.  They prefer to fail small, fail early, and recover fast.  And they have built their structure and process around it.  Again the building blocks we have referred to in this article – from test driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this mindset.

    Measure

    1. Metric: Cost & Capacity to Cost, Capacity, & Flow

    Silos work for Traditional IT because they are directly tied to its measurement model – cost and capacity.  IT is measured on how much can it get done (capacity) for the least amount of money (cost).  It is no wonder that cutting cost, while trying to keep capacity constant, has been all the rage in IT over the last two decades.  It’s one of the reasons why outsourcing became so popular.

    And while this model works great for stable legacy applications, the newer “system of engagement” applications require an additional metric – the element of time.  DevOps organization understand this paradigm shift and have added “flow” as an additional metric.  Flow forces an organization to take a look at its end to end cycle time, identify areas of waste, calculate true productive time, quantify quality, and focus on activities that add the most value.

    1. Definition of Done: “I did my job” vs. “it’s ready to deploy”

    Another source of measurement difference is in how the two types of organizations define done.  In Traditional IT, done is defined as the specialist doing “just” their part and handing it off.  In essence, their done is more focused on meeting the hand off deadline vs. making sure what is done is deployable.  This varying definition leads to some bad habits such as sub-par work, quality loss, finger pointing, and an overall sub-optimized process.

    To get way from the finger pointing mentality is another reason why a DevOps organization prefers to create a dedicated cross functional team.  With every member of the various parts of IT represented in this “cell” structure and all of them being held accountable for one and only one thing, they all end up having just one definition of done – bring quality software to market.

    If you would like to learn more about this topic, please join me at IBM InterConnect 2015.  I will be hosting several workshops on DevOps.

    About the Author/Mustafa Kapadia

    Mustafa KapadiaMustafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery.  He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

    Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

    Connect with Mustafa LinkedIn| Twitter

  • Change Management & Continuous Delivery: Clash or Harmony?

    Change Management & Continuous Delivery: Clash or Harmony?

    Introducing changes into running software systems is risky. There’s always a chance of failure, and that can have a negative impact on your business. It’s important to mitigate that risk, but there’s more than one way to do so.

    Change Management is all about ensuring that changes to existing systems and software in production are done in a controlled way. Change managers review the changes that are to be implemented, assess their impact, and perform due diligence, before giving or withholding their approval.

    Continuous Delivery (CD) is about improving the process of delivering software and increasing efficiency. It aims to cut down the time between ideation and realization. It’s about reducing the scope for error and increasing the speed to market to generate a competitive advantage.

    Is there an inevitable clash between CM and CD? Are they really compatible? At heart, the CD pipeline is intended to achieve the same thing as CM by enabling the smooth, error-free roll-out of changes, but today they are usually coming from very different angles.

    If CM is to have a future in the CD revolution, then it needs to change. Before we discuss how that might be achieved, let’s take a look at the current state of play 

    Is Change Management delivering?

    Ask yourself what impact change management is having on your organization. If change management was working perfectly then there wouldn’t be any production problems, and yet issues still crop up. There are a few things to consider:

    1. How many entries in the CM checklist could be automated? Is the change manager doing anything that really requires human input? Fact collection and verification could be a part of your CD pipeline.
    2. How much value do check boxes deliver in the first place? What is change management really bringing to the table?
    3. Is the process fit for purpose? Does it actually prevent problems, or is the change manager really just a fall guy, there to sign off on the release and take the blame if something goes wrong?

    Clashing with Continuous Delivery

    CM is intimately linked with the idea and practice of big complicated transformations, but with CD we’re not talking about enormous, disruptive changes. CD is about changing small parts of a system and doing it often. The risk to reward ratio is different when you think and work iteratively. Smaller change sets are less likely to cause problems and easier to rollback when they do.

    Does the process really require human input? Heavyweight validation will slow it down and speed is one of the key benefits Continuous Delivery offers. Streamlining and automating the checklist process makes perfect sense in many business contexts. The aims of CM and CD are in sync, but CD is better placed to deliver.

    A possible advantage of traditional CM is the presence of a human eye, offering intuition that a machine or a piece of software may not be capable of. By moving beyond the checklists and reimagining the CM role we can retain this advantage and push CD forward.

    From Gatekeepers to Masters of Continuous Delivery

    If the bulk of CM is really an administrative exercise in checking boxes then it could be an unnecessary overhead, but there is a way to retain that human oversight for organizations wary of full automation. Why not ask the former gatekeeper to own the CD process? For a CD pipeline to be truly effective someone has to guide the process, engage with all the teams involved and understand exactly why and how the process generates business value.

    A change manager, paired with a build or release engineer to handle the automation, could work to help organizations move towards CD. They are uniquely well placed to identify dependencies, map processes, and help construct the CD pipeline with minimal disruption. They can help sketch out what the pipeline should verify, what data and dependencies need to be taken into account, what automated tests are required, and how it fits with other components to automate that release procedure. They also have experience mediating between business and IT.

    The traditional CM approach of committee meetings and spreadsheet checklists is dated, and for many organizations it no longer makes sense. You can find much greater business value in focusing on an efficient rollback procedure that kicks in immediately and automatically when changes fail.

    Take a look at your CM and ask yourself – is it really adding anything to the process, or is it just another bottleneck? It may be time for a change.

    About the Author/Andrew Phillips

    andrew_PhillipsAndrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

  • Top Skills for Enterprise DevOps

    Top Skills for Enterprise DevOps

    There is one question I hear almost every time I talk with my large enterprise customers about DevOps:

    1. What skills do my people need to be successful with DevOps?

    Talk of DevOps almost immediately focuses on culture – like having empathy for fellow workers, being flexible and adaptable, seeking continuous improvement, building relationships, etc. However, while critically important in DevOps, culture is an outcome, not an input; and such attributes are mostly either innate or acquired slowly. Culture cannot easily be taught.

    At the other end of the spectrum, we often look at technologies – agile development, coding languages, virtualization/cloud platforms, management tools, etc. Again, these can be important, but while learning these skills might make you a better dev or op, they may not make you a DevOps practitioner.

    Like most transformations, DevOps will need new people, process and technology skills, so in this post I take a look at a handful of the most important, pragmatic, and trainable skills in all three areas for practitioners looking to adopt DevOps, especially in large and more complex organizations.

    People Skills

    • Communication skills – while this may not come easily to typically introverted IT staff, both written and verbal communications, within and across the organization, and even with customers, are essential to a strong DevOps approach. This skill is learnable from formal courses, groups like Toastmasters, direct mentoring, social activities, or informal sessions like ‘lunch and learn.’ For extra credit, jump on stage for open mic night at a local comedy club!
    • Business alignment – much of a dev or ops life is spent embedded in the technology of software delivery, rather than the business need. This can cause many issues, and is something DevOps tries to address. Understanding the business strategy, goals and priorities is essential to DevOps, helps drive and easily trainable. A ‘big picture view’ helps everyone collaborate on common objectives to deliver high-value business outcomes – a key tenet of DevOps
    • Workplace history – “an appreciation for history, both technical and organizational”* will help staff understand the critical antecedents of DevOps, so they can internalize its principles not just rote learn its actions. For example, understanding hierarchical and network structures, why we adopted open work environments (and why it hasn’t always worked), and the antecedents of DevOps in things like The Toyota Way will help staff not just to act predictably in known circumstances, but adapt appropriately to new ones. (*Thanks to @garethr for input on this!)

    Process Skills

    • Agile development – it may be possible to take a DevOps approach in a waterfall environment, but it is almost definitely less effective. DevOps arose out of the Agile movement, and they pair so well that it is sometimes referred to as ‘agile for infrastructure.’ Teaching on one or more specific agile approaches (Scrum, Kanban, Kaizen, etc.) will give staff an approach and mindset informed by agile practices to help teams work together to improve flow, delivery and quality
    • Lean approaches – ‘lean’ theories born in the manufacturing world (see ‘history’ above) give a pragmatic foundation for applied DevOps, not just new-age ideals. Concepts like Just-In-Time, Toyota Production System, Design for Manufacture, Theories of Constraint and Waste can be learned from original texts, courseware and many business courses. Some are also revised in modern IT-specific texts like The Phoenix Project or Implementing Lean Software Development
    • Process reengineering – DevOps is as much about refining and improving ‘flow’ in the SDLC as it is about encouraging a sharing culture, so classic learning about how to effect change in business processes is invaluable to early stage DevOps adoption. Understanding how and why to analyze needs, identify problems, and effect change in software processes, organizational structures, teaming etc. is essential to DevOps and a long-established body of knowledge

    Technology Skills

    • Coding/scripting – in its current iteration there is a lot of focus on helping operators to understand coding languages; and teaching developers to use script-driven operations tools. From JavaScript (eek!) to C# or .NET, any coding or scripting skills will help both dev and ops to have empathy for each other’s jobs, and help to develop the elusive ‘T-shaped people’ that are so well-suited to a DevOps approach
    • IT Automation – Enterprise DevOps especially will likely involve (require?) a raft of technologies in an automated tool chain. Because DevOps is concerned with end-to-end flow, such skills need to cover the whole SDLC. Look to provide training in Portfolio Management, Agile Project Management, Source Code Management, Version Control, Test Automation, Test Data Management, Service Virtualization, Continuous Integration, Release Automation, Cloud Provisioning, Configuration Management, Process Automation, Performance Monitoring and Event Management, whether through formal training or informal knowledge transfer
    • Workflow mapping – smaller organizations might get away with pen and paper, or even a whiteboard/flipchart, but to document, share, analyze and rationalize the SDLC for large, complex, enterprise applications, you will need to learn to use workflow mapping tools. This will allow team members across the org (and the world) to take ownership of the end-to-end process, and start to collaborate on continual improvement

    This is of course a (very) short list, but should give you a starting point for developing essential skills for enterprise DevOps practitioners. As always, please feel free to drop a comment and tell me what I missed, what is more important, or even what I got wrong. I am always interested in discussing ideas and learning new things!

  • Four key vectors for closing the DevOps feedback loop

    Four key vectors for closing the DevOps feedback loop

    A question I am asked frequently about DevOps – and a perennial challenge to all DevOps slide artists – is what exactly we mean by the idea of ‘closing the feedback loop’

    Q. What do you mean when you mention ‘closing the feedback loop’? Ops monitor our systems today, and alert devs when they are needed (next step, devs carrying pagers!). Is this the same Ac?

    Yes, this is definitely part of the equation. Feedback from operations (and more) to development (and more) – through alerts, monitoring, log entries, service tickets, or verbally – is essential. Good fast feedback helps, for example, to:

    • connect dev and ops more closely to user experiences
    • establish shared insight to replace fragile tribal knowledge
    • foster ownership from both teams in the entire service
    • provide rapid response to problems or new requirements
    • encourage faster experimentation and a tolerance for failure

    Feedback loops in smaller organizations tend to be relatively simple – just lean over the cubicle and tell your counterpart what is going on, or program the monitoring system to send you an alert. However for larger organizations the multiplicity of teams, volume of events, separation of duties, and variety of feedback can make this harder to handle.

    In larger organizations then, I see four key vectors used in closing the DevOps feedback loop:

    • Machine to Person (M2P)
    • Person to Person (P2P)
    • Machine to Machine (M2M)
    • Person to Machine (P2M)

    I have tried to illustrate these 4 vectors below (and not being any kind of graphic artist, I would love any feedback on this image):

    FeedbackLoops

    Machine to Person (M2P)

    This is more or less the alert cycle the original question posed – a machine process (performance monitoring, for example) picks up an error situation that it cannot identify it as a ‘known resolvable’ issue, so forwards it to a person to take charge. Machines can resolve many problems automatically with intelligent orchestration – rollback a release, increase capacity, modify a configuration, migrate a workload, restart a service – but sometimes a person needs to fix hard and/or infrequent problems.

    Person to Person (P2P)

    This is just the classic ‘hot line’ feedback, where application users (internal or external) talk directly to someone in IT (e.g. help desk staff). This can be the most powerful feedback of all. Not only can it help identify defects, but it can also identify opportunities for improvement – from new features to entirely new services. However, to be useful this personal feedback must be transferred into development or operations process – e.g. to update configuration scripts, or as input to future release planning.

    Machine to Machine (M2M)

    This includes automated problem resolution (e.g. monitoring detects a performance bottleneck, triggers a provisioning engine to add additional capacity), but also includes data mining that feeds back ‘real-world’ production data to build more realistic test and QA environments, or to inform more accurate capacity plans. This feedback makes prod more stable, makes QA and test more realistic, reduces human and process errors, and helps to build scalability into applications from the start of the SDLC.

    Person to Machine (P2M)

    This can take many forms, such as end-users logging problems into a self-service portal or Service Desk, or rating a mobile app, or complaining online. Automation can trigger resolution for known problems, so end users can address problems automatically; or feedback can reach product owners to resolve common user complaints. However, P2M feedback may waste resources and increase technical debt, as solvable problems are fixed repeatedly rather than permanently; or user complaints are ignored.

    Final Tips

    These four vectors of active feedback are just a starting point, but describe a framework that enterprises can work to when looking to close and accelerate your feedback loops. Remember too that in a DevOps mode, feedback needs to be faster, it needs to be more accurate, and especially in a large organization it needs to connect not just with dev and ops, but with other stakeholders too – business owners, test and QA teams, security teams, and of course end users.

    What are your feedback loops? Do you cover all four vectors? Do you have more that I am missing? Please let me know in the comments below – as always, I would love to hear from you.

  • Being Agile is all about controlling the flow

    Being Agile is all about controlling the flow

    For businesses, Agile is the new fashionable initiative as was E-business was in the 2000’s, Six Sigma was in the 90’s and Total Quality Management (TQM) was in the 80’s. If we look back across the spectrum of these initiatives, we need to examine how many of these initiatives met the goals the business was looking for and, more importantly, if they were fully adopted, remembering that many were derailed, did they deliver the value the business was seeking?

    Of course we cannot answer this question with generalities—it’s a spectrum. For some businesses, these initiatives were red herrings, some did not follow through to completion, but certain changes adopted as part of these initiatives had a very positive impact, while others succeeded wildly. It’s a Harvard Business Review case study as to why a business ended up at the point on the spectrum that they did.

    Clearly, one fundamental aspect that is consistent across all these initiatives is the ability for a business to embrace change. In a previous blog, “Cloud Computing, DevOps and Business Velocity,” I addressed the issue of business velocity—the rate at which an organization can embrace a particular disruptive technology change. Underestimating your business’ ability to adopt a new technology or process will most likely have a strong negative impact on the outcome. In a recent conversation with a client, the non-IT business areas has been very successful at adopting lean practices, but IT short-circuited shortly after starting. It didn’t outright fail, but after a strong start, they took a break from which they never started back up. Without a post-mortem, we don’t know why this specific initiative fell flat, but we can intimate that if something is difficult and feels unnatural, we’re more easily able to focus our attention elsewhere—think dieting and exercise programs in your own life.

    This seems to be the decade of Agility. Businesses want to be more dynamic and fluid. They want the organization to be prepared to respond to market changes more quickly, introduce new products faster, and respond to problems that occur more rapidly. To accomplish this goal, the business will have to let go of some of the ways that they do things today. They will have to re-engineer processes to be more compose-able and componentized so that they are less static. For example, a manufacturer goes through a lot of planning to deliver a new product to market. They go through steps for design, prototype, test, source, manufacture and distribute. The entire cycle may take upwards of a year and is highly automated once it reaches the manufacture stage. Tweaking the product between batches would cause  significant delays and pain because the process is static. It effectively amounts to restarting the entire cycle. So, what if they discovered that by simply elongating the body by 2 cm the product was far more useful their intended audience? In this case, agility might equate to being able to complete the design-to-manufacture process faster.

    In the previous example, this latter step of being able to tweak the product is called continuous improvement and is a key element of fostering agility. If a product isn’t selling well in the market, many companies will just abandon the product completely due to the costs associated with making the product viable. The reason being that most manufacturers consider that each product is a project and when it reaches the manufacturing stage it moves under the control of operations. That is, the creative team that includes marketing, finance, and other stakeholders view their position as “completed” with regard to getting that product to market and they then move onto other projects. With this mindset, there’s no room for the group to review the process, think about how it could have been done better and then iterate with the improvements incorporated. Hence, they just keep taking their “this is the way we do it” to the next project.

    Being agile means being able to control the flow. There’s no way to change how products come to market if there’s a culture of accepting finality once a product has reached a certain point in the cycle.

    The other aspect of controlling the flow is managing how products and services enter and exit the development pipeline.  I spend a lot of my time helping IT organizations unlock their potential to deliver by helping them to understand that there’s only so much that can be done to improve managing the work-in-progress (WIP). The only way to make gains beyond a certain point is to enable transparency with those that are responsible for inserting work into the pipeline or who are demanding that work completes and exits the pipeline at a certain time.

    The following two figures express well the reason transparency is a requirement. I believe most people would agree that managing production is a far easier task in Figure 1 than Figure 2. In Figure 1, there is one input and one expected output. Managing WIP for this pipeline is a matter of optimizing just the process for completing that one deliverable. In Figure 2, there are many more variables at work. There could be contention for resources within the pipeline to support the multiple inputs.

    wip

     

     

     

    wip2

     

    Likewise, there may be a request to reprioritize what gets completed when to meet certain business goals. Perhaps a particular report is required urgently to support a key business decision. I personally dealt with this very scenario when the business I was working for years ago was suddenly engaging in acquiring another company. All focus was transferred to producing the needed financial statements and everything else was put on the backlog. In that particular case, the CEO and CFO were the customer, so no one questioned the delays to their projects, but for many IT organizations reprioritization requests are never that clear making it very difficult to manage the control of the flow. Hence, transparency and communication between those consuming the services of the pipeline and those who own the control of the flow of the pipeline is an imperative for agility.

    Like TQM, Six Sigma, and E-Business, Agility will require changes to the fundamental way that a business operates. An analogy of this process can be as benign as a person recovering from knee surgery or as complex as recovering from a stroke depending upon what the business wants to accomplish and how they are organized currently. Along the way, people will most likely be the biggest hurdles to success.

    When change of this magnitude occurs, employees tend to become very myopic of their own situation. To mitigate this risk, it is critical that changes are reviewed with employees prior to change, during implementation and after the change is enacted. Management must gauge if employees are struggling because the complexity of the change or because they personally don’t agree with the change. The former can be overcome with education and support while the latter should be dealt with quickly through removal.

    Businesses can succeed with their Agility initiatives given they identify and mitigate key risk factors prior to starting and are willing to commit to looking critically at the way they have been operating the business. In agile software development, developers are trained to think about ways their code can break and then write unit tests to ensure that the software is sustainable in light of these possibilities. Creating an agile business means looking at ways to ensure sustainability in light of market changes, new competitors, supplier changes, and changing customer demands.

  • Five Top Tips for DevOps At Scale

    Five Top Tips for DevOps At Scale

    Since I published my blog explaining the topic of ‘Big DevOps’ late last year, the concept of Enterprise DevOps has exploded, with commentary, experience, and advice (some good, some awful) popping up from many different quarters. Fortunately, many pundits have now come to agree that application delivery in a traditional enterprise is very different from in a typical web-scale/startup environment.

    As a result, a growing body of work is filling a significant and important gap with unique insight into Enterprise DevOps, from many excellent commentators and practitioners who understand how large enterprises are different, and how to address those different challenges.

    As Enterprise DevOps takes off, I seem to field questions about it on an almost daily basis, so I decided to contribute to the body of work with a webinar explaining my answers to five of the most frequent challenges I hear about DevOps at scale. It was titled ‘Big DevOps: Five Top Tips When Implementing at Scale’ and you can catch the full replay on YouTube.

    The top five issues I chose to address are below:

    TopFiveIssuesDevops

    My five ‘Big DevOps’ tips for overcoming these issues center mainly on cultural change, because DevOps is primarily a cultural and attitudinal shift. However, it is also undoubtedly true that certain technologies can play a large part in enabling that change, especially in the more complex world of larger enterprises.

    With that in mind, my top five tips to address these challenges are as follows:

    FiveAnswersDevOps

    • Facilitate a culture of teaming and collaboration – this is at the core of devops in any environment, large or small, but especially for large organizations with bright lines between teams, this is essential to overcoming the tyranny of complex and dispersed silos
    • Establish agile development as a shared discipline – agile development is brings code to market faster, but can result in just filling the clogged pipes of PMO, HR, training, security, and other departments faster. ‘Agile’ needs to be common throughout the SDLC – in dev, ops, and beyond
    • Automate relentlessly to enable rapid devops response – automation allows faster execution throughout the system, enabling downstream processes to keep up with speed of DevOps. Automation also adds audit and governance, especially important to larger organizations
    • Push smaller releases faster, measure and remediate impact – large organizations especially are can avoid big errors by avoiding big releases. Adopt faster, smaller cycles with closer scrutiny to achieve a DevOps approach while managing the risk of faster cycles and fewer controls
    • Manage user identities to enable shared access – in smaller organizations it might be acceptable to give out root access like candy, but not for large enterprises. Identity and access management can enable a new DevOps approach that is safe, secure, auditable, and well-governed

    While these tips may not be the perfect answer for every organization, I have seen how these approaches have made the journey to DevOps easier, safer, and more effective for big enterprises. I hope they help you on your journey.

    If so, you will want to stay tuned to staging-devopsy.kinsta.cloud. Since starting work on Enterprise DevOps concepts, I have collected many questions on how large organizations can best succeed in their DevOps transformation. I will be sharing these questions along with my answers here on staging-devopsy.kinsta.cloud in a regular series, to give everyone the chance to understand how large enterprises can overcome the challenges of adopting DevOps at scale.

    So keep your eyes out for my posts over the coming days, weeks, and months in the brand new occasional series, ‘Enterprise DevOps Q&A’. I hope you will keep reading, and maybe even help me out by discussing your experiences and ideas in the comments below.

    Finally if you have any questions about Big DevOps, Enterprise DevOps, or anything to do with adopting a DevOps approach at scale, let me know in the comments below too. I would love to hear from you, and answer your questions if I can, or source guest opinions from the DevOps community.

    Meanwhile, please stay tuned for more Enterprise DevOps Q&A. I am excited to continue the discussion on what it takes to transform application delivery at scale.

  • Is SDN just DevOps for the network?

    Is SDN just DevOps for the network?

    The answer depends on your definition of SDN.

    (more…)

  • Infrastructure-as-code, new rules for the old game

    Infrastructure-as-code, new rules for the old game

    There’s been a lot of discussion regarding DevOps and its role in fostering quality and continuous improvement through increased levels of automation. In turn, this automation is fostered through emerging infrastructure and software architectures that are exposing control plane and management programming interfaces, which has been dubbed in many circles as Infrastructure-as-Code.

    This movement allows development and operations personnel to have real-time interactions with the underlying environment—the network, servers, operating systems, and storage—that are used to deliver applications and data. It also means that as the environment becomes more programmatic the business must treat this this effort as a first-class application development process and ensure that the corresponding assets follow common coding practices and meet with a consistent and sustainable architecture.

    Those that are unfamiliar with the activities of system administrators and data center operations may have an ill-conceived notion after reading emerging literature on DevOps that operators have been running the business on the backbone of purely manual processes. Even as a life-long IT person, until I started working with infrastructure and operations as part of my role, I was unaware of what happens behind the great curtain that keeps the digital bits flowing through the organization.

    The truth is that operations and administrators have been developing scripts for years to automate routine tasks. These scripts have typically been used to dynamically generate or update configuration files that were then applied after a restart. This change to Infrastructure-as-Code is significant because now these configuration changes can be made on the fly and are often immediately acted upon without having to take the system offline. If you’re now putting your hand over your mouth upon the realization of the potential disastrous effects this can have, you understand the impact of this power.

    Thanks to Software Defined Networking, hypervisors, and even Software Defined Storage, an operator can programmatically change the entire environment underlying the application without having to notify the application of this change. Architecturally, this is a wonderful thing. Pragmatically? Well, there’s a lot of bad code out there that makes assumptions about the underlying infrastructure and is about to be exposed when migrated to these modern infrastructures.

    Hence, building, testing and deploying infrastructure control applications must become part of the overall Software Development Life Cycle (SDLC) and begin to be built, tested, deployed and managed using the same controlled techniques that are hopefully being used for any applications that are being developed by application engineering.

    You may be asking, “Wait, doesn’t that mean my QA environment must have the same infrastructure components as my production environment?” To wit I respond, “Maybe!” In some cases this may be advantageous for ensuring that production deployment can support continuous deployment processes, which will ensure limited human intervention from the point of user acceptance through deployment, which will, hopefully, lead to limited opportunity for production failures. However, because we’re moving to a more programmatic infrastructure many vendors can also supply virtual instances of their infrastructure code for purposes of testing and this may be satisfactory for your needs.

    Of note, there are many businesses out there that do not develop their own software. They primarily use Commercial-Off-The-Shelf (COTS) applications. Thus, these businesses may not have an established SDLC practice. Likewise, these businesses may not believe DevOps is relative for them since they do not do “development”. If your business falls into this category, the move to Infrastructure-as-Code and Software Define Data Centers should be perceived as a driver for DevOps adoption as well as formally adopting one of the well-defined SDLC approaches. Even if these components won’t impact you for a few years until your next refresh cycle, the advantages DevOps brings around improving quality and consistency in delivery is of definite value to your business now.

    Infrastructure-as-Code is a significant change in capability that operations has for delivering compute and storage resources to the business. It will allow them to have greater insight into the hardware allowing them to do predictive maintenance, migrate applications to various resources based on demand, and tune the overall environment to meet the business’ needs at a given moment. In short, it will allow IT to deliver compute services with a high degree of availability and security. It will also expose shortcomings in existing applications that have made assumptions about the underlying environment on which it is operating. Thus, if operations and application engineering are not working together and the entire application environment inclusive of the infrastructure control apps is not tested and deployed as a unit, the likelihood for failure will be high.