staging-devopsy.kinsta.cloud

Category: Features

  • Q and A with Carmen DeArdo: DevOps at Nationwide

    Q and A with Carmen DeArdo: DevOps at Nationwide

    Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.

    DeArdo is no stranger here at staging-devopsy.kinsta.cloud, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.

    Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.

    staging-devopsy.kinsta.cloud: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?

    DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.

    What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.

    Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.

    These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?

     

    staging-devopsy.kinsta.cloud: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?

    DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.

    But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”

     

    staging-devopsy.kinsta.cloud: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.

    DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.

    We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.

     

    staging-devopsy.kinsta.cloud: What are some of the high-level challenges that you’ve had optimizing that last mile phase?

    DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.

    So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.

    I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.

    One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.

    That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.

    That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.

     

    staging-devopsy.kinsta.cloud: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.

    Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?

    DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.

    DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.

    Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.

    But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.

  • Comparing DevOps to traditional IT: Eight key differences

    Comparing DevOps to traditional IT: Eight key differences

    You can make the argument that in the Software Delivery Life Cycle (SDLC) value is created in two places.  First, when the software is created (development) and second, when the software is delivered to the customer and feedback received.  The rest is transportation.  The faster one can move code from the hands of the developer to the customer and receive feedback, the more value can be realized by both.  Don’t get me wrong, there is benefit to various “transportation” activities (testing, QA, staging etc.).  But those are not value creating activities, they are risk mitigating processes.

    A DevOps organization understands this value equation.  They focus on the processes that add value, organize themselves around those actions and do the best to minimize the risk.  Traditional IT gets it too – at least conceptually.  However they treat all stages in the SDLC with equal importance or worse emphasize the wrong stage(s).

    In general, when comparing DevOps organizations to traditional IT, DevOps will have policies and practices that differ from those of traditional IT along the following eight key dimensions,

    mustafa

    Planning & Organization

    1. Batch Sizes: Go from Big to Mirco

    Traditional IT has a bias for going big and for good reason.  First, most development shops grew out of the waterfall method which by its inherent nature takes a lot of time.  Second, since releases are costly and disruptive, operations allows developers only a few windows a year to release software.  And third, its simply not sexy – doing big is how you get promoted.  As a result, development organizations maximize productivity, by planning big projects, that involve a lot of code, bundled (hastily at times) into a release, and jammed into production.

    A DevOps organization takes the opposite point of view and believe that small is beautiful.  They understand that large batch sizes are inherently complex, risky (since there are so many moving parts), and hard to coordinate.  Small batches sizes, on the other hand, are simple, easy to understand and test rigorously, and less risky.  If things go wrong the impact is minimal and it is much easier and faster to fix.  In other words, by going small organizations are able to perform more frequent releases and become more responsive to the customer. 

    1. Organization: From Skill Centric Silos to Dedicated Cells

    Traditional IT is organized around skill centric silos.  For the most part silos work, they band like skills together, drive greater utilization, and benefit from economies of scale.  However, where these “cost optimized” silos break down is at the hand offs.  In a typical IT environment, a new feature has to go through at least 3 – 4 silos before the customer gets it.  It is not uncommon for an idea / code to spend 80% of its time waiting or ping-ponging back and forth between silos.

    A DevOps organization also operates in a silos but at a different cross section.  Here teams are arranged in “cells”, consisting of dedicated cross functional teams, and focused on only one application.  By creating this self-sufficient cell that consists of developers, testers, business analysts, and operators, an idea can move from one stage to another without hand offs, promotes cross-training / understanding (key to collaboration), focuses the team on the end goal, and encourages the “shift left” thinking.  As for utilization and scale benefits, the rules for response states that for every ¼ reduction in cycle time, productivity will improve 2x, and operating costs by 20%.

    1. Scheduling: Centralize to Decentralize & Continuous

    Efficient scheduling is at the heart of a Traditional IT organization.  Since resources are pooled, projects are usually clamoring for access to SMEs and / or infrastructure.  To get around this, enterprises have invested in sophisticated scheduling planning systems.  And while these systems are quite sensitive, they are inherently inaccurate, take up to much time to manage, and in some cases become the very bottleneck that they are trying to alleviate.

    In a DevOps organization scheduling is pushed to the local cell level.  The combination of smaller batch sizes, dedicated teams, and automated processes makes scheduling more simpler to operate. How? First, forecasting is limited to the very near future (2 – 3 weeks) where the teams have better insight.  Second, there is no fighting for people’s time since it is a dedicated team.  Third, there is no waiting for infrastructure since it has already been defined and automatically provisioned.  And fourth, it eliminates time consuming loop backs to management for escalations and decisions.

    Performance & Culture

    1. Release: Turn a High Risk Event to a Non Event

    In a Traditional IT organization, releasing software into production is a high risk proposition.  It is fraught with issues, escalations, and constant fire-fighting.  The process is tightly managed, governed from the highest levels, and requires participation from all parts of the organization.  It is not uncommon for IT to set up war rooms staffed 24×7 for weeks before and after the release.

    DevOps organizations, on the other hand, make the release of software into a non-event much as possible.  They reduce risk by daily integrating code into the trunk, automating testing, ensuring all environments are in sync, reducing batch sizes (as mentioned above) etc.  In other words, they only promote code from one stage to another after they are confident that it will work in production.  Thus eliminating all the hoopla about release windows and they are able to move new functionality into production at a much faster clip.

    1. Information: Focus is on Dissemination vs. Actionable

    Both types of organizations generate and share a ton of data.  The difference lies in how the teams uses data.  In Traditional IT, information is generated by specialists (e.g. operations team), bundled together with other data into a massive report, goes through management approval, is then shared with other managers, who then send it out their specialist (testers, developers etc.).  In most cases the report goes unread, simply because there is too much data, not timely enough, and / or not the right data.  In other words, information is shared but poorly consumed, and rarely used to take any actions.

    Within a DevOps organization, it’s the team cell that gather and creates the data.  Since the data is to be consumed locally they only collect the data (also automated) that the team deems is necessary.  And since the data is processed within the team, it eliminates the time lag of creating lengthy reports, manager approvals, and queue time (sitting in some ones email box).  As a result, the team is quickly able to read and react to the data – resulting in faster feedback time.

    1. Culture: Do Not Fail vs. Fail Early

    Traditional IT is fundamentally a risk averse organization.  A CIO’s first priority is to do no harm to the business.  It is the reason why IT invests in so much red tape, processes, approvals etc.  All focused on preventing failure.  And yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.

    A DevOps organization is risk averse too but they also understand that failure is inevitable.  So instead of trying to eliminate failure they prefer to choose when and how they fail.  They prefer to fail small, fail early, and recover fast.  And they have built their structure and process around it.  Again the building blocks we have referred to in this article – from test driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this mindset.

    Measure

    1. Metric: Cost & Capacity to Cost, Capacity, & Flow

    Silos work for Traditional IT because they are directly tied to its measurement model – cost and capacity.  IT is measured on how much can it get done (capacity) for the least amount of money (cost).  It is no wonder that cutting cost, while trying to keep capacity constant, has been all the rage in IT over the last two decades.  It’s one of the reasons why outsourcing became so popular.

    And while this model works great for stable legacy applications, the newer “system of engagement” applications require an additional metric – the element of time.  DevOps organization understand this paradigm shift and have added “flow” as an additional metric.  Flow forces an organization to take a look at its end to end cycle time, identify areas of waste, calculate true productive time, quantify quality, and focus on activities that add the most value.

    1. Definition of Done: “I did my job” vs. “it’s ready to deploy”

    Another source of measurement difference is in how the two types of organizations define done.  In Traditional IT, done is defined as the specialist doing “just” their part and handing it off.  In essence, their done is more focused on meeting the hand off deadline vs. making sure what is done is deployable.  This varying definition leads to some bad habits such as sub-par work, quality loss, finger pointing, and an overall sub-optimized process.

    To get way from the finger pointing mentality is another reason why a DevOps organization prefers to create a dedicated cross functional team.  With every member of the various parts of IT represented in this “cell” structure and all of them being held accountable for one and only one thing, they all end up having just one definition of done – bring quality software to market.

    If you would like to learn more about this topic, please join me at IBM InterConnect 2015.  I will be hosting several workshops on DevOps.

    About the Author/Mustafa Kapadia

    Mustafa KapadiaMustafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery.  He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

    Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

    Connect with Mustafa LinkedIn| Twitter

  • A DevOps Metamorphosis Echoes Through Reverb.com, Part Two

    A DevOps Metamorphosis Echoes Through Reverb.com, Part Two

    Last time, Dear Reader, we discovered how the Reverb.com e-commerce platform, which connects musicians who want to buy and sell gear sought to automate its workflow software development process, speeding deployment and bug fixes while satisfying customers who want to trade in an online environment that is as efficient and error-free as possible. Now, the conclusion of how Reverb.com adopted DevOps to achieve its goals.

    A Round Trip Through The Reverb.com CD Process

    Today, Reverb.com focuses the DevOps mindset on its development process using Chef [https://www.chef.io/chef/] for configuration management and Jenkins [https://www.cloudbees.com/jenkins/about] for continuous delivery and infrastructure automation tasks. Developers use Capistrano [http://capistranorb.com/] for rolling out new code changes and running database migrations, CircleCI [https://circleci.com/] for unit testing code, and DataDog [https://www.datadoghq.com/] for metrics and monitoring. “We also use Logstash [http://logstash.net/], ElasticSearch [http://www.elasticsearch.com/about/], and Kibana in unison for log aggregation and visualization,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com.

    In a typical deployment for Reverb.com, an engineer clicks the build button in Jenkins, which triggers a Chef converge on all the web heads. This rolls out any configuration changes that the developers have scheduled to go out, such as NGINX configuration updates or app config updates.

    Once this is finished, the system signals the application to perform a Zero Downtime Deploy in which the application reloads without dropping any requests. “If this process finishes successfully, Jenkins kicks off a code deploy using Capistrano and simultaneously deploys new code to our web application servers. If a database migration is scheduled to be run, that will also happen on one of our worker boxes,” says Enger.

    This may sound like a standard deployment so far. But that’s where DataDog and the ELK stack come into play. “That’s where we really gain the benefits of CD/CI and DevOps,” says Enger. Every time the development team converges Chef, they use a Chef Handler to submit an event to DataDog. And once Capistrano starts and finishes, another event goes to DataDog. The team has configured Logstash to watch the application logs on the web heads and to send an event to DataDog once the system renders an error. “The single point of information now becomes DataDog because we can see into our config and code changes and immediately know when a bad change goes out the door,” says Enger.

    The last thing the development team looks at is the performance logs in Kibana, to see how the application is performing based on the logs it is generating. “With the ELK stack in place, the team can answer questions like: Are there any 500 errors? If so, how many and how often are they occurring? Did development increase or decrease application response times? Do people like the new feature rolled out at /my/feature?,” says Enger.

    Hurdles

    The biggest hurdle in moving to DevOps is the mental shift you have to make from that of a traditional systems administrator or developer to a more DevOps-based mindset. This means the operations side learns coding and development learns about load balancers.

    “If we propagate continuous change in our computer systems, we should also expect to be continuously changing our mental models to adapt to new ways of doing things,” says Enger.

    Benefits You Can See Through

    Transparency is the number one thing the Reverb.com developers gained from this new CD approach. “Capistrano tells HipChat [https://www.hipchat.com/], “Hey, code is going out!” and I know to pull up my dashboards in Kibana and DataDog to watch for any anomalies,” says Enger. Humans can’t always be expected to communicate to each other efficiently. Computers help the Reverb.com team communicate across the organization through programmed notifications as to when changes go out.

    Code deploys are now predictable and automated. No more worrying about a failed deploy due to a bad connection. “When failure does occur, we are able to open up the Jenkins Build output and find the error pretty quickly,” says Enger.

    A Nice Surprise

    One really positive result that surprised the development team after implementing CD/DevOps is that they learned a lot about their user behavior and traffic patterns. “Now we know that 4.30am is the slowest period on the site,” says Enger.

    The team knows exactly when is the best time to roll out big configuration or code changes. They know immediately how deployments affect users and especially whether they ever break critical paths in the site, such as the checkout or signup functions. And the automated approach that DevOps provides enables them to fix bad changes within minutes of their original deployment.

    The Affirming Roar of Fan Feedback

    Reverb.com recently introduced more automation and monitoring to prevent site downtime and abuse by bots. The feedback: Customers don’t care about DevOps; they care about sites being operational and efficient. But it’s DevOps that makes that happen!

  • A DevOps Metamorphosis Echoes Through Reverb.com, Part One

    A DevOps Metamorphosis Echoes Through Reverb.com, Part One

    “Reverb.com is an e-commerce platform that connects musicians, enabling them to buy and sell used gear and instruments,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com. As an ecommerce site, it is critical to Reverb.com that its workflows such as checkout, offers to purchase, and messaging work in tandem with its selling workflows, which include creating new listings and managing orders.

    The Reverb.com team also develops a free, comprehensive online price guide and a complex in-house content and curation system that allows them to hand pick and recommend the best gear for their customers. In the Reverb.com marketplace, all software components must run smoothly for customers to have the optimum research, buying, and selling experience when conducting trade on musical equipment.

    To create those components, Reverb.com developers code primarily in Ruby on Rails on Macs. The etailer runs Linux in its production environment. “Our backend stack includes PostgreSQL  for our database, ElasticSearch powering our search engine, and Redis for queueing and key value storage,” says Enger.

    Not So Good, Old-Time Deployment

    Prior to DevOps approaches, Reverb.com deployed code from individual user workstations. “For configuration changes to our infrastructure we had to use SSH to connect into production. All while our fingers were crossed, of course,” says Enger. This loose-knit deployment process presented Reverb.com with some obvious challenges.

    “If a deploy failed due to an engineer being on a bad internet connection, we could potentially have had two updated hosts while three were still running an older version of code,” says Enger. This was especially problematic because these deployments often included database migrations.

    There was also no real way to know when a deployment failed or who started the deployment. “The process was devoid of any deployment log records, so errors and anomalies were harder to spot when the deployment was swallowed in the developer’s terminal,” says Enger.

    These risks made it possible for Reverb.com to face an even bigger risk: a lapse in faith on the part of its customers. Reverb.com’s reputation as a marketplace hinges on the perception of trust among its customers. Every aspect of the organization generates trust. From how Reverb.com interacts with customers as people, to how its systems present to users, the firm wants to exude confidence. “If we experience outages and slow performance, we may erode our trust with our audience,” says Enger.

    Goals For DevOps

    Reverb.com had many goals for DevOps in the hope that it would meet their challenges to fast, scalable, reliable delivery of code and configuration changes. Reverb.com needed to know when a change was going out. The firm needed to be able to deploy at a moment’s notice.

    “All processes must be behind the click of a button or an SCM post/pre trigger,” says Enger. All Reverb.com deploys must be logged and graphed. Scaling the team and the business should become easier if the teams have quick, repeatable processes that allow them to push new features and bug fixes quickly onto the new DevOps platform. This should allow for a scalable and dynamic workflow for the Reverb.com marketplace so it can be nimble, adapting to the users’ needs as they arise.

    Reverb.com’s Reasoning: An Automation Proclamation

    Reverb.com made its move to DevOps on the basis that the more the automation, the less the likelihood of human error. From that, Reverb.com development staff should create fewer issues for its customer base as consumers browse the website, make purchases, and sell hardware.

    “It’s extremely important to look at root causes here: if we only treat the symptoms every time the site slows or goes down, we will be playing Whac-a-Mole forever,” says Enger. By using the DevOps mindset, initiating repeatable and automated deploys, and measuring performance and application metrics in rea-time, Reverb.com developers should be able to trace when something was released, how it affected users, and whether customers are having a poor experience as a result.

    In the hopes that this would equip them to quickly pinpoint issues and fix root causes, the band of coders at Reverb.com set out on the road to DevOps, fired off its first continuous delivery power chord, and awaited feedback.

    Conclusion: Cliffhanger

    Tune in next time when staging-devopsy.kinsta.cloud reveals how Reverb.com adopted DevOps and used it to meet these development challenges and goals in the exciting conclusion of “A DevOps Metamorphosis Echoes Through Reverb.com”.

  • Why DevOps is like fitness or religion

    Why DevOps is like fitness or religion

    What is DevOps?

    I noted in a blog post last summer that limiting DevOps to a single definition is challenging. The Wikipedia entry for DevOps contains a variety of possible explanations that seem to disagree in some ways:

    • It’s a “software development method that stresses communication, collaboration and integration between software developers and information technology (IT) operations professionals.”
    • The goal of DevOps is “to maximize the predictability, efficiency, security and maintainability of operational processes. This objective is very often supported by automation.”
    • It “targets product delivery, quality testing, feature development and maintenance releases in order to improve reliability and security and faster development and deployment cycles.”
    • It “aids in software application release management for a company by standardizing development environments.”

    All of these are true, and yet none of them is necessarily THE definition of DevOps.

    A recent ZDNet post titled Why there will never be one do-it-all DevOps tool proclaims, “There is really no such thing as a specific DevOps “tool”—and don’t buy into the “DevOps” labeling many vendors are attaching to their solutions. Successful DevOps depends upon a multitude of tools and methodologies that can help organizations better align their software development and release cycles.”

    The question, “What is DevOps?” isn’t like asking, “What’s the temperature today in Phoenix, AZ?” or, “What’s the fastest land animal?” Those questions have precise answers that are based on measurable, irrefutable metrics. The answer is the answer and it isn’t really open to debate.

    “What is DevOps?” is more like asking, “What is the best fitness program?” or, “Which religious belief system should I subscribe to?” There is a vast array of answers and passionate beliefs on all sides. But, there really isn’t so much a “right answer” as a “right answer for you”.

    When it comes to fitness, there seems to be as many diet and exercise programs as there are people trying to get fit. Books, magazines, and infomercials are filled with extraordinary examples that “prove” that a given fitness program or philosophy works…at least for the subject of the story. There are some elements that are universal truths like exercise is better than being sedentary, and making healthy eating choices is better than eating an entire cheesecake for dinner. Almost everything else is subjective. There are too many variables involved that make people, and how they respond to different diet and exercise programs, different.

    DevOps is also like religion. Most people don’t choose their faith. They’re raised from birth to follow in the footsteps of their parents, and their religion is more or less predetermined. Religion is also personal. While some will insist that there is only one true faith and that theirs is the only real religion, most people recognize that religious faith is very subjective and it’s a function of finding the belief system that makes sense to you.

    DevOps is like fitness and religion in that what it means to an organization, and how it is first implemented, is influenced by the existing culture and infrastructure—the IT the company was “raised on”. DevOps is also like fitness and religion because there is no “one-size-fits-all” answer to the question “What is DevOps?”

    It isn’t a defined process—it’s more of a concept or philosophy. It isn’t a product or service that can be packaged and sold—it’s more a collection of the products and services that enable you to embrace the concept or philosophy. To some extent, DevOps is whatever works for your organization to automate routine tasks, reduce redundant effort, streamline operations, and remove unnecessary obstacles so people can get things done as efficiently as possible.

  • Clearing through the app backlog clutter

    Clearing through the app backlog clutter

    Development and operations teams today are under more pressure than ever to develop apps the business needs to make its workforce productive, its partners connected to the resources they need, and to deliver the apps to the marketplace that will make it competitive.

    The number of production hours for development is at a precious premium as demand for apps soar and resources remain tight. According to Gartner, the demand for enterprise mobile apps alone will be a $40 billion market by 2016.

    Of course, the inevitable happens and the list of designed applications gets backlogged. They get caught-up in budget requests that drag on, and then they get caught in the long list of requested apps in the development queue. A recent survey commissioned by PaaS provider OutSystems found that 85% of enterprises surveyed had an app backlog, with about half having backlogs of 10 to 20 apps.

    App Backlog, agile and DevOps

    Sure, development and operations teams move as quickly as they can to meet the enterprise demand for new applications, but when there is any deep app backlog, there is the potential for business costs when it comes to lost opportunity, untimely product development, and future missed sales. In a time when agile IT means competitive advantage, any amount of latency that can be cleared from the application backlog is crucial.

    Agile development approaches and agile IT departments tried to remedy some of these issues, but didn’t go far enough. Fortunately, DevOps picks up where agile left off in the evolution of enterprise development. And in the face of agile IT, continuous integration, continuous deployment, and the rise of DevOps – operations, development, and infrastructure bottlenecks become minimized – and infrastructure and applications move more quickly, and existing applications are iterated more swiftly.

    Of course, DevOps itself, can help to accelerate app deployment, in addition to continuous integration and continuous deployment efforts. And all of the associated deployment automation tools and deployment processes are essential to eliminating manual waste so teams can focus on pressing business demands. This waste can include time burnt on manual process, manual testing, and the configuration of the software development environment.

    DevOps-driven organizations have shorter app backlogs. A recent CA Technologies study in fact found that organizations embracing DevOps report a 17 to 23 percent improvement in certain business outcomes, such as increased revenue, and faster time-to-market.

    Cloud architectures and easier to master development languages also help, not just infrastructure services, but also platforms that help to free up operations teams and developers to focus on the applications that need to get out yesterday.

    Even with DevOps practices in place and automation running full throttle, the business development teams still find themselves having to make difficult choices about which apps to build and deploy. Not all of these apps will survive, in fact, many apps will go unused and dormant. This means that a lot of time can be lost developing apps people don’t want to use, or are too cumbersome to bother to use.

    Turning to the Minimum Viable Product

    In addition to DevOps, development teams are also turning to approaches such as the Minimum Viable Product (MVP) as a way to help identify apps that have real user demand and those that do not. Essentially, the MVP app is an app that consists of only the essential. The MVP provides the essence of the app and what users need to get done and no more. The MVP is a potent way to provide the essential attributes of an app to see if users embrace it before building the app out further. If the app is used, new features can be added over time.

    Not everyone is convinced that the gains made against app backlog aren’t necessarily illusion. David Mortman, distinguished engineer, says “DevOps and MVP both have the ability to create the impression that you are solving backlog issues. This is because in both cases you are enabling the development organization to get that first release as early as possible and then to keep iterating it and adding features as the dev group goes,” he says. “This however may not get you to a particular defined endpoint any faster since regardless of the organization there are always resource limitations.”

    Still, “One advantage of DevOps and Agile is that they let you interweave development cycles so multiple apps can be worked on faster and each can be incremented more quickly. They also make it easier to adjust priorities between apps more quickly and more quickly identify apps that aren’t meeting the business needs and either deprioritize or eliminate the app before overinvesting in its development,” Mortman adds.

    There’s no doubt, based on my discussions with enterprises, app backlog repeatedly comes up as a serious challenge to competitiveness. But by embracing continuous integration, deployment, and MVP development, organizations can go a long way to help clear through, at least the initial stages of their app backlog. What is your organization doing to help clear app backlog? We’d like to hear your experiences and lessons learned.

  • Shooting a Rocket Over Docker’s Bow

    Shooting a Rocket Over Docker’s Bow

    As momentum continues to build for the Docker containerization platform, some of the open source project’s biggest contributors argue that it has lost its way. Priorities have shifted at Docker from the ideal of building a standard container to instead supporting an entire containerization ecosystem. According to the developers at Linux disto company Core OS, it’s this move has left the entire Docker process “fundamentally flawed.” It’s why they announced this week a competing container runtime called Rocket that they hope will become the “simple composable building block” they’d originally envisioned Docker would be.

    “Unfortunately, a simple re-usable component is not how things are playing out,” wrote Alex Polvi of CoreOS in a blog about the launch. “Docker now is building tools for launching cloud servers, systems for clustering, and a wide range of functions: building images, running images, uploading, downloading, and eventually even overlay networking, all compiled into one monolithic binary running primarily as root on your server.”

    That’s an inherently risky prospect for security-conscious organizations, Polvi argues. What’s more, as Docker starts moving toward expanding the capabilities of its platform it will only increase its complexity and diminish its ability to integrate containers elsewhere. Which is why CoreOS is leading the charge for Rocket. Introduced as a prototype on GitHub this week, Rocket is a command line tool designed to run what it calls ‘App Containers.’ These can be a specification of an image format, container runtime or discovery mechanism.

    “Rocket is the first implementation of an App Container, but we do not expect it to be the only one,” Polvi says. “An open specification allows other systems do their own implementation of App Container without using Rocket at all. CoreOS fully supports and embraces alternative implementations.”

    It’s a big move considering that the company’s co-founder and CTO, Brandon Philips, is a huge Docker contributor and on the Docker governance board. However, CoreOS argues that the clients that run Docker on CoreOS are clamoring for something with the kind of flexibility that Docker seemed to promise early on.  So while CoreOS will continue to support Docker long into the future it hopes Rocket can help solve some problems that many of its clients have with the platform.

    “Our primary users have existing platforms that they want to integrate containers with,” Polvi says. “We need to fill the gap for companies that just want a way to securely and portably run a container.”

    Polvi says that as CoreOS designed Rocket around four major factors that its team believes are key for good design of a standard container. First, it should be independent and composable. Second, container images should be designed for easy, distributed retrieval. Third, it should be open so that independent implementations of tools can run in the same container consistently every time. That’s why CoreOS says that it’s developing an open specification so that other systems can implement an App Container without even using Rocket.

    And, fourth, security should be tantamount.

    “Isolation should be pluggable, and the crypto primitives for strong trust, image auditing and application identity should exist from day one,” Polvi explains. “Security primitives are very important to us, so we added an identity feature to the meta-data service. This means every instance of a running container is given a unique identity, coupled with a lightweight HSM-like service for signing.”

    This is a new approach for container environments, so CoreOS is especially seeking community feedback for this design. However, the entire project is essentially ready and waiting for community support. Those interested should  out the details on GitHub.

  • Automating Away the Regulatory Compliance Myth

    Automating Away the Regulatory Compliance Myth

    At the recent DevOps Enterprise Summit, we had a chance to sit down with Justin Arbuckle and discuss his concept of “compliance as code” and why it’s so important that security and compliance tests be built into DevOps practices at large enterprises. Recently, Arbuckle was chief architect at GE Capital where he introduced and sponsored DevOps at that organization – so he certainly knows about what it takes to “do DevOps” in large, highly regulated environments.

    Currently, Arbuckle is the vice president of EMEA at CHEF and also chief enterprise architect there. If you watch his talk from the DevOps Enterprise Summit, Hunting the DevOps Whale, you’ll see that he is passionate about the transformative potential of DevOps in large enterprises.

    At CHEF, Arbuckle is in helping large enterprises understand how to “de-risk” their migration to cloud with automation. We caught up with him at the DevOps Enterprise Summit to ask why that is so important. Here’s what he had to say.

    staging-devopsy.kinsta.cloud: When it comes to DevOps and regulatory and security concerns, many people say, “Well, just automate those tests.” What does that mean, to you? How do you “automate in” security and compliance?

    Arbuckle: I think, in terms of the specific issue of regulatory compliance, a lot of what we think of as the being compliant process today is a complete myth. There is just so much detail that needs to be checked that the organization is always in a position where they have to trade off against, “It’s good enough, we’re ready to go” versus, “We’re not going to go anywhere until we’ve literally crossed every T and dotted every I.”

    The reality is that a lot of the regulations, Dodd-Frank is a good example, are getting to a level of detail where they’re beginning to talk about process. They’re beginning to talk about how you design product. They’re getting very deep indeed.

    So the organization that has to govern that, can’t do it based on just this idea of oversight at particular points in the life cycle. Because if you’re going to do it properly you literally slow your velocity to zero, and so you’re making a trade-off between compliance and competitiveness.

    The way to break that cycle is to, and this is a big paradigm shift for regulators, a big paradigm shift for compliance people inside organizations, you have to start being able to express those compliance requirements in a form that’s codable. Using something like Chef for instance what we’re able to do is we’re able to specify a particular policy in a way that is automatable. In a way that we know is always going to be implemented every single time.

    staging-devopsy.kinsta.cloud: Would that include input from security people, compliance teams, design teams, system architects?

    Arbuckle: Absolutely. Honestly, in my experience I think the number of organizations that can count fully detailed, fully implementable — and that’s the key word — security policy by their infrastructure people is countable on one hand.

    Generally speaking, there are many obvious guidelines that are standards-based. But as the security organization tries to keep up with security threats, they’re obviously learning as they go. In this way policy always lags. And the only way for the organization to catch it is through this process of documentation, policy, and checks. And through it all they know that the standard is nonsense because it’s out of date by definition. So they have to create a point-in-time review. And these point in time reviews bring velocity to a halt.

    staging-devopsy.kinsta.cloud: Yes. Especially when something is wrong. They hit a red button and everything stops coming off the assembly line.

    Arbuckle: That’s right. The approach that we have to get to is the idea of policy as code. But you are not going to be able to, on the first go, be able to codify everything. Very often with automation, based on my experience, is that the difficulty at large enterprises has actually less to do with the tools and more to do with knowing what you want to automate. Knowing what you actually want to put in infrastructure as code.

    It’s the same thing with security teams. Ask them to go through the most important things that need to be tested in automation, and getting them to go through that process. It’s often difficult because they are not development people typically. They’re not infrastructure people, And they’re not operations people.

    They’ve kind of created their own “securocracy” if you like.

    What security teams have to do is understand their work is subject to source control as well. Their work is subject to variation and evolution over time and we should write tests to ensure that the following policies are being applied consistently.

    DevOps: And I assume these tests are designed over time, as one goes along? For example, if there’s a new malware or regulatory beast that you haven’t seen before, you can design a test and then implement that as part of the life cycle?

    Arbuckle: That’s exactly right. And the great thing about tests is they are additive. You’ve got a test that tests for the range of ports that are open on the operating system and then you add another test, maybe testing for the types of applications that are installed on the box. They are two separate tests and they are additive. So as you continue, you build up your library of tests.

    And for most organizations going through the adoption of test driven development, as an example, they have to do it that way. A large application that is beginning to develop in an agile way is going to have a lot of tests. If you take a typical large core banking application, for instance, it will probably have something in the region of about 5,000 use cases. That’s thousands and thousands of tests. Let’s just agree that in no short period of time are you going to write those tests.

    staging-devopsy.kinsta.cloud: That’s an enormous catalog of tests. How do you keep them up to date and fresh? I imagine as security changes, regulatory compliance changes, you may have to tweak these tests as you go along. How do enterprises keep up?

    Arbuckle: Like any requirement you have stakeholders that are responsible for those requirements. In just the same way, if you had for instance a mobile application and you have a product owner for that mobile application and that product owner is prioritizing feature sets you should also have a product owner for compliance.

    And the product owner should be detailing what the set of compliance requirements are for the product.

    Essentially compliance is embodied in the tests that are run against the appropriate applications. And so to manage automation and compliance as code, you need to have a product owner for compliance. And compliance as code, or automation, is the way it has to be in order for compliance to actually be true and consistent, as opposed to a myth.

  • Eight critical DevOps practices: innovate, deliver, repeat

    Eight critical DevOps practices: innovate, deliver, repeat

    DevOps has been garnering a lot of interest for some time. Discussions and debates on forums like staging-devopsy.kinsta.cloud are improving our collective understanding of the right approach and now we are seeing concrete proof points of fast-movers applying DevOps practices to disrupt entire industries with rapid-fire, digital-innovation based on Cloud, Big Data and Mobile technologies. I have personally been part of this transformation at IBM for several years and would like to reflect on what we have learned, and also share with you results of a recent study from Forrester Research.

    DevOps helps organizations address a critical business challenge – that of continuously capturing and responding to feedback (both internal and external) so we can rapidly and repeatedly turn innovative ideas into highly relevant, and desirable, products and services.  This involves not only development and operations, but all aspects of the organization, as it entails the collaboration across the entire organization including business and supply chain. Compared to single-direction,  traditional models, DevOps is bi-directional and is based on not just pushing something out to the customer but also on getting their feedback to make improvements.

    In a recent webcast, I discussed eight DevOps practices that speed up software delivery with Kurt Bittner, Principal Analyst at Forrester and a leading expert on agile and iterative software development. This webcast highlighted a recent Forrester survey on DevOps practices, cycle times and where enterprises are in terms of the maturity of DevOps practices adoption. 
    You can watch the webcast replay here to know more –     ibm.biz/Forrester-talk

    Their survey of 600 IT professionals with application development responsibilities found that about a third of teams consistently deliver at cadences of one to three weeks and that the fastest teams generated higher business satisfaction than slower teams, an indication they don’t sacrifice quality to deliver fast. The study reveals how these teams did it: by adopting the following eight DevOps practices:

    8_DevOps_practices

     

    1. Minimum Viable Product – Deliver the smallest viable product or application and then improve it in subsequent rapid releases, based on feedback, and delivering value incrementally.
    2. Dedicated Teams – Form dedicated, cross-functional teams, avoid over-specialization and ensure they are not interrupted by side projects.
    3. Loosely Coupled Architecture – Use loose architectural coupling within and between applications to reduce complexity and enable delivery in small increments.
    4. Minimize Hand-offs, Maximize Flow – Eliminate unnecessary steps, delays and friction between steps to increase the flow of work.
    5. Deliver in Small Batches – Deliver in smaller batches helps to expose the most uncertain issues first and enables feedback on the most valuable usage models earlier.
    6. Transparency – Transparency into progress helps everybody to see where you are at any point in time and also visibility into risk without decreasing productivity.
    7. Eliminate Overhead –Eliminate the overhead of periodic and manual status reports and status meetings which absorbs valuable time and distracts focus from delivering value.
    8. Automate Testing using APIs – Shift away from heavy manual testing to improve delivery speed, quality, and testing accuracy, and dramatically reduce cost.

     

    The study refutes the myth that releasing fast requires you to sacrifice quality and accept higher risk. Adopting these eight DevOps practices, especially automated testing and delivering in small batches, does exactly the opposite:  small incremental releases are more thoroughly tested, more stable and lower risk.

    The best part about DevOps adoption is the flexibility to choose which practices to adopt first, based on each organization’s priorities, and then to incrementally adopt the rest of the practices as needed to optimize the software delivery lifecycle and accelerate innovation.

    To learn more about the survey and the eight DevOps practices, download the Forrester study along with the approach that you can follow to adopt these as per your organization’s current state of delivery cycles.

    About the Author

    ashokAshok Reddy is responsible for Product Management & Strategy, product Design/user experience and Business Development for Rational Software Division of IBM Software Group and for the IBM DevOps portfolio.  He is based in Raleigh, NC, USA.  Ashok’s role involves all aspects of global product delivery including setting product strategy and for product line management, establishing product roadmaps and partnerships for IBM DevOps portfolio.

  • Being Agile is all about controlling the flow

    Being Agile is all about controlling the flow

    For businesses, Agile is the new fashionable initiative as was E-business was in the 2000’s, Six Sigma was in the 90’s and Total Quality Management (TQM) was in the 80’s. If we look back across the spectrum of these initiatives, we need to examine how many of these initiatives met the goals the business was looking for and, more importantly, if they were fully adopted, remembering that many were derailed, did they deliver the value the business was seeking?

    Of course we cannot answer this question with generalities—it’s a spectrum. For some businesses, these initiatives were red herrings, some did not follow through to completion, but certain changes adopted as part of these initiatives had a very positive impact, while others succeeded wildly. It’s a Harvard Business Review case study as to why a business ended up at the point on the spectrum that they did.

    Clearly, one fundamental aspect that is consistent across all these initiatives is the ability for a business to embrace change. In a previous blog, “Cloud Computing, DevOps and Business Velocity,” I addressed the issue of business velocity—the rate at which an organization can embrace a particular disruptive technology change. Underestimating your business’ ability to adopt a new technology or process will most likely have a strong negative impact on the outcome. In a recent conversation with a client, the non-IT business areas has been very successful at adopting lean practices, but IT short-circuited shortly after starting. It didn’t outright fail, but after a strong start, they took a break from which they never started back up. Without a post-mortem, we don’t know why this specific initiative fell flat, but we can intimate that if something is difficult and feels unnatural, we’re more easily able to focus our attention elsewhere—think dieting and exercise programs in your own life.

    This seems to be the decade of Agility. Businesses want to be more dynamic and fluid. They want the organization to be prepared to respond to market changes more quickly, introduce new products faster, and respond to problems that occur more rapidly. To accomplish this goal, the business will have to let go of some of the ways that they do things today. They will have to re-engineer processes to be more compose-able and componentized so that they are less static. For example, a manufacturer goes through a lot of planning to deliver a new product to market. They go through steps for design, prototype, test, source, manufacture and distribute. The entire cycle may take upwards of a year and is highly automated once it reaches the manufacture stage. Tweaking the product between batches would cause  significant delays and pain because the process is static. It effectively amounts to restarting the entire cycle. So, what if they discovered that by simply elongating the body by 2 cm the product was far more useful their intended audience? In this case, agility might equate to being able to complete the design-to-manufacture process faster.

    In the previous example, this latter step of being able to tweak the product is called continuous improvement and is a key element of fostering agility. If a product isn’t selling well in the market, many companies will just abandon the product completely due to the costs associated with making the product viable. The reason being that most manufacturers consider that each product is a project and when it reaches the manufacturing stage it moves under the control of operations. That is, the creative team that includes marketing, finance, and other stakeholders view their position as “completed” with regard to getting that product to market and they then move onto other projects. With this mindset, there’s no room for the group to review the process, think about how it could have been done better and then iterate with the improvements incorporated. Hence, they just keep taking their “this is the way we do it” to the next project.

    Being agile means being able to control the flow. There’s no way to change how products come to market if there’s a culture of accepting finality once a product has reached a certain point in the cycle.

    The other aspect of controlling the flow is managing how products and services enter and exit the development pipeline.  I spend a lot of my time helping IT organizations unlock their potential to deliver by helping them to understand that there’s only so much that can be done to improve managing the work-in-progress (WIP). The only way to make gains beyond a certain point is to enable transparency with those that are responsible for inserting work into the pipeline or who are demanding that work completes and exits the pipeline at a certain time.

    The following two figures express well the reason transparency is a requirement. I believe most people would agree that managing production is a far easier task in Figure 1 than Figure 2. In Figure 1, there is one input and one expected output. Managing WIP for this pipeline is a matter of optimizing just the process for completing that one deliverable. In Figure 2, there are many more variables at work. There could be contention for resources within the pipeline to support the multiple inputs.

    wip

     

     

     

    wip2

     

    Likewise, there may be a request to reprioritize what gets completed when to meet certain business goals. Perhaps a particular report is required urgently to support a key business decision. I personally dealt with this very scenario when the business I was working for years ago was suddenly engaging in acquiring another company. All focus was transferred to producing the needed financial statements and everything else was put on the backlog. In that particular case, the CEO and CFO were the customer, so no one questioned the delays to their projects, but for many IT organizations reprioritization requests are never that clear making it very difficult to manage the control of the flow. Hence, transparency and communication between those consuming the services of the pipeline and those who own the control of the flow of the pipeline is an imperative for agility.

    Like TQM, Six Sigma, and E-Business, Agility will require changes to the fundamental way that a business operates. An analogy of this process can be as benign as a person recovering from knee surgery or as complex as recovering from a stroke depending upon what the business wants to accomplish and how they are organized currently. Along the way, people will most likely be the biggest hurdles to success.

    When change of this magnitude occurs, employees tend to become very myopic of their own situation. To mitigate this risk, it is critical that changes are reviewed with employees prior to change, during implementation and after the change is enacted. Management must gauge if employees are struggling because the complexity of the change or because they personally don’t agree with the change. The former can be overcome with education and support while the latter should be dealt with quickly through removal.

    Businesses can succeed with their Agility initiatives given they identify and mitigate key risk factors prior to starting and are willing to commit to looking critically at the way they have been operating the business. In agile software development, developers are trained to think about ways their code can break and then write unit tests to ensure that the software is sustainable in light of these possibilities. Creating an agile business means looking at ways to ensure sustainability in light of market changes, new competitors, supplier changes, and changing customer demands.