staging-devopsy.kinsta.cloud

Category: Features

  • Web-scale IT drives Chef momentum

    Web-scale IT drives Chef momentum

    Chef today announced a series of metrics and features showing their support of and success as a result of what they call the shift to Web-Scale IT. As the picture below shows the Web-scale IT era has given rise to an exponentially larger, more scalable IT infrastructure.

    webscale

    What is Web-sclae IT? According to Chef:

    Web-scale IT brings the practices of large, web innovators into every business

    • Web-oriented architecture
    • Cloud capable including hybrid
    • Commodity hardware
    • Open source software
    • DevOps practices

    But I don’t want to give the impression that Web-scale IT is a Chef invention. Actually it is Gartner that is providing a lot of the analysis on the key drivers here and forecasting its heightened importance in IT. From a recent press release from Gartner:

    By 2017, Web-scale IT will be an architectural approach found operating in 50 percent of global enterprises, up from less than 10 percent in 2013, according to Gartner, Inc. Web-scale IT is a pattern of global-class computing that delivers the capabilities of large cloud service providers within an enterprise IT setting by rethinking positions across several dimensions.

    Barry Crist, CEO of Chef said, “Today, every business is a software business. Customers everywhere expect delightful and immediate services. Meeting these expectations is only possible through the speed, scale, and consistency provided by employing web-scale IT. Chef’s IT automation platform packages the success patterns of web innovators and delivers them to the enterprise,”

    Here are some of the elements or patterns of Web-Scale IT:

    patterns of webscale

     

    Having been through so many IT adoption cycles (I remember the big move to client/server), I am always weary of any new “paradigm changing” scenarios. If they were all true I would have retired an Internet millionaire back in the first dotcom bubble. However, what Chef is calling out here is certainly supported by the facts. IT today because of the Cloud, Big Data, virtualization, mobility and more has fundamentally changed. Automation, agile and lean principles are enabling the leveraging of these technologies more every day.

    Companies like Chef are capitalizing on these trends by providing solutions that both leverage and enable Web-scale IT. It is hard to argue with the numbers from Chef:

    • 182% Incremental Revenue Growth YoY
    • 100% New Enterprise Customer Growth YoY
    • Over 700 paying customers
    • 70% of total Sales to F1000
    • 600,000 Chef downloads per month
    • 55,000 Registered Chef Community members
    • Millions of Managed Instances

    As part of the announcement around their momentum Chef also announced some new product features. From their press release:

    –        New Analytics Platform, Acquisition of Tower 3: Chef’s new analytics platform provides full visibility into all activity on the Chef server and lays the foundation for audit and compliance capabilities. Chef recently acquired big data and analytics startup Tower3, adding significant engineering resources and expertise to its ongoing analytics development efforts. Now available to Chef subscribers is the new Chef action log, which publishes notifications on who is changing what on the Chef server and allows administrators to track cookbook usage, roles, environments, and changes to infrastructure, all through an easy-to-use dashboard.

    –        Test-Driven Infrastructure: Test-driven infrastructure ensures maximum IT safety and consistency with programmatic and automated testing of the entire stack. Chef is the first company to offer its customers commercial support for test-driven infrastructure. Chef has also created a single installation package – Chef DK – for the open source tools covering the entire test and development workflow, from the workstation all the way to production. Rally Software is among a range of enterprise customers using Chef for a test-driven approach to infrastructure and applications (see separate release).

    –        End-to-End Container Workflow Automation: Chef is the first automation platform to deliver end-to-end management of container workflows. Chef Container is a build of the Chef Client that integrates with all Linux containers (LXC), including Docker, to enable seamless management of container resources. In conjunction with Chef Container, Chef is also releasing a Knife Plugin-in for Docker integration with Chef Container, enabling users to launch, configure and manage Docker containers. Together, Chef’s new suite of container tools and content fully automate the creation, management, monitoring and testing of containers, making them versionable, testable, and repeatable within a production workflow.

    If you are interested in learning more about Web-scale IT, Chef have also released a white paper: “Automation and Web-Scale IT”

    All very impressive from Chef. But I don’t want to convey the idea that Chef is the only success in Web-scale IT or DevOps. Other companies like Puppet Labs, Saltstack, StackDriver, Ansibile, etc. are all experiencing rapid growth by riding the Web-scale IT express. Behind them are a host (no pun intended) of new startups that are offering solutions in this area.  Maybe this time we really are seeing a paradigm shift.

    In the meantime congratulations to Chef on some phenomenal growth numbers. For those of you who have made it to the bottom here, how is Web-scale IT changing your life?

  • Is Your DevOps Team Ready For The Real World?

    Is Your DevOps Team Ready For The Real World?

    Ten years ago I had the biggest cultural shock of my career. It wasn’t witnessing poor collaboration across IT development and operations or whatever tech fiefdom you care to mention. Nope, these were minor issues compared to those facing me on a really interesting project – integrating Industrial control systems with IT management.

    Working with a team of product managers, coders and sysadmins, our task was to develop a standards-based application for monitoring a physical research facility. When I say monitoring, I’m not only talking about network and systems management, but also managing equipment operating in the  ‘real world’ – like manufacturing robots, pressurized compartments, control networks, smart meters, alarms and controllers.

    It was a new experience for the team. Not only were we faced with a new set of technical challenges, we had to quickly become familiar with foreign sounding terms (to us anyway), like programmable logic controllers (PLC’s), supervisory control and data acquisition (SCADA) and human machine interfaces (HMI’s). Add to this a very different set of stakeholders – like industrial and plant engineers, and the initiative started off resembling something closer to a multi-language speed dating session (only without translators) than it did a traditional IT development project.

    At the time projects like this were a rarity because the worlds of operational technology and traditional IT was seen as separate. Process control and electrical transmission systems ran on separate control networks with proprietary protocols, whereas IT ran on a business network and primarily handed back-office business systems – the two domains were as different as ‘chalk and cheese’.

    But all that’s changing courtesy of the Internet of Things and a software-driven economy.

    Today, organizations are realizing that standards, unified architectures and software applications can be used to connect and integrate the devices and sensors in the physical world with IT systems – the benefit – more seamless end-to-end business processing.

    Take for example manufacturing. If plant production equipment can be better integrated with consumer demand analytics, then optimum maintenance windows can be identified. Similarly in energy and utilities, integrating the physical world of smart meters with back-end IT customer information applications is a must for efficient billing.

    This convergence will drive new approaches to developing and operating applications. The DevOps movement with its focus on teamwork and collaboration to shorten delivery cycles while improving quality might just be the answer, but there are some serious challenges to overcome – not least cultural.

    You might think you’ve solved the great IT divide with DevOps, but that’s nothing compared to the cultural chasm you’ll need to bridge across the world of industrial operations and IT. And no amount of IT ego will help, which I learnt the hard way when I was once given a sharp lesson in what ‘real-time’ processing actually is business by a group of chemical engineers – boy was I humbled.

    Technology beat up aside I believe there are many ways DevOps can help address the challenges associated with connecting and integrating operational technologies – especially:

    • Standards guidance – since operational systems work in isolation and use proprietary systems and communication protocols designed for specific control system networks (e.g. Modbus and BACnet), DevOps teams can help provide advice on new standards and integration platforms.
    • Knowledge and Expertise – DevOps teams can provide expertise in the selection of new software platforms that are now available to build and run Internet of Things applications.
    • Security and Risk Management – as more and more software finds its way into sensors and operational control networks become connected to the Internet, DevOps and security specialists will have a key role in identifying and mitigating many new security risks.
    • Data Management and Analytics– connected sensors and control systems may result in the need to capture and analyse huge quantities of data. DevOps teams can help in areas such as application architecture and design, testing, network capacity and cloud automation.

    As operational and information technologies converge, I believe the purview of DevOps will increase considerably.  Now teams might include technology constituents from other areas of the business – like engineering and logistics specialists – each bringing new skills to the table that when coordinated with IT expertise in areas like mobility and cloud computing has the potential to drive huge efficiencies.

    Application management will change also. Development work will increase, but so too will need to bake quality and resilience into every release. This is especially important because as processes become more integrated (from the factory floor sensor network to a mobile enabled materials management system or from healthcare diagnostic equipment to patient records applications) any minor defect, security issue or performance problem can trigger more catastrophic business failures.

    When I look back to my own teams foray into managing ‘real world’ technology, I realize that while we never had DevOps to guide us we still used many principles. Success came when we built a cross-domain business team that maximized a unique set of technical strengths, but also understood each other’s weaknesses.

    So DevOps teams take note – as technology convergence accelerates, the goal for many industries won’t be just bridging the IT development and operations divide and releasing more and more software, it’ll also be about developing a new range of end-to-end digitized business processes – from the factory floor to the boardroom – that’s a bigger deal and where success will ultimately be measured.

  • Video did not kill the radio star – IT Ops Continues on

    Video did not kill the radio star – IT Ops Continues on

    The Buggles classic “Video Killed the Radio Star” may have been a hit song, as well as hotly contested idea in the early 80s, but given the retrospect of the last 40 years, we know that video did not kill the radio star – it simply changed him. Just as MTV failed to kill the radio star, DevOps will fail to kill IT Ops.

    Today’s dynamic virtualized and cloud-filled world has significantly changed IT. With Software Defined “Everything” and virtualized compute, network, storage, etc., everything is fluid.  And to top this off, bandwidth is unlimited and inexpensive; making it easy to offload and move things in ways we couldn’t have dreamed possible just a few years ago. As a result, the infrastructure supporting these technologies is highly distributed, and can no longer be served from one place. All of this is significantly changing the way IT Ops, well, operates. In order to understand how IT Ops has changed, we need to look back at how it was. We don’t need to look back far, however.

    Most organizations have individual business units. Be it a large organization with separate businesses, or a smaller one with separate departments, these business units often find themselves competing over shared IT resources. Five years ago, business units vied for the time of shared application developers that lived in IT. Now, with the explosive growth and maturation of rapid delivery application development technologies, and with more and more revenue generated or influenced online, it makes sense, that these business units lobbied to the CIO to bring the app developer and application support functions within their control.  Application development has become a critical part of most businesses today..

    Once this happened, the distribution of budget also changed, taking full control of the IT budget away from IT Operations. Today, as much as 30% of the IT budget is directed by the individual business units. This happened because the business units were not getting the prioritization they wanted from the shared application developer approach. In the eyes of the business, IT couldn’t move fast enough for them, or didn’t have the same priorities that the business unit had. No one was satisfied.  So, the executive teams moved the application development and support budget to the respective business units, partly to stop the bickering.

    Some of these business units took their funds and used them to outsource development or to purchase software-as-a-service applications. Once cloud platforms and inexpensive, unlimited bandwidth arrived, the business units had alternatives, and they used them.

    This is all logical for business units, but it does not come without risk to the organization. With applications developed, controlled, and delivered outside of the purview of IT Ops, there were concerns about whether these applications would meet HIPAA or PCI compliance standards, IT’s security standards, business continuity guarantees, or back up schedules. A compromise was needed. What most organizations have come up with is to have IT Ops handle the general infrastructure, and allow each business to do what they want with the rest – about 30 percent of the budget. How does this look?

    Most apps today are componentized. If you take a retail app for example, the product browsing, the shopping cart, the payment page, and the confirmation and fulfillment are each separate applications. While the underlying infrastructure – how the app is configured, monitored, and supported – needs to be handled by IT Ops, each component lives on its own schedule of updates, and has its own team that maintains and fine tunes these apps to make sure they’re accountable to the business goals. But who, you might ask, comprises the team if not IT Ops?

    DevOps: The intersection of Development and Operations

    Enter DevOps. DevOps is one of those terms that can mean different things to different people, but in its most simple definition, DevOps is the convergence of development and operations. Forrester analyst Mike Gualtieri hit the nail on the head when he said: “The goal of DevOps to make the process of deploying applications faster and smoother.”

    Without “DevOps” all application issues (outages, etc.) will be routed to IT Ops. Because IT Ops is shared, has other priorities, and does not necessarily know the goals of each business unit, this is not the most practical option. Sure, IT Ops are sharp and talented people, but they specialize in maintaining infrastructure – they are not application experts by trade. Some organizations have hundreds of different applications spread among a dozen business units. There is no way that anyone could expect IT Ops to understand the specific needs and behaviors of each of these specific applications.

    The application developers know the app best, they know the requirements, and how to diagnose issues. They need a deep solution to trace transactions and end user experience. And, they know best what the app needs in a general sense….how much CPU, how much storage, how much RAM, etc. However, what they don’t have is expertise with VMware vSphere, Cisco UCS, VCE vBlock, FlexPod, IBM hardware, NetApp storage, EMC storage, etc. They are developers. They can manage their application operationally, but what happens when they run into an infrastructure problem? This is where IT Operations comes in.

    Gualtieri also said, “DevOps is a loosely defined set of emerging practices to get developers and operations pros to work together. Developers and operations professionals are often at odds. Developers want to release software more frequently; operations professionals want to protect the stability of the infrastructure.”

    This is why the portion of the IT budget that is moving away from IT and into individual business units continues to grow. It’s not necessarily surprising when people start talking about an end to IT Ops, or in some cases, and end to Operations all together (NoOps).

    However, despite the fact that organizations are moving away from IT Operations overseeing 100 percent of the budget, to a chunk of it now being managed by individual business units, without IT Ops each unit would control all of their own IT. Imagine unified communications without IT Ops. Do you really want each business unit in charge of that? No, you want economies of scale.  You want it to be shared – the network, the bandwidth, the storage – IT Ops will remain to manage all of that.

    As Gualtieri noted, “developers and operations need to stick together to best meet the demands of the business.” Recent research drives this point home. A study on the state of DevOps by Puppet Labs found that IT performance strongly correlates with DevOps practices like proactive monitoring, and continuous delivery and integration. In fact, according to the research, DevOps practices can boost IT performance and ultimately improve a business’s bottom line.

    IT teams have a more consultative and strategic approach, working in partnership with developers and other users of IT infrastructure to ensure that they all have access to what they need without bottlenecks or barriers to entry. Despite the fact that there appears to be room for both IT Ops and DevOps within organizations, it is clear that a shift is occurring.

    IT Ops: Making a move to large service providers

    Taking advantage of the latest technology innovation, as well as economies of scale, many organizations are now using service providers instead of owning their own infrastructure. One could say that this is pushing the IT Ops folks out, and they would be correct, but don’t worry they’re not becoming an endangered species. Instead of working for large enterprises, IT Ops professionals are working for the large service providers. They aren’t going away, they’re just moving off site.

    The budget will likely continue to go down for IT Ops on premise, but the cost of hosted infrastructure will go up, so the budget for IT Ops will even out. IT Ops is not disappearing, it’s transforming.

    In a time when Ops have become trendy and heated debates over terms have garnered the attention of the tech community, it comes as no surprise that the majority of us are left wondering what the future holds. One thing is for certain, be it DevOps, IT Ops, or SecDevOps, there will continue to be places for Ops within organizations.

  • Revolving door of IT employment threatens continuous delivery

    Revolving door of IT employment threatens continuous delivery

    As the IT job market heats up following a long lull, enterprises will increasingly need to cope with the difficulty of maintaining operational continuity in the face of personnel churn. Whether the revolving door agitates the employee pool for developers or operations staff, the length of time necessary to off-board departing IT staffers and on-board new ones can particularly endanger the fast-paced routine of continuous delivery and DevOps practices.

    Fortunately, the documentation and automation principles inherent to DevOps could actually also prove to be the very answer to a problem that threatens DevOps sustainability as job churn increases.

    “DevOps is not just a tools play, it’s a human play–how people eventually work has a very big impact on how well DevOps is put into place,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “We have an immense shortage of talent in the market, and as engineers come and go it is almost unrealistic to expect that everyone who is there today is going to be there through the end of a project. You have to build business continuity around that and DevOps can play a key role in that.”

    Suleman says he’s recently seen customers struggle with this problem first hand. In one particularly dramatic example, a smaller firm his company works with has been seeing its 10 person developer team ravaged with a churn rate of one employee turning over per month.

    Recent industry numbers show that this kind of turnover may not be the result of organizational problems, but due to broader trends. Earlier this month, the Department of Labor reported that May capped off four months of dramatic job growth in the IT sector. Whereas in 2013, the DOL reported an average of 10,700 IT jobs gained per month, in May that number hit above 15,000.

    “We’re now getting back on that track after some horrible months for tech professionals,” says David Foote, chief analyst at Foote Partners which has been tracking and reporting on IT labor trends since 1997.

    Meanwhile, Robert Half International reports that IT salaries will rise by 5.6 percent this year. And a survey out this month from Computer Economics shows that 52 percent of hiring managers plan to increase the number of full-time IT staffers they hire this year, the highest this figure has been in three year. These statistics show how industry trends will spur on more movement between firms as skilled staffers look for greener pastures. This is particularly true for jobs seeking staffers experienced in DevOps, particularly as those DevOps vets exhibit a cross-pollenization of ops, dev and people management skill sets.

    “There still remains a lot of volatility even in the IT employment marketplace which for some time been expanding mostly in the IT services industries according to these federal jobs reports. What they’re not reporting is all the new IT-business hybrid jobs that have been created and for which employers are feverishly mounting recruiting efforts,” Foote says. “They cannot find and hire them fast enough. There are probably ten or fifteen open positions for every one suitable candidate and that person is almost always already employed. They cannot find and hire them fast enough.”

    So how can DevOps principles help to solve the continuity issue as employees play musical chairs? According to Suleman, most important is documenting and automating as much of the deprovisioning and provisioning process as possible. In particular, organizations should be seeking to iterate in order to speed up their on-boarding process in much the same fashion as they have for release schedules. In his example, the 10-person shop he pointed to as an example was getting killed by a two-day on-boarding process that didn’t scale well with the amount of turnover it was experiencing. One of the big reasons is that the company had no documentation standardizing the process.

    “If you broke it down, a big chunk of that time was spent in just familiarizing the employee in what technologies the company used, with a lot of that shared through human interaction,” Suleman says. “That was often not a good use of time because they still needed to be relearned over the next few weeks as the engineer really deep dove into the code to figure things out on their own.”

    Additionally, though, there was the issue of having no automatic way to share all of software, libraries and so on necessary to set up the new employee’s system so he or she could get started learning the environment.

    “There are a lot of external dependencies as a part of that process. You may have an instruction saying go ahead and install the latest database version, but the version I’ll download today will be different from one you downloaded last month and something that will work on your computer won’t on mine,” Suleman says. “What ends up happening is those two days are spent figuring out all those discrepancies.”

    In this case, a document may be helpful but even more helpful would be to find a way to automate the creation of up-to-date images of different roles’ environments so that all it takes is one simple install to get a new employee running from day one. This is exactly what Suleman says his company helped the customer do, creating a single, pre-packaged USB stick to go along with newly created documentation so that a two-day process was slimmed down to 30 minutes.

     

  • Build fast, then build right 

    Build fast, then build right 

    This article is a part of a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef community. This article is by Kate Matsudaira, founder of Popforms

    Businesses have always needed speed and scale to meet customer demands. Today though, the kind of speed and scale businesses require to stay ahead of voracious consumer appetites and competitive offerings can only be achieved through a combination of: automation; internal culture shifts internally to integrate developers and IT functions closer together and more closely with the business; the use of commodity hardware; and a willingness to offer and embrace imperfect products. Then, and only then, will companies realize the benefits of moving quickly to deliver product, and also in unlocking innovation in this process to generate new revenue streams.
    For this to work, CIOs and CTOs must fully understand the business. They can no longer just be experts in technology. That’s not enough. For IT to evolve from a cost center to a revenue generator, IT must be integrated and engaged in the enterprise – large and small enterprises alike.

    Take a specialized candy company, for example. Before, it would not have been economical for this kind of business to scale at speed. But now, with minimal expense, it can acquire the compute cycles it needs, automatically deliver working code into its expanded production environment, and keep IT costs down through automation.

    Also, with the cloud and SaaS, costs for technology are way down – or, in some cases even free. This commoditization of technology is hugely positive for IT and its new role in many companies. Technology is no longer such a massive sunk cost. CIOs and CTOs can focus on meeting the quickly changing needs of internal and external customers to drive the business and growth forward and upward.

    That said, the pressure is on to deliver new code and new software solutions much faster, customer demand is relentless and escalating in intensity and speed. There is an almost immediate need that has to be fulfilled each and every day. And only automated and continuous delivery of code can meet that need in businesses of all sizes. It’s clearly a competitive imperative – and one that businesses must now approach differently as demand for “now” replaces demand for “perfect”. I advocate building v1 fast, then building v2 right. This gets product in the hands of consumers quickly and allows the team internally to answer pertinent questions that impact the long-term success of the product including:

    • How quickly will the data grow?
    • How fast will writes need to be?
    • Will it be feasible to write directly to the data store, or do you need a queue to manage

    writes?

    • What will the throughput be, and is it enough?
    • Will the system scale with usage?

    This “Lean Startup” approach to software development – which advocates fast iteration and data collection to hone in on the right product and requirements as quickly as possible – is all about figuring out what customers will use and then building those things.  And these same ideas can be applied to the technical ways we build products, too.

    Put another way – similar to creating a minimum viable product (MVP), build the minimum viable technical implementation that meets the business goals (the product can meet current usage requirements and performance).

    Most engineering teams have been taught to think about the software design and architecture from the beginning and build something that can scale. However, if you don’t have any customers yet, scaling really isn’t your challenge.  So, in some ways, designing for scale when you don’t have to scale yet, is solving the wrong problems.

    Of course, I am not advocating doing something careless, or writing bad code, but don’t over- engineer or solve for problems until you have them.

    In many ways, this process is about looking at the problems differently, and instead of solving the “big” problem, focusing on speed and efficiency – getting the highest ROI (return on investment) for your development resources.

    So you modify your development process to ship the first version quickly, and then once the v1 is out in the wild, plan to start on the v2 right after.

    Then your v2 can address all the problems with the first version of the product. You can improve the usability, cut or add features based on customer usage and feedback, pick the right technology and libraries to power the product, and spend time writing acceptance tests and unit tests that are unlikely to change. And, from a business perspective, you get the product out there faster, so it is easy to understand if it is going to drive revenue and success.

    This process makes sense because it:

    • Reduces the risk of over-engineering
    • If you build the wrong thing you can correct it
    • Chances are you will get to market faster

    So what are the downsides?

    • Total time to build is more than if you just got it right the first time. And operations can be a headache.


    If you do embark on this journey, here are some pointers:

    • Get management buy-in
    • Measure everything
    • Understand that the v2 will take much longer than the v1 and set expectations accordingly
    • Have a tight feedback loop with your customers
    • Make sure v1 and v2 are celebrated equally

    This is the formula organizations need to follow to be more competitive in today’s business of environment of rapidly evolving technology.

  • Part Dev, Part Ops

    Part Dev, Part Ops

     

  • Q&A with Gene Kim: Bringing the auditors to DevOps

    Q&A with Gene Kim: Bringing the auditors to DevOps

    As more enterprises embrace DevOps, organizational disconnects often are created between what controls DevOps teams have in place and what IT controls auditors believe need to be in place. And if the right controls are, in fact, in place they absolutely need to be communicated to the auditing teams. Bridging these gaps is often painful for both IT teams and the auditors.

    Helping enterprises to better understand what auditors need from them to do their work is the top-line goal of the DevOps Audit Defense Toolkit, the aim of which is to “define the authoritative guidance of how management and auditors should conduct audits where DevOps practices are in place,” according to the homepage for the effort. “By doing this, the DevOps Audit Defense Toolkit will elevate the state of the management practice, defining how to understand risks to business objectives, correctly scope and substantiate the of effectiveness of controls, which reduces the costs of audits and increases effectiveness of audits.”

    The review draft of the DevOps Audit Defense Toolkit was release earlier this month for review and comment. The authors, James DeLuccia, Jeff Gallimore, Gene Kim, and Byron Miller, expect the final version to be available in coming months.

    To get a deeper understanding of the Toolkit, we grabbed a few minutes of time with Gene Kim, author of The Phoenix Project.

    Why do you think something like the DevOps Audit Defense Toolkit is necessary?
    I think at its heart DevOps is transformative. It’s the way to increase the flow of work through the DevOps value stream. DevOps increases reliability, stability, and security. And it also helps the organization win in the marketplace.

    Yet, when organizations actually embark upon this journey, probably the number one obstacle they encounter is that their “compliance guys will never let us do this.”

    I think there’s some truth to this. Because of so much that’s being proposed when you embark upon the DevOps journey, you actually take away some of the key controls that traditional IT organizations have used – like separation of duty, like change approval processes.

    So when they hear, “Hey, I’m going to be letting our developers deploys whenever they want, without necessarily getting a change approval order from the change advisory board,” auditors, compliance managers, and security people often freak out.

    And you have first-hand experience of seeing this in action, right?
    It’s why I think it’s a very important effort. The more I talk to people, the more I hear about the friction between the audit teams and audit groups in organizations trying to move this way.

    How do you see the DevOps Audit Defense Toolkit achieving this?

    The DevOps Audit Defense Toolkit was designed to train. Since we can’t bring DevOps to the auditors, we must bring auditors to DevOps. What I mean is the goal is to train practitioners in DevOps how to think like an auditor and actually walk through what an auditor will do when examining a process and systems. That’s starting from the very highest level of the top-down risk assessment into the details, just as an auditor would go through the list asking what can go wrong.

    The DevOps Audit Defense Toolkit shows how an enterprise has the environment that can prevent bad things from happening, and if we can’t prevent it, we can detect and correct for it. Then, it shows how we evidenced it. I think by doing all of that, one is doing exactly what the best auditors who examine DevOps organizations do themselves.

    By training people in in the DevOps organization to think like auditors, we can actually bring the auditors along, show our work, connect the dots, and hopefully have the auditor become DevOps’ best friend along the way.

    So, this is training IT how to think like an auditor and view the systems and processes the way that auditor would?

    In many ways, yes. This is kind of a structured thinking process that any world-class auditor would do to actually create an audit plan and then audit fieldwork.

    The toolkit asks questions like: Why is this being audited? Describe the environment Walk through the process of how code moves to production, and where the development environment comes from. It also asks: How do you prevent developers from introducing backdoor code into production; how do we ensure that untended code doesn’t make it into production?

    I’ve learned a lot just working with the other team members, in terms of showing how we have a basis to say that we know what we’re doing, and that we’re really thinking through how to have an effective control environment.

    That takes it from sort of the high-level platitudes and claims down to well, no, really, this is how we enforce peer reviews of code. We ensure that all deploys are put into the build systems. We run all the automated tests. Every time someone commits code, we run all the security tests before it can ever get into the staging environment. We can confirm that what’s deployed actually passed all the tests.

    We communicate most effectively when we write down and show our work. And we’re exposing all that is required for an auditor to be able to look at it and say, “You know what? This is actually better than my best auditors would’ve done in the field, and you know what you’re doing.”

    I would imagine that simply learning the language of auditors and how they speak is very valuable itself.

    Yes, we’re now using a language that auditors use among themselves. Even just exposing what that language looks like, I think, is helpful to connect our own dots to be able to say, “Oh, I know why it’s being phrased this way. Tons of value, just by providing that.

    Did you learn anything along the way as you created the DevOps Audit Defense Toolkit that surprised you?
    It’s funny how much I learned by working on this. You would think the person who co-wrote the Phoenix Project would know what this would look like? But I learned quite a bit when I was looking at what were the actual business flows, and where that regulated data could reside, and what specifically are the applications in-scope for the compliance domains.

    The second thing was going through the management risk analysis – to walk through the structured thinking process and look at how a company is really doing the diligence that is required, deciding what are its top business risks and if it really has a control environment that can prevent, detect, and correct.

    Of course, we all know that DevOps makes everything better, but we need to know why and how to prove it, By what standards are you holding developers accountable? You talk about peer reviewing, but where is that documented? What is the standard? How is it enforced?

    You want to be in compliance, and that’s important. But you also don’t want people to insert backdoors. You don’t want systems to crash when changes are deployed. So you really need to name all the dots, connect them, and then have that processed challenged. I have learned so much in this project.

  • DevOps doesn’t end at deployment

    DevOps doesn’t end at deployment

    If you ask five IT admins the question “What is DevOps?” you maybe get eight different answers. We’ve attempted to define and explain the term DevOps here on staging-devopsy.kinsta.cloud a few different ways, but a precise definition is still elusive. One thing is certain, though, DevOps is about more than just automating virtual server deployment and configuration.

    At Velocity 2009, John Allspaw and Paul Hammond presented a session titled “10 Deploys Per Day: Dev & Ops Cooperation at Flickr”, which some credit with launching the concept of integrating development and operations—at least to mainstream IT. For many, though—especially those who are still trying to understand what DevOps is, and what it means for their organization—DevOps amounts to tools that simplify the process of implementing tens or hundreds of virtual server instances, and automating the initial configuration.

    Automating IT

    Tools like those offered by Chef and Puppet Labs are useful for automating routine tasks associated with deploying and configuring the IT infrastructure, as well as deploying and updating applications. These vendors can also help monitor and maintain a consistent, stable environment by enforcing a designated configuration.

    To be honest, that sort of automation is more or less table stakes for DevOps. It’s just one small facet of a much larger concept that involves the entire scope of development and operations—agile development and rapid iteration of new versions and features combined with an automated and easily-scalable infrastructure.

    Once the virtual server environment is up and running, though, and once the applications are deployed, the work is really just beginning. Part of merging development and operations, and keeping up with a more rapid pace of IT involves ensuring that everything is running smoothly. With frequent updates and rollouts, it becomes even more important to have tools in place to automate monitoring and alert IT admins when something goes awry.

    DevOps also involves tools like those from Stackify—tools that keep an eye on the apps running in your environment and let you know when things don’t quite seem to be firing on all cylinders. When something goes wrong, you want to know about it in a few seconds, not a few hours or days. Stackify can monitor and trend important metrics like CPU, memory, and disk utilization on your servers, and alert via email or SMS text when problems arise.

    Automating the creation, deployment, and initial configuration of servers is awesome—but you also need tools to address the challenges of maintaining and updating thousands of server instances as well. You need to be able to add or change user privileges, apply patches and updates to the operating systems or applications, change security configurations, enforce compliance policies, and more. When you’re talking about hundreds, thousands, or tens of thousands of servers, it’s imperative that you have tools like JumpCloud to help automate those tasks across the entire environment.

    More than just tools

    As great as all of the various DevOps platforms and tools are, though, DevOps is not just a matter of buying or implementing an array of tools to manage your IT environment. As the name implies, and as Allspaw and Hammond spelled out in 2009, it’s about instilling a culture that breaks down traditional barriers and enables developers and operations to work cooperatively and collaboratively.

    The traditional roles of developers and ops impose unnecessary friction that gets in the way of being effective. Developers blame problems on the implementation or IT infrastructure. Operations personnel blame problems on poorly developed code. Creating an environment where all appropriate parties work together, and where everyone takes responsibility rather than assigning blame creates an entirely different dynamic for productivity.

    When it comes down to it, it’s the culture of DevOps that accelerates the speed of changes and updates that necessitates tools and platforms to automate deployment and infrastructure management. The culture drives the tools, not the other way around.

    So, yes, one of the most visible components of DevOps may be tools that automate the deployment and configuration of servers, but that is just one element of DevOps, and it’s just the beginning. Everything that happens after that to monitor, maintain, and update that environment once it’s deployed, and everything that happens outside of the IT infrastructure to streamline productivity and increase efficiency is all part of DevOps as well.

  • National Renewable Energy Laboratory Uses DevOps For Better Compliance

    National Renewable Energy Laboratory Uses DevOps For Better Compliance

    One of the big stumbling blocks naysayers claim DevOps can’t overcome is the perceived incompatibility of DevOps with high levels of compliance regulations. But forward-looking IT practitioners at financial institutions, healthcare organizations and government agencies alike have already started to show how untrue that perception really is.

    The National Renewable Energy Laboratory (NREL) recently offered a perfect case study in this phenomenon. Not only do the key IT staffers in the trenches at NREL believe that DevOps is compatible with DevOps, but they actually used DevOps and continuous delivery principles to help the agency deliver compliance with a key piece of regulation around cloud integration within the NREL infrastructure.

    The success came from a year-long project designed to garner the lab a needed update to its ‘Authority to Operate’ (ATO) designation under the Federal Risk and Authorization Management Program (FedRAMP) for the new integration of Amazon Web Services across its IT infrastructure. The project successfully doubled as a pilot program for DevOps and continuous monitoring, says Ryan Kelley, systems engineer at NREL. He explained that the agency wanted to take lessons learned from staffers in a very limited separate department who had previously dabbled with AWS and DevOps practices and apply them for a fresh approach in a broader project within NREL.

    “The approach we took was, we’re not just going to try to jam all of our legacy toolsets into this and try to get them to work with the cloud,” he says. “We wanted to take a step back and say, ‘What’s the best approach for policy, what’s the best approach for toolsets and things like that for this new architecture?’”

    Part of the impetus behind this fresh approach was the lingering memory of how painful it was for NREL to gain its ATO five years ago. At that time the lab used outside consultants to shepherd it through the process and it turned out to be “extremely painful, drawn out and very, very expensive,” Kelley says, explaining that the ATO governs

    This time around, NREL hoped to find a way to use the expertise and resources it had in-house to streamline the process and not only comply with regulations but see a true return on its investment through a much improved infrastructure. According to Kelley, the project forced NREL to approach infrastructure almost like it was starting a new business.

    “We had a fresh slate and we kind of had to build almost everything policy-wise and everything toolset-wise,” he says, explaining that in order to comply with the security requirements around achieving an ATO, NREL was most concerned about what’s called customer-responsible controls, which are often very configuration-centric. “That’s where having a centralized configuration management system pushes you towards compliance because it’s just – it gives us that central place to audit all of our configurations. Out of the gate, to get cloud underneath our authority to operate, we needed the ability to centrally manage and have our configurations approved.”

    By approaching it as almost a new business, the IT team was able to make the cultural shifts necessary to achieve that through the improved teamwork and continuous delivery patterns that are the hallmark of DevOps.

    “Enterprise IT tends to think that they have to deliver these big, huge projects all at once and tied up in a nice bow, whereas we’re trying to take the approach of getting a minimally viable product and then rapidly iterating on it,” he says, explaining that it took about a year to get the ATO, but that NREL is still iterating to improve the cloud integration overall.

    Kelley says DevOps is providing a much needed shift for the organization due to the highly distributed nature of its programming resources. While the lab has a small cadre of IT staffers to run the backbone of IT infrastructure, the lab itself actually employs many programmers that actually work directly under what are essentially the lines of business for NREL, basically smaller sub-groups and departments working on individualized research projects. This ATO project showed that it is possible for the main IT group to interact with these smaller groups of developers to deliver quickly and effectively on projects and NREL plans to build on this success.

    “You know, we’re dealing with the same problems everybody else is: complaints that we’re too slow to deliver, not responsive enough, and not working with the business enough,” he says, explaining that past pundits’ warnings for IT to ‘align itself to the business’ never before offered concrete advice on how to do that.  “This was kind of one way that we did was just trying to find a project, develop a partnership and then sort of create a grassroots type of DevOps mentality.  It definitely wasn’t top-down. And we’re continuing to work with these groups in this particular manner.”

    Meanwhile, Kelley explains that the project also showed how continuous delivery toolsets can be instrumental in helping to with compliance demands.

    “We feel that these new modern toolsets that let you treat your infrastructure as code actually gives our cybersecurity team more insight into what we’re doing because once you do that and start treating it like software, then it can be audited a lot easier,” he says. “It’s a lot easier than building each server on its own and then waiting for security to scan it and then waiting for them to get us a report back and then we deploy it.”

  • The web-scale avalanche

    The web-scale avalanche

    Old-school businesses may not like it, but they’re competing with Google, Facebook, Amazon and other Internet powerhouses because those players have grown from day one with an eye on rapid, scalable, cloud-based and continuous IT operations.  

    Companies have been running application delivery aligned to an aging model based on low frequency release cycles. For decades, the major IT suppliers like Microsoft, Oracle, IBM, SAP and others have released software at the predictable, plodding cadence of annual or bi-annual releases. IT departments have adapted to the vendor release cycles with mirror-like processes: highly structured in the ITIL fashion with processes in place for working around lots of bugs that don’t get readily fixed by the vendors. Yet this delivery management model is now out of pace in today’s always-connected, Internet-based world in which apps and sites morph on a weekly or even daily basis.

    Suddenly, companies and vendors alike must change what they’ve been doing for years. They need to transform operations and products from the legacy-based delivery model to one which is faster, dynamic and proactive. But that’s hard for most companies. It’s like driving a 1979 Chevy pickup on the Autobahn. You can’t keep up with the pace of traffic, no matter how hard you try.

    Companies such as Facebook, Twitter and Netflix, born as Internet-age companies, don’t have the same problems adapting to modern times in which consumers expect instantaneous response times. They were built from the ground up to handle the relentless pace of technological change: their infrastructure and applications automatically shift according to network and CPU conditions and workload spikes. Such companies are masters at Web-Scale IT, and rely upon a multitude of open source and web-based technologies designed for distributed computing. More than just technology it also takes radically new processes and mindsets to become a Web-Scale IT company.

    Operating in the Web-Scale World

    Web-Scale IT consists of technologies and methodologies that enable developers to build, change and scale applications 10 to 100 times faster than in the past. Web-Scale IT relies upon many modern concepts, including cloud computing, peer-to-peer architecture, NoSQL, SOA, Agile development, DevOps, Continuous Delivery and Deployment and more.  Gartner describes Web-Scale IT as “a pattern of global-class computing that delivers the capabilities of large cloud service providers within an enterprise IT setting.” The research firm predicts that by 2017, Web Scale IT will be an architectural approach found in 50 percent of global enterprises, up from less than 10 percent in 2013. Below are a few areas of change that IT organizations must navigate to make the shift.

    1. Reevaluate IT partners. It’s not just CIOs that need to shift operations for Web-scale computing. The legacy IT vendors are struggling to walk away from profitable business models based on licensing to embrace the on-demand, services-based revenue that just a few years ago, catapulted startups like Salesforce.com into billion-dollar enterprises. The legacy vendors are unable to innovate fast enough and are having to rely on serial acquisitions to keep their technology relevant.  Yet rapid acquisition comes at a great price and is not sustainable. IT departments will need to look far and wide to find the best partners that can support their Web-Scale IT efforts moving forward.

    2. Acquire diverse, open-source based IT operations tools: CIOs will likely need to constantly review their technology sourcing and have a more critical method of choosing their technology partners. Relying on a few trusted long-term partners will likely not achieve the Web-scale IT goal. Consider that Netflix, which now runs its business almost exclusively on Amazon AWS, only a few years ago relied on its own datacenters full of legacy vendor equipment. Today, the company manages its high-availability video streaming business using a collection of inhouse-developed open source tools (coined The Simian Army) that test the fault tolerance of the company’s operations. Being flexible and open to change allows Web Scale companies to keep ahead of the competition. Standing still, even for a short while, is no longer an option.

    3. Foster iterative, collaborative processes: Strong DevOps cultures, where developers and IT operations personnel exchange information frequently to stay in step with changing business requirements, is also pivotal to making this shift. Companies with advanced DevOps practices are usually committed to running their businesses differently: more efficiently, more customer-focused and with the keen ability to respond quickly to change. CIOs and CTOs need people with strong Web technology skills yet also an unflinching commitment to collaboration and complex problem-solving. IT people today need to think on their feet and adapt as they go. Twelve-month planning cycles are out the window.

    The building of a storm

    Just a few years ago, it was considered evolutionary, if not revolutionary, to purchase CRM or HR as a service. That’s commonplace today. The next area for innovation is within core applications such as a retailer’s pricing application or a bank’s market forecasting tool. Those applications must now also run at Web Scale to be responsive, customer focused and speedy. Don’t count on your Oracle or SAP database to shift so easily or quickly—at least not yet. That’s why Internet leaders are managing IT infrastructure using the newer generation of Web Scale technologies including Puppet, Chef, AppDynamics, New Relic and yours truly, Boundary.

    If running at Web Scale sounds overly complex, it’s not. Since most of the latest, open source-based operational tools are highly automated, using them is actually simpler. Crowdsourcing is another strategy that companies are exploring for Agile development projects. The bigger hurdle is in moving from the old-fashioned software development style of over-engineering everything to the more intuitive, iterative style that characterizes modern Web application development. With Web-Scale IT, it’s okay to cut corners and take more risks, because the infrastructure will stand behind you.

    There are plenty of people who’ll disagree with me, but I can envision a world in which the Oracles and IBMs no longer run most of the companies in the world. — one that is adaptive, business-responsive and always forward-looking. The future is closer than we think. Will your business evolve?