staging-devopsy.kinsta.cloud

Category: Features

  • Microsoft strives to embrace open source culture

    Microsoft strives to embrace open source culture

    In many ways Microsoft is going through the corporate equivalent of a mid-life crisis—it has reached a stage where it longs to drop a few pounds, and turn back the clock a few years. After years of tremendous success, it’s now starting to show its age, and it is struggling to adapt to the new tech landscape and compete with younger, leaner, more innovative competitors.

    The shift began under ex-CEO Steve Ballmer. Ballmer’s approach, though, was misguided. Ballmer’s Microsoft suffered from hubris, and ignored the seismic shift in technology until it was too late. When Ballmer finally recognized the existential threat, and tried to turn the ship around, it was done from within the established bloated corporate culture. There was an effort to rearrange the chairs on the deck with a management reorganization, but it simply wasn’t enough.

    Then Satya Nadella took over. Nadella has a more aggressive vision. It’s still not an easy job, nor a goal that can be accomplished overnight with a behemoth the size of Microsoft, but Nadella has dramatically changed the culture, and the vision for where Microsoft is heading.

    Part of the new and improved Microsoft includes embracing communities that Microsoft has long ignored or alienated, and shifting the way things get done at Microsoft so the company can be more agile. It is evident in how Microsoft partnered with Chef and Puppet Labs to integrate popular DevOps tools into the Azure cloud platform, and it can be seen in how Microsoft is adopting open source.

    “Recent articles have shown that Microsoft is “open sourcing” code to their internal teams. The fact that that is advertised as innovation is telling,” declared Michael LaVista, CEO and Founder of Caxy. “That’s what smart organizations have always done. In fact, Microsoft has somehow found itself the underdog in this sense and probably needs to do things like this to attract talent.”

    Leveraging open source concepts internally isn’t quite the same thing, though, as actually making Microsoft code open source. Open source projects like Apache, Project Libre, or Docker share source code with the general public, and anyone can review the code, and submit changes to improve or fix the code. Open source communities collaborate to continuously enhance the code. Microsoft appears to be doing something similar, but without exposing its source code to the general public.

    LaVista suggests that Microsoft should take that next step, though, and work collaboratively with the open source community outside of Microsoft’s walls. If it does so, Microsoft will some issues to address. Will it accept code fixes from a 14-year old on the other side of the world? Will exposing source code for the Windows operating system, Microsoft Office, or other popular Microsoft applications open up a Pandora’s Box of security concerns?

    “They will have to embrace what happens to code when it is open sourced. Once the code is out there, everyone can see the flaws,” explains LaVista. “The benefit of the model is that you have so many more eyes looking at it from more perspectives. If Microsoft is used to running their own software and making their own decisions about what to work on and fix, this could pose problem.”

    Perhaps. LaVista makes good points about concerns Microsoft would face, but in and of themselves they’re not a reason not to adapt and evolve. Yes, exposing Microsoft source code to the general public would probably result in a spike in security concerns, but once the dust settles the collaborative approach of an open source community addressing those issues could result in quicker patch releases, and make the software more secure.

    LaVista recommends that Microsoft use companies like Red Hat, IBM, and HP as examples of how a larger corporation can be successful in an open source culture. Only time will tell. We’ll have to wait and see to what extent Microsoft makes the shift to open source, and we’ll have to wait and see how Nadella’s Microsoft adapts, and whether Microsoft can successfully reinvent itself to remain competitive in this new tech landscape.

  • How to Find Your Continuous Delivery Rhythm

    How to Find Your Continuous Delivery Rhythm

    This article is a part of a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef community. In this article we present an interview with Jez Humble, Principal Consultant, ThoughtWorks, on digital innovation.

    Focusing on speed and scale allows you to respond more rapidly to weak signals in the market, and to arbitrage those opportunities quickly. This is different from the way it used to be. In the past, we’d only respond to the strong signals in the marketplace. But now, with the lower cost of investment in product development, and the resulting optionality in that process, we can service subtle market signals effectively.

    Here’s an example of this. A company noticed that when people searched for bird-cages online, very few entries came up. So, to fill the void, the company began making bird-cages. And now this business has created a new revenue stream selling bird-cages. We’re also learning that the way we service customer demand – in all its varieties – is part of the product development process. This comes with the emergence of user-experience design. And it shows that marketing, as well as technology, is an essential part of product development today. Zappos is a great case study that illustrates my point. Many companies try to minimize the time spent on customer support calls. Zappos spent a maximum amount of time on customer calls every day. And, by deeply engaging with the customer, they learn about the customer.

    Code is really the key to exploring and exploiting new opportunities in an inexpensive and scalable way. And here, we have to give credit to Toyota, which has been using Lean to show us how to re-tool on the fly in response to customer demand. As a result, software is now following automobile manufacturing.

    A major part of all this is continuous delivery of code, which, from my perspective, is fundamentally about reducing the transaction cost of putting out incremental changes. And I want to emphasize the importance of incremental changes and small batches; continuous delivery of code really can’t be effective if it’s one-piece flow.

    Think about it this way. You don’t buy eggs at the supermarket once a year. And you don’t buy an egg as a replacement every single time you use one. No, you buy eggs on an ongoing basis in small batches. This is the kind of rhythm that powers our ability to move rapidly when there’s a weak signal in the marketplace. It’s the kind of rhythm that powers our ability to change code based on customer demand. And it’s the kind of rhythm that powers our ability to compete effectively – and with a decided advantage.

    Netflix, Etsy, Amazon, Google and Facebook are five of the best at this. And I also like the way the British government’s digital service has been re-tooled and re-built with Lean, customer experience, and continuous delivery principles in mind. The government site is, without question, all about customer demand and customer service, and it has saved citizens a lot of money as well. The take-away here is that software companies shouldn’t be thinking narrowly about software; instead, they should be thinking about how their software can better serve customers. To do that, the entire business has to be run differently and more effectively. And, again, technology people – as well as product marketing people, financial people and governance people – all have to join in and help make it happen.

    There are three practices of continuous delivery that can help you create the most efficient path from hypothesis to continuous feedback:

    • Comprehensive configuration management
    • Continuous integration
    • Comprehensive automation of build, test, deploy, and infrastructure management

    A new kind of “MVP”

    One of the key ideas that the Lean startup movement has popularized is the minimum viable product (MVP), which is defined as the least possible amount of work you can do to test your business hypothesis.

    Of course, people in manufacturing have been producing minimum viable products for decades – they’re called prototypes. I like the MVP because you’re not betting the company, or spending big wads of cash.

    Another key Lean concept is validated learning; here, you gather actionable metrics based on real usage of your product, without asking people. To do this, however, you’ll need to be able to answer questions like:

    • Did the changes we made to the product improve signup, retention, revenue? Or is it time to pivot?
    • Which version of our new feature came out better when we did A/B testing?
    • All our system metrics look okay, but a user reports that our site isn’t working. Are we down?
    • Which features of our product are generating the most revenue?

    These questions should be answerable by looking at a dashboard, and the information should be completely auditable. But this kind of learning is only possible if metrics are built into the “hypotheses” that are being tested. Once you have an idea for a minimum viable product, you need to start delivering software. The first step is analysis. My view is that you need to do just enough analysis up-front, and this approach needs to continue throughout the lifecycle of the project, with the emphasis on creating very small, incremental stories.

    Doing less and learning more

    Let’s say you’re working on a hotel booking site, and you want to add a feature to allow people to choose whether they want breakfast. Don’t create this feature for all hotels, or for all partner sites. Instead, start with a “hypothesis” that allows you to add that feature for a single hotel, with no configuration options, and get feedback for that possibility before you proceed further.

    Do less, so that you can focus your efforts on learning from your users.

    If you want to increase your release frequency, it’s essential to decouple your deployments from the act of taking a feature live. Feature toggles (also known as bits or switches) help you control who can see which features. This setup also allows you to release new versions of revised software that include partially complete features, containing perhaps a story or two of work, but nothing you’d want the general public using.

    In addition to enabling incremental delivery of functionality, feature toggles let you turn off a problematic feature when a release goes wrong, as an alternative to rolling back your deployment.

    The bottom line is that continuous delivery like this liberates us by reducing the cost and risk of releasing software, and by powering much-needed innovation at the same time.

     

    jez-humbleJez Humble is a principal consultant with ThoughtWorks and co-author of “Continuous Delivery” and “Lean Enterprise.” He began his IT career in 2000. Over the past 14 years, he has been a developer, system administrator, trainer, consultant, manager and speaker. He has worked with a variety of platforms and technologies, consulting for non-profits, telecoms, financial services and on-line retail companies. Since 2004, he has worked for ThoughtWorks and ThoughtWorks Studios in Beijing, Bangalore, London and San Francisco. He holds a BA in Physics and Philosophy from Oxford University and an MMus in Ethnomusicology from the School of Oriental and African Studies, University of London.

  • ChatOps: Communicating at the speed of DevOps

    ChatOps: Communicating at the speed of DevOps

    DevOps is about knocking down barriers between developers and operations teams, and in doing so reducing organizational frictions wherever possible. And that’s certainly the spirit behind ChatOps – the term for the hyper-collaborative way of running DevOps, including operating aspects of systems and infrastructure, through online chat.

    Proponents of ChatOps say it streamlines communication in a centralized and powerful way. Born within Github, ChatOps evolved from its open source chat bot, Hubot, which has changed the way the web-based Git repository runs operations through the automation of deployment, graphing, monitoring, provisioning, and even mitigating security events.

    Think of Hubot, a Node.js app that is commonly run on Heroku, as a robot that helps keep everyone informed in a centralized place and in real-time, or in a near real-time, way. There are many benefits to ChatOps, says David Mortman, David Mortman, contributing analyst at Securosis, including the integration of ops activities with notifications into one place, making it easier for everyone to know what’s going on across the enterprise, as well as everyone throughout the organization having the ability to see error messages on events as they occur, he says.

    Github, as a highly distributed organization, with its several hundred employees spread across 103 cities, can use all of the communication efficiencies it can muster. As Mark Imbriaco, Ops engineer at Github, in his presentation ChatOps, Augmented Reality for Ops, uses Github, email, internal apps, other web systems that they’ve built to help teams communicate asynchronously – and yet they needed an option for the real-time or “semi”-real time communication. “I say semi-real time because, when we talk about chat, the social contract at GitHub around chat doesn’t require you to be immediately responsive to everything you see. Chat is still asynchronous, but it’s more synchronous than GitHub or email or other sorts of notification mechanisms,” said Imbriaco.

    Without enhancement, Hubot doesn’t do a whole lot right away. But when added to chat with one of the dozen or so connectors that include access to Google Talk, IRC, Skype, and Hubot monitors for events that come from within chat or from one’s systems and applications. “The magic happens when you build additional functionality into him to automate, well, everything,” Imbriaco said.

    Hubot will do certain things, such as notify people not present in a chatroom that they were mentioned, phrases can be registered so that notifications to those interested in certain events will be notified. Github has also plugged Hubot into the GitHub service itself. Today, when anyone creates an issue or pull request in GitHub, according to Imbriaco, a link and definition of the event is sent to chat. While Github already has ways to send notifications, by entering events real-time into the chat stream, others can jump in and comment, help, or otherwise advise on events and interact as they’re occurring.

    As for the adoption of ChatOps, Github isn’t the only one: but it’s not yet widespread among DevOps practicing organizations, either, based on our reaching out to numerous organizations. Nor, is the concept especially well known.

    That’s not slowing Owen Zanzal a developer at systems management provider VividCortex down from adopting ChatOps. Zanzal says in his blog post, ChatOps at VividCortex, that, despite ChatOps being off to a slow rise, “ChatOps has really revolutionized how we operate,” he wrote.

    The first Hubot mission at VividCortex was to help improve how code is deployed. The problem Zanzal highlights is that encryption keys and new developers apprehension around code deployment, was slowing down production.

    To streamline deployment, Zanzal moved the execution of Capistrano commands to a Jenkins (a continuous integration server), and created Hubot commands so that jobs could be created using a Jenkins exposed REST API. When done, developers needed only run a chatroom command, such as “hubot stage webapp” or to deploy “hubot release webapp,” Zanzal explained. “Now, no one is blocked by the gatekeeper from pushing code into staging or production, and every developer can push code when they are ready. The unfamiliar Capistrano commands were simplified into much easier to understand Hubot commands,” he wrote.

    VividCortex is also using ChatOps to fetch logs and more quickly remediate issues as they arise, Zanal wrote.

    Chat-as-Ops
    GitHub has also pushed ChatOps much further than simply as a place to stage and deploy code. It’s using ChatOps as a way to help actually run ops. For instance, Imbriaco spoke to how Hubot and ChatOps is used for efforts such as rebalancing shards across its file servers. “Honestly, it took me a little while to get used to some of these. We have some [Hubot actions] that do things like rebalance shards across our file server. And that’s horrifying to me, because the consequences of failure are so bad. But the reality is, I’m either going to run it from a chat room, or I’m going to run a shell script from a terminal. If I ran it here, at least everyone saw it. So at the end of the day, this is repeatable, it’s tested, it’s commonly used,” Imbriaco said. “And if it stopped working we would notice,” he said.

    And one time when Github was fending off a distributed denial of service attack, Imbriaco needed only enter the name of their service provider to direct traffic and the attack was successfully mitigated.

    Moving to ChatOps isn’t without risk, however, Imbriaco explained. A lessoned learned when it became a little too easy to deploy to Amazon Web Services. “We went from zero to way too much spend on Amazon in about two weeks, when I turned up the ability for us to provision EC2 hosts from Chat,” he said. But that issue was resolved by giving users information at the time of provisioning, such as including metadata into the instance that details who created it, as well as informing users how much it’s likely to cost,” Imbriaco said.

    Despite the efficiencies of ChatOps, don’t expect a widespread embracing of ChatOps, but more likely a slow adoption. “GitHub is a flying unicorn. They’re like Netflix,” says Mortman. “People will pick up some of what they are doing but it’s too advanced for most,” he says.

  • Build fast, then build right 

    Build fast, then build right 

    This article is a part of a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef community. This article is by Kate Matsudaira, founder of Popforms

    Businesses have always needed speed and scale to meet customer demands. Today though, the kind of speed and scale businesses require to stay ahead of voracious consumer appetites and competitive offerings can only be achieved through a combination of: automation; internal culture shifts internally to integrate developers and IT functions closer together and more closely with the business; the use of commodity hardware; and a willingness to offer and embrace imperfect products. Then, and only then, will companies realize the benefits of moving quickly to deliver product, and also in unlocking innovation in this process to generate new revenue streams.
    For this to work, CIOs and CTOs must fully understand the business. They can no longer just be experts in technology. That’s not enough. For IT to evolve from a cost center to a revenue generator, IT must be integrated and engaged in the enterprise – large and small enterprises alike.

    Take a specialized candy company, for example. Before, it would not have been economical for this kind of business to scale at speed. But now, with minimal expense, it can acquire the compute cycles it needs, automatically deliver working code into its expanded production environment, and keep IT costs down through automation.

    Also, with the cloud and SaaS, costs for technology are way down – or, in some cases even free. This commoditization of technology is hugely positive for IT and its new role in many companies. Technology is no longer such a massive sunk cost. CIOs and CTOs can focus on meeting the quickly changing needs of internal and external customers to drive the business and growth forward and upward.

    That said, the pressure is on to deliver new code and new software solutions much faster, customer demand is relentless and escalating in intensity and speed. There is an almost immediate need that has to be fulfilled each and every day. And only automated and continuous delivery of code can meet that need in businesses of all sizes. It’s clearly a competitive imperative – and one that businesses must now approach differently as demand for “now” replaces demand for “perfect”. I advocate building v1 fast, then building v2 right. This gets product in the hands of consumers quickly and allows the team internally to answer pertinent questions that impact the long-term success of the product including:

    • How quickly will the data grow?
    • How fast will writes need to be?
    • Will it be feasible to write directly to the data store, or do you need a queue to manage

    writes?

    • What will the throughput be, and is it enough?
    • Will the system scale with usage?

    This “Lean Startup” approach to software development – which advocates fast iteration and data collection to hone in on the right product and requirements as quickly as possible – is all about figuring out what customers will use and then building those things.  And these same ideas can be applied to the technical ways we build products, too.

    Put another way – similar to creating a minimum viable product (MVP), build the minimum viable technical implementation that meets the business goals (the product can meet current usage requirements and performance).

    Most engineering teams have been taught to think about the software design and architecture from the beginning and build something that can scale. However, if you don’t have any customers yet, scaling really isn’t your challenge.  So, in some ways, designing for scale when you don’t have to scale yet, is solving the wrong problems.

    Of course, I am not advocating doing something careless, or writing bad code, but don’t over- engineer or solve for problems until you have them.

    In many ways, this process is about looking at the problems differently, and instead of solving the “big” problem, focusing on speed and efficiency – getting the highest ROI (return on investment) for your development resources.

    So you modify your development process to ship the first version quickly, and then once the v1 is out in the wild, plan to start on the v2 right after.

    Then your v2 can address all the problems with the first version of the product. You can improve the usability, cut or add features based on customer usage and feedback, pick the right technology and libraries to power the product, and spend time writing acceptance tests and unit tests that are unlikely to change. And, from a business perspective, you get the product out there faster, so it is easy to understand if it is going to drive revenue and success.

    This process makes sense because it:

    • Reduces the risk of over-engineering
    • If you build the wrong thing you can correct it
    • Chances are you will get to market faster

    So what are the downsides?

    • Total time to build is more than if you just got it right the first time. And operations can be a headache.


    If you do embark on this journey, here are some pointers:

    • Get management buy-in
    • Measure everything
    • Understand that the v2 will take much longer than the v1 and set expectations accordingly
    • Have a tight feedback loop with your customers
    • Make sure v1 and v2 are celebrated equally

    This is the formula organizations need to follow to be more competitive in today’s business of environment of rapidly evolving technology.

  • Citrix puts DevOps front and center

    Citrix puts DevOps front and center

    I had the privilege of attending Citrix Synergy 2014 in Anaheim at the beginning of May. Citrix unveiled a variety of new and updated products, but one announcement in particular surprised me a little. Citrix talked about new capabilities for automation with a very DevOps-ish approach.

    During a keynote presentation at Citrix Synergy 2014, Sudhakar Ramakrishna, senior vice president and general manager for the enterprise and service provider division at Citrix talked about automatic provisioning through Xen Desktop, and creating blueprints for deployment. He didn’t overtly call it DevOps during the keynote, but it all seemed very similar to what other DevOps tools and services accomplish.

    I followed up with Citrix, and had a conversation with Steve Shah. Shah told me that DevOps is absolutely front and center in the thought process for Citrix when it comes to developing products and services for its customers, as well as for mapping out its future strategies.

    Shah was at one point a bit of a DevOps naysayer. In an article in 2011, Shah questioned the credibility of DevOps and wondered whether it would survive as an actual thing, or if it was really just a flash-in-the-pan buzzword. Three years later and still gaining momentum, I think we can safely say that Shah’s doom and gloom assessment was off base.

    Citrix understands the need and the value for automating routine aspects of provisioning and managing virtual systems. According to Shah, though, one of the things Citrix has noted about the broader landscape is that there is no clear platform for doing DevOps. There are a wide variety of tools and services with a few leaders, but no clear “winner”.

    Shah told me that Citrix is only one part of a much larger solution, and it respects that it has to play nice with a wide variety of other products and services, or customers will find other solutions. There are too many unique, custom applications out there for Citrix to individually address them all, so Citrix has instead focused on creating open APIs that enable vendors or customers to develop their own methods of integrating with Citrix.

    Another area where Citrix has embraced DevOps—whether it is openly called that or not—is through the use of Blueprints. Citrix determined that there are certain deployment scenarios that are very common for customers across all industries. Rather than having each one of them struggle to develop their own solutions, Citrix designed Blueprints of common scenarios that customers can use as a foundation for their own environments—similar in many respects to the Recipes and Cookbooks used with Chef, a popular DevOps platform.

    My revelations from the Citrix Synergy keynote, and my subsequent conversation with Citrix made two things clear. First, many organizations may be embracing elements of DevOps without even realizing or calling it that. There are things that just make sense when managing a large, rapidly evolving environment that naturally lead to implementing aspects of DevOps. Flipping that point upside down, those same needs are more or less the origin of the DevOps movement.

    Second, DevOps is gaining momentum and becoming more mainstream as time goes on. While many still struggle to define DevOps, it shows up in day-to-day operations, and in the platforms and tools organizations use whether they understand it or not, and whether they call it DevOps or not.

  • DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow

    DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow

    In organizations trying to achieve agility with stability and quality, silos (of teams, tools, and processes) create major stumbling blocks to managing configuration drift between pre-production and production systems. Breaking down those silos, improving visibility into configuration drift and facilitating collaboration for improved flow are the keys to overcoming this challenge.

    The challenge of staying in sync when both sides are moving

    Production environments aren’t static. Whether you are applying patches for improved security, optimizing settings for availability and performance, or making changes for a technology refresh, production systems are continuously evolving.  As new releases are being developed, tested, and staged, pre-production environments are continuously evolving too. Ideally you want to develop with production in mind, but that gets harder and harder when both sides are changing and different teams are responsible for different environments and live in their “own worlds.” (e.g. the common split between QA teams, release teams and production operations).

    Even within pre-production, staying in sync is a significant challenge. For example, because applications ride on middleware and middleware settings are unique to each application, keeping systems in sync from QA to staging can be a real challenge.

    QA and staging environments are regularly reconfigured to support multiple releases that are in process at the same time.  The costly way, often employed to minimize changes, is to “protect” QA environments for a certain application or release leading to low utilization and bottlenecks.  The rapid proliferation of mobile applications to front end legacy systems magnifies the challenge by adding another dependency layer.

    Disconnected teams, information, and perspectives (those pesky silos)

    Why are these challenges so hard to overcome if we know the issues?  Many IT teams operate in their “own worlds” with tailor-made but disconnected tools and information resources.  Silos have evolved because they permit the specialization required to deal with the immense, growing complexity IT deals with everyday, but they have now become the biggest impediment to agility and quality.

    For example, the operations team has access to systems management alerts that most others don’t, engineers have access to build systems and scripts that others don’t, developers have access to functional requirements and stories that others don’t, and change approvals are stored in a system of record that only certain people (often managers) have access to – just to name a few silos.

    Coordinating work and sharing information across these silos of teams often equates to lots of meetings, conference calls, and email threads.  This is not the most efficient way to get work done and it’s not particularly effective in sharing information to keep environments in sync.

    “Didn’t you see my email from yesterday with the attachment listing all the middleware settings required for today’s test?”  “Weren’t you in the meeting three weeks ago when I said a Unix server was probably going to be needed in addition to a Windows machine?”  “I told so and so in an instant message on Friday to run this script before he went out on PTO, but he must have forgotten. Who else has root permission to run it while he’s out?”

    Sadly, these kinds of statements are the norm, not the exception, in most IT shops.

    The reality is there is inefficient coordination to manage all the steps and handoffs between application owners, development, QA, release and production functions.  Instead, I recommend thinking about the software development and deployment lifecycle as an “uber change” and “uber collaboration” process that must be viewed and managed from end to end; from business requirements, to development, to QA to staging to production.  All of the different silos of teams must have a way to work together in a unified way with access to critical information from upstream and downstream teams to ensure a smooth lifecycle end to end.

    Automation ≠ DevOps

    The heart of DevOps is collaboration and managing change, yet the conversation often seems to start with automation.  The fallacy here is the belief that we can eliminate the need to collaborate and manage change by removing people from the process and relying on the machine.  This out of balance perspective between the machine and people creates more problems than it solves.

    Automation tools, while important and necessary, are not all that’s needed to truly solve these synchronization and deployment issues. In addition, to have environments only changed by automation tools and never by people during the firefighting of incidents is unrealistic.  You need your people armed with intelligence and tribal knowledge to drive your automations, but all too often we rely on individuals with disconnected perspectives and limited awareness so things get missed or overlooked.

    Automation does what it’s supposed to do, but it makes a critical assumption; what’s being changed is configured how it’s supposed to be configured.  If drift has occurred because critical settings were modified to remediate an incident or compliance issue, without the automation engineer aware of the change, the automated deployment will result in you hearing a loud “boom!”

    As one popular DevOps Twitter handle has said, “To make a configuration error is human – to do it across a thousand servers is DevOps.”  While that may seem like an extreme statement designed for comedic effect, it isn’t extreme to say that thousands of deployments go south every day because systems aren’t configured the way we expect them to be given that the complex software stacks that multi-tiered applications now depend on are touched in so many ways.  Automation alone doesn’t solve the DevOps challenge.

    Negative consequences to business

    Of course, the issues I’m raising here aren’t just IT issues; they are real business issues.  Time to market is typically the number one impacted area. For example, if a two-week development sprint is used to get a business critical need addressed, but then it takes weeks and weeks to get the sprint through the quality cycle and into production due to resource constraints and configuration issues, what have we achieved?  

     Stability and quality are the next big area of impact.  When change is rapidly introduced through new releases and unexpected drift occurs in environments, performance and availability impacts occur.  It is one thing for IT to be perceived as slow and unresponsive to the business, but it is quite another when they are perceived as slow and quality is poor resulting in business impact.

    Just when IT thinks it can’t get any worse, the business asks a presumably simple question, “What happened?”  IT’s typical response, “Let me get back to you on that.”  And then begins a scramble of hours, days or weeks as IT’s best people desperately try to figure out what went wrong.  This is obviously not the best confidence builder for the business.

    Compliance is typically the other casualty as there is often very little awareness of operational policies on the development side of the house so deployment packages and instructions aren’t sensitive to compliance issues.  Building and testing an application without full awareness of policy governance across the whole, multi-tier stack is bound to introduce delays and errors when attempting to deploy in production.

    Visibility, configuration transparency and collaboration; a better way!

    1. Understand drift for the entire application service. Rather than everyone having their own disjointed perspectives on databases, app middleware, network or firewall, and server configurations, provide team members with an understanding from the business service (or application) perspective with direct visibility into the differences between how things should be configured and how they are actually configured for the entire multi-tier application stack.  This enables you to create smarter automations and initiate change processes to bring configurations back in line when necessary rather than waiting for a failed deployment to occur, identifying the root cause, and correcting it while the business waits.
    2. Optimize for flow not individual processes.  As stated in the previous example, making one process faster (e.g. development) without considering its impact on QA, staging and production will generate new constraints downstream.  Instead, view DevOps as one continuous process from goals and requirements all the way through to deployment into production and define success for the team in those terms. When you optimize for flow instead of silos, you will be amazed at how teams share more and collaborate better.

    The benefits of breaking down silos through better visibility, understanding drift and collaboration are significant:

    • faster time to market;
    • quicker ability to react to competitive pressures;
    • deployment with fewer errors;
    • better compliance with policies; and
    • greater productivity for everyone.

     To sum it up, if you break down silos, identify and resolve drift, and manage for flow (while avoiding islands of automation, information, and people) you can unlock agility, maintain stability and ensure the quality your business demands.

  • DevOps: Getting Past Audit

    DevOps: Getting Past Audit

    New business processes (and technologies) too often throw auditors into a tailspin. It’s no one’s fault, really: the decision making on what IT and process controls that need to be in place moves slow – and it is the tail to the rapidly moving technology dog. In fact, while there is no way a standard or regulatory body can predict technological change from year to year, there are ways to help ease the relationship between auditors and organizations embracing DevOps and continuous deployment practices.

    “DevOps audit is an interesting problem,” says David Mortman, distinguished engineer at Dell (formerly Entratius). “Audit is something you just can’t avoid, especially in any industry that’s regulatory controlled. And auditors are a lot like security people, in my experience, which is to say that they have a lot of responsibility and they are justifiably concerned about things they don’t understand,” Mortman says.

    And concern their certainly is when it comes to any organization that must deal with acronyms like HIPAA, PCI DSS, SSAE 16, SOX, FedRAMP, among others means few things will raise auditors’ eyebrows more than hearing that developers are performing deploys on their own code.

    That’s why it’s important to know how to manage these audits, and continuously improve the audit process.

    Rapid technological change, glacial control placement
    One of the first things to note is that these standards and management programs move slow. “If you are familiar with ISO 27001, which is a security management program, they just released a new version at the end of September [2013]. From 2005 to now, the standard hadn’t changed,” said James J. DeLuccia, a senior manager in the advisory services practice of Ernst & Young LLP in his Flowcon presentation, Successfully Establishing and Representing DevOps in an Audit. But technology certainly has changed in that time: increased use of virtualization, the rise of cloud computing, ubiquitous mobile, as well as changes in development and operational processes such as continuous deployment, continuous integration, and DevOps.

    One of the biggest issues that come up with disciplines such as DevOps and continuous deployment is separation, or segregation, of duties – that development, testing and operations should be separated duties. Not only do many government and industry mandates require it, so does many enterprise security policies. That’s why DevOps, when not explained or approached properly, can create bug-eyed expressions in unfamiliar auditors.

    The goal of separation of duties comes down, essentially, to proper change management. If the developer who wrote the code can deploy the code into production, and they are given too much access (such as root) or shared access, there’s a lost control check, and potentially no log of the event. But it’s a concern that is overstated, experts say.

    “In a proper DevOps environment, the person deploying the code doesn’t need to have root access. There’s no reason why, when properly setup, developers can’t logon, push the deploy button, and create an auditable log that says at 2:32 on Friday the 18th, David deployed a version of the code. And you have it down to the hundredth of second who did what,” says Mortman.

    As DeLuccia explained in his talk, when auditors are asking to see change control logs they are really trying to see that these controls are in place. “They’re really asking for help to understand this linkage,” he said. Depending on the policy and requirements, the auditor needs to see reports that demonstrate, in a statistically relevant way, that everything in IT is working the way it needs to and is said to be working. “Because remember, when we’re putting together these reports, we’re trying to say, everything is working the way you say it’s working,” he said.

    Auditing DevOps: An education process
    When done right, Mortman says DevOps and continuous deployments create more accurate logging and change management. “You have a log of every change that’s ever happened and it’s all generated automatically. You don’t have to worry about someone changing something. When setup properly, they don’t have the access to change anything other than the very limited roles of what that application they can deploy to,” Mortman says. “So in a lot of ways, in fact you’re actually maintaining better control over your environment.

    The challenge there, as DeLuccia explained, is that audit programs are built a year in advance. “When they come to see you, they’re already have it planned out based off last year’s risk. That’s already too late [to keep up with rapidly changing technology and processes]. You really want to plan this a year in advance,” he said. The goal, he explained is to become part of the process and show how the procedures they are asking the IT team to execute against are in line with reality of the processes and the technology on the ground.

    “Much like any other industry, you have an education problem,” says Mortman. “Security people run into this all the time. The business will say, “Hey, we want to do this new thing, and the security people reply, “Whoa, hold on a second. You can’t do that. That’s not secure. Or, we can’t audit that.” So a lot of the challenge with DevOps is an educational challenge. And you need to educate auditors that the control processes they are looking for are in place and that there are adequate logs that can be audited and evaluated,” says Mortman.

  • Automating code helps businesses adapt to customer demand quickly

    Automating code helps businesses adapt to customer demand quickly

    This is the first article in a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef (click here) community.

    How does the need for speed and scale affect businesses today?

    Let’s start with scale. As a technology company grows, its infrastructure grows – more machines (or machine images), more networking, more data storage needs, more code, more developers, and more users. The only way to scale with growth is with automation. And, as more and more of the world’s interactions happen online (be it commercial transactions, email, social networking, video streaming, etc.), that scale of infrastructure is breaking records all the time. With these new levels of scale come ever-increasing automation requirements.

    On the speed side, first-to-market has always been an important part of business – but what that means has evolved. It’s no longer simply about producing that new cool item before the other guys. Now it also means you must be able to iterate quickly, adjust quickly for changing user tastes, and adapt quickly to the ever-increasing pace of new technologies. To be able to do this effectively, instead of just developing fast, you need to be able to deploy quickly, react quickly, troubleshoot quickly, and update quickly. For many companies, this is no longer icing on the cake, but instead, the structure underpinning their business – in the same way that having an e-commerce site was 10 years ago.

    Speed is becoming ever more critical for business in the digital age. With the rise of social media, a lot of customer feedback is public or semi-public, allowing people to unite behind specific complaints or desires and provide more clear and unified messages to companies. This helps companies get more feedback of higher quality and – hopefully – to react to it more quickly. This is also dramatically accelerating the cycle of getting features/products/releases to users, and, perhaps more significantly, prompting users to provide more useful feedback faster. As I mentioned, the scale and speed of business operations in every industry are growing daily, and, because of that, automation has gone from being beneficial to critical. In the technology world, this automation comes in the form of code.

    Pushing your code faster and faster means you have to be able to update and deploy infrastructure faster, which in turn means infrastructure/systems teams must also be able to deploy, react, troubleshoot and fix faster.

    But code, in and of itself, isn’t necessarily sufficient to meet the challenge of escalating speed and scale. How quickly you deliver that code is what’s important. Continuous Delivery is one technique used to address this need.

    Continuous Delivery (CD)

    People talk a lot about CD and it’s important to remember that this phrase refers to a methodology that has pros and cons.

    For most technology companies, especially those whose product is primarily Web-based, CD is often the best choice to enable speed of delivery. The main benefit is that the closer to “continuous” your release-cycle gets, the easier it is to have an iterative process. Faster feedback from users means you can more rapidly deliver updates based on that feedback. As you approach CD, you approach near-real-time feedback, which allows you to hone your product or feature far faster than if it were sitting in long dev/qa cycles. Another benefit is best described with a comparison. In troubleshooting, a key rule is make one change at a time, so you know which change fixed the problem – or made it worse. CD is an extension of this idea, although it comes earlier in the lifecycle. If you are pushing/delivering a codebase more often, there are less changes per push, and so breakages are easier to track down – and it’s easier to push fixes.

    A common misconception is that an organization either achieves CD or it does not. CD is a spectrum with frequent releases on one end and less frequent on the other. A CD solution for a bank may mean something very different than a CD solution for a gaming Web site. The idea of the CD methodology is continuing to push toward faster deployments, in order to reap the benefits that come from doing so.

    In the end, no matter what part of the CD spectrum a company aims to achieve, they must be willing to adapt and change.

    From an internal IT perspective, this means realizing that this isn’t the old world, where businesses lock users into small boxes to make things simpler. We’re in a world where everyone has the power of the Internet in his or her pocket and there’s enough information out there for anyone to break out of his or her box. Businesses need to provide engineers, designers, assistants, VPs, etc. the flexibility and freedom to work and think in ways they do best, so they can take the business to the next level.

    From an external (production) IT perspective, companies must be able to quickly adapt to new growth and new technology. It took the Internet a (relatively) long time to start adapting to the small displays on mobile devices, and it adapted only slightly faster to tablets. But new form-factors are just the beginning. Broadband is becoming ubiquitous in first-world countries, and companies are just starting to rely on that fact. Yet there are still many users the Web isn’t taking into account very well. Mobile is becoming the default interface for a variety of locales and ages, and many countries have limited bandwidth and/or high latency. Further, it seems likely that we will see drastically different user input mechanisms, as well as display mechanisms, in the near future.

    Technologists need to be agile and readily adapt to technology and infrastructure changes, as well as speed and approach changes, such as CD, which will be required for the next generation of user and technology demands.

    About the author

    philPhil Dibowitz has been working in systems engineering for 12 years and is currently a production engineer at Facebook. Initially, he worked on the traffic infrastructure team, automating load balancer configuration management, as well as designing and building the production IPv6 infrastructure. He now leads the team responsible for rebuilding the configuration management system from the ground up. Prior to Facebook, he worked at Google, where he managed the large Gmail environment, and at Ticketmaster, where he co-authored and open sourced a configuration management tool called Spine. He also contributes to, and maintains, various open source projects and has spoken at conferences and LUG’s on a variety of topics from Path MTU Discovery to X509.

  • Microsoft bridges the gap between Azure and DevOps

    Microsoft bridges the gap between Azure and DevOps

    DevOps can be easy for a startup. Many of the concepts and principles of DevOps come quite naturally to a fresh company just getting started. It’s a different story, however, for large, established enterprises trying to wrap their arms around this DevOps thing. For IT admins working in Microsoft’s Azure cloud platform, though, things just got a lot easier thanks to the integration of Chef and Puppet Labs.

    Many organizations are heavily invested in the Microsoft ecosystem. IT teams and administrators are familiar with Windows domains, Active Directory, Hyper-V, and Azure. They know how to write scripts and leverage PowerShell to get things done, but they’re much less familiar with the core principles of DevOps, or common DevOps automation tools like Chef and Puppet Labs.

    One of the driving forces of DevOps is the ability—or perhaps necessity—to automate those tasks that can be automated so IT resources are freed up to focus on more important tasks. Deploying and configuring virtual servers and applications is necessary, but it can also be tedious, and it is a task that can be easily automated using DevOps tools.

    The arrangement also works the other way. Smaller companies that are already familiar with, and invested in DevOps may need to scale to a platform like Azure. They should be able to build on what they’ve started, and continue to use the tools they’re familiar with as the company migrates to Microsoft’s Azure.

    One of the main attractions of Microsoft Azure is the degree of flexibility, choice and control it gives customers. Microsoft Azure’s partnership with Puppet and Chef allows customers to choose from an array of existing technologies and communities to easily deploy, configure, automate and manage their VMs.

    According to a Microsoft spokesperson Puppet Labs and Chef complement and extend Microsoft technologies such as Resource Groups and PowerShell Desired State Configuration (DSC) to offer more options for customers to migrate existing or deploy new applications to the cloud. The partnership offers Microsoft the ability to support Linux and UNIX environments, and also alleviates the concerns that organizations already using Puppet or Chef will have about abandoning their investment. Now, they can continue to use their Puppet modules or Chef recipes in Azure.

    Luke Kanies, CEO of Puppet Labs believes the integration of Puppet Labs and Azure will help Microsoft IT admins by making it easier to deploy applications across any platform or cloud environment. Kanies explained in a press release, “Our goal is to help you automate the configuration and ongoing management of your machines and the software running on them, so you spend less of your time firefighting and running simple scripts across a thousand machines, and more time getting great software in front of your users.”

    I spoke with Kanies about the difference in dynamics between a small company and a large enterprise. He told me that in a small company it’s much easier to communicate and coordinate resources, and the cost of failure of any particular project is much lower. For a large, established enterprise, however, there is assumedly a large base of existing customers, and there is more fear and worry about possibly disrupting normal operations.

    I also had a chat with Ken Cheney, VP of business development for Chef. His philosophy mirrors a lot of what Kanies described. Cheney told me that emerging companies have the luxury of starting from ground zero with a clean slate. There aren’t yet any established processes to navigate. The challenge is trying to transform an organization with established processes, and a team of existing developers over to a DevOps mentality. Integrating DevOps into Azure helps organizations to meet that challenge.

    DevOps adds an element of speed and simplicity that makes it easier for larger enterprises to be more agile. Using tools like Puppet Labs and Chef enables IT admins to add speed as a core requirement for infrastructure projects without disrupting or conflicting with other requirements—like security and stability.

    Ultimately, the new alliance illustrates Microsoft’s commitment to empower Azure customers with the best-in-class configuration and automation tools by embracing the technologies, communities and ecosystems that developers know and love. Combining Azure with powerful and popular platforms like Chef and Puppet Labs bridges the gap between traditional IT and DevOps—enabling old school IT admins to embrace DevOps, and DevOps startups to mature and evolve into larger enterprises.

  • Don’t let automation be the tail wagging the dog

    Don’t let automation be the tail wagging the dog

    There is quite a bit of talk right now about DevOps and how important automation is.  This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot.  In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market.  This failed with famous stories of robots painting each other and others welding doors shut.  Ultimately this project failed and the robots were scraped.  It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

    Automation dominates the conversation when it comes to DevOps.  If you do not learn how to build cookbooks you are not doing DevOps.  I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there.    Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand.  The best way to understand it is in two key ways, change and performance.

    Let Science set you free

    This was the secret that Toyota used to dominate the automobile industry.  They were able to transform all of their employees into little scientist.  Constantly tinkering to squeeze our inefficiencies in the system.  Every worker was empowered to think how things could be done better.

    At the heart of the agile and DevOps processes is the practical application of the scientific method.  Think back to your 8th grade science fairs.  No not the volcanos but the ones that won best of show.  These had a clear hypothesis that stated what they were going to try and prove.  They were able to understand what they were going to change and measure the difference between the before and after.  They could clearly state what changed and if it was a successful experiment or a failure.

    This thinking is key for companies trying to transform to a DevOps IT organization.  If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

    Tracking your experiments

    Any good scientist knows exactly what he is working on.  He knows every experiment that is running anticipating the outcome.  He knows all the details about each change that has been made to see how they impact the outcome.  This is the same way that IT should be tracking every change happening.

    Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application.  Developers and Operations will be more informed about the cause and effect of their changes.  There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

    Measuring the outcome

    There is no need to track the experiments if there are not being measured.  A scientist is very deliberate in how they measure the outcomes.  Good enough just does not get this done.  This is beyond just monitoring.  Monitoring in a lot of cases is after the fact, in production.  This is about measuring the difference for every experiment regardless of where it is being tests.  This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update.  Following it through all of testing and to production.  Understanding how each experiment impacts the system as a whole.

    Efficiency through automation  

    Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit.  Because every experiment it known the lower priority ones can be identified and automated.  Risk is better assessed.  Processes can start to speed up because there is visibility and accountability.  Applications are no longer classified as high risk or low risk only change is classified that way.  Determining what can be automated just becomes another experiment that is tried and measured for success and and failure.  Automation becomes just another outcome of DevOps to the business and not the route to achieve it.