staging-devopsy.kinsta.cloud

Category: Features

  • DevOps: The Last Great Hope for Application Security?

    DevOps: The Last Great Hope for Application Security?

    Once upon a time, there was a great battle between speed and security.  Development wanted to go fast.  But, Security wanted to slow down and be safe.

    “We must protect our gilded apps”, cried the application security team.

    “Speed is cherished by our people”, declared the development team.

    For years, they endured the pain of testing late in the lifecycle, sorting through reams of false positive reports, and dealing with the added cost of pushing bad software out the door.  They knew there had to be a better way…

    And then came, The DevOps Revolution.  The DevOps team had an answer.  “Let’s bring Application Security and Development closer together — and shift its focus further to the left”.  The DevOps team knew that by introducing awareness of security vulnerabilities and policies early in and across the software development lifecycle — without creating a time-consuming tax on development — that both teams could win.

    During an evening event named “Wining Not Whining” at the RSA Conference 2014, we gathered some of the top DevOps experts and influencers and asked them “Why is application security so important to the DevOps revolution?”

    Listen to what they had to say:

  • Q & A with Kevin Behr on DevOps, its impact on security, and the challenges of bringing change to the organization

    Q & A with Kevin Behr on DevOps, its impact on security, and the challenges of bringing change to the organization

    Kevin Behr has been in IT game for 26 years and is the author of seven IT management books, including the bestselling The Visible Ops Handbook, which he coauthored with Gene Kim and George Spafford, and The Definitive Guide to IT Management, published by Hewlett Packard. His most recent book, The Phoenix Project, also coauthored with Kim and Spafford, has helped to spark discussion about DevOps globally. Behr is also well-known for his technology and management keynotes for such organizations as the National Academy of Sciences, Hewlett-Packard, the SANS Institute, AFCOM, and the IT Service Management Forum.

    Behr is also the founder of the Information Technology Process Institute (ITPI) and chief science officer for the CXO and Board Advisory Practice at Praxisflow (formerly Assemblage Pointe), where Behr consults, mentors, and coaches IT organizations and executives on how to increase their business effectiveness and competitive advantage.

    We caught up with Kevin Behr to discuss DevOps, its impact on security, and the challenges of bringing change to the organization.

    staging-devopsy.kinsta.cloud: We’re now hearing a lot about continuous deployment, but what does that mean compared to traditional enterprise thinking about deploys? And what does it mean for IT disciplines like security?
    Behr: With continuous monitoring, you can catch failure quickly, even if the issue that caused the failure makes it into a production state. With today’s SaaS and cloud models, failures don’t matter as much because you are not shipping software as much as you are managing a service. And one fix works for all of your users instantaneously.

    This is the difference between traditional enterprise IT thinking and the thinking around continuous deployment, which really is starting to creep into the enterprise. The new thinking is: how do we do this smaller, faster, and more safe to fail. And organizations aren’t ready to make everything safe to fail, but pushing things in that direction becomes an actual security strategy—versus the notion of, let’s just see what happens.

    That’s what we’re getting into with continuous deployment. You’ve actually made the deployment safe to fail. You almost don’t need very much testing because you can actually deploy, then wait to turn on a new feature; and if it doesn’t work, you can turn that feature off. What we’re starting to see is this kind of radio button style of deployment: they’ll deploy software and it’ll be turned on progressively, versus an old-school Big Bang.

    That’s really kind of neat when you think about it. Software is being treated more like infrastructure. You hear that phrase a lot: Code is infrastructure, and infrastructure is code. You start to see the different meanings of that everywhere. And I think that’s right. I think the security practices have to be a part of the safe to fail approach. You want to have very good monitoring and detecting controls, because they’re going to help you be more resilient.

    staging-devopsy.kinsta.cloud: I think these are concepts that may have challenges when working with some security and IT professionals and others who are used to having control over the process. What are you seeing in regard to the balance between an organization’s need to oversee and protect itself and the benefits of continuous deployment?
    Behr: You’re getting at the fear that security practitioners and others may have, that this breaks down controls that they’ve put in place. They’re realizing that it could actually add more resiliency—consider continuous authorization to do certain things, cross-functional teams, continuous monitoring, and all these things that could very possibly increase resiliency.

    You’ll see more success if security could embrace DevOps, from the standpoint of, “Oh my: This is amazing and so what are the top ten things we must do to make DevOps work? Not “here’s why it can’t work.” Security teams that roadblock like that are going to get steamrolled right now.

    But it’s not good news everywhere. I hate to say that some companies out there are more stupid than others. I’m seeing places use DevOps as a way to excuse all the reduction in force that they’re planning: “We need to get rid of five people because we’re doing DevOps.” What sense does that make? I think mini-backlashes are coming in some organizations. And one of them is going to be where organizations call things DevOps that aren’t, just because it serves their ideas.

    For instance, I’ve been into a couple of banks that have created DevOps departments, which is sacrilege to this whole movement, because the whole idea is to get rid of silos, not to create a new one.

    Some days I get bothered by the notion of organizations calling it DevOps, and then other days I don’t because it’s just the result of the success of an idea.

    staging-devopsy.kinsta.cloud: What steps do you see organizations taking to deal with the change? Are you seeing schisms in organizations where some people are resilient to change?
    Behr: I think that the folks who own these systems that they work in have a long way to go. And I think the people who are having a hard time with DevOps are getting steamrolled right now. People in vice president of operations positions, or the director of operations: They may not love the concept of DevOps because it creates cross-functional challenges for them. The first is: “All of these people who don’t report to me and now are in my data center. What are they doing? Why are they on keyboards? How do I control this?” [Story]

    Think about it. Their whole working role is judged on this notion of reliability. Their heuristics tell them that having lots of unaccountable folks doing stuff isn’t consistent with reliability in the long term. In the shops where DevOps has been successful, the security folks have been able to get in and say: this is awesome. They love the reduced cycle times. They love that the dev guys are coming down and helping with patching. Getting things done whenever they need it. And that they actually know what this stuff means now; it means that their security feedback loops directly into dev.

    I think that’s the other thing that people don’t get. DevOps, when really done right, creates parallel feedback loops, not just serial feedback loops. In the old way, someone asks someone to do something, they do it, and then the provider asks how they liked the service? That’s the feedback loop most operations have today. But when the devs are brought in, all of a sudden the devs are adding another feedback loop. It goes something like, ”Hey, I just watched you do this thing that I don’t do every day. Man, that’s a lot of extra steps. Why are you doing it that way?”

    That’s one feedback loop that comes from just teaching somebody something new, and getting the feedback from the outside. And then there’s the feedback loop of the quality of what you delivered. When you’re getting both of those at the same time, parallel feedback loops are how evolution happens. That’s one of the hallmarks of all evolving systems and adapting systems.

    If security can become one of those parallel feedback loops, and all the documentation of what you’re doing when it happened, and all that related stuff is in one place, then security becomes part of the refactoring.

    I think the other thing is, because the dev and ops people are collaborating more, you’re starting to hear more contextualization happen with the developers. They’ll say things like, “I’ve always wondered why they yell at me whenever I hard-code IP addresses. Now I get it: because you guys have to change these all the time, because we’re now down here changing a bunch of them.” That contextualized understanding goes a long way.

    This is a huge thing. Because there’s a ton of infosec ops: patching, log parsing, and different things that happen at an operation level that are really security driven. And if you can get the devs involved in helping out with some of that stuff, they start to understand why you’re doing it. And in many cases, we’ve got technologies in place that we spend a lot of money on, and security, that are just to prevent devs from doing things that we didn’t think we could change.

    The organizations that get this—and foster it—are the ones that are going to succeed.

  • DevOps Adoption – Startups

    DevOps Adoption – Startups

    This is part 1 of a 2 part article

    DevOps is somewhat unique in that it has applications to both large enterprises and to startups.  The way DevOps is adopted in startups is incredibly different than how it happens in large organizations. The profile of how DevOps is adopted in each of these cases is the focus for this two part article series.

    In this first installment, we will study how DevOps is adopted within startups and in the second, how enterprises adopt DevOps. There are lessons for each in the way that DevOps is adopted. Organizations can leverage this information to enable better, faster, and easier adoption of DevOps.

    Much of the DevOps way is rooted in startup culture. From methodologies such as Agile, lean principles, and just in time manufacturing, startups adopted DevOps as a means to compete, innovate, and move faster. DevOps is inherently a quicker, better way to get product to market – a perfect fit for startups. They could adjust quickly, iterate again and again to get to the point where customers en masse were interested in their technology.

    In order to understand how DevOps is adopted in startups, we need to look at how startups evolve. In the high-tech world, a startup gets going when a founder or group of founders has an epiphany for a new product or service. Generally one of the founders will start to develop the solution by writing some code. The startup may add a couple of developers but it is usually just a few folks cranking out the first version of the product. There is often a business or product focused function where requirements and feedback are gathered from potential users or customers.

    In this first phase, the development team often adopts Agile as a way to quickly iterate on the development process. These short-term sprints build the initial functionality of the product. While initially, there isn’t really much to release to potential customers, the core developers are starting to build their development and release cadence. The short sprint help to keep the functionality requirements small and the team on-track to deliver a small piece of functionality in a short amount of time. This foundation also ends up being an important part of the future development process. And, just as importantly becomes the kernel of DevOps.

    As a small team building the initial releases of the product, the culture and environment of the startup is generally fast, highly communicative with little to no process. Often the development team is sitting side-by-side cranking out code quickly. No specs, just a list of what needs to be done often stored in Pivotal Tracker or some similar tool. Sometimes mockups and screens designed to describe the functionality. There isn’t a rigorous process in the choice of technology platforms – language, database, etc. are chosen often based on what the core team knows or thinks is the right solution rather than on research. This small, agile environment is critical to understand when thinking about how DevOps is adopted.

    There is little formal process at most startups. Things are evolving and moving so quickly that there just isn’t time. The goal, after all, is to get a product into market and start selling. While startups are high adopters of Agile as a development methodology, there is generally little rigor in its implementation. There doesn’t need to be because the team is so small and trying to move quickly. Adding in product management, sales, and marketing functions are often ad hoc. There isn’t a well-developed end-to-end product release process. The movement from development to operations is also less than well defined. Often, the developers building the product are the operations people deploying and managing the product or service. Process gives way to the practicality of just getting things done.

    The first phases of DevOps adoption start to occur at this early stage in a company’s life. Developers as they are building their product often encounter some basic challenges and begin to solve these issues. How they solve them often dictates whether they take a DevOps path or not. One of the first things that developers do is build an environment to test their software. Often, they will find a cloud provider and start spinning up a few instances to host their application. They also may need a database server to round out their testing. They might manually do this a few times, but as they test more and potentially on different platforms they will start to look for some tools to automate this process. Of course, at this stage, there are likely zero operations people, so the tasks fall on the developers. And, developers wanting to spend more time coding pick-up some tools to do this work.

    The tools might include infrastructure providers like AWS, Rackspace, or SoftLayer to configuration automation tools such as Chef or Puppet. There may be a variety of other toolsets chosen as well including project management tracking, libraries, language frameworks, and others. The goal with this first wave of tools is just to save time. That could be by not having to code basic functions, the setup of test environments, or manually tracking tasks. As the organization launches and brings their product or service to market, they may adopt some management and monitoring tools to help them gain visibility into their product or service. Generally at this early stage the decisions are focused on driving short-term problem solving. A developer needs a test environment quickly and repeatedly. A solution to that need is quickly found and implemented. The underlying paradigm is to move fast, solve problems, and get to market.

    While tools are hardly a complete DevOps methodology, they are the catalyst for the next step in a startup’s journey to adopt DevOps. Many of the tools chosen in this first period have APIs, can create automation, and are highly configurable. As a startup keeps moving forward and increasing their users, the flexibility of these tools and the various ways that the developers utilize them starts to drive some conversation around process. A few hiccups in the production environment with some bugs, outages, or failures, and the technical leaders begin to start to say the ‘p’ word. Process. While there may be some Agile components already embedded, the growth of the startup starts to create a desire to add in some control, accountability, and guard rails around the product development process. Many organizations will also start to loop in the business side so that there is a more comprehensive process around releasing a product to market. Agile now starts to extend up and down the organization in a more rigorous way. DevOps is starting to take hold and become more fully formed.

    The early stage DevOps lifecycle wraps up with it becoming a part of the culture. The organization now shifts into a mode where this methodology becomes a fundamental underpinning of the business. Values such as agility, intense customer focus, rapid iteration and others are a part of the fabric of the company and encompass the entire organization. Hiring decisions are made based on whether the candidates match the philosophy and culture of the company. DevOps is viewed as more than just a process that the company follows, but a competitive advantage in fulfilling its mission to its stakeholders. At this point, DevOps is firmly embedded into the organization and becomes its way of life.

    Startups are unique in their approach to adoption of technology and methodologies because of their size, focus, and evolution. DevOps adoption, as we have seen out in the wild, seems to go through some natural stages. While the stages may overlap, it generally follows a tools, process, and then culture sequence. It could be said that some startups just start with the culture of DevOps, and that can be true for some, but many that we see are often so small at that stage that the culture is largely undefined. IT leaders who understand this sequence can become more intentional about the way that DevOps is adopted, perhaps accelerating it or even shaping it earlier in the company’s lifecycle. A little bit from now, our hypothesis is that startups won’t even start without DevOps embedded into their plan.

    In part 2 of this series, we will examine the adoption of DevOps at larger companies.

  • Netflix, the Simian Army, and the culture of freedom and responsibility

    Netflix, the Simian Army, and the culture of freedom and responsibility

    What do you do if you’re entire business relies on delivering streaming video content over the Web to millions of customers, and there are no examples to follow or tools to use? If you’re Netflix, you set the example and build the tools.

    Netflix had tremendous success with its original business model shipping movies on DVD to customers in trademark red envelopes. The precision of the shipping logistics, and the efficiency with which Netflix was able to meet user demand and ensure speedy delivery is a case study of its own. But, Netflix also saw the writing on the wall and became one of the first to embrace delivering movies via streaming video over the Internet.

    Networking protocols are fairly resilient, and able to get data from Point A to Point B eventually, but streaming content like audio or video is less forgiving. The data needs to be transferred at high speed, and the packets need to arrive in order to avoid buffering or glitchy playback.

    Netflix chose Amazon Web Services (AWS) to provide the cloud server infrastructure for delivering its streaming content. Amazon is reliable and resilient, but it is not infallible. When Amazon experiences a massive outage—as it has a few times in the past—it can mean down time for Netflix customers. Netflix, however, works very hard to guard against being affected by issues like that, and—thanks to the Simian Army—it is quite successful.

    Simian Army

    I spoke recently with Ruslan Meshenberg, director of platform engineering and Josh Evans, director of operations engineering for Netflix to learn more about how Netflix has embraced both DevOps and open source to develop tools that help it deliver high quality streaming video content to customers around the world.

    Simian Army is the name Netflix has given its suite of automated tools engineered to stress test the Netflix infrastructure and proactively identify weaknesses so Netflix can resolve them. The Simian Army is comprised of tools like ChaosMonkey, ChaosGorilla, and ChaosKong.

    Josh explained that when you’re dealing with a massively distributed system like the Netflix cloud infrastructure, it’s inevitable that you will encounter bugs. Rather than waiting for issues to surface, Netflix uses the Simian Army to intentionally induce failure on its own terms, so that issues can be identified and fixed before they impact customers.

    ChaosMonkey wreaks havoc on an individual server or cluster level—randomly shutting down servers to make sure that automated resilience works, and that redundant systems pick up the slack so content delivery is not interrupted. ChaosGorilla works on a larger scale by taking out entire AWS zones, and ChaosKong takes the concept to a national level by randomly shutting down the East Coast or West Coast regions of AWS to ensure traffic is automatically redirected to the remaining region.

    There are also other tools in the Simian Army. JanitorMonkey keeps everything tidy by cleaning up unused EC2 resources. As the name implies, its job is to automatically clean up after developers. Then there’s ConformityMonkey, which checks for EC2 instances that are not conforming to predefined rules for best practices.

    Netflix lets the Simian Army roam free causing random chaos, but only Monday through Friday between the hours of 9am and 3pm when managers and developers are present to address any urgent situations that might be caused inadvertently. In some cases the tools take automated corrective action, and in some they simply generate an alert and escalate the issue to the appropriate group or individual.

    Embracing Open Source

    One of the things that makes the Simian Army unique is that Netflix created these tools out of necessity. Netflix was blazing a trail on the cutting edge of delivering content from the cloud, and there simply were no commercial tools available to do the things Netflix needed to do.

    In the early stages of scaling cloud services to meet demand, Netflix frequently stumbled and hit a variety of walls. Netflix was a pioneer in delivering content through the cloud at this scale, but the open source community has experience, and an established understanding of working with a distributed environment.

    By marshaling the resources of the open source community, Netflix was able to get more eyes on the code, and enlist the volunteer support of hundreds or thousands of developers to help refine and improve the tools in the Simian Army, and other tools Netflix has developed. Ruslan and Josh also explained that there is an unintended benefit of cooperating with the open source community: in-house developers are more diligent when they know that the code they write will be exposed to the general public.

    Netflix doesn’t just take advantage of the open source community; it also makes its tools available to the general public through the Netflix Open Source Software Center on Github. There are tools for availability (like the Simian Army), cloud management, persistent systems, infrastructure services, developer productivity, and more—all available for free for anyone who wishes to use them.

    Culture of Freedom and Responsibility

    Lots of organizations talk a good game. It’s easy to come up with a profound mission statement, or an insightful set of company values. Actually living up to them, however, is a whole different story. Netflix prides itself on building a culture of freedom and responsibility.

    The Netflix culture is part of what makes the company so successful, and why Netflix has been able to achieve what it has in the realms of DevOps and open source. Netflix believes in finding highly skilled, seasoned engineers and developers, and allowing them the autonomy to get their jobs done and own responsibility for what they produce.

    Other organizations can learn a lot from Netflix. Netflix has paved new paths for DevOps, set the bar for collaborating with the open source community, and established a culture of success that any company could benefit from adopting.

  • Google Starts Cloud World War I, DevOps Wins

    Google Starts Cloud World War I, DevOps Wins

    No matter the outcome, a cloud pricing war will leave DevOps the winner

    By now you probably saw the news and all of the buzz about how Google dropped its drawers eh, pricing on its cloud hosting. This is of course a direct shot at Amazon’s AWS. It’s about time someone with big enough stones (and a big enough data center) took on Bezos and his band of merry cloud makers.  But the real winner in all of this may in fact turn out to be DevOps.

    Of course this is just the first salvo in what promises to go down in history as Cloud War I. Amazon is not going to stand still.  They are perfectly willing to compete on price and in fact part of the AWS mystique is that it is the most cost effective solution (cheapest) out there.

    This will not be just a tit-for-tat move though. In announcing the price cuts Urs Hölzle, VP of TI at Google said, “”We think cloud pricing should track Moore’s Law, so we’re simplifying and reducing prices for our various on-demand, pay-as-you-go services by 30 to 85 percent.”  That means we are going to see cloud prices drop as the speed and power of chips goes up. That is good news! You can see Hölzle’s complete presentation including a cameo by SnapChat’s Bobby Murphy below (be warned this is about 1 hour and 20 minutes long).

    [youtube=http://www.youtube.com/watch?v=qokEYBNWA_0]

    Of course this begs the question, what about the other cloud providers.  How long can Rackspace sit up on its OpenStack perch before diving into this fray? What about Microsoft’s Azure? Redmond wants its piece of the cloud pie, does new CEO Satya Nadella have the stomach for this fight? Can they afford not to? The fact is that Microsoft has the girth and arsenal to compete here. Maybe Google starting this war is the opening they need to climb out of the tech cellar and be relevant again. Cisco just announced a billion dollar initiative to become a cloud powerhouse as well.  As big as a billion is though, that is penny stakes in this game I am afraid.

    Here is another one to watch, IBM’s SoftLayer. Fresh off of a 2b+ acquisition and pledging another billion in cloud dollars, IBM via SoftLayer has the profile to be a winner in this war as well.  They also have IBM’s minions out pushing the solutions. In fact Microsoft and SoftLayer/IBM may have the channels to win this thing in the long run (OK maybe not).

    So where does this leave DevOps? Think Switzerland.  From a DevOps perspective it really doesn’t make a difference which cloud provider emerges dominant. The cheaper and more attractive cloud becomes the more it will drive better DevOps tools, DevOps culture and philosophy.

    Of course there are those that say that if PaaS becomes the dominant cloud platform (and I think it will), that could spell trouble for DevOps.  As if somehow PaaS doesn’t need DevOps.  I say poppycock to that too. Maybe you won’t configure servers in a PaaS environment (and I am not conceding that either), but there will be plenty of places along path that will call for automation, continuous delivery, continuous monitoring and continuous management.  PaaS does not obliviate DevOps.

    That is not to say that applications are not king.  Applications are driving the cloud, DevOps and everything else tech today. A large reason why Google is so goo-goo over Cloud is because it drives more Android apps.  I think we will see more DevOps tools and practice around application development, configuration, management and monitoring. New Relic is already on it.  Anything related to Docker stands to do as well as munition and tank manufacturers did during WW II.

    So buckle in as it may get rough out there as these tech giants battle for cloud supremacy, there an will be winners and there will be losers.  It won’t be just the cloud providers either.  Wars create opportunity and those willing to take the chance will gain the most.

    But for DevOps it is all good. It is like we have a stash of chocolate and gold while we make watches here in Geneva. The war rages on around us but we need at the end of the day no matter what we win.

  • DevOps Is For Horses: Stop Making Excuses For Starting

    DevOps Is For Horses: Stop Making Excuses For Starting

    While DevOps practices may be spreading throughout organizations both large and small, often times the most visible success stories are also the most exceptional. Organizations like Netflix, Etsy and Google are brought up in conversations about DevOps time and again because they offer the most stark examples of the benefits that the model can bring to businesses. But the scale of their operations can often be intimidating to the everyday IT person.

    “People will tell me, ‘Well, these are unicorns—special companies. We’re not like that. What works there will not work here,’” says Jez Humble, a principal for the consultancy ThoughtWorks and co-author of Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. “I think that’s a fallacy.”

    According to Humble, two phrases that he hates the most as excuses for not adopting DevOps or continuous delivery principles are ‘That can’t work here,’ and ‘But we’ve always done it that way!’ He’s one of a growing contingent of DevOps pundits who argue that it is plenty possible for even the most run-of-the-mill organizations to engage in DevOps.

    “DevOps is also for horses,” says Gene Kim, author of The Phoenix Project. “In fact, every unicorn was once a horse. Amazon in 2001, Google in 2004, Twitter in 2009 and LinkedIn in 2010—they all had huge monolithic code bases and inconsistent deploys.”

    Humble agrees that these so-called unicorns “have the same problems as everyone else.” But no matter how remarkable the scale of their solutions to those problems, other organizations should be able to learn at least one very valuable lesson from their case studies.

    “The critical piece is they’re always working to improve what they do, and that’s something that works everywhere,” Humble says.

    That’s Nick Galbreath’s philosophy. As the former director of engineering at Etsy, Galbreath saw first hand the transformation that the organization was able to make using DevOps patterns. He’ll assure any organization that it didn’t all happen in a day. Which is why when people ask ‘How do you get started with DevOps?’ his answer is simply to start.

    “If you have a sympathetic management ear, ask ‘Why don’t we try to cut deployment time in half?’” says Galbreath, who is now the vice president of engineering at IPONWEB philosophy. “Because the other way is to say ‘Deployment time takes too long, let’s do half as many.’ Wrong direction.”

    Setting up a goal like cutting deployment times in half will get an organization streamlining and weeding out junk processes. It’ll also have the team working to move those processes from inside a select few peoples’ heads and into things like a batch script or even just a simple document so that they’re easily repeatable across the team and that more can be done in parallel.

    “Cutting that work that adds no value in half, you’re really going to empower your group and your team to come up with innovative solutions,” he says.

    In a similar vein, DevOps for normal folks could also be chunked up incrementally on a project-by-project basis or automation tool-by-automation tool basis, says Josh Corman, CTO of Sonatype.

    “You don’t have to do an enterprise-wide shift hard left to DevOps,” he says. “You can do it in little projects or little small automations that free up time so you can achieve more competence.”

    In addition to incrementalism, Humble suggests providing a leadership environment that encourages taking chances and learning from mistakes.

    “Create safe situations so that when people make mistakes we can actually learn from them rather than engage in finger pointing,” he says.

  • DevOps Contrarian

    DevOps Contrarian

    I admire and appreciate the ideas of DevOps as a result of living through two decades of working in IT and experiencing the pain of what I would call the “failed state of IT”. The failed state of IT being something inherently broken, something that couldn’t survive on its own, something that eats its budget and provides no value and something where the people working in such a system don’t even see the worth of the system and stop challenging it all together. In order to correct or prevent these “failed states” I often apply my experiences learned through DevOps and Lean processes and values.

    DevOps to me has been about recognizing and pulling yourself out of this “failed state” concept of IT and doing so with a little bit of empathy, a little bit of common sense and a lot of science. I don’t necessarily mean pure science in trying to come up with an underlying law that describes and predicts your system of work, but the foundations of science – The ability to question everything, the ability to collect data, analyze data, measure data and develop analysis thereof. By thinking “DevOps Science” you choose to develop models that describe your work, define your flow, show the value of your system and tailor it to your business needs and to research the best models to achieve your end goal and strive to continually improve those models. DevOps is about putting this science to work, and doing so with collected intelligence & experimentation behind it.

    Where I find myself being a DevOps contrarian of sorts is in the often over simplified representation of DevOps as an applied model. I don’t have any issue whatsoever with the concepts described below by themselves; they create amazing discussions and speak to very core foundations of DevOps, but they’re often only spoken of superficially.

    DevOps is not just about Culture.

    I often see people speak of DevOps as an applied “Cultural Invention”, an innovation found to be useful to a group of people in their behavior. While that is a great description from a 30 foot view as if you’re looking from the outside, it really doesn’t help one understand culture in any sense to help foster it. DevOps culture isn’t about copying what others do and hoping for success, it’s about developing it within realizing that its affected by both forces encouraging change and forces resisting change and finding the balance there of is that is important. You can’t simply force DevOps on an organization and call that culture. The real power of  DevOps as a cultural invention is being able to develop DevOps values that directly apply to your organization so that it can be passed down generation to generation and survive on its own.  Thinking scientifically, we learn from these shared examples of cultural invention and their successes by experimenting with them to see how they fit our organization as there is no universal law of culture but merely models we can research and identify with. It’s great to build upon the success for others, but the only way to validate those successes is to test them yourself!

    Another danger of thinking applied culture is the development of cultural optimums, I parallel this to what Garret Hardin coined “The Tragedy of the commons”, whereby individuals, acting independently and rationally according to each one’s self-interest, behave contrary to the whole group’s long-term best interests by depleting some common resource. I won’t claim to define what your resource is that you’re depleting, but it’s something to keep in your mind as you think of culture because it’s important to understand culture external of yourself as much as it is within. The value of culture isn’t in how you rationalize it for your own goals, but how it operates on its own and if that culture is appropriate for the good of the entire system (business). The strongest metaphor in DevOps for depleting the wrong resource is wasted resources not working towards your common goal. Are you actively engaged on the right tasks? Right automation? Right flows? Are you merely creating more work or are you adding value? DevOps isn’t something you apply methodologically, its something you experiment with methodologically.

    DevOps is not just about automation.

    Automation is the process for operating with minimal or reduced human intervention.  If you’re focused on automating processes are you still improving that process or did you think to yourself “I automated that, now I’m free to work on other stuff”? Automation can tempt you into doing “more work” if not kept under control and that creates a real danger of losing focus, having too much work in progress and actually hurting your overall performance.

    Often times the best value in the systems we call “Systems Automation“ – the Chef, CFEngine, Puppet, Ansible & Saltstack systems  is not necessarily their amazing capability to automate processes but their ability to document processes. Those cookbooks, manifests, scripts and DSL’s you’re using are absolutely the best documentation of your system you can have. They provide a great resource when done in such a way to easily describe the processes that they’re automating so that they’re well enough documented for improving them. You can’t automate that which you don’t understand and the best way to understand them is to make sure the processes that describe them are easily readable, documented, tested, and validated. They should be part of your experiment, not just testing for failure but testing for improvement.

    Through the lens of science, we can look at automation more as standardizing on a process and applying the best resource to that process.  Sometimes it is compute power instead of human power  but don’t fall into the trap that the process is said and done (Automated all the things!) and time to move on! Science is a never ending process by where you build upon your conclusions!

    One conclusion I and many others have come across is that people are the most flexible resource you have! It may be a better value to your business to have the flexibility of a human resource if the process you’re trying to build requires flexibility. Even when planning for automation, requiring human intervention is a perfectly valid result and a justification you can sell to your business that has been proven through experimentation & data. Don’t ride the automation bandwagon when your data shows otherwise!

    DevOps is not just about measuring.

    What are you doing with the data you measure? What value does measuring give your organization? When you have the flu and you take your own temperature, what are you measuring for? Do you have the cultural aptitude in your organization to accept the ramifications of what you measured? Can your organization stop working when it’s “sick”, take the time to heal and start back up after taking corrective or remediation actions?

    Measuring is extremely important, but measuring isn’t what DevOps is.  Measuring is a part of what makes your organization responsive, culturally adept, and accountable for itself. It’s what allows you to convey value, to compare itself against and strive for improvement. If you’re not measuring, what are you improving against? Inversely, If all you’re doing is measuring, what is the value of your measurements? What are they the basis of? I like to think of biological systems when I start thinking of computing systems where there is a cost of energy to do something. Are you utilizing that cost the best way you can by measuring what you are or should you evolve the system to spend that energy elsewhere? Are you collecting the right metrics to even comprehend your systems? Do your metrics allow for capturing exaptation and evolution thereof? When you experiment, measure, rinse & repeat you get patterns; measuring helps identify those patterns. Creativity, experimentation and the process of science allow you to act upon those patterns and create value from those patterns. Some people call that innovation.

    DevOps is not just about sharing.

    There are people who believe that unless you are open source, have a GitHub repo and publish your code, you’re not really sharing and you’re not DevOps. I certainly hope we’re not going down this path and certainly hope we can appeal to the broader ideas of sharing. Sharing is about the joint use of a resource and sharing is knowledge charity. Sharing goes hand in hand with culture in not just expressing your needs and sharing your knowledge, but having the ability to listen, feel empathy and accept what others share as well.  When it comes to collaboration & sharing it’s much easier to accomplish if everyone sees it through the lens of science, not only do you speak to DevOps values more rationally but you can even accept opposing views and ideas much easier because you have evidence to them and aren’t emotionally tied to their success, you’re tied the process that challenges their success as a way to improve them!

    DevOps through the lens of Science.

    The beauty of DevOps through the lens of science is that science is about challenging ideas,  accepting failure, learning from history/prior experiments and having the strength to not only say those successes and  failures are a valid output of your experiments but also come up with a new experiment or new hypotheses and do the process all over again. This helps define your culture, this helps setup what you should automate, this helps you validate what to measure and this develops what you should share.  Through the lens of science you learn through experimentation, testing & verification what works for you, what creates value, what brings you closer to your goal. When honestly done you also learn what doesn’t work for you, where you failed, and how you can use those lessons in failure to continuously improve.  A failed hypothesis after all is still good data!

    Why so much focus on the sciences? Personally, I enjoy astrophysics not because there is direct monetary value in understanding the cosmos around us but because of the philosophical nature of being, the beauty of knowing and the questions it gets you asking as the more you know, the more you wonder. That wonder, is what makes DevOps so great to me. That wonder is what drives continuous improvement. That wonder is what drives the “what next…”. That wonder is what pushes “question everything”. That wonder is always what pushes me to accept and understand the science we do know and trust it because it’s based upon the values that are respected by thousands of scientists all over the world through peer review and consensus. That foundation is not the assumption that we think we know it all, but that we know enough to describe something and we trust that description good is enough that we challenge you to break it.  I would highly recommend that any DevOps practitioner cross pollinate their technology passion with the sciences as there is nothing more exciting than realizing commonalities of biology, physics, astrophysics and the laws of nature within your own thoughts and patterns as they’re applied to DevOps.

    So be a DevOps contrarian; for its not accepting DevOps as the answer, it’s the challenge there of that will make it aspire to bigger and better things. It’s not that Culture, Automation, Measuring and Sharing is wrong by any means, but they’re just scratching the surface of what you need to understand to make DevOps “Work” for your organization.

    I’m not only a DevOps contrarian, but I’m an OS contrarian, a Developer contrarian, a Systems Contrarian. I’ve made the best of running Windows, Oracle Linux, Oracle Database, eBusiness Suite, Weblogic and much more. Not because I’ve personally identified or I am idealistically aligned with any of these tools & systems, but because they solve the needs of the business; they create value, they operate and perform admirably and we have proven their worth. It’s no utopian system by any stretch of the imagination, but I find great reward in the continuous experimentation of making these systems work and improving upon them in a very DevOps way.

    After practicing DevOps & Lean principals and having  over two decades of IT experience, I’ve realized the “failed states of IT” and people who thrive in them are people who stopped critical thinking, stopped challenging the status quo and stopped experimenting.  They stopped having a contrarian.

  • Config Management & F***ing Shell Scripts

    Config Management & F***ing Shell Scripts

    Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

    Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

    Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

    Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

    We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

    People either missed this, or chose to ignore it:

    “This is simply brilliant! I’ve dreamt about such tools.”
    – artellectual

    “I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
    – sdegutis

    It provided a perfect platform for many to voice their frustrations with current tools:

    “Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
    – mattjaynes

    “In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
    – s_kilk

    “We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
    – sdegutis

    “Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
    – sp1982

    In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

    Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

    Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

    Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?

  • What “Programmable” Means

    What “Programmable” Means

    One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

    Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

    When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

    The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside. 

    programmabilty-illustratedWhat that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

    Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside. 

    To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership  to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside. 

    So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic  necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

    Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

    Okay, so that’s data path programmability. What about control path programmability?

    Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

    So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

    But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

    SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

    Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

     

  • Don’t let automation be the tail wagging the dog

    Don’t let automation be the tail wagging the dog

    There is quite a bit of talk right now about DevOps and how important automation is.  This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot.  In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market.  This failed with famous stories of robots painting each other and others welding doors shut.  Ultimately this project failed and the robots were scraped.  It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

    Automation dominates the conversation when it comes to DevOps.  If you do not learn how to build cookbooks you are not doing DevOps.  I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there.    Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand.  The best way to understand it is in two key ways, change and performance.

    Let Science set you free

    This was the secret that Toyota used to dominate the automobile industry.  They were able to transform all of their employees into little scientist.  Constantly tinkering to squeeze our inefficiencies in the system.  Every worker was empowered to think how things could be done better.

    At the heart of the agile and DevOps processes is the practical application of the scientific method.  Think back to your 8th grade science fairs.  No not the volcanos but the ones that won best of show.  These had a clear hypothesis that stated what they were going to try and prove.  They were able to understand what they were going to change and measure the difference between the before and after.  They could clearly state what changed and if it was a successful experiment or a failure.

    This thinking is key for companies trying to transform to a DevOps IT organization.  If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

    Tracking your experiments

    Any good scientist knows exactly what he is working on.  He knows every experiment that is running anticipating the outcome.  He knows all the details about each change that has been made to see how they impact the outcome.  This is the same way that IT should be tracking every change happening.

    Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application.  Developers and Operations will be more informed about the cause and effect of their changes.  There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

    Measuring the outcome

    There is no need to track the experiments if there are not being measured.  A scientist is very deliberate in how they measure the outcomes.  Good enough just does not get this done.  This is beyond just monitoring.  Monitoring in a lot of cases is after the fact, in production.  This is about measuring the difference for every experiment regardless of where it is being tests.  This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update.  Following it through all of testing and to production.  Understanding how each experiment impacts the system as a whole.

    Efficiency through automation  

    Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit.  Because every experiment it known the lower priority ones can be identified and automated.  Risk is better assessed.  Processes can start to speed up because there is visibility and accountability.  Applications are no longer classified as high risk or low risk only change is classified that way.  Determining what can be automated just becomes another experiment that is tried and measured for success and and failure.  Automation becomes just another outcome of DevOps to the business and not the route to achieve it.