staging-devopsy.kinsta.cloud

Category: Doin’ DevOps

  • Zen and The Art Of Infrastructure Maintenance

    Zen and The Art Of Infrastructure Maintenance

    There’s a famous passage I’ve quoted from time to time when I see someone “stuck” with a problem, flailing potential fixes and solutions wildly in hope that one of them does the trick.

    “Assembly of Japanese bicycle require great peace of mind.”

    Just like that. Improper grammar and all, quoted verbatim from Robert Pirsig’s philosophical classic Zen and the Art of Motorcycle Maintenance. The quote is in reference to the first printed words on a set of technical instructions, which the narrator discusses with friends and colleagues on a long-distance motorcycle trip.

    And the response I get is the same every time, likely the same response you’re having right now…

    What in the world are you talking about?

    Well good, now I have your attention. And I’d be willing to bet that attention is on 1) the ludicrous loss of translation from Japanese to English and 2) debating whether or not something as seemingly superficial as “peace of mind” is really required to assemble a bicycle.

    Language barrier aside, let’s abstract away from modes of bipedal transport into a more generic “machine”. Do we really require “peace of mind” to assemble or create a machine?

    cory2

    Reliable? Maintainable? Usable?

    We certainly all want to receive peace of mind from our machines. We want them to do the job they were created to do without fail so we can move our attention onto other things.

    What words do we use to describe such machines that give us this peace of mind? Reliable? Maintainable? Usable? When we think of the reliability, maintainability or usability of a machine, we’re really just describing a machine that can produce peace of mind. I’d argue if we want that machine to produce peace of mind, then it must be created under peace of mind.

    Why is that? If we, the creators of machines, are in a state of peace of mind when we start our endeavor, we’re highly motivated to do that which will continue our state of peace of mind. We don’t want to lose this state. Since if the machine does break, we the creators will be forced to lose our peace of mind and fix it, we’re motivated to create a machine that’s resilient against failure and resilient against our loss of peace of mind.

    Without this key precondition to creation, we’re likely to build our existing personal problems right into the machine itself! Under time pressure? We’ll cut corners to make deadlines. Dealing with a different machine breaking? Likely to gloss over a key failure condition in the machine we’re actively building so we can fight a fire elsewhere. Worried about the problems of today? Then I can guarantee there’s no thought going into a machine that will be resilient to the problems of tomorrow.

    Quality creations produce serene creators as much as serene creators produce quality creations.

    Those in a state of serenity will continue to create that which maintains the state of serenity or peace of mind; those that are not in a state of serenity are unable to escape a vicious cycle of build-fix-build-fix, never achieving peace of mind. Ultimately, the serenity of the creator is the key test for the quality of creation.

    And ultimately, this passion for peace of mind from our machines is what you’ll find when you meet the folks in the DevOps community. We’re guided by a desire to maintain our peace of mind — for ourselves, for our families and for the folks that count on us the most: our customers.

    If the machine produces tranquility, it’s right. If it disturbs you, it’s wrong until either the machine or your mind is changed.

    The material object of observation, the bicycle or [in this case, the global infrastructure], can’t be right or wrong. Molecules are molecules. They don’t have any ethical codes to follow except those people give them. The test of the machine is the satisfaction it gives you. There isn’t any other test. If the machine produces tranquility it’s right. If it disturbs you it’s wrong until either the machine or your mind is changed. The test of the machine is always your own mind. There isn’t any other test.
    – Robert Pirsig

    So the next time you find yourself struggling with a problem, take a lesson from Robert Pirsig: step back, achieve peace of mind, combat that which threatens peace of mind and strive to maintain peace of mind.

    You’ll be a better (and happier) creator for it.

  • Docker as a framework for your DevOps culture

    Docker as a framework for your DevOps culture

    If you’re like me and spend a lot of time evaluating new technologies with the goal of “doing more faster” with your engineering organization, then you’re certainly aware of the many choices of DevOps tools like Puppet, Chef, Ansible, Salt, etc. In my opinion each of these tools is amazing at one piece of the DevOps puzzle – configuration management, and some are OK at other DevOps tasks. Each of the tools has a slightly different approach to configuration management but after comparing them I find that you’re usually picking a tool based on stylistic reasons like the DSL, existing recipes/playbooks, or existing familiarity. While I have my opinion on which of the above tools is “the best”, each tool only solves the configuration management piece of the puzzle.

    If DevOps is truly a culture change and not just the codification of operational tasks then we need tools that help foster that culture and I think Docker goes a long way towards doing that.

    Coming from the perspective of a developer looking at development tools, I see the above tools as APIs for DevOps, where Docker is more of a framework for DevOps. This is because Docker is both a runtime environment and a configuration management tool in one. Best of all Docker eliminates many common tasks in your development cycle and creates a tighter feedback loop between your development and operational tasks. I believe creating fast effective feedback loops between development and operations is critical weather development and operations are entire departments or simply tasks in within your organization.

    In my experience developing software as a service platforms, one of the most problematic parts of the process is the management of the environments, how many times have you heard “works for me” or “works on our test servers”? Generally issues like this arise from subtle dependency changes between the environments introduced by both developers, QA and operations throughout the deployment lifecycle. To fix this problem I’ve seen, and attempted, several approaches. For example I’ve tried having a base VM and using Vagrant with Puppet or Chef to create local environments that were the same as test and production environments. This seems to work at first but it proved to be cumbersome to manage the differences between development and test environments as time went on. I found that this mostly because the DevOps (configuration management) tools are geared towards maintaining a perfect operational state based on strict server roles defined in the tool. While this approach was better than manual management of development and test environments it was still quite easy to have the configurations between environments drift apart over time and end up with the same issues mentioned above.

    So how does Docker help and what makes it better?

    • Uses a layered approach to dependency management, making the configuration of environments easier to maintain.
    • Provides various runtime options allowing the same image to be run in multiple ways without extra configuration.
    • Can use any or no DevOps tools to build up your Docker images reducing the learning curve across engineering.
    • Lightweight runtime makes running multiple Docker images on a single machine more feasible than with VMs.
    • Decoupled network and storage layers allows a single Docker image to be run on a single system or multiple systems easily.

    These features allow developers to run complex multi-service architectures on their laptops and with the same runtime images operations can support any number of load or availability scenarios with as many or as few systems as necessary. This combination of predictable dependency management and runtime flexibility is like having all the ease of a Heroku environment with the flexibility of AWS or Softlayer.

    Docker has many other impressive features, and few shortcomings, but in my opinion it’s a great framework to build your DevOps culture around as it makes maintaining a predictable production environment and a flexible development environment easy with one tool. Over my next few posts I’ll cover more specific examples of using Docker in development, test, and production environments. In the mean time visit the links below to read more about Docker and it’s growing ecosystem.

    Docker, Deis, Shipyard

     

  • Don’t Be Afraid of the “R” Word

    It seems that many people are afraid of the word refactor.  I think in many cases, if someone admits they need to refactor, it means they either didn’t do their job right.  They either missed requirements, wrote inefficient or buggy code, etc.  Alternatively, there is something fundamentally wrong with the product or system.  The fact of the matter is that sometimes there are big problems and a major refactor/rewrite is required.  I think that is rarely the case, but if embraced by all and done judicially, the outcome can be significantly better and the major refactor avoided.

    It doesn’t matter if it is a feature, deployment script or architectural decision, at some point, everyone must admit they are not clairvoyant and cannot predict every way end-users or systems will behave.  Therefore, there will be missed/new requirements and unforeseen problems.  As such, I very much embrace the philosophy of build for what you know now and adapt as you go.  Typically, that means the output is simpler, it got done faster and enhancements and bugs are easier to address.  That philosophy also means you are constantly in a state of refactoring, or said differently, incorporating new learnings, making individual methods more efficient, improving the readability of the code, filling in missing tests, etc.

    Now, just because we have embraced refactoring and we have noble intentions of, in general, leaving the code better than we find it, that doesn’t give us the license to go crazy and start rewriting everything we see. I think there are right and wrong times to refactor.

    First and foremost, don’t let those yucky bits of code distract you from your real purpose which is to deliver business value by building those new systems, features and fixing bugs.  So instead of just jumping right into refactoring, quickly note the yuck as a TODO in the code, story or in your notepad so you don’t forget about it, but also stay on task.  That said, bad code that is directly in your way is fair game and I even think that bad code near the edges of your task are also fair game.

    Secondly, follow good engineering practices.  Make changes in small, bite size chunks, test and deploy your changes often and communicate them with the team.  In the end, you will end up with a better product, happier end-users and engineers which should all translate into a more successful business.

    If you are one of those clairvoyants, then email me and let’s talk over a beer because I have lots of questions.  Otherwise, if you are a mere mortal like myself, then embrace the “R” word and seek out companies and teams that also embrace it.  At the end of the day or after a few releases, it is pretty impressive to reflect back on how adaptable the team and code can be while solving real problems very quickly.

  • Having DevOps In Your Job Title Is Doing You Harm

    matrixThis post was originally published on my personal blog in May 2013.  But I think this topic is important and wanted it to be my first contribution to staging-devopsy.kinsta.cloud

    I used to be Director of DevOps…twice. Both times, I changed my title later. I even ran a DevOps team…although the team was already called that when I took over.

    I have fallen victim to the the abuse of the DevOps title, and I see it all the time. In the DevOps twitter hashtag, people contacting about DevOps job opportunities…it’s got to be a real thing, right?

    I mean, we totally have people that are doing DevOps these days, right?  I know that I can find all kinds of job descriptions for DevOps Engineers from many types of companies.

    But, there is one very important reason why you might want to think twice before using “DevOps” as part of your job title.

    My friends (rightfully) give me grief for the fact that I self-assigned my title at Dyn to be the Director of DevOps (I still feel bad about it). I really hate job titles in general and this is what happens when you don’t care about what your title is.

    How the story goes is that when I received the offer from Dyn, my original title was Director of System Automation and Release Engineering. Since I was pretty sure that wasn’t going to fit on a business card, I took the job and asked that we change the title later. Fast forward a month, as after talking and trying to formulate my job description & title for about 20 min, I say simply enough, ”F this. Just call it DevOps, I’ve got other things to do.” I was far too busy to really care much about an arbitrary title.

    Fast forward a few weeks to Monitorama, and I was chatting with Mike Rembetsy. After talking for two minutes about how the new job was going, he said, “Oh, that reminds me, you need to change your job title. It’s terrible.” I tried to convince him for about three more minutes, mainly trying to express to him that it doesn’t matter. He then dropped the knowledge bomb that gave me the aha moment.

    When you are the Head of DevOps, you “own” DevOps. If DevOps fails, it is your failure, when it should be a failure of the entire company to change, adapt, and accept the cultural shift.

    That is the major problem with being the “head” of DevOps, or the “Director/Manager/VP/etc. of DevOps.” If DevOps fails inside your company, it’s a really easy way for the executives to say “Well, you are the head of DevOps so obviously it failed because you failed.”

    The reality is that DevOps is a culture that should be accepted by every facet of your organization. Everyone needs to embrace the changes of a DevOps culture, and there can not be a single owner of it.

    Soon after that conversation, I relaunched my team and title in general to be the “DevTools Team” (a brilliant Etsy recommendation). The more I thought about it, it accurately describes the work that we do. We build the tools needed for Developers (and Ops/QA/NetSec) to spend more of their time writing code, and less time dealing with distractions.

    I know some people want to have a DevOps team, or want to call their team the DevOps team. Jez Humble wrote a wonderful blog post last year that I highly recommend that you read. He summarizes that there are scenarios that a “DevOps” team label works, and I do agree with what he says. But don’t just label a team DevOps, assign a new Manager of DevOps and wonder why three or six months go by and things are still bad. Things are still terrible because all you did was change a team name, or give a brand new team a name, or in some cases , just created a new silo called the “DevOps Team Silo”.

    Well, what am I supposed to call myself?

    Here are some of the examples that came up during the Open Space:

    • Site Reliability Engineer
    • Automation Engineer
    • Release Engineer
    • DevTools Engineer

    I don’t know why recently there is a need to move away from the same job titles we’ve used in the past, as if there is some negative connotation towards them. It’s not that the jobs have really changed much; it’s that we’re just getting better and more efficient at doing them with the tools at our disposal. The tools have grown so quickly, it has enabled us to be much better at our jobs, but only if you are willing to learn something new.

    • System Administrator
    • Operations Engineer
    • Network Engineer

    In the end, titles really don’t matter very much, and they shouldn’t matter much. But they can actually have a more powerful (negative) impact internally by making you the owner of a culture that should be accepted by everyone. But there is one point that I believe is so important it needs to be repeated:

    If DevOps fails, it should be a failure of the entire company.

    Check out my talk “How to Keep the People You Need,” where I discuss recruitment and retention of your hard to fill positions. Also check out the panel on DevOps with Twitter’s Rich Paret from Dyn’s Geek Summer camp.

  • Improving Communication Through Autotmation

    Improving Communication Through Autotmation

    One of the great traits of an organization that adopts DevOps principles or even just an Agile methodology is that everything, and I mean everything, is more dynamic.  While each company will have a different level of fluidity in their processes, Clip Interactive’s source control and deployment processes are very fluid.  As such, we attempt to over communicate when it comes to these parts of the engineering process.

    The primary methods of communication the engineering team and rest of the company have adopted are chat/instant messaging and email.   There are several chat clients out there, all of which have their pros and cons and there is no one right answer except what works best for your team.  At Clip, we have chosen to use HipChat.  Other options include IRC, Flowdock, GTalk, Skype and the list goes on. HipChat was a good option for us because it is cross-platform, supports custom plugins and integrates with a wide variety of tools. 

    In HipChat, each time an engineer pushes code into source control, a build is started, finishes or deployment runs on a QA system a message gets posted to the “Engineering Feed”.  The messages for code pushes include the commit summary and link to the code diffs.  Messages regarding builds and deployments contain a link to their respective outputs.  More significant events, such as deployments to staging and production, get posted to the general chat room where more conversation happens and other non-engineers are more likely to be.  Those notifications are pulled “out of the noise” of the feed.  Additionally, these activities are typically chatted about more before they happened, so they fit more in the flow of conversation.  Last but not least, system notifications, including warnings, problems, recoveries and acknowledgements from Nagios, are sent into the “Operations” room.  To keep everyone informed and spread the knowledge, we post updates and resolutions to issues in the chat room.  All of these messages enable everyone to keep a pulse on the state of the system without being intrusive and should problems arise, to very quickly put together a chronology of events.  Aside from the automated items, there are additional rooms for in-depth feature discussion, production problems and sometimes just for fun.

    Another key form of communication is good old email.  Our build and monitoring systems sends the standard emails and text messages directly to the engineering team.  Each production deployment sends out the list of commit messages, naturally with links to the code diffs.  Since our deployment process is very fluid, this extra notification has gone a long way toward keeping everyone informed on the features being deployed into production.

    Going forward, we intend to add more automated communication around the product management process and system performance metrics.  These automated communications, when added to our normal cadence of meetings, have had a significant impact on getting us closer to the goal of improving transparency in and out of the engineering team, while not making people go out of their way gain value from the tools.

  • Changing Organizational Culture – A Sweaty Use Case

    Changing Organizational Culture – A Sweaty Use Case

    When we think about changing organizational culture our immediate reaction is usually – yeah right, go fight city hall – and we take the Homer Simpson route, and don’t even bother trying.

    Over the course of the last year, I managed to achieve what felt like an impossible change within my organization, and I took some valuable lessons from the experience.

    It dawned on me afterwards that there were a lot of parallels in this process that we also underwent when trying to instill a DevOps culture in our organization, and I thought I’d share what I learned.

    The Epiphany

     I love riding my bikes – mostly mountain biking, and I usually am only able to do this over the weekend.  It’s something I really love to do, but in addition to my full time day job – Head of Product at my company, I’m a full time dad – a proud father of three.  All of this doesn’t leave me a lot of time to do the things I love.

    On a good day I spend more than an hour stuck in traffic, and on bad days more than an hour and a half.  This is basically dead wasted time.

    So I had an epiphany one day. Why shouldn’t I take advantage of this time to my favorite thing?  I could easily bike to work – and use this wasted time to my benefit, doing the thing I love most.  The problem lies in the way I look after an hour bike ride.  Pretty sweaty.  Here’s a pic.

    And, I know what you’re thinking – the answer is – no it wasn’t raining that day, in case you were wondering.

    The challenge though, is that I work with people, so I needed to find a way to clean up after my rides.  But, alas alack, we don’t have a shower in the office.  The plot thickens.

    So you might be wondering at this point, how this all relates to DevOps.

    DevOps is about pushing change in organizations – and not necessarily just technical ones. There are cultural organizational practices that often times need to be changed before you can even start to think about technical changes.

    So I believe this story represents a real parallel case study on the organizational changes often needed, to create a DevOps culture within an organization.  These processes many times involve a change in hardware – in this case, the shower itself.  As well as require a change in culture, actually motivating the management to realize the importance of it, and have people start choosing to do athletics before work (and preferably also ride to work).

    The Method

    And then I ran into Jesse’s Robbins’ rules. For those who don’t know, Jesse is one of the founding fathers of DevOps and web operations, and happens to also be the founder of Chef (OpsCode at the time). In one of the recent CloudExpo conferences in Europe, Jesse gave an awesome keynote about how to hack organizational culture to drive change. So working on the premise of “Jesse’s Rules” – I got to work:

    • #1 – Start small – Build trust

    Until the new shower was built, I’d shower daily with a HelioPressure shower  (you can find it on Amazon for less than $100), and I’d shower every day on Nati Shalom’s [LINK TO DEVOPS BLOG] balcony in the office.  Demonstrating I was serious and dedicated to my biking to work. Yes, it definitely raised a few eyebrows, but then again, better to shower on a balcony than walk around smelling like a sweaty rug all day…

    • #2 – Create Champions

    Doing crazy stuff yourself is the easy part. Finding like-minded individuals that will do the same, well that’s a different story. But after working hard at convincing others, I managed to find at least two people that would occasionally use my improvised spa, and this really helped me drive change in my company.

    • #3 – Use Metrics to Build Confidence

    For me there were actually a few metrics that contributed to the driver for change.  The first was that, I right off the bat, lost quite a few pounds – which was an incentive for a number of people to jump on the bandwagon – or rather bike – in this context.  The next metric was hard research that shows that people that are in shape tend to be absent from work 80% less often than employees that are not in shape. That helped me in convincing my CEO that it’s actually an investment and not just thoughtless spending…

    • #4 – Celebrate Success

    After committing to our makeshift shower for months, budget was officially allocated, and a shower was built for all the employees looking to participate in sport activity on their way to work.  Needless to say that I’m not the only one riding my bikes to work today 🙂

    Lessons Learned

    So the lessons I learned throughout the process, and my key take aways as I see them are:

    •  #1 You’ll run into a lot of skeptics along the way. IGNORE THEM.

    Many people will attempt to discourage you from your goals along the way – don’t listen to them. If I had a dollar for the amount of times I heard “it’s never gonna happen”, I wouldn’t need company budget to build a shower.  But I stuck with the plan, and it ultimately paid off.

    •  #2 Change takes time

    Rome wasn’t built in a day.  It’s a cliche – but cliches are born from common experience.  We’d been discussing building a shower for 2-3 years at the least.  It’s true that until I didn’t demonstrate that I was truly dedicated, rule #1, it wasn’t taken seriously. It took time for the company to come around, and digest the idea, until it gradually happened.   So again, if you stick to your guns, and have faith – a change ‘gone come.

    •  #3 Take it one small step at a time

    Basically start small, and be creative. Until you can achieve the big results, celebrate the small achievements too.  Adding champions to your cause, increasing awareness, possibly a meeting to discuss the changes you propose.

    •  #4 Be prepared to improvise

    I owe a lot to my colleague that came up with the pressure shower idea.  I think until I actually demonstrated that there was a really dire need for such a change, it wasn’t really considered seriously.

    •  #5 Find a management sponsor

    Luckily our CEO is athletic in orientation, and is an avid bike rider as well, and he was committed to help making this happen.  Many times it won’t necessarily be the CEO, but find a management champion that believes in your cause, and is willing to help you bring about the change.

     Bottom Line

    No matter what change you’re trying to push, these lessons and rules have proven to hold true for me in any circumstances, and I often times use this method to push significant changes in work and in life.

    Oh, and here’s a pic of the new shower.

    If you’re ever in Israel, and need a place to work, and want to bike there…you’re welcome.