Category: Blogs

  • Around the DevOps Blogs and News Scene

    Around the DevOps Blogs and News Scene

    It was another busy week in the DevOps media world; not just here on staging-devopsy.kinsta.cloud either. We had almost 30 new original features and blogs published this week. You should take some time and read them before running off and reading any of the articles and blogs I mention here. But then again I am biased, so take it from where it comes.

    Besides the staging-devopsy.kinsta.cloud articles, I wanted to call out to two different articles over on Information Week. One is a guest post from Rackspace CTO, John Engates. It is called DevOps: ITs only chance of keeping up. In it the Engates calls out that breaking down barriers between development and operations, as well as other barriers in IT is the only hope for IT departments to keep up with the pace of business.  I spoke about this with one of my friends and he thought of it as a chicken and egg question.  Does DevOps make business go faster or does business going faster make us do DevOps?  Of course the answer is probably “C”, all of the above.  Anyway it is a great article by John and after you read enough here on this site, you should head over and take a read.

    The other article on Information Week is by my Facebook friend Lorna Garey.  Lorna was a participant in a recent DevOps State of the Union event I attended in Boston some weeks back.  We both heard firsthand from some companies where DevOps allows IT to perform at a high level. Lorna writes that DevOps is creating a digital divide. It is creating a class of haves and have nots. Much like those with fiber internet and those with sub-5mbps DSL.  This is going to create inherent advantages to those who have DevOps over those who do not.  For me there is an easy answer, everyone should do DevOps if they want to.

    everyone-has-a-voice-that-deserves-to-be-heard_designAnother topic that has been active here on staging-devopsy.kinsta.cloud and on other DevOps discussions is whether or not DevOps is a title and real job. I don’t care what you call it, there are certainly people doing it for a living, but that is just IMHO.  According to this article in ITproportal, the number of DevOps engineer jobs in the UK has tripled over the last two years. In another report based on the same Rackspace data, Computer Weekly declares that “It is official, DevOps is a real job“. So I guess that ends that discussion, said no one ever.

    Finally I have been having a good time monitoring the Twitterverse as the @devopsdotcom stream grows over these past weeks. I would say overall the reaction has been pretty positive. The DevOps community has embraced what we are trying to do with open arms and have been more than eager to help out and even be involved. But you can’t please everyone. One person in particular  thought we have come “late to the conversation” with “little credibility  beyond buying staging-devopsy.kinsta.cloud”.  Hey there will always be some of that. But for the record we didn’t buy the domain and as I said in my first post three weeks ago, we aim to post a big tent here at staging-devopsy.kinsta.cloud.  There is plenty of room for old DevOps folks and even more room for all of the new DevOps folks to come. We all don’t have to agree, but this site is here to give everyone a platform to add their voice to the discussion.  I don’t think you need a union card or have to pass any literacy test to particiapte.

    Have a good weekend!

  • DevOps Adoption – Startups

    DevOps Adoption – Startups

    This is part 1 of a 2 part article

    DevOps is somewhat unique in that it has applications to both large enterprises and to startups.  The way DevOps is adopted in startups is incredibly different than how it happens in large organizations. The profile of how DevOps is adopted in each of these cases is the focus for this two part article series.

    In this first installment, we will study how DevOps is adopted within startups and in the second, how enterprises adopt DevOps. There are lessons for each in the way that DevOps is adopted. Organizations can leverage this information to enable better, faster, and easier adoption of DevOps.

    Much of the DevOps way is rooted in startup culture. From methodologies such as Agile, lean principles, and just in time manufacturing, startups adopted DevOps as a means to compete, innovate, and move faster. DevOps is inherently a quicker, better way to get product to market – a perfect fit for startups. They could adjust quickly, iterate again and again to get to the point where customers en masse were interested in their technology.

    In order to understand how DevOps is adopted in startups, we need to look at how startups evolve. In the high-tech world, a startup gets going when a founder or group of founders has an epiphany for a new product or service. Generally one of the founders will start to develop the solution by writing some code. The startup may add a couple of developers but it is usually just a few folks cranking out the first version of the product. There is often a business or product focused function where requirements and feedback are gathered from potential users or customers.

    In this first phase, the development team often adopts Agile as a way to quickly iterate on the development process. These short-term sprints build the initial functionality of the product. While initially, there isn’t really much to release to potential customers, the core developers are starting to build their development and release cadence. The short sprint help to keep the functionality requirements small and the team on-track to deliver a small piece of functionality in a short amount of time. This foundation also ends up being an important part of the future development process. And, just as importantly becomes the kernel of DevOps.

    As a small team building the initial releases of the product, the culture and environment of the startup is generally fast, highly communicative with little to no process. Often the development team is sitting side-by-side cranking out code quickly. No specs, just a list of what needs to be done often stored in Pivotal Tracker or some similar tool. Sometimes mockups and screens designed to describe the functionality. There isn’t a rigorous process in the choice of technology platforms – language, database, etc. are chosen often based on what the core team knows or thinks is the right solution rather than on research. This small, agile environment is critical to understand when thinking about how DevOps is adopted.

    There is little formal process at most startups. Things are evolving and moving so quickly that there just isn’t time. The goal, after all, is to get a product into market and start selling. While startups are high adopters of Agile as a development methodology, there is generally little rigor in its implementation. There doesn’t need to be because the team is so small and trying to move quickly. Adding in product management, sales, and marketing functions are often ad hoc. There isn’t a well-developed end-to-end product release process. The movement from development to operations is also less than well defined. Often, the developers building the product are the operations people deploying and managing the product or service. Process gives way to the practicality of just getting things done.

    The first phases of DevOps adoption start to occur at this early stage in a company’s life. Developers as they are building their product often encounter some basic challenges and begin to solve these issues. How they solve them often dictates whether they take a DevOps path or not. One of the first things that developers do is build an environment to test their software. Often, they will find a cloud provider and start spinning up a few instances to host their application. They also may need a database server to round out their testing. They might manually do this a few times, but as they test more and potentially on different platforms they will start to look for some tools to automate this process. Of course, at this stage, there are likely zero operations people, so the tasks fall on the developers. And, developers wanting to spend more time coding pick-up some tools to do this work.

    The tools might include infrastructure providers like AWS, Rackspace, or SoftLayer to configuration automation tools such as Chef or Puppet. There may be a variety of other toolsets chosen as well including project management tracking, libraries, language frameworks, and others. The goal with this first wave of tools is just to save time. That could be by not having to code basic functions, the setup of test environments, or manually tracking tasks. As the organization launches and brings their product or service to market, they may adopt some management and monitoring tools to help them gain visibility into their product or service. Generally at this early stage the decisions are focused on driving short-term problem solving. A developer needs a test environment quickly and repeatedly. A solution to that need is quickly found and implemented. The underlying paradigm is to move fast, solve problems, and get to market.

    While tools are hardly a complete DevOps methodology, they are the catalyst for the next step in a startup’s journey to adopt DevOps. Many of the tools chosen in this first period have APIs, can create automation, and are highly configurable. As a startup keeps moving forward and increasing their users, the flexibility of these tools and the various ways that the developers utilize them starts to drive some conversation around process. A few hiccups in the production environment with some bugs, outages, or failures, and the technical leaders begin to start to say the ‘p’ word. Process. While there may be some Agile components already embedded, the growth of the startup starts to create a desire to add in some control, accountability, and guard rails around the product development process. Many organizations will also start to loop in the business side so that there is a more comprehensive process around releasing a product to market. Agile now starts to extend up and down the organization in a more rigorous way. DevOps is starting to take hold and become more fully formed.

    The early stage DevOps lifecycle wraps up with it becoming a part of the culture. The organization now shifts into a mode where this methodology becomes a fundamental underpinning of the business. Values such as agility, intense customer focus, rapid iteration and others are a part of the fabric of the company and encompass the entire organization. Hiring decisions are made based on whether the candidates match the philosophy and culture of the company. DevOps is viewed as more than just a process that the company follows, but a competitive advantage in fulfilling its mission to its stakeholders. At this point, DevOps is firmly embedded into the organization and becomes its way of life.

    Startups are unique in their approach to adoption of technology and methodologies because of their size, focus, and evolution. DevOps adoption, as we have seen out in the wild, seems to go through some natural stages. While the stages may overlap, it generally follows a tools, process, and then culture sequence. It could be said that some startups just start with the culture of DevOps, and that can be true for some, but many that we see are often so small at that stage that the culture is largely undefined. IT leaders who understand this sequence can become more intentional about the way that DevOps is adopted, perhaps accelerating it or even shaping it earlier in the company’s lifecycle. A little bit from now, our hypothesis is that startups won’t even start without DevOps embedded into their plan.

    In part 2 of this series, we will examine the adoption of DevOps at larger companies.

  • DevOps Lessons Learned – The “DevOps Engineer”

    DevOps Lessons Learned – The “DevOps Engineer”

    Back in 2008 I was part of a startup that had success and sold to large company three years later. During the startup years, there was three of us in IT plus an offshore team that did our web development. The three core resources owned everything from the API down while the offshore team owned the front end. Since we were small, we each owned our own part of the architecture and managed the deployments and operations of the system with few issues. Once we were purchased, our world changed drastically.

    Our team quickly ramped up to over 20 people while my original two architects soon left for greener pastures. Now we had a lot more people involved in the process and none were domain experts. Things quickly started spiraling out of control as server sprawl started creeping in and deployments were becoming increasing challenging and error prone. We hired our first “DevOps” person to help us reign all of this back in. Then things got really ugly. For those who have followed my DevOps writings and my rant called “No you are not a DevOps Engineer“, what I am about to describe is the basis for my position on DevOps.

    DevOps as a Role: Fail Pattern

    We never intended to create a DevOps silo or position. We just wanted someone with expertise with cloud and automation since everyone else on the team was a developer and many of them were new to both the cloud and the platform. Unintentionally, this person became labelled as DevOps and he was soon joined with three other people which became an official DevOps team. We now were the proud owner of a new silo which became the new bottleneck.

    The DevOps team had their hands full to regain control over the environment. Permissions had to be pulled back to prevent server sprawl from continuing. Development teams now had to request environments from the DevOps team. When we were a startup, this was easy and manageable. As the team got larger and more concurrent projects were being run, this became challenging. The DevOps guys were so busy keeping up with the demand that they were not always in tune with the goals and the deliverables of the different projects. The app dev teams were so focused on “agile” that they were not including the DevOps guys early enough in the projects.

    The DevOps team became drastically overworked. They were monitoring and operating the systems, they were being pulled from their deliverables to provision environments, and they were trying to put controls around a platform that experienced sudden growth from a 3 person team of founders to a 20+ person team of new people. To make matters worse, we knew what we wanted to do to fix these problems but there were not enough hours in the day to get there. We had mapped out an entire continuous integration and delivery strategy, but we just could not get enough time to focus on it.

    DevOps started to get a bad rap from the app dev team because there was often delays in getting environments provisioned in time. We experienced environmental issues between dev, QA, and stage because of the lack of timely communication of the environment requirements from the app dev team and because the different app dev teams were not being consistent (e.g. different version of Python and different libraries).

    All of this led to frustration from the business, especially from the business people who came from the startup and was used to frequent deployments with high quality.

    Lesson Learned

    Eventually the team turned this around. Although they still have an official DevOps team, the team works much closer with the app dev teams. A high level of standardization is now in place for the environments. The teams share operational requirements early in the project life-cycle and developers assist in the environment creation. The lesson learned here is that creating a DevOps role or organization is not the correct approach. Creating a new silo just creates new bottlenecks. DevOps is not a role, it is a way of getting quality, reliable software to market faster. The sooner an organization realizes this the sooner they can get to a successful state.

  • From Simple Automation to DevOps Like a Boss

    From Simple Automation to DevOps Like a Boss

    Automation as We Know It Today

    If you haven’t been hiding in a cave in the last year or two, you’ve probably heard the terms DevOps and infrastructure automation more than once… But even today, infrastructure automation is mostly focused on setup and deployment of complex systems. For example, if you’d like to deploy your application to the cloud, you would likely automate the steps of provisioning the cloud resources, installing the right components on top of these, and then orchestrating the startup of your components – or better known as – cloud orchestration. Take even the simplest application that has a web server and database. After installing and configuring everything, you’d first need to ensure that the database is started, and only then the web server. You’d also need to propagate specific runtime information from the database to the web server, such as the database’s host and port. This stage, for the most part, is where most automation processes focus on today.

    DevOps: Stage Two

    Automation, however, can do much more for you. Let’s imagine, following your initial automation of the setup and deployment of our your application, that you want to make changes to your infrastructure or application. For example if you have a patch you’d like to install to the database or operating system, or if you want to update a piece of your code on your cluster of web servers. Both of these concerns involve pretty complex processes that you need to automate. For an infrastructure upgrade, you would typically roll out the upgrade server by server, take down one server at a time, update it, then restart it. Even though it doesn’t happen very often, there’s still a lot of benefit in automating this process and minimizing the chances of human errors. Pushing new code is a bit different. For starters, you do it much more often, even a few times a day. Also, (no offence), it is likely that the stability and quality of your code is lower than that of an approved patch to the OS or database. So the risk of something going wrong if you don’t automate is even higher with this process. That’s why there are common automation strategies that help you deal with this risk. Two such strategies are the Canary Instance strategy and the blue/green strategy. The former derives its name from the mining days, when a canary bird was sent out into the mine before the miners, to check if the air was toxic or not. So in this fashion, a “canary instance” is chosen, the new code is pushed to that instance alone, and then a number of sanity tests are performed. If these pass, the code is then pushed to a few more instances. If all goes well again, the code is then sent to the rest of the infrastructure. In the latter strategy (A/B), the code is sent to only a part of the servers. In an even more sophisticated approach, the code is only enabled to a portion of the users, for example, a certain geography or time of day. All of this ties into how to setup and manage an ongoing system. But although this process, as well the infrastructure upgrade, can get quite complex, especially when dealing with large scale, live systems, in most organization code pushes and upgrades are still not automated. But there are more than a few companies that are blazing the trail for the rest of us.

    The Plot Thickens – Monitoring Like a Boss

    So once you finally saw the light, put in your sweat and tears and managed to automate your setup, upgrades and code pushes. What’s next? This where it really becomes interesting, and where you can really take your business to the next level, with the monitoring of the entire system, and eventually, what you do with the aggregated data.

    More to come…

    In my next post I’ll discuss the different levels of monitoring and logging to get the to promised land of proactive DevOps.  And then…take that even further.  So stay tuned.

  • How to laugh at DevOps

    How to laugh at DevOps

    Talking about DevOps is all the rage these days. But with that conversation, it’s easy to become inundated with the amount of serious discussion around the topic. I do agree that DevOps is not something we should joke around about, but since humor is an inherent part of our culture at VictorOps, it makes sense that we would be drawn to the humor of the situation.

    Below are the ten of the funniest Twitter accounts that make light of the DevOps movement. Some are obviously one-joke wonders while others continue to make us laugh. Either way, the list below is proof that you can’t take DevOps too seriously:

    DevOpsBane

    DevOpsYakov

    • @UncleDevOps – Everyone’s favorite continuously-deploying uncle. Also a little crazy.

    UncleDevOps

    • @DevOps_Jesus – Continuously delivering from evil since 4 BC.

    DevOpsJesus

    • @DevOpsGrouch – I complain. I swear. I automate. Deal with it.

    DevOpsGrouch

    DevOpsTourettes

    • @DevOpsBorat – Cultural Learnings of DevOps for Make Benefit Glorious Teams of Devs and Ops.

    DevOpsBorat

    • @DevOpsX – Always fighting the good fight for DevOps equality.

    DevOpsX

    DothrakiDevOps

    DevTylerDurden

     

    Feel free to add any that I might have missed to the comments below. And remember to laugh…it’s just DevOps after all.

  • Trust & the trusted image

    Trust & the trusted image

    Graphic No BGWhat is Your Trust Model?

    Information security conversations often start with the question “What is your threat model?” This blog asks: “What is your trust model?”.  Trust is a complex subject and an integral part of managing DevOps-oriented organizations and highly automated IT infrastructures. Who (developers, ops) or what (code, process) is trusted to accomplish specific tasks in the infrastructure can sometimes be difficult to characterize to management, auditors, or in operational/incident reviews. The goal is to describe relationships among people and systems at work in the DevOps world and discuss approaches to making systems more transparent (to coders and non-coders!).  

    The Trusted Image

    Organizations like Amazon, Canonical, and Red Hat provide virtual machine images of popular operating system distributions. It’s common to launch these images directly “as-is” and install software onto them after launch.

    However, there are advantages to specializing one (or several) of these images and using it instead of the base OS image. A base image which has been supplemented with configuration and installed packages that suit the needs of your application is referred to as a “Trusted Image”.  And there is definitely value to putting your trust in a purpose-built and well-characterized image.   

    Why Make a Trusted Image?

    The reasons are several:

    Reliability: The more downloads that a VM requires in order to provision and configure, the more likely that something will go wrong. Network errors, outages at package repositories, and random download errors can render your VM unusable or unstable.  Building base packages into a trusted image minimizes problems.

    Secure Configuration: If there are configuration options that you want to ensure on every VM, you can build these settings into your trusted image. For example, you can configure firewall (iptables) in a trusted image to protect every VM that you launch.

    Secure Communication: If you want to use SSL or TLS inside your application, you can establish trust in a root certificate via a trusted image.

    Built-in Service “Wiring”: Do you want to ensure that the syslog of every VM you launch is sent to your logging server? Or to ensure that a specific version of Python, Ruby, Java, Node.js, etc is pre-installed on every VM that you launch? How about client libraries? A configuration management agent?  Build these into your trusted image.

    Speed: No more waiting for package installations; build them into your trusted image, and launch the end product faster.

    VM Identity and Authorization: You can build trusted images that will create an identity for each VM, and help the VM to get permissions to operate accordingly in your infrastructure. Bootstrap credentials can be used to access resources that the VM needs to join the application, such as database credentials, system passwords, API tokens, and other sensitive information.

    The Counterpoint..

    The use of trusted images can have drawbacks, for example:

    Security Patching:  Security updates from vendors are frequent. Any new image that you build will have all the latest security patches built into it; but your images will quickly get out of date as patches are released by your OS vendor.  One approach is to configure VMs to patch themselves on a schedule, for example, nightly during a “quiet period”. [ Note: Patching is tough. Please share your experiences with this problem! ]

    Management: Once you have built a trusted image, you will need a way to manage them. Image tags and other metadata features provided by your cloud can be helpful here. Treat your images like any other product of your build system: version them strictly and keep track of the provenance (history). Lastly, be sure that the system you use to build trusted images is itself trustworthy!

    Weighing the pros and cons

    Overall, for many organizations the security, performance and reliability benefits of the trusted images outweigh the drawbacks.  Would love to hear more experiences from those of you who use trusted images and who’ve considered trusted images but taken another path.

  • Protect and Defend: Repositories

    Protect and Defend: Repositories

    When you’re creating work that gets deployed into production, there is typically a repository involved. It could be a code base, a deployment hub, a definitive software library, a library of VM or application templates, etc.

    Often, you may not think twice about what’s in these repositories or whether you can trust their content – after all, you and your teams built these repositories, right? That may be true, but what if your trust is misplaced?

    The recent Target breach illustrated (among other things) that you may be incorrect in assuming the “inbox” of your deployment process is trustworthy. In the Target example, there was a queue of patches ready for deployment but someone added additional, malicious code to that queue without being noticed. The automated deployment process grabbed both the authorized and malicious code and distributed it to every point of sale device in the network. Not a good outcome.

    Their workflow included a trustworthy process to test code that before it was checked into the deployment queue, but there was also a flaw in the process: there were no controls in place to detect when anyone went around the process and made arbitrary changes to the repository. In other words, even if you put trusted packages into the queue, the packages could be modified and additional code could be added without those alterations being noticed.

    What we can learn from this kind of situation is the value of the tried & true principle of “trust but verify:”  don’t assume you can trust your repositories without validating what’s in there.  With that notion in mind, think about your environment – are there repositories that people could alter without being noticed? If so, that could present a problem both for security and reliability in your organization.

    To correct this, we need to treat our repositories like bank accounts – account for everything that goes in or out, and monitor the contents of the repository continuously to look for any undocumented or unauthorized “transactions” (adds, removes, or changes).  We must also add validation checks to the deployment process to make sure everything is accounted for before the deploy occurs.  Yes, this will likely mean you need to do some extra work on your deployment workflow, add technology for instrumentation, etc. but it will more than pay for itself through better security, accountability, and availability.

    By the way – this same principle applies to the scripts and automation tools you rely on, as well.  If someone were to alter these tools without your knowledge, you may get results you don’t like.  You should also consider that your automated deployment repositories aren’t the only vector for untrusted code to enter production – continuous monitoring of the production environment is also a crucial activity.

    Protect and defend your repositories – it’s important.

  • staging-devopsy.kinsta.cloud Podcast

    staging-devopsy.kinsta.cloud Podcast

    Mitchell Ashley and I have been doing podcasts together since about 2007 or so. Back then the subject matter was security. But now with the launch of staging-devopsy.kinsta.cloud we are launching a new series of podcasts on DevOps. In doing so we are joining a small field of quality podcasts on DevOps. Podcasts like DevOps Cafe and Arrested DevOps have set a high bar of quality. Eventually Mitchell and I hope to join them.  Our podcasts will be shorter than some of the others though. They will be supplemented with Mitchell or my notes from the show.

    So here is our first episode. We really just introduce Mitchell and I to the audience and talk a little about some of the stories on staging-devopsy.kinsta.cloud over the last week.  In the future we will have guests and dive a little deeper into the world of DevOps.

    [soundcloud url=”https://api.soundcloud.com/tracks/142516025″ params=”color=ff5500&auto_play=false&hide_related=false&show_artwork=true” width=”100%” height=”166″ iframe=”true” /]

     

  • Operational Overhead – or why hubris is bad.

    Operational Overhead – or why hubris is bad.

    Stop me if you’ve heard this one: A guy walks into a meeting and says “We need a system that does something”, and the engineer in the room replies “Oh, we can totally build that”. Most of the punchlines to this joke go “we can totally build it, and now we have to own its lifecycle and we have sacrificed valuable time and resources for something we should never have built.” ba-dum-ching

    I am guilty of playing my part in these conversations, as I’m sure many of you are. About two years ago, as our budding startup was really starting to take off, I was in one of those meetings. Some of the developers wanted to try a recommendation engine, and were going to use Mongo as part of it. Frankly, I thought they wanted to screw around with Mongo and needed an excuse as much as I thought they were going to really do something. At the time I was drowning – running our cloud systems, our LAN, facilities, purchasing, and I had just been tasked with bringing some sanity to the tech side of our video post-production process. I. Did. Not. Have. Time. For. A. Mongo. Setup.

    See, here’s the thing. Most of the time the system administrator in that conversation, trying to be a good devops partner, replies “oh sure I can stand up mongo”. But let’s be honest, it’s not just standing up Mongo. It’s handling upgrades, security, failover/HA, backups and restores. If this recommendation engine actually worked it was quickly going to become a key part of our infrastructure. I had just enough presence of mind to know that while I probably could find an hour to stand up mongo, I didn’t have the several other hours needed to do it well. And really, I didn’t have an hour. So I quipped a now-famous-and-often-used-to-troll-me response “Look guys, we need to reduce our operational overhead. Let’s find an outsource.”

    Ten minutes and a credit card later, we were setup with MongoLab. Two years later? The recommendation engine is awesome, and I have not cared one ounce about mongo. No upgrades. No middle-of-the-night pages. No architecture discussions. It just works. I’d like to spend a few words dealing with the standard objections:

     

    • OK we can try outsourcing, and if it works we’ll bring it in house. Right? Because you never run key pieces of the infrastructure out of your control. Right? Because, if you do… then… that’s bad. Right! Ye Olde World IT Wisdom is pretty clear on this point, you can only ever be sure that something is good if you’re running it in-house. In my experience however, you’re probably more likely to fumble a backup, fail an upgrade, or bungle a failover than an outfit whose sole job is to do those things.
    • But! it’s expensive! Right? Just about any *aaS provider has an entry level pricing structure that lets you screw around for less than the cost of that team lunch you’re buying on Friday. If you’re in production, and you are making money, or even better yet more money because of this infrastructure choice… justifying the cost is stupidly easy. As long as the cost of the service is less than the money you are bringing in, good job. If you aren’t actually making money, then what exactly are you doing?
    • We can’t trust Service Provider X. Guys, these companies are in business to provide 1 thing for you. They literally have 1 job. You probably have 14 jobs. Are you asking for a 15th? A 15th thing that you have to watch, and care for, and feed, and love, and hate? A thing that is not core to your actual business? Shouldn’t you make that 15th job something directly beneficial to your team/code/organization? Isn’t that what you’re here for?
    • OH GOD NO NOT VENDOR LOCK-IN. Hey genius, you know what? You have to write code, and whether you are writing code talking to a database in your environment, or a queuing system in someone else’s… it’s just code. I hope to spend most of the rest of my life writing about how utterly stupid this argument is. Yes! If you fire a vendor you have to refactor. You know what? If you upgrade your in-house stuff you probably have to refactor. If you change the in-house system you have to refactor. Anyone prescient enough to foresee all possible outcomes, and who can make perfect, refactor-free decisions can talk to me about vendor lock-in. The rest of you should move past this objection and get back to doing something productive.

    You know what my favorite operational overhead story is? Amazon SQS (Simple Queue Service). Different day, different developers, same idea. Wow, our little website is now processing enough orders that we need asynchronous fulfillment… I know! Let’s setup a queuing system! I used ActiveMQ once! Or RabbitMQ! Or… we can use the same queuing system that runs the largest ecommerce site on the planet. Where there is literally a horde of highly trained monkeys watching every teensy little thing that makes it work 24x7x365. Someone else has already thought of every imaginable permutation of scale/resiliency/security. And you know what? It costs 0.0000005 per request. I can process 14 million orders at a net cost of $7. Yeah, let’s totally spend time standing up something better and cheaper.

    Good engineers are good because they know they can build things. Even when they don’t know they can, they are pretty sure they can figure it out. DevOps should be about making smart decisions, and hubris is not smart. Use the best tool at hand. Rebuilding every wheel, administering every platform, doing everything yourself… is just adding to your operational overhead, and subtracting from your ability to be impactful to your business.

  • My Journey into DevOps

    My Journey into DevOps

    Like many developers, I started my career in a waterfall-based development environment. I was at a very large telecom, working on a product that had been around literally for decades. We’d spend weeks or months in the requirements phase before any code was written. We’d throw the code over the wall after our development period, and QA would file defects that would take weeks to prioritize and triage. It took about a year to ship a new version of this product. All progress came at a crawl, and I felt so disconnected from the customers that I would forget they even existed — the work I did was to get the project manager off my back, not to please users.

    Moving to a startup that practiced Agile development was a relief. Far less mature product, smart people contributing at every level, rapid development — and a rapid feedback loop, too. It felt so natural and right that I forgot there was ever another way of building software. We talked to real customers and heard about how they wanted the product to work. Our priorities were based on actual need, with estimates based in reality and with a focus on iterative improvements to functionality. My teams designed and implemented features and had it tested and accepted quickly… software development was thrilling again.

    However, it wasn’t perfect. While we could rapidly build features, and get those features right, we still had the problem of getting those features to customers rapidly. We’d design and implement and test great functionality in a matter of days, but that great stuff would sit on the shelf for weeks or even months before anyone would actually use it.

    The problem was this: we feared disrupting a working production environment. Every time we’d push to working systems, we’d bring in the operations team and the senior developers on a weekend, sit tensely while the new code was fired up, and then spend hours fixing things that had somehow crept in. Sometimes interactions between different new features hadn’t been properly vetted. Months of changes had accumulated so the changes were huge, and deployment was a nightmare. And because it was a nightmare, we didn’t want to do it often… so releases got bigger and bigger over time. It’s a vicious circle.

    I’m sure you know where this is headed — the answer to this problem is continuous deployment. Over time, we reduced our multi-month release cycles to weekly and finally daily, releases. The things that were most painful, we do more often — addressing the problems and automating the process. By releasing more often and in smaller chunks, we were able to clearly know what had changed if a problem occurred, and with that isolation of code changes, we started to fix problems much faster. Classic continuous deployment story.

    It wasn’t easy, of course. We struggled, for instance, with distributing the necessary keys to remote systems automatically — otherwise each deployment (and individual machine affected by that deployment) required hands-on interaction. We also wrestled with getting the ability to roll-back to prepare for the inevitable “bad release”. We tried redeploying the code from a tagged branch in the repository, but it was just too slow and led to several minutes of downtime. Ultimately we ended up duplicating almost the entire infrastructure in order to build a “Blue-Green” environment, letting us swap back to the last known good state as easily as re-pointing our load-balancer.

    In making that transition — in bringing that deployment pain forward – our development team needed to do a lot of tasks that were traditionally operational. We set up servers that mirrored our production environment. We learned that they needed to be reproducible, so we worked with the operations experts to make servers that could be spun up from scripts and effortlessly deployed to any environment. We felt the pain of manual installs and updates of our product, so we simplified it, hardened it, automated it – we did everything we could to make that part of our life simple. We learned firsthand the ugliness of needing to manually make changes to point to the correct database and roll the logs via cron jobs, so we made these things configurable. We uncovered the engineering assumptions and design decisions that made our product difficult to maintain and manage in production. We organically evolved into a DevOps organization. We learned to manage the servers and take on operational tasks, and we improved our process in order to make those tasks simpler.

    In retrospect, it was natural – just like the transition to Agile from waterfall. To me, this is the way software should be created. Agile allows us to quickly discover and develop the products that our customers really want. DevOps extends that concept, and teaches us how to take that product and get it into the hands of our customers more quickly, and how to maintain and support that product most effectively.

    I’m thrilled to be working in DevOps and can’t imagine ever going back.