Category: Doin’ DevOps

  • The Dark Side of Infrastructure as Code

    The Dark Side of Infrastructure as Code

    We spend a lot of time extolling the virtues of infrastructure as code. All of it is true. Treating infrastructure as code can add great value in terms of its ability to promote consistent, predictable and repeatable results during the application deployment process. One of the ways in which infrastructure is treated as code is through the use of programmability.

    Programmability is an integral component in software defined architectures and a part of the DevOps approach is automation through integration with traditionally development lifecycle focused tools and technologies. APIs, app templates and the ability to programmatically interact with the data path are enabling rapid provisioning of services in the network. That’s important to the business. Improving service velocity in the network is critical to achieving faster time to market and enabling the operational scale necessary to meet growing pressure on networks arising from the growth of applications driven by cloud, mobile and the Internet of Things.

    Speeds and feeds of processes are now as important to gain a competitive advantage as speeds and feeds of packets.

    Those who see DevOps and SDN as having a strategic impact on their organization view programmability as having even greater importance.

    But that does not mean that treating infrastructure as code doesn’t have its dark side. Because adopting a programmatic approach to anything means you get both the benefits and the risks.

    I happened to be cogitating on this topic when I stumbled across a post reminding us of the importance of code reviews. The content was so-so, but the premise was spot on: code reviews are a must for those embracing the notion of treating their infrastructure as code.

    Why is that (or should it be) an imperative?

    Consider some stats from InitialState, “a technology startup helping engineers capture and understand product data. Our cloud-based log visualization-as-a-service platform helps engineers make sense of the exploding number of logs being produced by “smart”, complex products and applications.”

    According to their data, developers have to find and fix 100 bugs per 1000 lines of code. And it takes 30 times longer to fix one of those bugs than to write one line of code. Overall, developers end up spending 75% of their time (about 1500 hours per year) debugging code.

    Now spend a moment thinking about what that means when you start treating infrastructure as code, writing scripts to automate provisioning and integrating with other systems to orchestrate deployment processes.

    2011.09.18_code_reviews

    I won’t lie, code reviews can be tedious. But the first time someone points out an error, particularly one that could have resulted in a security breach or an undesirable service outage, the pain is more than made up for. In the long run, code reviews are beneficial to eliminating disruptive errors. A SmartBear survey in 2012 found that “70% of organizations indicated that code reviews are very important” and “Those that do code reviews clearly have a greater level of satisfaction in the overall quality of their software – they were twice as likely to indicate that they were very satisfied with their software quality.”

    In other words, code reviews are an important part of the software development lifecycle (SDLC). Which means it should be a part of the software deployment lifecycle when it leverages programmability to treat infrastructure as code.

    Therefore it is important in the early stages of DevOps (or SDN if you’re going that way) adoption that you put into place processes to check (and perhaps double check) the scripts, templates and integrations you’re using to automate application and service deployment along the entire operational chain – from the app to the network.

  • Why Continuous Integration Doesn’t Work

    Why Continuous Integration Doesn’t Work

    Continuous integration is easy. Download Jenkins, install, create a job, click the button, and get a nice email saying that your build is broken (I assume your build is automated). Then, fix broken tests (I assume you have tests), and get a much better looking email saying that your build is clean. Then, tweet about it, claiming that your team is using continuous integration.

    I’ve seen it multiple times in multiple projects. The start is always bright and smooth. Problems start later when the build gets broken again and we simply don’t have time to fix it. We’re simply busy doing something else and a few broken unit tests shouldn’t be a distraction for us. After all, we all know that unit testing is not for a team working with deadlines, right?

    Wrong. Continuous integration can and must work. The latest build should always be clean. Always.

    What is Continuous Integration?

    Nowadays, software development is done in teams. We develop in feature branches and isolate changes while they are in development. Then, we merge branches into master (Git usage is assumed). After every merge, we test the entire product, executing all available unit and integration tests. This is what is called continuous integration, abbreviated as CI.

    Sometimes, some tests fail. When this happens, we say that our “build is broken”. Such a failure is a positive side effect of quality control because it raises a red flag immediately after an error gets into master. It is a well-known practice, when fixing that error becomes a top priority for its author and the entire team. The error should be fixed right after a red flag is raised by the continuous integration server.

    # Why Continuous Integration Doesn’t Work?

    CI is great, but the bigger the team (and the code base), the more often builds get broken. And, the longer it takes to fix them. I’ve seen many examples where a hard working team starts to ignore red flags, raised by Jenkins, after a few weeks or trying to keep up.

    The team simply becomes incapable of fixing all errors in time. Mostly because the business has other priorities. Product owners do not understand the importance of a “clean build” and technical leaders can’t buy time for fixing unit tests. Moreover, the code that broke them was already in master and, in most cases, has been already deployed to production and delivered to end-users. What’s the urgency of fixing some tests if business value was already delivered?

    In the end, most development teams don’t take continuous integration alerts seriously. Jenkins or Travis are just fancy tools for them that play no role in the entire development and delivery pipeline. No matter what continuous integration server says, we still deliver new features to our end-users. We’ll fix our build later. And it’s only logical.

    # What Is a Solution?

    The solution I propose is simple — prohibit anyone from merging anything into master and create scripts that anyone can call. The script will merge, test, and commit. The script will not make any exceptions. If any branch is breaking at even one unit test, the entire branch will be rejected.

    In other words, we should raise that red flag before the code gets into master. We should put the blame for broken tests on the shoulders of its author.

    Say, I’m developing a feature in my own branch. I finished the development and broke a few tests, accidentally. It happens, we all make mistakes. I can’t merge my changes into master. Git simply rejects my push, because I don’t have the appropriate permissions. All I can do is call a magic script, asking it to merge my branch. The script will try to merge, but before pushing into master, it will run all tests. And if any of them break, my branch will be rejected. My changes won’t be merged. Now it’s my responsibility — to fix them and call the script again.

    In the beginning, this approach slows down the development, because everybody has to start writing cleaner code. At the end, though, this method pays off big time.

    In my projects I automate this merging operation using www.rultor.com, a free hosted DevOps service. Its usage is explained in this article: Rultor.com, a Merging Bot. I would be glad to answer your questions, just shoot me an email.

  • Walgreens reminds us to keep our database healthy

    Walgreens reminds us to keep our database healthy

    It’s clear that the super app you developed with your team, is really something, and it has high chance to be the “Bread and butter 3.0”, but are you prepared for the worst? There is a good chance that some combination of data, events and circumstances out of your control will eventually collude against you, to cause a failure or a software defect. You can try to anticipate and reason about all of the things that can go wrong with your application to prepare, but in a complex system it’s impossible to predict everything.

    (more…)

  • COBOL: Completely Obsolete But Omnipresent Language

    COBOL: Completely Obsolete But Omnipresent Language

    The programming language COBOL is a nuisance, something everyone wants to get rid of; however, I’m about to advise you to learn it. It might be a gold mine.

    The IT world is a fast paced one where obsolescence is the rule and where things evolve so fast that any person who wants to understand it has to continuously learn and train. Languages, methodologies and technologies keep on changing, evolving to make programming easier, safer more robust and above all, faster. As to every rule, there is one exception.

    In 1959, convinced that the English language would be more suitable for programming than machine code or the assembly language, Grace Hopper creates the first ever human-readable programming language that would survive for over fifty years: COBOL

    COBOL, 55 years old and still in its youth, not ever thinking about retirement, has become a nuisance, something everyone wants to get rid of but no one has the courage to try.

    Are we ready to replace COBOL?

    Nowadays, 80% of financial transactions are made through COBOL programs and most of those programs have been written decades ago and have proven to be reliable, effective and efficient. To top it all most companies which are still using it have invested a lot of money in those systems and if it is not broken, why fix it?

    So why would one want to change them? Well, it’s not Java. COBOL was created in an era where IT didn’t have the meaning it has today. A program was mainly designed to replace simple repetitive tasks such as retrieving data from a database, performing a couple of operations and generating a report or updating some other data. Today’s programs have to present a front-end to thousands of users, go through different layers of security, logic and quick data retrieving before accessing the back-end to ultimately alter data and send some information back to the user. Alternatively, a program can be a tool to design objects and buildings, communicate with your friends, frag your opponents, enhance your reality, you name it… The whole philosophy behind the conception of programs has changed so much in the last two decades that having such an outdated technology that almost no one knows anymore represents a potential danger. Only few people learn COBOL these days and even though those systems are not supposed to evolve much, there is always maintenance to be performed. But are we ready to replace it just yet?

    Fast, robust and future-proof language

    COBOL has been on the brink of death for years but its presence overall hasn’t changed. Not only are the cost to migrate from one technology to another huge but the choice of the replacing technology itself is a million dollars question. What better than COBOL to perform typical COBOL operations? While not designed to be an Object Oriented Language such as Java or C#, COBOL has slowly adopted some of their aspects, extending itself and its functionalities to a point where almost everything is possible. Everything with the exception maybe of creating user friendly or good looking Human-Machine Interface. In that matter, nothing has changed much since Hopper’s time. Many of us still remember the frustration of spending hours to get a simple screen with a couple modifiable fields and those who still use them must be dreaming every day to replace F5 and F6 by simple buttons…

    But that was never the point; the point was to have a fast, robust and future-proof language.

    COBOLs main contenders

    The discussion is still opened as to which language could replace COBOL but here are some pros and cons to the main contenders:

    –       Java is easy to learn, well spread and has a huge community of developers already but is rather slow compared to other languages despite the existence of some specialized compilers and massive improvements in the last few versions. It is also arguably still in its youth and the yearly evolutions and changes to the language don’t play in its favor. Not to mention the recent security issues.

    –       C++ is fast, flexible and robust but it takes time to learn and understand and has what is maybe its biggest flow, potential memory leaks. Something COBOL and Java programmers have, almost, never heard of. By conception, it offers a lot of freedom to developers but that also implies few ready-to-use components and thus it’s less cost-worthy.

    Let’s say that one day we manage to find a clear winner, another major problem will suddenly emerge. COBOL was made to perform all kind of operations in one block but we have learned throughout the years that modularity is the key to success. So if we replace COBOL itself by another language, we will also need to have databases and SQL, a front end and HTML, most likely a Business Process Management Software to externalize workflows… Basically, replacing COBOL programs means creating whole new ones.

    The ultimate piece of advice

    While it is possible to replace this aging language, this comes at such an extremely high cost that we can anticipate its presence in at least the next twenty years which brings us to the ultimate piece of advice of this paper: if you have the opportunity to learn it and enough motivation to write white on black interfaces, go for it. The pool of competent COBOL programmers is getting smaller by the day and the demand doesn’t change, meaning that this personal investment might reveal itself a gold mine.

  • A Culture of Trust

    A Culture of Trust

    Trust is by far the most critical element of a DevOps culture.  It is also the most difficult to achieve.  IT was not built for trust.  The silo structure that is common in most IT organizations was built for specialization and territorial ownership.  The silo walls are thick.

    While DevOps may not be able to break down the silos, but it can encourage and groom a culture of trust.

    Trust is difficult to define.  We know when we feel it, we know when we don’t.   In 1993, Dr. Duane C. Tway, Jr. published a dissertation called “A Construct of Trust”.  In it, Dr. Tway defines trust as

    “The state of readiness for unguarded interaction with someone or something.”

    Dr. Tway believes that trust is actually “constructed” from three basic elements:

    • The capacity for trusting
    • The perception of competence
    • The perception of intentions

    The capacity for trusting is how your total life experiences have developed your current capacity and willingness to risk trusting others.

    The perception of competence is your perception of your ability and the ability of others with whom you work to perform competently at whatever is needed in the current situation.

    The perception of intentions is your perception that the actions, words, direction, mission and decisions are motivated by mutually-serving rather than self-serving motives.

    While trust is individual, a culture of trust is organizational.   Both are earned.  DevOps is a great opportunity to look at Dr. Tway’s construct of trust and identify small but meaningful opportunities to steadily increase the trust between Dev and Ops including

    • Improved and honest communications
    • Increased personal interaction
    • Honored commitments
    • More listening than talking
    • Frequent knowledge sharing
    • Welcomed input and feedback
    • Admitted mistakes
    • Intolerance of blame
    • Mutual respect
    • A celebration of mutual achievements

    What else could you do to increase the capacity for trusting, perception of competence and perception of intentions in your organization?   Perhaps you can start with an honest dialog with and between your Dev and Ops teams.  The answers may be surprising and insightful.

  • Moving to DevOps in a traditional enterprise

    Moving to DevOps in a traditional enterprise

     

    Introduction

    One of the challenges of introducing DevOps in the setting of a traditional (i.e. siloed, waterfall etc) Enterprise is where to begin.

    It’s certainly something I wrestled with alongside the CTO for quite some time.

    It’s not possible to impart years of experience of moving a large company to a DevOps ways of thinking but I hope to give the main 3 pillars of how I’ve approach it previously.

    Note…none of it mentions technology and tools!  DevOps is primarily a cultural idea and therefore affects people above anything else.

    Step 1 – bring in the Rock Star

    If you are moving to DevOps in your organisation, then bringing together different people from different silos and then expecting them (under their own steam) to magically cooperate and agree on everything is a little unrealistic.

    If you are dealing with historical teams and employees – which I suspect most of you are – then breaking down silos is likely to be a significant challenge.

    Aligning incentives makes things a lot easier. If both your devs and ops guys are paid a bonus depending on how much “change” they do in a year is a simple way of getting them to agree on whether they should automate releases for example.  That’s a little too simplistic but hopefully you get what I mean.

    This may well come at the cost of something else though.  The first thing that springs to mind in the last example is quality.

    Let’s put everything straight into live!  We get paid for it!

    Ok so that isn’t going to work.

    Sometimes, at least to begin with, you need to bring people together and ultimately empower someone to make decisions when one can’t be reached through consensus.   I’ve seen a real danger in DevOps of death by committee.  The more silos you bring together then potentially the greater the possibility of them not making a decision.

    You may have heard about this idea of DevOps rockstars.  Guys (and gals) that get DevOps inside and out.  People that understand all sides of the coin (development, DBA, infrastructure etc).

    They try and influence through education and persuasion and point people in the right direction.  They help the silos reach the right decisions themselves rather than a top-down “do what I say”.

    All said and done sometimes this open inclusiveness doesn’t always work and someone needs to just stand up and be counted.

    So what’s my point?

    If you are trying to move towards DevOps then hire a rockstar (I doubt Rod Stewart is suitable btw).  Someone that has done it before.  Someone with a very mature sense of what DevOps is and preferably someone that has done most of the jobs he/she is trying to influence.

    Moreover, hire someone who’s first instinct is to try and bring people around to their way of thinking.  And lastly, someone who isn’t ultimately afraid to put their privates on the line and make a goddam decision.

    Step 2 – look at your people structure

    The traditional definition of DevOps is to break down the silos between Development and Operations so that they accomplish shared goals.

    However, I’m a keen advocate that DevOps isn’t just Dev and Ops (see http://enterprisedevops.blogspot.co.uk/2014/07/devbiztestreleasethingyops.html).

    It’s about all the functions that come together to achieve meaningful change.

    The devs, the ops guys, the business and techies should all have shared goals and incentives. Where possible they should be sat around the same desk. The team should rise and fall collectively (there is no “us and them”). The team should manage change from cradle to grave (requirements, build, test, deploy, support).

    I think we should rename Devops to “DevBizTestReleaseThingyOps”.   Catchy.

    To that end, DevOps is as much to do with “people” than it is about new tools and processes.

    With that in mind, potentially you are talking about some fairly radical changes across a wide number of people of various skills, experience and personality types.  I guess it really depends on what your organisation is structured like, the skills of your team and many other factors.  For my company (a large UK insurance company) however, it’s meant some fundamental changes.

    We started by aligning change functions (devs, testers, BA’s etc) to individual business units (rather than monolithic shared group services).

    That fundamental change has been sponsored on the back of some early and small wins by implementing things like Continuous Integration and labeling it “DevOps”.  Ok so it’s not DevOps but if using that term starts to get visibility and movement within your organisation then I don’t think that’s a bad thing. I always believed we needed a banner for the modernization of the department to fly behind.  If that banner reads “DevOps” then so be it.

    That early small success was really vital to us.

    When we started out we had a lot of push back and negativity….

    •     Why are you telling me how to do my job?
    •     You are using the wrong tools
    •     I don’t see the benefit
    •     I know more about this than you do
    •     What you are trying to do is easy
    •     We don’t have the right people
    •     This is a hipster idea for the likes of Facebook
    •     There are regulatory reasons this won’t work

    Basically everything under the sun.

    That’s stressful when you are trying to do the right thing and move the incumbent forward.

    I don’t think we helped ourselves too much to begin with.  There was too much of my team trying to pick up and change other team’s technologies and processes.  Not enough leading the other team to water and helping them implement something themselves.  Lesson learnt.

    Step 3 – create a team to help others

    I mentioned earlier that DevOps is cultural idea and not a job title.  Definitely not.  Unimaginable.

    So how is it I run a “DevOps” team?   Surely that’s a collective of “DevOps engineers”?!

    My team “Platform Services” supports the idea of the change platform at my company.  The change platform consists of tools such as Jenkins, Puppet, uDeploy, Teamcity, TFS and a few others.

    Someone ultimately needs to “own” the tools making sure they are maintained, creating best practices and establishing a roadmap for the future.

    My team goes beyond this and helps different change functions adapt to the change platform and are evangelists for things like DevOps and Continuous Delivery.

    With DevOps being concerned with breaking down silos its contradictory to have a team that bottleneck of knowledge and skills.  To this end, Platform Services is the silo-buster – we help others pick up the tools & processes and implement them.

    For example, we don’t do deployments ourselves but we do sell the benefits to other teams and help them to adopt the tools and processes.

    I think this is a good model and helps spread the success of DevOps around the IT function.

    It helps that consultancy from my guys is effectively free to other parts of the IT team (thanks to an enlightened CIO & CTO).

    I hope you found some of that useful.  Feel free to check out my blog on my move to DevOps: http://enterprisedevops.blogspot.co.uk/

    Good luck and get in contact if you have any questions or comments.

  • The Double-Edged DevOps Job Title

    The Double-Edged DevOps Job Title

    Last month at GWO in Atlanta I met Nathen Harvey with Chef. I liked his DevOps presentation, “DevOp With Me!”, especially how he used “Walk This Way” by Run DMC and Aerosmith as an example for how collaboration in a DevOps environment often ends up playing out. They literally break down walls in the music video.

    While not all of Nathen’s presentations are safe for work ( many include some explicit language. It’s safe to assume so unless clearly labeled otherwise) they are all entertaining and informative. I’d encourage you to check his stuff out, especially if you’d like to know more about how and why Chef and DevOps mix well.

    I asked Nathen if he minded me linking his deck and giving him a shout out in a blog post. I even promised to say nice things about him. The answer he gave me only reconfirmed he does really get it:

    Of course! (even if it’s not all nice things, as long as they’re honest)

    If I’m being completely honest I have to admit this changed the topic of this blog post. I liked the idea of setting devOps to music so much I was going to post a DevOps playlist. I may still write or talk about that another time. It ended up being pretty easy to think of other songs that would fit for one reason or another. It actually even makes for a decent drinking game in the right crowd.

    Instead I’m going to talk about the one thing I know Nathen and I  don’t see eye to eye on. Nathen’s an advocate for doing away with using the word “DevOps” to describe a role or a team. I, being a DevOps Engineer, strongly disagree with this perspective. When I asked Nathen about this he responded with a single question:

    Can you describe what you do without using the word DevOps?

    It’s true there are a lot of folks with the word DevOps in their title that are adequately described as a systems administrator or a build automation engineer. Coming up with job titles that don’t contain the word DevOps is also a decent drinking game in the right company. A good serious answer for me would be ombudsman or public advocate. I think KTLO (keep the lights on) custodian would also accurately describe how I’m earning a large part of my keep these days. Still, the job I interviewed for was DevOps Engineer, and for now I’m going to keep using those words to describe what I do. I like Nathen’s Socratic response to why we shouldn’t be describing people and teams with the word DevOps, but what I am doing isn’t accurately described by other job titles where I work. If another title is a better fit for the work you do where you’re at, then I would encourage you to take Nathen’s advice.

    The more common reason I’m given as to why I should have changed my job title on day one is really pretty disappointing. I’ve read more than once that I should be concerned that my head will be the first one on a pike should the devOps movement fail. This really isn’t a concern I have.

    First  of all, DevOps does work. I’m not sure who would dispute that the CAMS model for DevOps is made of things any organization should be trying to improve Kaizen style. I’d even say that improving Culture, Automation, Measurement, and Sharing together results in a sum that is greater than any of its parts.

    There are however places that aren’t ready to embrace the change needed for this type of improvement. I caught a quote from @filler when @RealGeneKim retweeted it last November:

    If you’re concerned about a lack of progress in echo chamber, go offer yourself up in a different forum. And bring cookies.

    Advice I took quite literally. I found a job with a DevOps title at a place that was actively trying to change. I regularly bring in food. Just this past week I baked six dozen toll house cookies from scratch for a DevOps round table meeting I hosted. Why? Because I genuinely give a damn. I want people to come to that meeting to listen and to share. People who are passionate about technology and have faith that there’s always a better path. This group should find such a round table conversation quite enjoyable. The cookies help us remember this is supposed to be fun. They help keep the mood collaborative instead of conflicting. Such a job shouldn’t be left to baked goods that have been sitting on a shelf for weeks. If I want the best from the people I’m working with then I should give them my best whenever I can. It’s often the smaller details that could be neglected or ignored without issue that are noticed and have a more lasting impact.

    Some of you right now are in an environment where the bigger details that are making life difficult for everyone are actively being ignored. You may have been told not to worry about them or that the real solution is for everyone to work weekends. This is garbage and you know it. Please do not change your title to something safe and hide under your desk while in the office. Be the change you want to see in the world. There are lots of opportunities to help organizations make this transition. Find one that’s a good fit for you and make the jump.

    My advice for businesses is to be very careful when you describe a position as a DevOps role. We know this to be a very specific thing, but if you’re simply shoving the word DevOps into a job title in an attempt to attract talent there’s a good chance you’re in for a rude awakening. The person applying for that position knows there’s always room for improvement. They know that any transition is going to cause some friction but are willing to work through it. They’re also not scared to rock the boat. So, if your organization is not ready and willing to collaborate as one big team it won’t take too long for your devOps staff to go somewhere else on their own. Threatening their job security in an effort to maintain the status quo will only accelerate the process. You may very well end up with a lower head count at the company picnic as warned, but if it’s because all of your talent moved on to greener pastures you’ve effectively turned a bad situation into a terrible one.

    If you’re a DevOp-er that has a work environment in that gray area between heaven and a coal mine I’d encourage you to push the envelope. Share your honest opinion with someone you know doesn’t agree with you. Set up a meeting to discuss conflicting perspectives diplomatically. Drop an F bomb in a conference room if it will wake the room up and get some blood pumping (use sparingly and with caution). Don’t expect people not to get upset. Instead, try really heard to shut up, listen to everything they’re trying to say and do your best to give them what they need. If DevOps is anywhere near your title then you have some responsibility in getting these conversations started. If your current employer didn’t know this when they used DevOps as an adjective then shame on them.

  • Meet Infrastructure as Code

    Meet Infrastructure as Code

    Maybe you have already heard the terms “Programmable Infrastructure” and “Infrastructure as code”.  But even if you have not, if you are interested in DevOps it’s something you need to know.   Most new lingo causes me to roll my eyes.  But in this case, it’s not just new lingo, it’s an entirely new concept. Besides making the nerdy feel cool when they say it, these terms describe a new idea of what an application consists of. (more…)

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.

  • Don’t Be Afraid of the “R” Word

    It seems that many people are afraid of the word refactor.  I think in many cases, if someone admits they need to refactor, it means they either didn’t do their job right.  They either missed requirements, wrote inefficient or buggy code, etc.  Alternatively, there is something fundamentally wrong with the product or system.  The fact of the matter is that sometimes there are big problems and a major refactor/rewrite is required.  I think that is rarely the case, but if embraced by all and done judicially, the outcome can be significantly better and the major refactor avoided.

    It doesn’t matter if it is a feature, deployment script or architectural decision, at some point, everyone must admit they are not clairvoyant and cannot predict every way end-users or systems will behave.  Therefore, there will be missed/new requirements and unforeseen problems.  As such, I very much embrace the philosophy of build for what you know now and adapt as you go.  Typically, that means the output is simpler, it got done faster and enhancements and bugs are easier to address.  That philosophy also means you are constantly in a state of refactoring, or said differently, incorporating new learnings, making individual methods more efficient, improving the readability of the code, filling in missing tests, etc.

    Now, just because we have embraced refactoring and we have noble intentions of, in general, leaving the code better than we find it, that doesn’t give us the license to go crazy and start rewriting everything we see. I think there are right and wrong times to refactor.

    First and foremost, don’t let those yucky bits of code distract you from your real purpose which is to deliver business value by building those new systems, features and fixing bugs.  So instead of just jumping right into refactoring, quickly note the yuck as a TODO in the code, story or in your notepad so you don’t forget about it, but also stay on task.  That said, bad code that is directly in your way is fair game and I even think that bad code near the edges of your task are also fair game.

    Secondly, follow good engineering practices.  Make changes in small, bite size chunks, test and deploy your changes often and communicate them with the team.  In the end, you will end up with a better product, happier end-users and engineers which should all translate into a more successful business.

    If you are one of those clairvoyants, then email me and let’s talk over a beer because I have lots of questions.  Otherwise, if you are a mere mortal like myself, then embrace the “R” word and seek out companies and teams that also embrace it.  At the end of the day or after a few releases, it is pretty impressive to reflect back on how adaptable the team and code can be while solving real problems very quickly.