staging-devopsy.kinsta.cloud

Tag: maintenance

  • You Still Have to Make Time for Maintenance

    You Still Have to Make Time for Maintenance

    I’m going to take a side trek today into home computing with applicability to IT. Bear with me.

    We maintain a pretty complex home network, as home networks go. With three NAS servers, a small one-box SAN, some web servers and connections to several hosted environments. We both work from home, and my business is contained in one of the NAS servers and some of those hosted environments, so it’s all necessary. Scheduling downtime with several people using the network darned near 24/7 is difficult. Not medium enterprise level of difficult, but still an issue.

    Two of our NAS servers each have a disk out. It happens, that’s why they’re NAS servers. In general, the only day that replacing the disks makes sense is Sunday, when things are a little slower (both in schedules and network usage) than normal. One of our web servers is just plain dead–but there’s info on there we have to get off. The code is mostly being replaced, but the data is collected over years and we need to see how much we can salvage.

    None of these things are shiny DevOps things. Yet, they still have to get done. Importantly, they have to get done when automated tools aren’t running across the network or counting on the NAS to be responsive. The rush isn’t huge, because there are off-site backups, but restoring a remote backup is a long-term proposition–weeks, not days in our current environment–so it must happen as soon as practicable.

    Don’t forget that with the drive to automate everything. If you own the hardware, it’ll need maintenance. If you own the software, same thing. If you need data to move platforms, it will take time. All things we kind of hand-wave in the DevOps world. They’re seen as bottlenecks, if seen at all. Generally, we don’t even consider maintenance issues until they impact DevOps, but we know they are going to happen so we should have a plan for how to deal with them.

    Those few of you whose entire environment is hosted in a cloud provider have an easier job of it, but eventually you will be asked to move cloud providers. As sure as the sun will rise tomorrow, something will drive a move away from your chosen platform. Normally TCO, but other factors play in, too. When that happens, the work will be more than “Find time for the NAS to rebuild without overly impacting everything else.”

    So, spend some time thinking about it. Do you have plans to make room for maintenance? Are you scheduling maintenance into release cycles? Are you coordinating releases with updates that are not required by this specific release?

    This is where dev (typically Agile/Scrum) meets ops (typically system/network maintenance), and it is every bit as important as what is normally seen as DevOps. Get it right and have fun, keep cranking stuff out. Get it wrong and struggle with the problems infrastructure issues can inflict upon your cool new update. One of our new websites outright stopped development when the web server went down–for a variety of reasons, but the catalyst was the loss of the original. We’re working out what to do about that.

    Meanwhile, we didn’t make time yesterday, so I’ll be replacing disks and tearing apart a web server in a much smaller window that is available today. See you when I poke my head back up. Hopefully, you’ll have better planning than we did.

    — Don Macvittie

  • The Beauty of Inheritance Architectures

    The Beauty of Inheritance Architectures

    Spoiler alert: This article is not about inheriting your great-grandparent’s estate. It is rather about how embedding the principles of inheritance into the architecture of your chosen DevOps toolset can yield exponential results. And yes, I realize that you, the customer, have little to do with what a vendor builds its tooling architecture upon. However, you, the customer, can vote with your feet, to search out products that offer features such as inheritance, rewarding vendors that have it and punishing those that do not. The simple question to address here is, Why should the customer care?

    Like anything that can grow, sustainability on scale is a key concern. How much will it cost me when I am maintaining 6,000 applications on this particular DevOps toolset, as opposed to the six applications I started with? You can also infer maintaining a higher number of technologies as having the same need. For example, if I need to maintain three different types of databases, two different middleware, four different hardware platforms on “x” different versions of operating systems … you get the idea. The larger the application portfolio (in sheer numbers of apps), or the more technically complex the portfolio (in sheer numbers of technology components in an app), eventually will drive you to worry about how easy it is to maintain these applications within your chosen DevOps toolset.

    The capabilities of your chosen set of DevOps tooling will govern the costs of both the initial adoption of your services as well as the ongoing maintenance of the application portfolios you serve. It is hard to sell the value of DevOps to the business when it begins to see a one-for-one increase in DevOps engineering staff for every new application or technology venture under consideration. However, one of the chief culprits in driving large DevOps support staff organizations boils down to the lack of an inheritance architecture in the tooling that was originally selected.

    Setting a Common Language

    So before we discuss how inheritance will benefit us, first let’s agree on a common vernacular of what an “application” is. When you say the word “app” to someone, you get a wide variety of responses on what that means—from something that sits on your phone to a cluster of things that sit on the mainframe. Let’s begin by examining what an “app” is by the effort it takes to onboard and then maintain it on our DevOps sets of services (regardless of platform).

    Consider for a moment the hierarchies by which we could describe software constructs. It is relatively easy to discuss how software works in terms of hierarchy. First, at the lowest layer, we could describe having multiple technology components (or the lowest level of a particular piece of technology that must be addressed in a build). Next, assembling these components together forms the picture of what a single application looks like. Finally, assembling multiple applications that have a common symmetry or common purpose together form a system or series of apps.

    An imperfect but quick example could be the Microsoft Org Chart component of its Office application(s). For purposes of example we will argue that Org Chart, Clip Art Manager and Equation Editor are all components of the application Microsoft Excel. To build a new version of the application Excel, we would have to deal with each of these individual components that, once assembled in our build, would create a new version of Excel. When you combine more of the apps—Excel, Word, Outlook and PowerPoint—you get the system, or series, called MS Office. (And, yes, I know this is a less than ideal example, but it is meant only to illustrate a point.)

    Assessing the Work Effort

    The reason for discussing software in this way allows us to begin to discuss the level of effort to automate it. By addressing the work effort to automate each component (by type), we can begin to estimate how long it will take to complete “the automation of software.” For example, if my homegrown application has a source code component, a configuration file component, a database component and a middleware component, then I have four components I must automate (or include in the build and deploy process) to assemble a new version of my application.

    This gives us a “least common denominator” language for assessing how much effort it takes to automate a given app. I have seen most common applications have only two to three components. But I have also seen several apps that, due to their unique architecture and construction, have 30 to 50 components a piece. Obviously, it is easier to automate an application with only three components than one with 50 components. The other factor we need to apply on the ease of automation is whether I have encountered this particular component technology before in my prior automation efforts.

    Let’s say, for example, that the application I am estimating contains Java component(s). If I have encountered Java automation prior to this, I should have constructed a template for Java type component automation that I can use for a situation just like this one. In this situation a familiar technology component is far easier to automate than a new one (or, rather, new to me). However, if this is the first time I have ever seen Java, then it may take me a bit longer to construct the template I want for this app, and for future apps I encounter, that also use this technology type.

    The Beauty of Inheritance

    Here is where inheritance begins to play its critical role. If my DevOps-build construction tooling includes inheritance, then I could construct a “master” template or framework for each type of technology as I encounter it. A master template for SQL databases or Oracle or DB2, as an example. Or, I could make a master template for WebSphere or SharePoint types of technologies—one master for each kind of programming language we have in my shop. You get the idea.

    Having done this allows me to have a limited number of master templates, in which I can maintain all the critical steps and processes my company wants me to do for a given technology type. If different business organizations or development organizations need a variation, they can create a “child” of my master and vary it according to their own needs. As long as the linkage is maintained to my parent/master template, I can make the big, overarching changes driven by corporate mandate, and the business unit can continue to vary items to their unique needs.

    Having inheritance in this instance allows me to roll out changes to an entire organization, such as, for example, a security patch or process, for a given technology by simply adjusting the master template of that component. Inheritance automatically flows my change down to all the “children” of my master and possibly to all their “children,” should the need arise. The alternative is having to go through the specific automation of each application that has the targeted technology component and updating it manually, hoping I don’t miss one, and hoping the steps I may need to adjust do not break the old automation instead of alter it the way I needed.

    Imagine having a WebSphere subject matter expert able to adjust every client application that uses your corporate installation of WebSphere by simply adjusting one master template. Or having a network guru, or database expert, be able to do the same for all applications that use their particular technology component all at once. Inheritance preserves the real-world need of having limited alterations of the same technology type within the same company, flowing down changes from parent to child.

    Having a one-size-fits-all is simply not practical. Using copy/paste requires repeating updates to each impacted template or framework manually. But having the ability to create offshoots from a main, and not lose the connection to the main, allows corporatewide alteration from a central point of control. The costs of maintenance or administration are formidable; inheritance can reduce them. This allows a much higher ratio of applications or technology components supported by a much lower number of engineering staff.

    A Wider Impact for Inheritance

    Inheritance is keenly useful in release orchestration software as well. Most vendors offer the cut-and-paste approach: you copy the last the release you did, alter it slightly and use it again. That might work for some organizations. But having an inheritance feature creates entirely new dynamics on how a release is constructed for a given line of business and how corporate governance can be implemented with minimal effort. In this case, a corporate master release template can be constructed, and child releases can inherit from the master all the main governance required without risk of omission.

    I realize inheritance architectures might not be at the top of your list when evaluating DevOps tooling solutions. But I would submit it is a strategist’s job (if you need one, let me know, I am looking 🙂 ) to point out how long-term sustainability can be greatly benefited by this architectural feature—even if the vendor would love nothing more than to steer you away from it. If you already have committed to a tooling solution, you may want to bring this up at your next vendor features performance meeting. If not, you should at least consider what having it means to your internal cost structure, and what not having it will add to the true cost of ownership of any proposed solution.

    To continue the conversation, feel free to contact me.

  • Resurrecting the CMDB Through a Singular Path to Production

    Resurrecting the CMDB Through a Singular Path to Production

    Change consistency (or the predictability of how change is deployed) followed by change transparency (or the visibility into the state of change at any stage in the software development life cycle [SDLC]) are two key benefits of embracing a highly functioning enterprise-class DevOps set of services. Most organizations are attempting to transform themselves from a widely varied manual delivery system to an automated, template-based delivery model that DevOps relies upon.

    Along this journey a key fact soon emerges: We cannot allow “manual” tweaks to our environments and maintain the predictability, visibility or the stability we want. To be truly consistent, process must be fully automated. A process that relies upon human interaction (whether planned or unplanned), is subject to human variability that can impact its success. In short, we do not want a deployment process that has humans coming along after it, to “clean it up.”

    Thus a key principle of DevOps, is that it becomes “the singular path to production.” We want all changes going in to production to have gone through our DevOps systems and processes. Our goal becomes to reduce and eliminate human reliance in every interaction. As an example, we automate test execution, then test evaluation, then test gated change progression. We use “smart” deployment technologies to overwrite what is different in the destination environment, forcing it into compliance with our stated version of changes that “should” be there.

    In effect, we “step on” variations, overwriting them every time we initiate a deployment. This quickly teaches our freestyle engineers who make changes on the fly that their work will get “stepped on” with each new deployment (perhaps daily or hourly). Most engineers quickly get the hint that to preserve their work, they need to include it in the deployment process itself (updating the proper configuration file, or adding the right series of steps in the deployment execution automation itself). Otherwise, they will face a daily onslaught of repetitive tasks that inevitably will overwhelm them.

    In this sense, a properly implemented DevOps system is “self-cleaning.” Paired with an executive mandate to have all change go into production through DevOps, another benefit emerges that may bring the CIO glee. Just a few years ago, IT organizations were bent on the idea of documenting change (therefore understanding it better) through the means of a configuration management database (CMDB). Those projects tended to be large, costly, and doomed to minimal success. Typically, the CMDB effort was great for 90 days after its first instantiation, then over six months its data became “suspect,” then after a year or two its data became prehistoric and close to useless.

    The simple truth is, engineers (of nearly every variety) are toxically allergic to documentation. Asking engineers to have the human discipline of keeping up with documentation is like asking the sun not to shine. No matter what set of inducements is offered, or punishments threatened, the engineer simply remains toxically allergic to documentation and its maintenance (probably a DNA thing).

    Industry tried to solve this problem by producing “discovery tools,” or tools that could determine automatically what our state of change was environment by environment. The idea being that if we ran discovery tools on regular intervals we could at least renew our CMDB baseline from time to time. It was an improvement, but discovery still lacks context for change and often is not thorough enough to generate real value beyond asset location and basic configuration.

    Enter DevOps, which inherently has an artifact database capability to track versions of builds, deployments and releases over time. DevOps understands the need for artifact identification, articulation, compilation and history. Most early DevOps tooling has some form of it built into the products. But this is where a perfect storm becomes possible.

    If our DevOps systems have achieved transparency; that is, if we are able to connect the context of change outlined from ideation (in the business requirements) through production release (including the history of testing, etc.) by version, we now have a great deal of wealth we could plug into a resurrected CMDB. And we could maintain the CMDB automatically (requiring no engineers to have to do manual things to keep it up to date). Our CMDB becomes a repository, or master repository of the artifacts of change we implement through DevOps.

    This is more than just baseline or the ability to establish a “golden copy” of what change should look like. It is the ability to establish “relative” golden copies by development version while it progresses through environments of even highly complex SDLC systems. We have thus increased the value of a CMDB exponentially beyond its use in just production, but can integrate it, into the SDLC process and watch change become documented in real time no matter how fast or how often we move change.

    This should be groundbreaking. Even if the CIO has burned all his funding and political capital on a former CMDB project that has had less-than-desirable results, he or she can salvage what is left of it and quietly use DevOps to resurrect the CMDB, making it an integrated step of moving change. Once the CIO has demonstrated success doing this, he or she can offer this reclamation project to the business and begin exploring ways of creating CMDB valuation to the business (beyond asset location and valuation).

    What emerges is a DevOps-centric CMDB. DevOps, having become the singular path to production, has both insight and context into all change progression and simply updates the CMDB (master artifact repository) as needed automatically. In this way, an integrated CMDB becomes part of the DevOps continuum. And perhaps the greatest benefit or win is the simultaneous recognition of engineer’s allergy to documentation already baked into this sustainable solution. I would call that win-win—or, at least, a recognition of reality.

    To continue the conversation, feel free to contact me.

     

  • DevOps doesn’t end at deployment

    DevOps doesn’t end at deployment

    If you ask five IT admins the question “What is DevOps?” you maybe get eight different answers. We’ve attempted to define and explain the term DevOps here on staging-devopsy.kinsta.cloud a few different ways, but a precise definition is still elusive. One thing is certain, though, DevOps is about more than just automating virtual server deployment and configuration.

    At Velocity 2009, John Allspaw and Paul Hammond presented a session titled “10 Deploys Per Day: Dev & Ops Cooperation at Flickr”, which some credit with launching the concept of integrating development and operations—at least to mainstream IT. For many, though—especially those who are still trying to understand what DevOps is, and what it means for their organization—DevOps amounts to tools that simplify the process of implementing tens or hundreds of virtual server instances, and automating the initial configuration.

    Automating IT

    Tools like those offered by Chef and Puppet Labs are useful for automating routine tasks associated with deploying and configuring the IT infrastructure, as well as deploying and updating applications. These vendors can also help monitor and maintain a consistent, stable environment by enforcing a designated configuration.

    To be honest, that sort of automation is more or less table stakes for DevOps. It’s just one small facet of a much larger concept that involves the entire scope of development and operations—agile development and rapid iteration of new versions and features combined with an automated and easily-scalable infrastructure.

    Once the virtual server environment is up and running, though, and once the applications are deployed, the work is really just beginning. Part of merging development and operations, and keeping up with a more rapid pace of IT involves ensuring that everything is running smoothly. With frequent updates and rollouts, it becomes even more important to have tools in place to automate monitoring and alert IT admins when something goes awry.

    DevOps also involves tools like those from Stackify—tools that keep an eye on the apps running in your environment and let you know when things don’t quite seem to be firing on all cylinders. When something goes wrong, you want to know about it in a few seconds, not a few hours or days. Stackify can monitor and trend important metrics like CPU, memory, and disk utilization on your servers, and alert via email or SMS text when problems arise.

    Automating the creation, deployment, and initial configuration of servers is awesome—but you also need tools to address the challenges of maintaining and updating thousands of server instances as well. You need to be able to add or change user privileges, apply patches and updates to the operating systems or applications, change security configurations, enforce compliance policies, and more. When you’re talking about hundreds, thousands, or tens of thousands of servers, it’s imperative that you have tools like JumpCloud to help automate those tasks across the entire environment.

    More than just tools

    As great as all of the various DevOps platforms and tools are, though, DevOps is not just a matter of buying or implementing an array of tools to manage your IT environment. As the name implies, and as Allspaw and Hammond spelled out in 2009, it’s about instilling a culture that breaks down traditional barriers and enables developers and operations to work cooperatively and collaboratively.

    The traditional roles of developers and ops impose unnecessary friction that gets in the way of being effective. Developers blame problems on the implementation or IT infrastructure. Operations personnel blame problems on poorly developed code. Creating an environment where all appropriate parties work together, and where everyone takes responsibility rather than assigning blame creates an entirely different dynamic for productivity.

    When it comes down to it, it’s the culture of DevOps that accelerates the speed of changes and updates that necessitates tools and platforms to automate deployment and infrastructure management. The culture drives the tools, not the other way around.

    So, yes, one of the most visible components of DevOps may be tools that automate the deployment and configuration of servers, but that is just one element of DevOps, and it’s just the beginning. Everything that happens after that to monitor, maintain, and update that environment once it’s deployed, and everything that happens outside of the IT infrastructure to streamline productivity and increase efficiency is all part of DevOps as well.