staging-devopsy.kinsta.cloud

Tag: change

  • Our Infrastructure is Still Expanding

    Our Infrastructure is Still Expanding

    You would think that sooner or later, marketers and early adopters would take a rest, but it seems that if one group slows down for a bit, another picks up. Our infrastructure is expanding in almost every possible way, and this creates more of a burden on every aspect of DevOps, specifically in IT.

    We have more security needs because we have a larger footprint, for example. We have long had APIs, but now we have massive API groups that are used and re-used, and while standard code scanning works with the source, the companies doing “API security” start with security of the interface, not the code. That is slowly changing, but API management is also impacted, and organizations that determined they didn’t need API management because adjacent markets were serving enough of their needs are slowly discovering that with proliferation comes the need to look at API management again.

    I stay up to date on most new technology because, as an industry analyst, what is coming can inform my discussions of what a given technology space looks like today. But even for me—someone who is not spending a huge chunk of my day coding or working on the toolchain or perfecting the new deployment platform—it can be hard to stay abreast of everything. While one of the foundational tenets of DevOps was no specialization and a shared workload, I think everyone now agrees with what a few of us were saying from day one: Some specialized skills are still needed, except in corner cases. But we certainly are sharing workloads more broadly and far more responsive, so DevOps is absolutely a massive success. Site reliability engineering (SRE) extends that success but focuses more on Ops. I’m really not a huge fan. I like that DevOps wanted to merge the processes and make the management of applications more holistic; I don’t think re-dividing it is the best idea out there. Like every title, “SRE” is interpreted pretty broadly. Some roles really involve DevOps (but SRE sounds cooler), and other organizations are just replacing ‘developer’ with DevOps and ‘operations’ with SRE. However, most are using SRE to manage the entire environment in which the app runs.

    Where are we going? Constant change. That’s the answer. I’ve waited for the breather for a couple of decades, but since VMs were popularized, it has been non-stop and will continue to be.

    This means a lot to you, to your viability as an employee, and to hiring. With the glut of people on the market, we’re seeing a resurgence of “must know every tool in our toolchain” type position postings. Of course, you must not, in a world where the combination of languages, frameworks, CI/CD tools, alerting, and monitoring tools would make a great college-level combinatorics equation. You add value by being adaptable. Both to your current employer and to future ones. The key thing you should seek to demonstrate is your ability to pick up and master new technologies when required. Do it on your current job to show worth, and keep a record of it for your next job. I’ve been a hiring manager off and on forever, and when the market is flooded with good candidates, that is the type of differentiator that gets the team’s attention. Do it; make it something worth doing and document that you did it. You won’t regret it.

    A group of us were helping a recent grad figure out what to do with their career a couple weeks ago, and the consensus was “avoid AI until it shakes out.” This is not the first time I’ve heard that advice—it’s not even the first group I was a part of giving that advice. And I agree in the general sense, but certainly in places where it is being implemented by all vendors in the space and it adds value, you do need to be aware of how it works and what you can do with it, and a project is one way to get that knowledge.

    It is still a rolling opportunity, and you are still keeping the lights on. I’ve seen people who worry they’re not doing enough. Any employer worth working for wants you to keep the systems running first, and everything else after—so if you’re doing that, you’re doing enough. Now go beyond. And keep rocking it. This new technology hasn’t implemented itself, and it isn’t self-maintaining (yet) … That’s all you.

  • Forget Change, Embrace Stability

    Forget Change, Embrace Stability

    DevOps is about change. It was introduced to accelerate change, and it has opened up a whole new world of tools and possibilities that are all based on the premise of changing IT. Starting with Agile, we now have DevOps variants and subprocesses that run the entire software development life cycle (SDLC) process.

    Want to make more frequent updates that can regularly show demonstrable progress to business owners? Agile and CI! Need to get the app settled in its inevitable home with automation? You need CD! Want testing and zero-day protections? Start with DevSecOps built into the CI/CD process and add in application and API security! Want to fully automate deployment in the whay builds are automated? Grab a GitOps tool and let’s goooo! The list goes on and on. Literally. We’ve almost got more tools than we have names for spaces at this point.

    And all of those tools introduce change. Most of them introduce a massive amount of change. You don’t get to GitOps without breaking some of your existing processes and reworking them. We have fallen in love with change in IT.

    But there is a funny thing that’s happened along the way; I’ve mentioned it before, but never from this perspective. All great changes require time for consolidation. Be it stock market swings or revolutions, afterward, there is a period of consolidation. In IT, containers are a good example. When they became capable of actually running workloads, IT experienced a period of time when making all the various machines run in VMs was the thing. People weren’t (generally) looking for the new big change in the process, they were just glad that those one-off machines that had custom hardware could now be moved to something more manageable and that spinning up servers could be done in days or weeks instead of the months it used to take when you had to order hardware. But that consolidation created organizational standards and stability that the organization enjoyed for years – even after containers rose to challenge VMs and some organizations even today.

    So every once in a while, take a breather, consolidate gains, create standards that have meaning for your organization, then continue on with whatever the shiny new change is. Make sure that once you have identified, for example, the need for code scanners that can work within or that are complementary to the integrated development environment (IDE), that you have one scanner—or one scanner per programming language—to maintain and monitor instead of one tool per project. That saves time and creates stability, both of which will serve the organization moving forward.

    Change is here to stay; it’s the only constant. IT is caught in a cycle of change followed by change, with the next wave driven by generative AI. But in times of change, seek stability and, in times of stability, seek changes that can have a positive impact. So go standardize and organize before the next wave hits.

    Mostly, the changes will serve the organization by freeing up your time and letting you spend more of your time rocking the IT world. Those systems stay up because of you. More time is more innovation, and more stability means less burnout. All of which is good for the organization.

  • Some Change is Bad

    Some Change is Bad

    Some time toward the end of the year, I start considering the volume of change we, as practitioners, have been under for … literally years, even decades. And I start to try and think of ways to keep you all from drowning in change. And then I write something that I hope helps you all focus on good change, and keeps moving your environment forward.

    This year I’m going to say “consider the reasons for the change.” I will attempt to find examples that do not require mentioning a specific product or vendor, so apologies if these read a bit convoluted as I attempt to illustrate this point without castigating.

    There are three basic reasons that a market or marketers want you to consider a change. There are, of course, variants and cases where it is a bit of this reason and a bit of that, but most demands that we change what/how we are doing things can be lumped into “mostly one of these”. They are:

    1. A great idea that is really a game-changer for some subset of the market.
    2. A shifting of who is in control.
    3. Locking customers in or expanding revenue.

    All of these happen at the vendor/project level, and all of them happen at the market level.

    Number one is best illustrated by any of the waves of server virtualization that have come through. From virtual machines to containers, each has solved problems and changed the way we do business. Notably, I’m only talking about server virtualization efforts here. Most of the other virtualization attempts have really not fit the “great idea that is a game-changer” category, though some have limited usefulness. Personally, I think the new version of storage virtualization will be in the game-changer category but am waiting to see it in use.

    Number two is one to watch out for and is most obvious when big-name OSS projects are forked. It is rarely about the technology being implemented (after all, a fork is a copy of the original; at first it is not at all about the technology), but rather about who is calling the shots. There is a bit of this in every public cloud vendor implementing their own version of container management (though this example is more about lock-in, in my opinion).

    Number three is multi-pronged. If your vendor went from selling you hardware or software to selling you a service, they want that recurring revenue and they want you to be linked into that service so deep that you can’t walk away. The much-reviled 1990s mantra of “embrace and extend” is the poster child for this kind of change. There might be a value-add in there somewhere and sometimes the added functionality makes the move a no-brainer. Often, though, the change is simply to the vendor’s advantage, not their customers. As most major Linux distros have shown, open source is not immune to this one, as most of them are making OSS users the de facto beta testers for their paid offerings.

    None of this is to say that the change in front of you is bad—even if the reason it was originally spawned is bad. It is reminding you that, after a constant stream of change, we’re more open to it than maybe we should be and you should remember the only important thing about new changes—do they make our environment more secure, faster, more agile? Everything else is just noise. Yes, all the cool kids are switching to coding in [insert flavor du jour], but does it offer your organization anything new? Yes, it seems everyone is using [insert current CI/CD flavor], but does it address the weaknesses of your current toolchain?

    You all are nailing it. Keep doing it. And don’t be afraid of change, but make sure you know if that new OSS database system is really new, or just a lot of installing the same code under a slightly different name. Because someone didn’t like the new owners.

  • The Third Way of DevOps: Stacking the Cards in Your Favor

    The Third Way of DevOps: Stacking the Cards in Your Favor

    In the Third Way of DevOps, change is as much attitude as it is behavior

    In our first post, “The Third Way of DevOps – From Knowing to Being,” we shared our thoughts on effectively applying knowledge to put the Three Ways of DevOps into practice. This blog explores a complementary dimension: How to build confidence when faced with a seemingly impossible task.

    This is the all-too-familiar setting. You’ve seen an opportunity to improve things and genuinely want to do something about it. But you soon feel the clarity and anticipation draining out of you. First there’s the pressure of the day job. Then there’s the inertia of your co-workers, who are oblivious of or indifferent or hostile to your ideas. And finally, as if these concerns weren’t enough, you’re faced with the difficulty of unlearning old habits and developing new ones. In short, you don’t take action on what you have learned and know.

    Self-efficacy

    Expressed in the somewhat abstract language of the theories of Planned Behavior and Reasoned Action, there’s a gap between behavioral intention and actual behavior. A plausible explanation for the gap is low self-efficacy, defined as the conviction that one can successfully execute the behavior required to produce desired outcomes. It’s the belief we have in our own abilities, specifically our ability to meet the challenges ahead of us and complete a task successfully. This is a crucial precondition because investigations have shown that peoples’ behavior is strongly influenced by their confidence in their ability to execute that behavior.

    If-Then-Else

    So why doubt your ability to change? Well, when a change seems large, the gut reaction is to feel the need to come up with a project plan. Because a plan creates the confidence that when the steps are taken, the desired results will be achieved. Without a plan, we’d be irresponsible risk-takers. That’s how we’ve been taught to think. And rightly so—at least, in a predictable environment, where you can determine which steps will have which effects. If you work with IT, you are likely to think in terms of If-Then-Else. This is how we would like to believe IT always works. Predictably.

    If-Then-Maybe

    However—and this might be the biggest “however” you’ve come across for a while—things are not always predictable. Not unpredictable in the sense that it should have worked but something just went wrong because we didn’t do enough homework; unpredictable as in simply unknowable. Unpredictable as in possibly seeing a pattern of behavior in hindsight, but never having been able to predict it, let alone control it. Unpredictable as in “complex adaptive systems,” as they are referred to in the world of complexity thinking. It’s no longer If-Then-Else; the name of the game is If-Then-Maybe.

    Complexity Thinking 

    Complex adaptive systems are all around us but if you’ve been used to thinking that things are—or could or should be—predictable, you tend not to see them for what they are: one of kinds of systems that exist in nature. The Cynefin sense-making framework describes systems in terms of five domains: obvious, complicated, complex, chaotic and disorder. Obvious and complicated are both predictable domains, where something in the complicated domain needs a bit of work to discover the predictability than when it is obvious. Complex and chaotic are unpredictable domains, the main difference being that something chaotic is dangerously unpredictable and therefore needs direct action. There is a fifth domain: disorder. You are in this domain when you don’t know which of the other four domains best describes the current situation. A word of warning: People’s cognitive bias will often lead them to think that they are in their usual domain, in which they feel most comfortable.

    Step by Step 

    The way to introduce changes in complex systems successfully is to take it step by step. Because you can’t predict the results of a step, you have to closely monitor what effects—both desired and less so—each step creates. Then, based on your assessment of the new situation, you determine which next step to take. Rather than moving in a straight line toward a “to be” end state, you move each step to the “adjacent possible.” You make good use of the disposition of the system; in other words, where the energy is tending to flow.

    No Big Answers, Just Little Questions

    So, it’s not a case of looking for the big answer. It’s about a series of small questions. The Toyota Kata approach recommends asking ourselves (and others) these questions:

    • What are we trying to achieve?
    • Where are we now?
    • What obstacle is now in our way?
    • What’s our next step, and what do we expect?
    • When can we see what we’ve learned from taking that step?

    A key concept is that we should work towards the next “target condition.” This is an interim goal on the way to whatever we want to achieve. It just describes where you want to be next, not how to get there. It labels the future set of circumstances that lie just beyond our current level of understanding. It will typically have to be achieved between a relatively short period between a week and a few months; otherwise, it will be ineffective. How we will get there will emerge though the process of experimentation.

    Stack the Cards in Your Favor 

    So the next time you feel inspired to make a (little) difference, don’t be discouraged by the prospect of having to think everything through in detail. This is often even contra-productive. The liberating realization that you can take things step by step, is good for your self-efficacy, increasing the chances of actually achieving results.

    This article was co-authored by Mike Orzen.

    — Mark Smalley

  • The Code Doesn’t Lie, and Other Operations Mantras

    The Code Doesn’t Lie, and Other Operations Mantras

    As engineers, we spend a lot of time talking about things such as release processes, QA environments and deployments. But at the end of the day, most software systems fail because the software itself is faulty. The code doesn’t lie—you can almost always find the solution to your problem within the code itself. Of course, looking to the code for answers is a time-consuming process if you’ve deployed too much in one batch. After all, sometimes the tiniest mistake can derail a whole string of code, and having to sort through a haystack of code looking for a tiny needle is not fun.

    There are a few corollaries to this mantra that “the code doesn’t lie”:

    • “Know your data like you know yourself.” Sometimes the quality of the data itself can cause a system to behave differently.
    • “A feature is a bug in a tuxedo,” which is largely for you product folks out there.
    • “Where there is one bug, there are many.” When you find a problem and it leads to a systemic set of problems, it may seem terrible at first, but the reality is, you have the opportunity to fix a lot in a short amount of time.

    I had a chemistry teacher who used to say, “A mole is a mole is a mole,” referring to Avogadro’s number. My version is: “A bug is a bug is a bug.” A feature that behaves incorrectly (even if built to spec) is a bug. A badly performing system is a bug. A transient failure is a bug. A race condition is a bug. No matter what they look like, it’s our job to squash the bugs.

    Branch Readiness is a Joke

    A very talented engineering director once made the comment, “Branch readiness is a joke,” about the branch/development model. As 23 different teams were preparing their branches for integration onto the main trunk, code changes were flying around with very little testing/certification. We even had code that would not compile.

    When all 23 branches hit the main release branch, all hell broke loose. And this is where the actual functional, integration and performance testing began. The key problem with this model is that integration and certification happened late in the game. The other problem was that we did not abide by our own criteria of what constituted branch readiness; instead, we were constantly softening the restrictions until bad code made its way into the system.

    We moved to a trunk-based model of development where check-ins are made directly to the trunk, and all code must compile and be pretested. Continuous integration and testing are run against the trunk all day and all night. We can ship from the trunk at any moment.

    Now you may be asking yourself, What does this have to do with operations? The answer is: everything. We can control what changes get to the site. We can certify what changes get to the site. We are assured at every moment of a minimum quality in the changes that do get to the site. These smaller changes give us the ability to move bits onto the site more frequently. We are not forced into large-scale, monolithic deployments, which inherently create more risk to the site itself, often with little ability to roll back.

    Learning from ‘Branch Readiness is a Joke’

    Just because the code doesn’t lie does not mean that you can always understand all of its implications. The key is limiting the amount of code being changed in any individual release to an amount an engineer or automated system can understand. By avoiding large, monolithic releases containing thousands of changes and instead testing, verifying and considering each change individually, we can understand what the code is telling us. We can release to the trunk with confidence that what we build and deploy is going to work.

    The One-Character Change

    We live in a world of bits and bytes (eight bits per byte). One byte is the equivalent of one character. In some cases, a one-character change is benign. For example, changing the number of members of a site on an informational page from 134M to 135M (4 to 5) will not cause any harm to the site.

    But sometimes, a one-character change can be disastrous. Consider a DNS change that has a one-character mistake for www.yourcompany.com. Get it wrong and you are off the air. It is very important to understand the impact of a change going awry. If it can cause a large impact, then we need to ensure we understand the change and have a clear plan to roll it back if we need to.

    Here’s a great example of a very small change going sideways very fast: We once scheduled a maintenance to test routing traffic through a European point-of-presence. Unfortunately, the time-to-live parameter was set to a large number (hours instead of minutes) on the DNS entry. The result was that the change, once implemented, couldn’t be fully undone for hours. It was a one-character change with a bad outcome.

    Learning from the One-Character Change

    The code doesn’t lie, but that only matters if we are paying attention. Put simply, not all changes are equal. By taking the time to think about what a change is intended to accomplish and the various ways it could go wrong, we will always be better off than if we didn’t do due diligence. The one-character change can be benign, or it can be catastrophic. It’s important to figure out the possible ramifications of any changes (no matter how small they may seem) before proceeding.

    Operations tends to be the first group to get called after hours when something goes wrong. As a result, we have a vested interest in quality of code shipping to our site. By working to improve the code quality before it hits production, you can significantly reduce the number of problems you encounter.

    Start with where the code gets committed into source control. For each change made ask the simple questions: Does the code still compile? Does your application still build correctly? Does it still work as intended? By automating the process of asking these questions for every change, we gain the ability to deploy new versions of our application based on any commit.

    Once we have good code quality making it into source control, we can begin making further improvements through the use of a canary process and start focusing on things such as performance. Remember, the code doesn’t lie. If we can understand each change, then we can start preventing problems before they begin.

    This post is part of the series “Every Day Is Monday in Operations.” Throughout this series we discuss our challenges, share our war stories and walk through the learning we’ve gained as Operations leaders. You can read the introduction and find links to the rest of the series here.

    About the Author / David Henke

    David Henke has more than 35 years of experience working in technology, including senior engineering leadership positions at LinkedIn, Yahoo!, and AltaVista Company. He’s also been a founder at two different software companies, both of which were acquired. Currently, David serves in a variety of board and advisory positions with organizations like NerdWallet and UC Santa Barbara.

  • An obituary to a Managed Test Service

    An obituary to a Managed Test Service

    1st October 2020, Enterpirseborough , F500 Inc street

    Managed Test Service, 66, died Friday, 3rd October, 2020 at the Delivery governance war room in the IT head quarters of F500 inc. following a terminal illness caused by a lifetime burden of 34 SLA’s, debt burden of 44 KPIs , 29 painful claw backs by procurement due to failed deliveries and overdose of turnaround specialists and consultants

    MTS leaves 4 programs , 45 projects, 12 application centres and a number of small changes behind.Born and raised in the HQ , MTS was a lifelong resident of the borough.

    In the early years, MTS would help do functional testing for the core system portfolio and eventually taking ownership of non-functional testing and user testing. After graduating from the delivery transformation high school , MTS attended the SLA University, graduating with a degree in cost optimisation and skill super-speciaisation. MTS eventually joined the IT Board, where he was the voice of quality(or so he believed)

    His attention to detail and obsession with centralisation of anything that had Test in it’s name was legendary. He was proud and preached the virtues of , albeit bloated and inflexible, test service

    However,he struggled with newer delivery approaches of Agile and DevOps which spread like cancer throughout the delivery organisation. Devops challenged the fundamental idea of skill super-specialisation and independent testing that MTS championed and built his empire on. DevOps challenged the way of living of MTS as unsustainable 

    DevOps valued people and interactions over process unlike MTS which obsessed about process

    DevOps valued working software while MTS was worried about comprehensive documentation

    MTS got caught up in time-wasting contract negotiations instead of allowing his team to focus on customer collaboration

    MTS preferred a predictable plan while his delivery friends had already moved on with adaptable plans

    A funeral service was held on Monday 6th October from the funeral Home of Procurement , with a private burial.

  • Lean for DevOps: Is it OK to move the banana?

    Lean for DevOps: Is it OK to move the banana?

    There is an urban legend about a Lean deployment which, in the process of applying the 5S to a workshop, drew lines on the desk to indicate where each worker should keep their banana.

    Katie 2 5 S deskKatie 3 5S desk 2

    5S, for the uninitiated, is a set of principles by which Lean practitioners organize a workspace – which might apply to a virtual, shared workspace, or Sharepoint repository just as well as a back office space or factory floor.

    • Sort – identify and get rid of unnecessary items
    • Set in order – organize what’s left so that everything has a logical place – including deciding on the optimum spot for banana placement
    • Shine – tidy up and clean your new workspace
    • Standardize – document and put boundaries around your newly organized workspace so that it’s obvious where everything should go – including drawing a line around that banana
    • Sustain – establish systems, tasks and behaviors to ensure your work place remains neat, organized and free from clutter

    Katie 1These are great principles to apply to any shared workspace – and standardization will speed your adoption of DevOps principles. Standardizing the way we organize information allows for fungibility, so a Quality Manager can switch teams at short notice and quickly pick up the way their new team works, and access the information they need. Standardizing storage in a way is the principle behind Docker, and other modular code concepts. Standardization can reduce errors: if user requirements are captured in a consistent way it is easier to see at a glance if any essentially information is missing before we start work. And standardization ensures consistency, for example, applying a pre-determined decision tree and criteria to enhancement requests will allow prioritization decisions to be made quickly and equitably.

    But you can have too much of a good thing, Creative activities and problem solving tasks are often intuitive or require people to take a variety of approaches until they find one that fits. A checklist just may not help here. Furthermore, one of the aims of DevOps is to increase flexibility and speed to market. So if your standard start to get in the way of that, you’ve lost your way.

    As a Lean Coach I have struggled with the amount of churn I see in requirements during the development lifecycle: at first glance it reads like rework. But it is, in fact, a healthy feature of an iterative, responsive process. Many IT departments have sought to manage churn by trying to standardize it out – setting deadlines for new intakes, and arduous change control procedures. As soon as you layer governance on top of a process, you create a need for a shortcut, and an expedited route.

    I recently ran a Lean workshop for an IT team during which we identified that 30% of change requests were going through an ’emergency’ process.  Clearly the standard process wasn’t working.

    DevOps requires standardization on an ‘as needed’ basis. This includes a standard way to handle rapid change, and variation. That standard approach must be light touch governance: make the right way the easiest way. And I’ll stick my neck out and say that standardizing where a worker keeps their snack is a step too far.