Tag: ci/cd devops

  • The Life and Times of Feature Flags

    The Life and Times of Feature Flags

    A while back, I had the opportunity to look closely at DevSecOps tools (as the market currently defines them). There was a lot to like, and they are moving forward to a “Find all of the vulnerabilities, all of the time” future … Except there will be new vulnerabilities that aren’t as easy to catch once we get there, and our journey will continue.

    More recently, I’ve had the pleasure of diving into application and API protection, which I can’t (actually, I’m not certain if I can or can’t, so won’t) say too much about at this time, but which are also moving forward in a similar direction, if not in a similar manner.

    And as a developer, feature flags have intrigued me since they first appeared.

    Sing the Sesame Street song with me: “One of these things is not like the others.” Feature flags are … a feature.

    When they first hit the scene, it was literally developers writing #if and #ifdef statements in their code to turn bits on and off at compile time. That reminds me of nothing so much as the Seinfeld episode where they’re explaining to network execs what the show is about. “A developer does his job … that’s a product!”

    The market has come a long way since then. But, honestly, I just don’t see these as stand-alone products. Take a look at the breadth of things most products are doing these days—CI/CD tools, either of the two markets I just mentioned or just about any other market in a similar space. Feature flags are still just about “turn this functionality off, turn it back on”. Newer iterations allow this without rebuilding the entire app. But it’s just not up there with the potpourri that current tools markets offer.

    My take is that these tools will end up as part of a larger toolset. Microsoft, Atlassian and Cloudbees (to name the ones I’ve seen without looking too hard) already implement feature flags, shrinking the available market. And they implement them because it’s a perfect fit with several markets. DevOps tools—which is where each of these vendors implements them—are one of the best market fits.

    Which market will end up with them? I have no idea, and that’s part of the fun as an analyst. The best on-the-surface fit would be DevOps toolchain systems like Atlassian offers, where no matter your toolchain, you can use them to speed acceptance testing or save a release without re-releasing. But I can make valid arguments for other locations too, so I guess you all will decide as options come available—decide with your dollars, that is.

    Watch for it. I’m not at all saying “Avoid feature flag companies because it’s not a real product!” I’m just saying that your feature flag vendor is likely to become part of a larger solution at some point, so make sure you have a plan to move off of it in case the larger solution is something your organization just doesn’t want to deal with.

    I am saying, “If you don’t have a feature flag solution—even a homegrown one—in place, it’s time to put one in place. The ability to say “Oh, yeah, that feature is mucking things up, let’s turn it off, quick!” is huge. For larger sub-projects that take multiple sprints, the ability to have the code right in the stream with regular check-ins, but feature-flagged out of the build, is also huge. No merging or other craziness, just “BuildNewFunction=FALSE” in the source or config file. The more advanced the feature flag system in place the better—the newest ones can turn things on and off with an environment variable from a dashboard. This makes feature flag code inclusion far easier and more intuitive than, “Did we account for that database update not happening?” of hand-rolled solutions. Still not perfect—because this is a journey—but store-bought is now better than anything you’re likely to hand-roll.

    And keep rocking it. Feature flags or not, you’re making systems that power the world—or at least your corner of it—and not getting the recognition you deserve. So once again, thank you. For all those people who don’t even know that they should thank you.

  • CI/CD Is Still All About Open Source

    CI/CD Is Still All About Open Source

    One of the lessons I’ve learned in my years at the helm of staging-devopsy.kinsta.cloud is that the heart of DevOps is CI/CD. And at the heart of CI/CD is open source. While the names have been changed to protect the innocent (and the obsolete), these lessons are as true today as they ever were. CI/CD is still all about open source.

    When we first started staging-devopsy.kinsta.cloud, folks said that DevOps was just a symptom; CI/CD was the cure. For many of you, that cure was delivered via open source—probably using Jenkins. Jenkins, which itself was a fork of the open source Hudson project, was created by Kohsuke Kawaguchi.

    Jenkins was a CI tool at heart and later morphed into a CI/CD tool. Many people think that this fork in the road may have hurt the continued evolution of continuous delivery in the long term. But that is an argument for another staging-devopsy.kinsta.cloud article (or maybe even a panel discussion at an upcoming DevOps live event).

    Regardless of where you stand on that issue, as an open source project, it is hard to argue with the success of Jenkins. Driving a lot of that success is the Jenkins plug-in architecture. There are literally thousands of plugins that allow Jenkins to work with just about anything. That is the engine that powered Jenkins, yes; but its secret superpower was and is open source.

    That said, Jenkins has grown a bit long in the tooth over the years. It’s not that it doesn’t do what it always did, it’s that what we do and how we do it has changed. Microservices, Kubernetes and even cloud have changed the very fabric of the tapestry in front of which Jenkins sits. The open source community that supports Jenkins should receive enormous credit here: It has tried mightily to keep up with the many changes. 

    Jenkins X sought to bring Jenkins to Kubernetes, for instance. But ultimately, nothing lasts forever and, as it says in the Bible, “For everything there is a season, and a time for every purpose under heaven: A time to be born, and a time to die.” Now, I am not saying Jenkins is dying, but I think its time as the leader of the pack is behind it.

    The good news is there are a host of new CI/CD tools that have risen to fill the vacuum left by Jenkins. Some of them are entirely open source; some are based on open source. Solutions such as Buildkite, Harness, Codefresh, GitLab, GitHub, CircleCI, JFrog and Tekton to name but a few. 

    Another major development has been the rise of GitOps. Weaveworks has led the way in adoption, not to mention coining of the term GitOps itself, but clearly, they are not alone. What these solutions all have in common, though, is open source.

    I used to think it was sort of a chicken-or-the-egg question when it came to CI/CD and open source. But I realize now that it is not. I firmly believe that, if not for open source, CI/CD would not be the force it is today. Make no mistake, it is truly a force. CI/CD has become the dominant method for software development.

    Of course, you could say “Why stop at CI/CD? Open source has become the dominant software development foundation in observability, OS, cloud and so much more!” Yes, I agree—we are living in the open source age. This rising tide may in fact lift all boats, including CI/CD.

    This wave is forcing all vendors to either go entirely open source or at least support open source. What’s more, for the foreseeable future, I don’t see that changing. Open source has also wrought other changes in the market. How software tools, including CI/CD, are distributed and consumed. Who is hiring enterprise software salespeople anymore? Doing large POCs? No, open source has resulted in getting software into the hands of developers and DevOps teams directly and delivering delight. If your product works, they buy it. If it doesn’t work, they won’t, either—and they will let you know, too!

    This meritocracy is another benefit of open source and is helping to speed the development of better, next-gen CI/CD tools in the market. And at the end of the day, we are all better off for that.

  • The Promise of AI for DevOps in 2021

    The Promise of AI for DevOps in 2021

    DevOps is a natural target for AI-driven efficiencies, as it involves frequently repeated processes that generate mountains of data. It seems reasonable to expect that, like other domains that require decisions to be made based on large volumes of data, AI will play an important role in DevOps, too.

    Definitions of AI vary considerably, so you can’t be blamed if you’ve sat through a discussion of AI and DevOps and still don’t understand exactly how the two intersect. But the bottom line is that AI will prove most useful in situations where there’s lots of data generated by, or passing through, a repeatable process. Humans are pretty good at identifying heuristics to help them make reasonable decisions based on patterns in data. But machine learning (ML) techniques bring with them the promise of teasing out inherent characteristics that underpin the data, and that are often impossible for humans to observe.

    There are two main points in the DevOps process where large amounts of data are generated, and where machine learning would be most useful when applied: testing and release and monitoring.

    1. Testing and Releasing

      Deployment packages: AI can analyze trends in what is actually being deployed. My company, Gearset, is a DevOps solution for development on the Salesforce platform. In that context, AI could track the items of Salesforce metadata a team usually deploys, and optimize requests for retrieving that metadata from its org based on those items, speeding up comparisons and deployments, and, ultimately, release cadence.
      Deployment errors: At Gearset, increasing deployment success rates is a constant focus. To that end, we aggregate and analyze data from deployment errors to identify the most common deployment-blocking errors. We then build problem analyzers that suggest automatic fixes to users. With machine learning running on telemetry data, we could more rapidly identify other patterns of use that trigger common deployment errors, and suggest new problem analyzers we should build, boosting deployment success rates still higher.
      Static code analysis: Static code analysis generates data about the quality and security of new code, judged against a ruleset. Within Gearset, Salesforce developers can choose and customize the PMD rules they want in their ruleset, so they’re warned about the code quality issues they care about. But richer insights would be possible with AI, such as high-priority areas for improvement and refactoring based on trends in code quality and security.

    2. Monitoring

      Infrastructure monitoring. Huge amounts of data can be generated from DevOps monitoring. It’s easier to know what to focus on with AI-driven insights into performance log monitoring. Pattern recognition can also predict usage growth and help with capacity planning.
      Security. This has been a key area for innovation in recent years, with the development of products that use machine learning to enhance threat detection, intrusion detection and vulnerability database compilation.
      Unit testing. Gearset provides tools for monitoring code in Salesforce, such as automated unit testing, which reveals when code is no longer executing as intended. AI could identify patterns in the logs and suggest areas of the codebase that appear to require attention.
      Change monitoring. Changes in the codebase, whether expected or unexpected, are tracked by change monitoring and backup jobs. Gearset currently has smart alerts on backup jobs that users can manually set up to be warned of changes or deletions that are unusual, both in terms of the kinds of records being deleted and how many. But AI could automatically spot anomalous and unusual patterns of churn that should be investigated.

    In summary, AI promises to accelerate innovation in the areas where we currently use data-driven insights to improve the performance of DevOps tools and processes. The result will be even higher deployment success rates leading to increased agility, better code quality and performance, and enhanced security.

  • The Chaos Mindset: Teaching Your Code to Cope

    The Chaos Mindset: Teaching Your Code to Cope

    Chaos engineering sounds alluring and exciting—it’s fun to experiment, right? But what some misunderstand about this approach is that it’s not about moving fast and breaking things. It’s about designing and introducing disruptions in the software production process that tests the resiliency of the code, much like crash testing in the automotive industry.

    If you think about it, this is the logical extension of the way developers like to think anyway: we’re designing software and systems for the real world. Shouldn’t they be able to handle real-world situations?

    To design effectively for the real world, developer teams don’t necessarily need to become chaos engineers. This isn’t about training for an entirely new set of skills. Rather, it’s about adopting a chaos mindset—a set of systems and perspectives that lets your team engage in controlled experiments to deliver products that can cope with whatever end-users throw at them.

    So, how do you know if your team is ready to adopt chaos engineering principles? And what should you do to help them?

    Chaos Engineering: The Basics

    This approach is all about using distributed systems as a “safe” experimentation environment, so that dev teams can ensure their products will withstand unexpected turbulence in production. Netflix engineers get the credit for coining the term, based on their experiences developing for Netflix on AWS.

    The fun part of chaos engineering is that it lets developers find loopholes in existing systems that allow them to develop for resiliency. By running experiments within existing platforms in a distributed environment, or within a sandbox development environment, teams can supervise and control the experiments without impacting end users.

    Some applications, such as transactional applications in finance or trading, should not be disrupted in production, and developers should test in a sandbox environment instead. Chaos engineering is very easy to put in place if your team uses containerization techniques already – which should be part of any modern DevOps CI/CD process anyway. It is common to setup a DevOps CI/CD pipeline to run a battery of tests on an application before it gets to the publish step. Within that testing step, you can easily add some disruptions; some may be harder to put in place and require a more elaborate environment.

    Dev teams can use existing environments and systems to show each other how unexpected issues could impact production of a software product, allowing them to collaborate and solve the coding mystery before it slows or stops production entirely.

    Applications designed by engineers are subject to common issues, whether the applications run on-premises or in the cloud. We can group these issues into the following categories: hardware limitations, sub system failures, external system failures and software bugs. While we could define more categories, these are the main ones, and the ones that are easy to test.

    Chaos Mindset: The Tao of Chaos

    Like Agile, chaos engineering is more than a set of activities and workflows—it’s also a state of mind. Your people and your culture must be ready and able to adopt chaos principles, as well as chaos processes.

    For the DevOps leader, adopting a new mindset might sound a little, well, vague. But this shift is based on concrete actions, not just philosophical musings.

    Consider an example from the world of cloud infrastructure: a mission-critical application that is hosted within a cloud service could be at risk for failure if, say, that cloud service is centralized in a single location, or within a limited number of microservices within the cloud infrastructure. But if the app is hosted in a distributed way, you can create greater opportunity for application-level availability and resilience, and you can test for that resilience within the existing production environment.

    This kind of distributed architecture isn’t brand-new for most enterprises, and, therefore, the process of developing applications in way that tests for availability in a variety of infrastructure scenarios also shouldn’t be a foreign concept. As a DevOps leader, you can build a culture of resilience-centric thinking by empowering your teams with the tools they need to adopt chaos-style testing, and then showing them how to build that thinking into every sprint and every standup.

    It takes work to train your teams, but you don’t have to do it alone. Netflix’s Simian Army has tons of tools and guidance; Facebook Storm and Amazon Gameday, both war room-type experiences to simulate failures in the cloud, also have helpful examples and ideas.

    Chaos Engineering in Real Life

    Deploying a chaos-inspired experiment strategy isn’t easy. Start with a simple set of operating principles as a guide for designing your chaos strategy:

    • Hypothesis: How we think the system should work, so we know when it doesn’t work.
    • Scenario planning: What are the possible failure events or crisis points that you know can happen in the real world?
    • Set up the experiments: Design and put in production the tests that measure the code’s strength.
    • Configure monitoring: Monitoring will help test the hypothesis and help you see, in real time, the impact and resolution of the disruption.
    • Make the experiment run itself: Automating the tests lets you continuously assess both performance and resilience.
    • Contain the experiment: You don’t want software that’s already out in the wild to get disrupted by your lab experiments, so make sure to design with guardrails in place.
    • Shut-off switch: Even when contained, have a shut-off switch to prevent the disruption from further impacting some users.

    The fundamental irony of chaos engineering is that it’s anything but chaotic. It’s a disciplined approach to breaking things that lets your team get smarter, faster, about how their products and applications will work in the real world. It’s a strategy that harmonizes beautifully with Agile practices. And even though it won’t make your apps bulletproof, it can definitely help you dodge the bullets of failure with greater grace and finesse.

    You must look at the software architecture as a whole to find the most appropriate chaos engineering testing tool for each portion of the architecture. There isn’t a single tool that can test all the components of your architecture; you will have to use different tools for different parts – and don’t forget monitoring! It’s only through proper monitoring that you can assess how the software is behaving in response to a disruption.

    Sometimes, the best approach, and the simplest one, is to build some chaos engineering steps into your software so that the software itself can be disrupted given specific inputs without exposing a security weakness in an obvious way. This is very much like how cloud applications create heartbeat and health check endpoints. We like to think about chaos engineering as a great complement to unit testing and integration testing, except we run it on production systems.

  • CI/CD Best Practices for Software Development

    CI/CD Best Practices for Software Development

    Continuous integration (CI) and continuous delivery (CD) are popular software development practices for automation and shortening feedback times. However, set up improperly, your CI/CD pipelines could instead cause delays in development. For that reason, review  CI/CD best practices to ensure that your pipelines are effective and efficient.

    What You Need to Know About Continuous Integration

    Continuous integration is the practice of automating the build and testing of code every time a change is made and committing that code back to a central repository. One of the fundamental cornerstones of continuous integration is that it encourages breaking up development tasks into small bite-sized pieces that can be performed frequently by every developer on the team.

    The Benefits of Continuous Integration

    There are four key benefits of continuous integration:

    • Easier Bug Fixes
    • Reduced Project Risk
    • Improved Software Quality
    • Higher Productivity

    What You Need to Know About Continuous Delivery

    Continuous delivery is a software development practice that enables continuous process and software improvement through automation. Without continuous delivery, you would have to manually develop, test, and deploy code—which can often take months. That is why continuous delivery is important as it can save you and your team a great deal of time.

    The Benefits of Continuous Delivery There are four key benefits of continuous delivery:

    1. Streamlining Workflows
    2. Lowering Staffing Costs
    3. Improving Operational Confidence
    4. Enhancing Teamwork

    Static Code Analysis Complements Continuous Integration and Continuous Deployment

    Static code analysis is a natural addition to any continuous integration development process. Done correctly, static code analysis adds the possibility for almost immediate feedback of new coding issues, specific to the branch or commit containing them.

    In addition, a static code analysis tool can provide your CI/CD pipeline with the following benefits:

    1. Detection of common security vulnerabilities, potential runtime errors, and other general coding errors.
    2. Compliance with safety-related coding standards, which includes MISRA and AUTOSAR.
    3. Enforce your coding guidelines or naming conventions along with your maintainability requirements.

    To read more, please visit: https://www.perforce.com/resources/kw/ci-cd-best-practices-software-development