Category: Doin’ DevOps

  • Engineering Practices Can Overcome DevOps Challenges

    Engineering Practices Can Overcome DevOps Challenges

    When I think about the challenges of DevOps transformations, I love the viewpoint expressed by Ryan Harmon, strategic accounts manager at Trace3: “DevOps is a complex maze that has many leaders frustrated. In many large enterprises, ‘DevOps’ is simply a buzzword they are striving to achieve but struggle with what the end state looks like or even where to start. There are many layers across the organization that are instrumental to truly executing on DevOps. DevOps is not something you go get a quote for and simply buy. It’s an evolving journey.”

    In my previous article, The Time Has Come for an Engineering Approach to DevOps, I agree with Ryan.

    Too many organizations launch into a DevOps transformation initiative without considering the full scope of what it takes to engineer a successful DevOps solution. DevOps is more than “just” a CI/CD pipeline or toolchain, as complex and challenging as this alone may be to accomplish, for even one application. DevOps is more than simply a culture change, as difficult as that always is.

    There are 24 DevOps practice categories that need to be mastered to realize and evolve well-engineered, high-performance enterprise DevOps solutions. Not all 24 need to be mastered to begin the DevOps transformation, but, ultimately, all of them do need to be mastered before the solution will achieve the high performance levels to which many enterprises aspire. The good news is, successful enterprises have demonstrated that the investment in people and technologies required for the journey is worthwhile, and a strategic approach will get you there systematically if you follow a disciplined engineering implementation roadmap.

    As explained in my book, Engineering DevOps, the core of every successful DevOps practice includes nine core pillars. These are the pillars to focus on as you start on your DevOps journey. The following paragraphs illustrate requirements for each of the nine pillars.

    Leadership – Leaders set an inspiring DevOps directional vision for the organization, and proactively stimulate and sponsor team activities that help progress toward goals.
    Collaborative culture – The organization encourages cross-functional collaboration, shared responsibilities and works to reduce restrictive boundaries between organizational functions.
    Design for DevOps – Products are architected to support service-oriented, modular, independent packaging, testing and releases, in accordance with the principles of twelve-factor apps.
    Continuous integration – Changes to software code and artifacts needed for releases are frequently merged, built, tested and packaged into executable artifacts for application releases.
    Continuous testing – Software tests that are most important to verify the results of pipeline stages including pre-flight, integration, regression, performance and release acceptance tests are automated as much as possible.
    Elastic infrastructure – Infrastructure-as-a-service (IaaS) resources (i.e., software, computing machines, storage and networks) are integrated into a toolchain for automating builds and testing environments to efficiently support variable workload demands.
    Continuous monitoring – Applications, pipeline tools and infrastructure components are instrumented with metrics, and the collected data is analyzed to manage health and performance.
    Continuous security (DevSecOps) – Security practices, processes and technologies for applications, databases, pipelines and infrastructures are integrated into the culture and pipelines to assure security.
    Continuous delivery – Release artifacts are validated, prepared for deployment to production and may be automatically deployed to production once acceptable deployment measures are achieved.

    A Comprehensive Engineering Blueprint

    While the above nine practice categories are the core of every successful DevOps solution, 14 additional practice categories are essential parts of a comprehensive engineering blueprint for accomplishing high-performance DevOps.

    DevOps infrastructure practices are used continuously by the nine core practice categories as follows:

    Version management systems – Every stage in the end-to-end DevOps value stream depends on keeping track of versions of software source code, configuration data and executable images using source code management systems, artifact repositories and configuration management systems.

    Infrastructure-as-a-Service – Virtualized cloud and dedicated resources must be served up ephemerally to meet variable workload and storage demands, which requires mastering engineering practices for orchestrating dedicated and cloud, hybrid cloud and multicloud services, and deployment of applications to the infrastructure.

    DevOps pipeline practices, which constitute a layer just above the nine core practices, include two additional categories that make use of the nine core DevOps practices.

    Application release automation – Directing and monitoring software changes through the CI/CD pipeline requires process orchestration and data analysis working at a level above the CI/CD pipeline itself.

    Value stream management – A deployed service, composed of multiple application microservices and data services, requires orchestration and analysis of a federated set of services, end-to-end, across the value stream from planning through to operations.

    Management Practices

    DevOps management practices include four categories that make use of the DevOps Pipeline practices, as follows:

    Service catalogs – Organized collections of software stacks, tools and pre-configured pipelines, made available through portals, enable DevOps-as-a-Service operations and simplify access, configuration, control and evolution of DevOps resources needed by enterprise application teams.

    Disaster prevention and recovery – Proactive disaster monitoring coupled with high-availability configurations and disaster mitigation practices, such as automated processes, work together to minimize the blast radius of disastrous events, and orchestrate rapid restoration of essential DevOps services, when needed.

    Site reliability engineering (SRE) – A combination of practices which include service level objectives (SLOs), error budgets, monitoring, work-sharing policies, toil reduction, proactive anti-fragility practices and blameless post-mortems help to “shift left” production wisdom from operations into development, thus improving the efficiency, readiness, security and resiliency of applications and infrastructures in production.

    Governance – The scale and scope of enterprise DevOps solutions are controlled using governance policies-as-code implemented through service catalogs, management dashboards and error budget policies.

    Project Practices

    DevOps project practices include four categories that guide enterprise adoption and evolution of DevOps, as follows:

    Adoption – Enterprise-wide DevOps adoption strategies that engineer and build DevOps capabilities in logical phases. This begins with a small set of representative applications that serve as a model for other applications; this method has been shown to be more successful than “boil the ocean” strategies that attempt to transform all applications at the same time.

    Model applications – Applications – that can serve as a model for greater enterprise adoption – are selected and monitored according to criterion that spotlight the value of DevOps for other applications.

    Return on investment (ROI) – DevOps investments must compete with other IT projects for resources. An ROI model that demonstrates measurable cost savings is an approach that has proved to achieve greater numbers of DevOps project approvals compared to other methods.
    Continuous improvement – Competitive forces and industry trends require that DevOps solutions continue to evolve to keep up with enterprise requirements.

    The Human Element

    Human development practices include two categories, as follows:

    DevOps engineering – The development of DevOps engineer roles, skills and the structure of DevOps engineering within an organization is critical to the success of designing, building, maintaining and evolving DevOps solutions.

    Continuous learning – A DevOps continuous learning strategy is critical for the development and transfer of DevOps knowledge needed to develop and continuously improve DevOps for the enterprise.

    In summary, the importance of practice categories – and specific practices within each of the categories – will evolve as each application does, alongside the changing needs of the enterprise. DevOps is complex, but a comprehensive engineering strategy that considers the 24 DevOps practice categories described in this article can help enterprises achieve the high-performance DevOps solutions they need to justify the investment and to meet business goals.

  • DevSecOps Implementation: Patch Management

    DevSecOps Implementation: Patch Management

    Patch management has been a thorn in IT’s side for decades. While it is undeniably necessary for security and stability purposes, the staggering array of patches that are available make managing them effectively nearly a full-time job – or a job better left to users. Few IT professionals want to be the “patch person,” and yet, nearly all IT professionals want their systems protected by the latest patches. No one wants to find out an intrusion was easily preventable via patching. It goes without saying that the SolarWinds attack has left quite a few IT departments scrambling for an alternate solution.

    The world of patch management has gone through several evolutions that we won’t cover in much detail here. Instead, we’ll focus on what’s available, what you should look for and where the market is headed. Suffice it to say that patch management as a practice has grown from an individual applying patches to servers and server apps while employees were encouraged to patch their equipment, to complex systems that manage the patching themselves, with IT oversight. Currently, patch management is often bundled with endpoint management, simply because many of the patches are going to end points (user machines and devices).

    Out of this growth have come several market differentiators that you should be aware of when looking into a way to manage the avalanche of updates the average organization is seeing.

    • Centralized or de-centralized? Increasingly, patch management is run on servers/cloud and pushes changes out to those machines/devices that need them, or agents on each machine manage updating themselves. Which is best for a given organization really varies. If a lot of machines are running at the edge of their capability, an agent running updates in the background might be too much to ask. In that scenario, a server-managed install environment is probably better. Conversely, if a centralized location to run updates from is considered an architectural weakness (because it requires access to the update pushing engine), agents might be the better plan.
    • OS support. We’re way past the days when single-OS solutions were considered useful. List the operating systems used in your environment, and choose a product accordingly. Most products on the market support Windows/Mac/Linux, but there are variances. First off, make certain your chosen product supports all three. Then ask, “Which flavors of Linux?” Make certain the flavors running in-house are supported by the software.
    • App support. There are a lot of applications out there. Which are supported by the tool? How much leeway does the customer have for adding applications by pointing the tool at update locations? Does it support FOSS updates? There are a lot of those too, and which ones matter. Make certain the tools the organization uses are supported or can be added.
    • Many tools will scan machines for update issues. Understand what your chosen tools scan for, and how the scans impact performance on the target machines.
    • Updates are not 100% perfect. There will inevitably be problems. Make certain the tool can roll back changes. Updating 50 core infrastructure machines, only to find, hours or days later, that the update breaks something demands roll-back capability.
    • Inter-relationships. Updating a piece of software can require that another piece of software or a driver be updated, which can demand that UEFI/BIOS be updated. Or the CMOS on a RAID controller. Understand how cascading updates are handled by the tool, and if the cascading updates can be packaged, so that a failure automatically rolls back the updates applied thus far.
    • Approvals. Speaking of driver and BIOS updates, what tools are available to allow a human to approve changes? No one wants willy-nilly updates to the drivers on core systems – or even to drivers on user systems.
    • Security of the tool itself. We’re in perilous times where IT security is concerned, and frankly, an update command-and-control server is an attacker’s dream machine – take it over, put your back-door update into the mix, and tell it to update every machine it manages. It cannot be understated: an update management tool must be as protected as possible – both by the customer locking down the installation and by the developer making sure the code is solid.

    Once the system is chosen and worked into your DevSecOps environment, updates will be more responsive, which leaves one more thing to consider – does it feed into your agile tools? If the update management tool notifies a project management tool like JIRA when it updates a server, then if code on the server breaks, the developers aren’t in the dark about changes, and will have breadcrumbs to help them track down the source of the problem.

    Imagine if manual updates became a rarity instead of the norm. There is a cost associated with this process, but most of the products on the market can be had for a reasonable price – if you waste a couple hours a month (or for some products, a year) on updating, they’re paying for themselves. Of course, your time is a sunk cost, and the cost of these is an operational cost, so there is an evaluation your organization will have to undertake. Given how busy everyone in DevOps and DevSecOps is, it is likely worth the investment for all but over-staffed organizations. What? There are a few!

    Find what suits your organization, and put it to use. Faster updates, less time investment. The idea is sound, you just need to figure out what suits your needs. And keep rocking it. You are managing updates already, this is just a way to make your life easier. And we could all use a bit of easier in a busy, complex IT environment.

  • Rapid Strategic SRE Assessments Accelerate IT Transformations

    Rapid Strategic SRE Assessments Accelerate IT Transformations

    A Strategic SRE Assessment is critical in helping organizations successfully transform traditional IT operations to SRE

    Site reliability engineering (SRE) is rapidly becoming more important to organizations that need to scale IT operations while maintaining or improving reliability and security, and to modernize operations to match the growth of DevOps in the organization. Without sufficiently thorough consideration given to best practices for people, processes and technologies, the transformation from traditional IT operations to SRE is unlikely to succeed. Despite challenges, the transformation to SRE is compelling because it provides an opportunity to reduce total operations costs and improve security and reliability while scaling services.

    To be successful, IT leaders need a comprehensive understanding of SRE best practices and need a well-engineered, yet affordable strategy and road map that effectively implements best practices and balances the many aspects needed to transform people, proces and technology. A Strategic SRE Assessment approach, based on best practices, is a key tool to help leaders understand, obtain the strategy and create a road map that will successfully transform traditional IT operations to SRE.

    There is no “standard” SRE assessment approach in the industry. This article enumerates a practical and proven Strategic SRE Assessment approach. The heart of the Strategic SRE Assessment approach is a survey of SRE best practices categorized under nine pillars: Culture, Work Sharing, Toil Reduction, SLAs/SLOs/SLIs, Measurements, Anti-Fragility, Deployments, Performance Management and Incidents Management. The survey is used to collect importance and capability scores for each of the SRE practices. As part of the assessment, a survey determines the current state and preferences of SRE practices for an organization for each of the nine SRE pillars.

    The following are examples of best practices for each pillar that are part of the Strategic SRE assessment survey:

    Culture – Leaders need to understand and sponsor a clear vision for SRE.

    • The culture embraces working practices to proactively shifts left the “Wisdom of Production.”
    • The culture encourages learning from failure and continuous improvement.
    • The SRE role is clearly defined and understood and SREs are considered a vital part of the organization.
    • Rules of engagement and principles for how SRE teams interact with their environment are codified—not only the production environment but also the product development teams, testing teams, users and so on. Those rules and work practices help to maintain SRE focus on engineering work as opposed to operations work.

    Work Sharing – Successful SRE requires that work be shared between SRE teams and development.

    • SRE teams are composed of software developers with strong operations knowledge or IT operations people with strong software development skills.
    • SRE workloads are managed to a budget with typically 50% or less spent on operations work.
    • SRE on-call work budgets are typically 25% or less. Developers share on-call work.
    • Technical debt is worked in small increments.

    Toil Reduction – SREs work to reduce non-value-added work through automation and standards.

    • Work standardization and automation—of both tools and processes—is employed to improve scalability, repeatability and other important goals.
    • Work standardization is used to reduce time spent on non-value-added work.
    • Automation is used to reduce time spent on non-value-added work.

    SLAs/SLOs/SLIs – Service level metrics, based on user perspectives, combined with error budget policies, are critical to managing flow and reliability.

    • SLOs are selected to measure availability, performance and other metrics in terms that matter to an end user.
    • An error budget (= 1-availability) internal service level objective (SLO) is established for each service. SLOs are set to be more stringent than external service level agreements (SLAs) promised to customers. The SLO budget considers what level users will accept, alternatives for users who are dissatisfied, and usage patterns of the product at different availability levels.
    • An error budget policy prescribes consequences if an error budget is spent. For example, the service freezes changes (except urgent security and bug fixes addressing any cause of the increased errors) until either the service has earned back room in the budget or the count period resets.

    Measurements – Smart monitoring implements observability to accelerate response times to incidents.

    • Monitoring solutions are designed to produce service level indicators (SLIs) from which SLOs can be calculated automatically.
    • SLIs do not require a human to interpret an alerting domain. Instead, software observability algorithms do the interpreting, and humans are notified only when they need to act.
    • Monitoring systems use telemetry and instrumentation to ensure the data being monitored is accurate, relevant and available in real-time.

    Anti-Fragility – Systems of people, processes and technologies are continuously tested and improved to assure they are resilient enough to serve applications as they scale.

    • Anti-fragility strategies are practiced to proactively assure the resilience of applications, infrastructures and pipeline services.
    • Fire drills are periodically conducted to proactively determine weaknesses in systems of people, processes and technologies that could potentially impact SLOs.
    • Chaos monkey or equivalent infrastructure testing strategies are used to proactively determine weaknesses in infrastructure systems that could potentially impact SLOs.
    • Security testing and DevSecOps practices reduce the risk of security vulnerabilities.

    Deployments – Deployment strategies reduce the risk of deployment failures.

    • Non-emergency deployments use progressive rollouts in which changes are applied to small fractions of traffic and capacity at one time. If unexpected behavior is detected, the changes are rolled back first and diagnosed afterward to minimize mean time to recovery.
    • Automation is used to implementing progressive rollouts, detect problems and implement rollback changes safely when problems arise, because outages are primarily caused by changes in a live system.
    • Deployment strategies such as blue-green, feature flag rollouts, A/B testing and canary rollouts are used to reduce the blast radius of failed deployments.

    Performance Monitoring – Proactive testing and monitoring ensure applications and infrastructures have the capacity necessary to flex and scale when needed.

    • The SRE team is responsible for provisioning and monitoring utilization because utilization is a function of how a given service works and how it is provisioned.
    • The SRE team is responsible for capacity planning and provisioning. Regular load testing of the system is used to correlate raw capacity (servers, disks and so on) to service capacity.
    • Provision to handle a simultaneous planned and unplanned outage without making the user experience unacceptable. This results in an “N + 2” configuration, in which peak traffic can be handled by N instances (possibly in degraded mode) while two instances are unavailable.

    Incident Management – Incidents are managed in an efficient, blameless manner with a goal to use incidents to improve responses.

    • Playbooks record the best practices for human response actions because playbooks produce roughly a 3x improvement in mean time to recovery.
    • Emergency response performance is measured using mean time to repair (MTTR) metrics instead of mean time to failure (MTTF) because the most relevant metric in evaluating the effectiveness of emergency response is how quickly the response team can bring the system back to health—that is, the MTTR.
    • Postmortems are blameless and focus on process and technology, not people.
    • Postmortems are conducted for significant incidents regardless of whether they resulted in a page; postmortems that did not trigger a page are even more valuable, as they likely point to clear monitoring gaps. Find all root causes of the event and assign actions to correct the problem or improve how it is addressed next time.

    SRE Assessment Surveys Are Not Enough

    The survey alone is an essential tool to collect information about the current state and preferences for each of the SRE practices. However, this is only part of the comprehensive Strategic SRE Assessment approach.

    The Strategic SRE Assessment approach starts with workshops to get alignment between leaders and teams regarding big-picture goals prior to conducting the survey. This is necessary so that everyone who answers the survey has a common perspective and can better respond with considered views of importance and capability scores for each of the practices. The results of the survey include an analysis that points out specific pillars and individual practices that have a high gap score in which the practices are scored as important but not yet practiced well. The high-priority gaps are then used to organize a value-stream map working session, that is designed to find the highest priority bottlenecks in people, processes and technology bottlenecks in the value steam, relative to the organization’s goals and gaps. With the goals, gaps, and data from the value stream map in hand, the next step is to formulate an implementation road map that directly addresses the goals, gaps and bottlenecks in the value stream.

    The final step is to obtain strategic alignment of the big-picture plan by reviewing the findings and road map with the leadership and implementation teams. This entire Strategic SRE Assessment approach, from goal alignment through surveys, value stream mapping, road map creation and strategic alignment, varies depending on several factors such as the size of the organization, number of applications and size of the deployment domains to be included in the assessment.

    A Rapid Strategic SRE Assessment for a sample set of model applications typically is conducted in 21 days. More complex Strategic SRE assessments involving many applications may take up to 90 days.

    In most cases the Rapid Strategic SRE Assessment is the best place to start. Most often, it is not a good idea and is too expensive to “boil the ocean” and make changes across all applications and deployment domains at once. By starting with a small number of model applications, it is easier to implement, measure and demonstrate success and use the results at each step to justify ongoing investments needed to complete the transformation across the organization. Another advantage of the Rapid Strategic SRE Assessment is that it is a quick and affordable approach to check-point progress as SRE is implemented progressively across applications and deployment domains.

    Key Takeaways for SRE Transformation Leaders

    SRE offers immense value for modernizing and scaling IT operations reliably and securely and to complement DevOps transformation. Adherence to SRE best practices is essential to reduce risk and assure security as the transformation is implemented.

    Each organization is different and has different requirements and priorities for its applications and deployment domains. The specific SRE practices that are the highest priority vary according to the priorities of applications and deployment domains. The Rapid Strategic SRE assessment approach, based on nine pillars of SRE best practices, is quick, effective and affordable.

    The Rapid Strategic SRE Assessment is a preferred approach to kick-start the SRE transformation. This approach is also a preferred and affordable tool to reassess the direction as SRE is implemented progressively across the organization.

    Summary

    There is no “standard” SRE assessment approach in the industry. The Rapid Strategic SRE Assessment approach described in this article has been tried and proven with large and smaller organizations. The author invites your comments regarding this approach and the SRE best practices.

  • Design Systems: Development Choices for Meeting User Expectations

    Design Systems: Development Choices for Meeting User Expectations

    Modern software development trends involve design systems and modernization, and if we drill down further, the trend within the trend is about creating better user experiences. This means improving usability, emerging technology and performance enhancements. This is easier said than done. What does it mean for developers?

    Employing Design Systems

    What a designer sketches, a developer tries to replicate, often with varying degrees of success. When we introduce multiple screen sizes and form factors, the experience can vary greatly if the collaboration between design and development is not in sync. To combat this, many modern software development practices are beginning to employ the use of design systems to provide an outstanding experience to the end user and streamline processes internally.

    Typically, a design system consists of UI components and visual style guidelines, both in the form of reusable code and reusable design artifacts. The system could include UI patterns, foundational elements and design principles or guidelines. Many go as far as defining what tools should be used for optimal UI and UX. For example, one of the most widely known design systems—and most used, due to its open source nature—is Google’s Material Design.

    With a design system in place, developers and designers are singing from the same sheet of music. This eliminates many delays and inconsistencies and provides prescriptive guidance for both the visual and functional experience of the app. It allows developers to deliver the beautifully styled, feature-rich responsive experiences the end user has come to expect.

    Choices for Modernization

    For .NET developers, this could mean exploring Blazor, Microsoft’s answer to JavaScript, which allows developers to build interactive web UIs using C#. It could mean diving into.NET 5, which is the next step in building a unified platform, allowing developers to use it to target Windows, Linux, macOS, iOS, Android, watchOS, WebAssembly and more. Or, it could mean getting a jump on .NET MAUI, the welcome evolution of Xamarin.Forms, combining the best of Mono and .NET Core and allowing developers to build truly native desktop and mobile apps for any operating system from a single codebase and project system from their preferred IDE.

    For the JavaScript developer, modernization could mean looking toward Vue 3, which is a faster, smaller and more maintainable framework. Or, it could mean exploring Angular 10 and its warnings for CommonJS imports. Or maybe it’s truly understanding and appreciating the downplayed release of React 17, which now allows multiple versions of React on one page, making it easier to use React with other frameworks. Modernization is all about staying relevant, learning and leveraging the latest the industry can offer.

    Popular Frameworks

    For front-end development, React, Angular and Blazor are some of the most popular frameworks at the moment. Blazor is so new enough that you aren’t going to find anyone with years of experience, but it is a framework that is incredibly adaptable and easy to learn if you are C# developer.

    React and Angular are more mature and with their latest releases provide even more compelling reasons to build your web apps with them. The soft skills—the ability to communicate and collaborate—are some of the most in-demand skills today.

    Regardless of your front-end preferences, the proliferation of frameworks and new versions means that developers have more choices than ever for creating rich, interactive, modern experiences for multiple platforms and devices.

  • Winning DevOps in the Finance Industry

    Winning DevOps in the Finance Industry

    DevOps is making its impact on almost every software-powered company these days. Companies in the finance domain, however, have been perhaps the slowest to embrace DevOps because of concerns regarding governance, security, compliance and regulatory regulations. But it is safe to say that they are currently leading the digital innovation in this era of cloud-native technology.

    Why DevOps for the Finance Industry?

    Higher customer engagement and continuous transactions make the finance industry one of the busiest and naturally requires uninterrupted infrastructure. DevOps implementation helps companies develop fast, fail fast and learn fast so they can fulfill customer expectations and deliver features faster to market than their competitors. Technology in the finance sector has not evolved much, as many companies have assumed it is too risky to embrace DevOps because of the amount of legacy code and old methodologies still in practice. As a result, digital innovation has lagged.

    Challenges of Implementing DevOps in Finance

    Unlike other industries, regulated industries imply several challenges:

    • Strong restrictions on secured networks.
    • Fine-grained audit trails.
    • Strong ACLs models.
    • Full lifecycle governance.
    • Integration with third parties.

    Security and compliance are top concerns in financial companies, and DevOps practices are, in fact, viewed as risk factors to security. Plus, the increased frequency of software releases in DevOps is seen as a threat to governance and regulatory controls.

    Is the Needle Moving Toward DevOps?

    According to the “2020 State of Database DevOps report“ by Redgate, financial services organizations report the highest levels of database DevOps adoption this year, with proportionally higher rates than any other sector. That shows financial sector maturity in terms of digital innovation. 

    Image source credit: Redgate

    Case Studies of Financial Companies and Their DevOps Journey

    Barclays DevOps Adoption

    In 2015, Barclays announced it was adopting DevOps for its digital transformation journey. Barclays processes payments that equate to around 30% of the UK’s gross domestic product. The developers’ morale and quality of code increased because of DevOps and the leadership team at Barclays credited DevOps with significantly reducing the complexity of its codes, which has allowed the company to reduce delivery risk and ultimately, improve the quality of services.

    DevOps the Lunar Way

    Lunar Way’s journey shows you don’t have to be big to use Kubernetes. The company started its cloud-native DevOps journey by splitting its massive monolith application into smaller microservices. To spin up these microservices, Lunar Way used Ansible, Terraform and Jenkins and to deploy these microservices as a whole unit. 

    Then Lunar Way started to experience scaling issues with microservices and didn’t experience any of the microservices benefits. 

    The company started looking for ways to move past this complexity by shifting its focus from machine-oriented to application-oriented architecture. It chose Kubernetes as the abstraction layer along with AWS, not worrying about where the containers are running, and this is how it was able to manage and unlock the velocity of microservices. Lunar Way also chose Kubernetes from a security perspective and to specify how the applications should run. Now the company runs around 80+ microservices in production with the help of Kubernetes. Watch and learn how Lunar Way did it: “Two years in production with Kubernetes.”

    Italy’s Biggest Bank Embraces DevOps

    A conventional bank running its real business on such a young technology? No way, are you kidding me? Nope, I am not kidding. 

    Italy’s banking group, Intesa Sanpaolo, has made this transition. Banks still run their ATM networks on 30-year-old mainframe technology, so embracing the hottest trend and tech is nearly unbelievable. 

    Even though ING, the banking and financial corporation, changed the way the banks were seen by upgrading itself with Kubernetes and DevOps practices very early in the game, there was still a stigma attached to adopting Kubernetes in highly regulated and controlled environments including health care and financial services. The bank’s engineering team came up with an initiative strategy in 2018 to throw away the old way of thinking and started embracing microservices, container architecture and migration from monolithic to multi-tier applications. Today, the bank runs more than 3,000 applications. Of those, more than 120 are now running in production using the new microservices architecture, including two of the 10 most business-critical for the bank. 

    HSBC Going the DevOps Way

    HSBC is changing the way typical banking works with cloud-native technology and DevOps, as an early user of Google’s Cloud Services Platform to provide core banking services to its business customers. 

    HSBC plans to build its all-new business banking service to run on a Kubernetes-managed container infrastructure using Google’s toolset. HSBC also has excellent partnerships with AWS and Microsoft, favoring the multi-cloud strategy. The company’s engineering team believes that moving to a container model under the umbrella of Kubernetes is significant because it means the environment is similar across different clouds. HSBC also hired people from outside of the banking sector because it wanted to change its culture from within. HSBC is said to have invested $10 million so far in its DevOps initiative.

    Capital One’s Agile to DevOps Journey

    Capital One’s Agile journey started in late 2011, with just two teams, which expanded slowly as more teams were trained in Agile development. The developers at Capital One were following the Scaled Agile Framework (SAFe). Initially, since the teams were new to the automation methodologies, integration testing, security testing, unit and performance testing were all done outside of development sprints by separate test teams. Later, once they understood how important it is to have a collaborative culture, they integrated testing into the dedicated DevOps teams and automated it well. Then slowly, they moved all testing into development sprints, adopting a culture of DevOps and uniformity and wiring integration, security and performance testing into a continuous delivery pipeline. As of 2016, the company had more than 700 Agile teams following continuous delivery.

    DevOps at Bank of America

    Bank of America is one of the world’s biggest financial institutions following DevOps practices, and its digital enterprise operations support nearly 4,300 retail financial centers and approximately 16,600 ATMs worldwide. The award-winning digital bank has nearly 38 million active users and approximately 29 million mobile users.

    Automating Security and Compliance

    As DevOps continues to evolve and become mainstream, it will continue to be of interest for many senior managers across financial services. The above case studies show they are already making DevOps a priority. But security and compliance remain barriers slowing the innovation in the financial companies. 

    Organizations should start small and keep security high priority, first using continuous integration and then adding continuous delivery and deployment using the available tools in the market. Over time, the more a company automates, the higher return on investment it can achieve.

    Automating security permissions and controls will help financial companies remove some of their most common software barriers and get their releases out quicker than before, while still maintaining the necessary governance and compliance. 

    Take a Step Forward

    Financial services companies need complex and rigorous software development processes to meet intense corporate and regulatory requirements. Yet, they are expected to release software faster than ever to meet expectations and beat the competition. By starting small and keeping security and compliance in mind, DevOps in finance can be achieved with great success.

  • Maintaining Progressive Delivery Quality With Feature Flags

    Maintaining Progressive Delivery Quality With Feature Flags

    How does Netflix know exactly what you want to watch? How do social networks ensure major software updates are smoothly deployed across millions of users? The answer, in part, lies in progressive delivery. Progressive delivery, the incremental introduction of new features, has become key to rolling out new features. By testing unproven features on a small subset of users, platforms can advance quickly while keeping rapport with the main userbase. But, where does QA fit into the progressive delivery cycle?

    As software development teams face the need to iterate rapidly, the role of QA engineers is evolving as well. QA teams are increasingly assigned to support new customer experience initiatives and meet developer consumer needs. They must do so while understanding which customer metrics to monitor as developers introduce new features.

    I recently chatted with Jeff Sing, lead software QA engineer at Optimizely, recently acquired by Episerver, to explore how feature flags could be used to maintain quality and respond to user feedback throughout a progressive delivery life cycle.

    Defining Progressive Delivery and Feature Flags

    Progressive delivery is is a software release style that emphasizes very early code pushes and rapid feedback. By testing features among small subsets of a greater user base, developers can quickly gauge what works and what doesn’t.

    Feature flags are a necessary component to make progressive delivery work. Also called feature toggles, feature flags are switches that enable developers to turn a feature off or on, helping engineering teams test unfinished features or alpha features while avoiding maintaining multiple branches. “Feature Flags are a powerful technique, allowing teams to modify system behavior without changing code,” wrote Pete Hodgson in a post on MartinFowler.com.

    Feature Flags Enable Experimentation

    Adopting feature flags for progressive delivery allows for a lot of experimentation, said Sing. For example, companies could test a new user interface adjustment on a small subset of users. If it works well, the team could further and optimize it for specific user tastes.

    “Why would you want to build something that no one wants to use? How do you know you’re building the right thing?” asked Sing.

    Iterative release strategies are nothing new—and they go by other names: canary release, A/B testing, targeted rollouts and feature gating. However, what seems unique about feature flags is the ability to optimize granularly.

    For example, if you compare a friend’s Amazon home page with your own, you will likely find the dashboard, product recommendations and overall UI vastly different. These companies are continually “running active experiments on a per-user basis,” said Sing. This enables them to personalize experiences to niche user tastes, in real-time.

    Many other companies are utilizing progressive delivery tactics. Blue Apron, for example, adopts experimentation to find and refine business value. Experimentation is core to how HelloFresh distills its customer experiences. Compass also uses progressive delivery tooling to enable feature flagging within a suite of apps serving various platforms.

    For QA teams, this means shifting right—from business testing to nano-testing at the pace of constant delivery. Thus, QA teams must have an active role—“continually test customer feedback and continually reiterate on that feature,” said Sing.

    Implementing Feature Flags: QA Shifts Right

    By adding customer feedback into the continuous delivery loop, feature flags aid incremental deployment strategies and become essential for DevOps. So how do we implement feature flags in our application development processes?

    If we’re testing a single function within a small product, an obvious idea would be to comment out or uncomment code. But, that doesn’t scale well. Developers could, instead, program a conditional variable. However, there is still more dynamic feature-toggling options. More advanced solutions may involve a toggle router and/or a UI-driven alternative.

    Sing described a system using switches written as YAML files and adopting a visual dashboard to toggle on/off features. This setup can enable a process whereby the developer checks in code and passes it to site reliability engineers (SREs) or QA managers. They can then initiate testing in production, view data from beta users in real-time and respond accordingly.

    Integrating with a CI/CD pipeline would be the next logical step. Each build could carry many potential features within each build. If a flag in the code is on, the system would execute all code and components can easily be turned off or on remotely. This way, engineers “can control how things are represented in real-time without a code deploy,” said Sing.

    Melding feature flags and progressive delivery returns extremely valuable data, Sing said. If teams can see metrics such as system behavior, bounce rate, customer NPS score and business conversions, they can better increase the quality of future deployments. “You can’t argue with hard data,” he noted.

    Benefits of Feature Flags

    Utilizing feature flags within continuous development appears to afford many benefits. In the process, QA plays a more present role in how engineering teams roll out (or roll back) new features. “QA must understand the business impact and customer story,” said Sing. “It’s not just about uptime; it’s the experience, too.”

    As you can see from the use cases above, progressive delivery with feature flags brings many benefits:

    • Quick validation: Bring new product features and proof of concepts faster to market faster by improving testing.
    • Experiment and iterate: Release incremental software versions. Experiment with new or unfinished features that consider fringe cases.
    • Development: Avoid constant merging of branches.
    • Reflect customer journey: Gather more heuristics on the customer journey to test and measure customer data.
    • Optimize per-user: Test and optimize for user tastes. “Why would you want to build something that no one wants to use?” For Sing, targeted rollouts can answer that user imperative immediately.
    • Limit blast radius: If new features introduce outages, easy togglable capabilities mean they can quickly be turned off and put back into development, thus increasing service reliability.

    Drawbacks of Feature Flags

    Feature flags introduce many efficient methods for testing new features, but they don’t come without side effects in other areas. One potential drawback of feature flags is supporting a vast number of intricately defined user scenarios.

    With each segmentation comes new optimization requirements and support must adapt to a quickly growing archive of various feature sets. Companies need advanced tooling to support the easy addition or removal of features. “You need the power to understand what’s going out and how you manage it,” Sing noted.

    A fractured ecosystem with thousands of microservices can become difficult to support continuously. Engineering teams must draw the line somewhere between endless optimization and retaining a cohesive product. Ideally, developers can reuse base components but represent them in different ways.

    Another negative side effect is public opinion. With the release of Social Dilemma and related tech industry critique, many user-facing tech networks are under harsh criticism due to the unethical use of user data to develop hyper-targeted ad campaigns. Optimization tactics that disrespect users should be frowned upon.

    This is not to mention regulations around user data and privacy. If users do not grant consent to share user data to “improve the user experience,” engineering teams won’t have the necessary metrics and tracking data required to decide which features to implement.

    While feature flags appear to help optimize business value, there are many potential “gotchas” to carefully consider before implementation, from technical logistics to service fragmentation, user ethics and legal ramifications.

    Final Thoughts

    “As with any new technology, if you don’t use it correctly, it’s almost better if you don’t use it,” said Sing. If progressive delivery isn’t managed correctly, with a QA mindset, it could come back to bite them.

    As we continually build new tech to ship faster and smarter, it’s also important to consider customer needs. “Automation is great, but a computer can’t understand what a customer really wants,” Sing noted. To avoid the technical hurdles and social dangers of hyper-personalization, QA should be customer champions that consider customer happiness.

  • How to Scale Microservices CI/CD Pipelines

    How to Scale Microservices CI/CD Pipelines

    As opposed to a traditional monolith, microservices architecture can theoretically bring many benefits. Microservices decouple software elements, enable reusable components and allow independent development cycles. However, in practice, microservices are prone to many issues.

    For example, teams accustomed to a traditional monolithic build process may find it difficult to reshape their established practices. Furthermore, as the number of microservices increases into the hundreds, managing separate CI/CD pipelines for each service becomes untenable, especially as DevOps components aren’t always optimized to fit every pipeline precisely.

    These are some of the real-world problems that Dan Garfield, chief technology evangelist at Codefresh, brought up in episode five of the DevNetwork Dev Professional Series. In this article, we follow his talk to examine the benefits of microservices, their hangups in a real-world CI/CD case study and solutions for scaling delivery as complexity and numbers increase.

    Monoliths in Space

    To get a better idea of why we need microservices, Garfield uses an analogy from space—the Apollo 13 mission. Apollo 13 was made of two ships: the lunar module and the command module. In software terms, these were essentially two giant monoliths.

    When the command module experienced an oxygen failure, its crew needed to repurpose oxygen filters from the lunar module. However, the lunar module engineers designed their oxygen filters differently, causing a major headache to reassemble and retrofit these parts.

    This analogy goes to show how monolithic-style development can create tunnel vision. When teams adopt guidelines that work only for the monolith, they are usually incompatible externally. This stifles the possibility of reusable components, thus decreasing code value and extending overall development effort.

    “Being able to reuse components and scale independently is the benefit of microservices,” said Garfield.

    Expedia’s Microservices Journey

    To see real-world microservices architecture in action, take Expedia. Previously, Expedia services had existed within monoliths, which meant much duplication. Services were detrimentally tethered, meaning if a service broke you had to fix the entire monolith.

    Garfield describes how Expedia isolated its architecture into individual microservices to represent abilities such as car search, car sorting, bookings, airline search and others. While this aided elasticity and usability, the microservices journey came with issues.

    Problems in Microservice Heaven

    Expedia ran into multiple problems within their microservices journey. Though Expedia had valid business reasons to adopt microservices, “they didn’t have a plan for how to scale their CI/CD that was realistic,” expressed Garfield.

    Foremost, as the number of microservices expanded into the hundreds and thousands, standardizing CI/CD became quite difficult. The organization was using one CI/CD pipeline per service, meaning they now had to support thousands of pipelines, each connected to an individual Git repository. A microservice may also have different pipelines for production, internal use or testing environments, leading to even more duplication.

    When you scale microservices, you also encounter a new set of roadblocks related to network and storage. This became extremely evident when Expedia realized the sheer network bandwidth and related costs from running validation and integration testing on thousands of separate CI/CD pipelines.

    “The scale of automation itself is where most people have the biggest bottlenecks,” said Garfield.

    Consolidating codebases with plugins and shared libraries became a common issue. Expedia developers experienced issues with Jenkins and were forced to programmatically or manually edit each pipeline, copy and pasting code from one to the next. In short, the flow was not easy, and differing approaches between teams led to shadow IT.

    Solutions For Scaling Microservices

    To help ease large-scale microservice environments and CI/CD pipelines, Garfield recommended taking a few approaches:

    • Use a container-based pipeline: The first principle Garfield suggested is using a container-based pipeline. If a pipeline is containerized, you can run it independently with different language versions. According to Garfield, “shared libraries are not the solution,” as they create version-specific conflicts. Using a Docker image is better than shared libraries because users can self-serve images with whatever version they want.

    There are many resources on Docker Hub to ease common CI/CD steps such as building a Docker image, git clone, running unit tests, linting projects and security scanning. Using these images can help avoid wasted development effort.

    • Consolidate to a single pipeline that operates with context: Instead of managing thousands of pipelines for all your microservices, Garfield suggested you use a single malleable pipeline for all integrations and deployments. His method involves Triggers, which carry information to direct actions.

    In this setup, Triggers carry context, or metadata, allowing a pipeline to change its behavior accordingly. These Triggers could be time-based or centered around actions like a repo Git commit, or pushing a new image. When a trigger is initiated, the trigger brings in relevant dependencies to perform an action, such as cloning a repository or cloning a codebase. Further steps can then consume these unique criteria.

    Garfield demonstrated how this is accomplished in practice within the Codefresh CI platform with native Kubernetes integrations. The platform also enables you to filter and view by certain criteria; a window into commits and changes.

    Garfield acknowledged this overall strategy does mean making microservices a bit more uniform, but perhaps the ease of deployment could outweigh the development constraints.

    • Adopt canary release testing: If an organization has hundreds or thousands of microservices, spinning up each microservice to run independent tests can quickly become cost-prohibitive. As Garfield noted, “testing early becomes less useful as infrastructure complexity arises.”

    To solve this issue, teams can adopt a canary release strategy. Canary release is a practice in which new software is first released and tested among a small subgroup of users. It’s a good way to “limit the blast radius of a change that goes wrong,” said Garfield.

    Lasting Image: Pipeline Reusability

    Adopting a single container-based CI pipeline could help resolve the expanding set of issues with large-scale microservices testing and deployment. By templatizing pipelines, many microservices could abstract complexity and reuse the same modular components. Essentially, make CI/CD reusable in the same way microservices are.

  • How to Secure Your Kubernetes Cluster on GKE

    How to Secure Your Kubernetes Cluster on GKE

    Google Kubernetes Engine (GKE) is easy to get going with, but requires additional security controls. The documentation is hard to grasp as there are many features and changes that tie to specific Kubernetes versions that will require using beta feature enablement, as well as some out of date documentation that can catch people out.

    There is also an outstanding bug that we have raised on Kubernetes that is hit by GKE. This is an edge case situation if you end up running pods directly and not deployments, and only happens if you enable pod security policies (which we recommend doing for security reasons).

    If you are looking at production and have sensitive workloads, we advise that you implement everything inside of this article.

    Securing Your Kubernetes Infrastructure

    When you deploy a default GKE cluster with no additional options provided, you’ll get some sensible security defaults:

    1. Basic authentication is disabled.
    2. Encryption at rest for the Operating System (OS) image is enabled.
    3. Client certificate issuance is disabled.
    4. Secure Operating system (Using Container-Optimized Operating System; COS).
    5. VM Integrity monitoring to protect from malware/rootkits.
    6. Restricted Service Account.

    So, why did Google enable these things by default? Essentially it means:

    • You can only authenticate via Google Cloud Platform (GCP) Single Sign On (SSO) to the Kubernetes API. That means, no basic authentication or client certificate can be used.
    • The OS is locked down and container optimized with a limited and read-only root filesystem with integrity checks. Integrity monitoring of the OS is enabled, to protect against rootkits and malware.
    • The service account that the Kubernetes nodes are using is restricted and will only have privileges to access cloud services it needs, such as Stackdriver for logging and monitoring. 

    The things you need to make sure you enable:

    • Enable secure boot to validate the authenticity of the OS and kernel modules. (If you decided not to use the default OS, then it might cause issues).

    Using the virtual trusted platform module (VTPM) to sign kernel images and the OS, means that authenticity can be established. It also guarantees that nothing has been tampered with, and kernel modules have not been replaced with ones containing malware or rootkits inside.

    • Enable intranode visibility so you can see the data flowing between pods and nodes.

    Making sure all traffic is logged and tracked between pods and nodes will help you identify any potential risks that may arise later on. This isn’t necessarily something you need to do for development, but, something you should do for production.

    • Put the Kubernetes API on a private network.

    • Put the node pool on a private network.

    • Provide a list of authorized networks that should be allowed to talk to the API.

    Making your nodes and Kubernetes API private means it isn’t subject to network scans that are happening all the time by bots, hackers and script kiddies.

    Putting it behind an internal network that you can only access through a VPN is also good, however, this is a much more involved process with GKE and isn’t as simple as a feature flag like the others. You can read about what is involved here.

    Securing Your Application Containers

    Again, Google enables some default features to get you started. However, there are still gaps that you will need to fill. 

    What do you get?

    • Managed certificates
    • Role Based Access Controls (RBAC)

    What you will want to enable:

    • Pod Security Policies. This is Beta and requires you to inform Google that you’re wanting to run the cluster creation in beta mode:

    • Network Policies.

    This sounds great, but what does it actually mean? Basically, some of these roles are split into two, one is for application development teams to own and the other is for the cluster administrator.

    Application Developer

    • Set network policies at an application level so you only allow the right level of access, i.e., AppA can talk to AppB but AppC can’t talk to either of them.
    • Have encryption to your service endpoint for your desired domain, i.e., https://www.mydomain.com.

    Kubernetes can be a bit of a learning curve, there are technologies that make it simpler in terms of dependencies such as helm, that will allow you to deploy application dependencies with pre-defined deployments. But there is no real substitute from understanding the main components of Kubernetes;  Network Policies, Ingress, Certificate Management,  Deployment, Configmaps, Secrets and Service resources.

    The main security components are network policies, secrets and certificate management. Network policies will allow you to control the traffic to and from your applications. Secrets are base64 encoded, so there is no real security in terms of how they are stored; therefore making sure the cluster administrator has enabled secret encryption, (as mentioned further down), will add that additional layer.

    Certificate management will make sure the traffic to your service is encrypted, but if you’re communicating between services, then you should also add TLS between your applications. Having the cluster administrator install something like cert manager, will allow an easier way to encrypt between services. There are also services like Istio, but as that product does a lot more than just certificates, it can add more complexity than necessary.

    Cluster Administrator

    • Have the ability to control what users can do and in what namespaces (RBAC), i.e., maybe Bob the developer can do deployments, (create, update, delete), but can’t view secrets in namespace our-teams-app1-dev.
    • Implement a deny all network policy for both egress and ingress (outbound and inbound). This can be controversial, as it can catch teams out, so making sure you’re communicating this is key. It will, however, force teams to define network policies to get their application working.
    • Force an application deployment security stance with Pod Security Policies. This means preventing containers running in the cluster escalating privilege, mounting in devices (including binding to the host network) on the nodes or running more privileged kernel system calls.

    You want to make sure that development teams can’t deploy insecure applications or make attempts to escalate their privilege, mount in devices they shouldn’t or make unnecessary kernel system calls. Pod Security Policies offer a way to restrict how users, teams or service accounts can deploy applications into the cluster, enforcing a good security posture.

    RBACs and Pod Security Policies go hand in hand. Once you define a good Pod Security Policy, you then have to create a role that references it and then bind a user, group and/or service account to it, either cluster wide or at a namespace level.

    Note: GKE uses a webhook for RBAC that will bypass Kubernetes first. This means that if you are an administrator inside of Google Cloud Identity Access Management (IAM), it will always make you a cluster admin, so you could recover from accidental lock-outs. This is abstracted away inside the control plane and is managed by GKE itself.

    We recommend the below to be a good PSP.  This will make sure that users, service accounts or groups can only deploy containers that meet the criteria below.

    If you wanted to create a role to use the defined PSP above, then it would look like something below, this is a cluster role as opposed to a standard role. To then make this enforced on say all authenticated users, you would then create a role binding to apply to the “system:authenticated” group.

    Remember that as this is cluster wide, any applications that may need more privileged permissions will stop working, some of these will be things Google adds into kubernetes; such as kube-proxy, that runs in the kube-system namespace.

    You can read more information on RBAC and PSP’s on kubernetes.io.

    Securing Sensitive Data and Application Cloud Services

    We’ll break this down into two.

    1. Encrypting Your Secrets in Kubernetes

    The recommendation for encrypting secrets using Google Clouds KMS service, is to have segregated roles and responsibilities, and to define a new Google project outside of the Google project that will host Kubernetes and applications. This is to make sure the encryption keys and, more importantly, the key that is signing the other keys (envelope encryption) isn’t residing in the same project that could potentially get compromised.

    For encrypting secrets you need to:

    • Setup a new Google project to implement segregation of duties (if necessary).
    • Create the master key ring and key inside that project.
    • Only allow the GKE service account in the Kubernetes hosted project access to the key in the dedicated key project.
    • Only allow just the right permissions (encrypt and decrypt) of the service account.

    The documentation on how to do this can be found here. But the main things to remember are:

    • The key has to be in the same location (zone) as the cluster (this is to reduce latency or loss of a zone).
    • Access permissions have to be correct for the service account.
    • The binding is a KMS IAM policy binding.

    Once this is setup you can pass the path to the key, to the “gcloud container clusters create” command:

    Note: If any of the above is incorrect, you will get a 500 internal server error when you go to create a cluster. This could be the path is incorrect, the location is wrong or the permissions are not right.

    2. Consuming Cloud Service Inside of Kubernetes

    There are four different ways to allow application containers to consume cloud services. All of these have limitations in some way, i.e., being less user friendly and unautomated for developers (making developers wait for the relevant access to be provisioned), or have less visibility and more complexity in tying together auditability for Kubernetes and cloud administrators.

    1. Workload identity (this is still in Beta and is the more long-term direction Google are going in): Require a constant process of managing google IAM as well as Kubernetes Service Accounts to tie them together. This means managing Google IAM roles, policies and service accounts for specific application service lines as well as Kubernetes Service Accounts. It does, however, improve auditability.
    2. Google Cloud Service Account keys stores as secrets inside Kubernetes namespaces (where the application will be living): Similar to the above, but without the binding. This means provisioning a service account and placing it as a secret inside of the applications namespace for it to consume natively. This has the downside of not having full auditability within Google Cloud and Kubernetes.
    3. Use something like Vault to broker between Cloud and Applications: Using Vault provides an abstraction to cloud and will generate short lived access keys to applications. However, there is still a secret required to be able to speak to the Vault service, hence, the permissions are still the same but just abstracted down one. It also disjoints audibility between Google Cloud and Kubernetes.
    4. Using the default service account node role of GKE: Much simpler and more developer friendly, but more risky. It would mean allowing applications to use the default node service account and modifying the role to cater for all the service policies applications would need, increasing the scope and capability of the node service account to most Google Cloud services.

    Having Good Visibility; Current and Historic

    Note: As of today, there is no access transparency on when Google accesses your Kubernetes cluster (Google Access Transparency). This could be problematic for a lot of organizations who want assurances around the providers data access.

    When the cluster is provisioned, all audit logs and monitoring data is being pushed to Stackdriver. As the control plane is managed by Google, you don’t get to override or modify the audit format, log or extend it to your own webhook.

    It does mean that you can search your audit logs for things that are happening inside of Kubernetes in one place, for example; to query all events against a cluster inside of a specific Google project against your user ID, you can do the below:

    From this point on, you could take it further and add additional security rules such as setting custom metrics to alert when cluster admin changes are made or specific modifications are happening on roles (cluster-admin roles).

    Summary

    Security is something everyone wants but, as you can see, it can be quite an involved process to achieve. It also requires domain knowledge to get enough context to assess the risk and what it might mean to your applications and business.

    Security without automation can slow down productivity. Not enough security can put your business at risk. Enabling security features that require Beta feature enablement may also not be suitable for the business and only General Availability features are acceptable, which compromises on security.

    As a general rule, hardening your clusters and enforcing secure ways of working with Kubernetes, containers, cloud services and your applications will get you the best outcome in the end. There may be frustrating learning curves, but as the industry matures, these will slowly be remediated.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

    — Lewis Marshall

  • Predictions 2020: 7 DevOps Predictions Shaping 2020

    Predictions 2020: 7 DevOps Predictions Shaping 2020

    2019 was an exciting year for DevOps practitioners, with DevOps adoption continuing to grow at a fast pace. Most organizations were either executing or evaluating their DevOps strategies. So, what is 2020 going to bring in DevOps, infrastructure and application automation? Here are seven DevOps predictions for emerging trends in the new year.

    Customer Experience

    CX will become the guiding light of the DevOps journey, and not necessarily just technology. This is a big mindset shift from the early days of pure-technologist enthusiasm. As DevOps matures, we have to remind ourselves that it’s all about making business faster, not just the cool tech.

    Take advantage now:

    • Tie DevOps achievements to business needs and results to get executive buy-in and budgets.
    • Successful QA teams will start looking at their role as guardians of customer experience and not just testers.

    Trust

    Everyone will be talking about trust—not only in the context of zero trust networks but also in the context of making it possible for Dev and Ops to trust one another as the foundation for DevOps.

    Take advantage now:

    • Proactively invest in creating a culture of trust. Identify causes for lack of trust in your value stream and make trust a 2020 goal.
    • Encourage product-level conversation and define methodology and ownership when it comes to value stream management, application and infrastructure automation. It is the foundation that everyone is working on.
    • Invest in staging environments—Staging environments are needed to ensure things work before you put them in production. For many of us, jumping directly to automatically deploying to production is a big step. The risk can be mitigated by investing in staging environment automation, which will be the first step toward deployment to production.

    Value Stream

    The DevOps toolchain is going away. In 2020 we will think value stream. The industry is maturing beyond the notion of chaining together dozens of open source tools and trying to make it work together. In previous years many felt comfortable saying that if they have a CI system in place it means they “do DevOps.” This is no longer the case. DevOps is not just CI, and DevOps cannot be established in silos. Mature DevOps needs to take into consideration the entire value stream—from planning a new capability and the business requirement, all the way to the customer getting more value and a wonderful experience.

    In 2019, Gartner announced it’s the last year of the ARO MQ, as it’s not the right perspective. In 2020 we finally gain a system-level view on the value stream. More players are announcing end-to-end value stream management including Atlassian’s new Forge platform, GitLab and CloudBees. Companies are maturing and adopting system-level solutions. 

    Take advantage now: 

    • Plan for platform operations. System-level value stream perspective requires roles that take ownership of orchestration and automation throughout the value stream and not just in silos.

    The SRE Role Will Continue to Mature

    The SRE role that started with Google bringing software development practices into Operations will continue to evolve. Increasing volume, velocity and variety of data will make AIOps solutions thrive. Also, increasing levels of automation will require smart ways to handle dynamic infrastructure and applications without losing control, and being able to track changes all the way back to the coder. 

    Take advantage now:

    • If you haven’t implemented the SRE role yet, consider how SREs can be part of your teams in 2020. Define SRE areas of responsibility and KPIs, and how the SREs communicate with the teams. Is it a centralized position or are the SREs part of the team?
    • Make sure you provide SREs with the tools they need to succeed including:
      • What are the observability solutions you use? Who uses them today? How do you analyze it and how is it communicated?
      • What are the production environment orchestration, deployment and configuration tools that you use today?  How do you connect them to application and infrastructure automation that happens before production?

    Seamless Infrastructure

    We will have to make infrastructure seamless for our teams. Internal customers are also customers, and their experiences will be just as important. Self-service and empowering teams will be the main themes in the bridges that need to be built between Dev and ITOps. Public cloud, Kubernetes and serverless technologies will grow. And while it makes automation more feasible than ever, the emphasis will shift to self-service and to the end users. With more companies adopting similar approaches, the need to develop internal expertise will decrease, and we will see more off-the-shelf solutions.

    Take advantage now:

    • Map how infrastructure is used throughout your value stream, and work on unifying the approach in product-level.
    • Make sure you build self-service solutions with strong focus on the end-user in order to enable and balance both speed and control. Make sure developers and ITOps are equally represented when determining an approach to infrastructure automation.

    Infrastructure Automation

    Infrastructure automation is going to get smarter, with a lot of AI and ML. Making sure automation is properly governed will also increasingly be important. Take into consideration security and compliance, as well as cost.

    Take advantage now:

    • Assess infrastructure automation practices throughout your value stream. Who builds the automation? What teams and systems consume it? How do you enforce security best practices (e.g. no passwords in public repositories, using proper data sets for testing, compliance of automation blueprints)? How do you enforce cost control (e.g. tagging, reviewing cloud spend reports)?
    • Invest in control mechanisms before you lose control, especially in public cloud automation. If you don’t have governance in place, look at baking governance into automation, rather than adopting traditional cloud management approaches. Invest in automatic de-commissioning of infrastructure, automatic tagging, analysis of infrastructure consumption data and connecting it back to the business requirement, secrets and key management and access control.

    SaaS

    Open source is nice. SaaS is great. While big open source projects set the pace (K8S, Github) and are strongly backed up and funded by the industry’s giants, smaller projects are increasingly difficult to scale and glue together, especially when trying to assure enterprise readiness. The need to “do it yourself” is gradually fading, as more vendors offer products and services for supporting DevOps with clear business and support models and, as important, with OOB integrations with the ecosystem. Just Rest API is no longer enough.

    Take advantage now:

    • If your enterprise-readiness requirements make it difficult for you to scale DevOps initiatives, consider evaluating new SaaS offerings that eliminate much of the heavy-lifting and gluing that were typical to the skills-heavy early-days implementations. If you were concerned with security, cost and skill requirement, today you can find tools that are purpose-built to support DevOps for enterprise at scale. The good news is that most of the vendors are well-aware of the criticality of integrating to your ecosystem, making it easier to achieve your end-to-end value stream vision.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Maya Ber Lerner

  • The Automated Data Center Emerges

    The Automated Data Center Emerges

    Hyperscale and automation seem to go hand-in-hand, at least when it comes to the data center. Synergy Research reported that, at the end of 2017, there were 390 web-scale data centers worldwide, up from 300 a year earlier. IDC identified that hyperscale data centers “have a minimum of 5,000 servers and are at least 10,000 sq ft in size but generally much larger.”

    Today, hyperscale data centers are driving a sense of urgency for automation, but many smaller corporate data centers are just as complex, even if they are less physically expansive. In either case, automation takes data from across the data center, applies analytics and machine learning to that data and helps to identify both inefficiencies and trouble spots in the data center–as well as their origins. 

    Some data center functions are well on their way to becoming automated, making new deployments and changes to existing deployments simpler, faster and more efficient than they are today. Among those functions, orchestration, change management, software-defined architectures and DevOps stand out. 

    Meanwhile, another data center function is lagging: problem resolution. The challenge revolves around the siloed nature of the data center, where the multiple, disparate tools and methods used to sleuth out inefficiencies and pinpoint trouble spots lack a unified view of the data center infrastructure. In addition, silo-centric infrastructure monitoring tools lack application context. Short of automation, today’s data centers are stuck in the past. Especially as a data center grows and must interact with cloud-based services, no amount of human intuition in using those tools and methods can overcome the barriers of siloed systems. 

    However, data center infrastructure planners are making strides in identifying the requirements for accelerating the move to the automated data center. 

    Three Keys to the Automated Data Center

    For years, the automated data center was just a dream. And as recently as late 2018, Forrester spoke of islands of automation.

    “For the enterprises I’ve talked to, it’s not like any of them are completely manual at this stage, but they’re all at varying degrees of trying to integrate all these pieces,” said Chris Gardner, Forrester senior analyst. He added that fragmentation was due to a lack of holistic solutions that could connect all aspects of the data center.

    But now, three forces are shaping up to help make the automated data center a reality. Let’s look further. 

    Real-Time Visibility 

    Real-time infrastructure visibility has become a critical element of automating the data center. Why? Because an issue can occur anywhere on the virtualized, hybrid infrastructure, and the issue can ripple to create problems elsewhere, making issue resolution a perennial challenge. 

    For that reason, your best defense is to build deep, real-time visibility into the compute/virtualization stack and the storage/storage network stack. This will help you quickly ensure the performance of mission-critical applications accessing the data from those stacks.

    For ITOps, it is about proactively pinpointing the source of the issue and resolving it before application users are impacted. That requires stitching data across the full stack of technologies supporting the business: application services and infrastructure services, both on-premises and in the cloud. 

    Real-Time Analytics

    Why is real-time analytics crucial in the data center? In short, it’s because real-time responsiveness is critical for supporting anomaly detection, root cause analysis, remediation, prevention and planning use cases, where processing and correlating massive amounts of information is critical.

    For ITOps, it means distributed real-time architectures must combine cloud-based and on-premises analytics that leverage AI and machine learning. It’s the analytics and correlation that helps you make sense of all that real-time data so it becomes actionable.

    Closed-Loop Systems

    IT organizations are acutely aware of the perils of introducing changes to critical production environments. In fact, they take great pains to establish a governance of change. Closing the loop is about humans delegating decision-making to technology.

    For ITOps, that means ensuring integrated control of change processes across the entire technology stack. Data and analytics should lead to optimal real-time decision-making that empowers the automation for closed-loop operation.

    How Automated Is Automated?

    In 2016, researchers at the University of Pisa argued that hyperscale data centers would have to rely completely on analytical, automated systems to have to any chance to manage the huge number of day-to-day issues their servers generate. They added that “large computer systems in general, and data centers in particular, will ultimately be managed using predictive computational and executable models obtained through data-science tools, and at that point, the intervention of humans will be limited to setting high-level goals and policies rather than performing low-level operations.”

    Now, more than three years later, critical technologies and methodologies are in place to make the automated data center a reality, even if humans are still required for guidance, if not for sleuthing out infrastructure issues.

    — Len Rosenthal