staging-devopsy.kinsta.cloud

Tag: database devops

  • Achieving Business Maturity with Database DevOps

    Achieving Business Maturity with Database DevOps

    Companies have seen unprecedented change over the past year and a half, with IT teams trying to help businesses weather the storm by meeting customer demands for services and products to be delivered in new ways while adjusting to remote work. As more companies introduced hybrid approaches to their workplace, a particular trend began to emerge: DevOps adoption. 

    Many adopted DevOps to navigate the new landscape, boosting software delivery and operational performance with advanced application and database development practices. In fact, research from Redgate Software indicates that DevOps adoption reached its peak earlier this year, with 26% of enterprises leveraging DevOps across all projects—a significant increase from previous adoption rates.

    Although not all companies have been quick to adopt for fear of disrupting existing workflows, a technical skills gap or lack of buy-in from management, there is a strong case to be made for why adopting DevOps practices, including for the database, can help businesses better manage the evolving tech environment.

    Adopting Automated Deployments

    Companies everywhere migrated databases to the cloud over the course of the past year, and that trend shows no sign of slowing. Analysts at IDC predict that by 2022 more than 90% of enterprises will rely on a mixture of on-premises/dedicated private clouds, multiple public clouds and legacy platforms to meet their needs. One of the best ways to help maintain that data is to implement DevOps processes for those databases and automate deployments. This provides a host of benefits, including: 

    • Preventing software errors: Manual database migrations often come with a host of errors that lead to downtime, rollbacks and rework. It’s far better to rely on an automated process that is robust and repeatable and can be implemented across all databases to ensure they’re updated in the same, reliable way. 
    • Staying in the loop: Knowledge of how to release updates and changes is often limited to a handful of people or even just one person. If, instead, it is part of the system and development process, it ensures everyone is in sync with updates and developments and allows anyone on your IT team to handle deployments.
    • Guaranteeing quality and consistency: For companies with security and compliance concerns, the predictable and repeatable processes that automated database deployments introduce ensure nothing is left to chance, while providing an auditable record of exactly what was changed, when and by who.

    Although the research shows 20% of companies cite cost as a roadblock to automation, avoiding automation will leave businesses at a disadvantage as they prepare for the coming year. 

    Creating Communities of Practice 

    When asked how they decide the strategic direction and planning for DevOps, the most popular answer from respondents was communities of practice. This concept was proposed for the first time in the early 1990s, and the practice has seen increased popularity among IT teams recently. It’s described as groups of people who share the same passions or concerns for something they do and often learn how to do that task better based on their regular, frequent interactions. 

    These various communities—regardless of whether in-person or virtual, given the current environment—help foster intrinsic motivation in employees and improve overall decision-making and encourage innovation. According to Redgate’s research, over a third of respondents noted that a community of practice makes decisions, provides recommendations or is consulted before DevOps strategies and planning are put into place. Another interesting point from the research is that the adoption of communities of practice correlates with company size. Enterprise and medium-size companies reported adoption rates of 38% and 36% respectively compared to only 28% adoption in small companies.

    As companies grow and teams mature, these communities not only benefit the organization by bringing more diverse minds to the table when making decisions but benefit individuals by supporting development, enabling networking and improving recognition across the organization. 

    Managing Cross-Database Estates

    Redgate’s research found that 70% of organizations now have more than one database management system in use, and 48% have teams working on three or more different databases. One possible reason for this is the increased adoption of cloud databases. With a large range of cloud databases and platforms in use, each offering different advantages, teams are now making choices based on specific use-cases. There was an increase in the share of respondents using cloud-native databases such as Azure Cosmos DB, MongoDB and Snowflake compared to last year, for example.

    With cloud adoption accelerating due to the pandemic and the need for technologies that can be managed, operated and accessed remotely, the advantages of introducing DevOps processes to maintain database changes and ensure version control have never been clearer. Teams that have automation and advanced development practices in place can continue building and deploying software as smoothly as before the pandemic, even when working remotely.

    Instead of working to build up processes and procedures, mature DevOps teams can focus their efforts on solving problems and ensuring higher quality of code, without the need for laborious manual tasks or code reviews that automated systems perform more reliably. 

    Making the Case for Database DevOps

    When teams are physically separated, working disparate schedules and dealing with their own challenges at home, the more companies can do to reduce workloads and help them focus on value-added work creates a happier workplace and improves the chances for business success. The rise of DevOps for both application and database development helps mitigate many of the challenges organizations face as part of their digital transformation journeys. It’s going to help IT teams remain productive when working remotely and encourage them to explore new database platforms, both on-premises and in the cloud.

    With remote and hybrid work the new normal for the future, companies that set their sights on adopting DevOps across the company will only see these benefits increase. 

  • A New DevOps Database Platform is Needed

    A New DevOps Database Platform is Needed

    The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

    When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

    In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

    In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

    If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

    Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

    During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

    So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

    Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

    Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

    Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

    While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

    Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

    Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

    The following are examples of the capabilities required for such a new DevOps database platform.

    • Automated database provisioning
    • Distributed repositories support
    • Simple declarative builds
    • Dynamically scalable
    • Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
    • Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
    • Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
    • Many clones per virtual machine (VM)
    • Security features including access management, data masking and encryption
    • Compatible with and easily integrates with CI/CD platforms
    • Compatible with multi-cloud (AWS, Azure, GCS)
    • Built-in data security and governance into DevOps data.

    I will explain these more completely in a subsequent article.

    What This Means

    Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

  • Survey: The State of Database DevOps

    Survey: The State of Database DevOps

    Enterprises are undergoing rapid changes. Throughout the past year, companies have been pressed to learn a diversified set of technologies and methods to remain agile in a changing economy. In this digital-first world, data plays a heightened role; in tandem, so does the delivery of data and the persistence of the database layer itself.

    Applying DevOps principles and automation throughout the software life cycle is critical to maintaining a high level of efficiency. As with application development, databases are not immune to the DevOps takeover. Database DevOps is a discipline that considers the processes used to automate database changes and deployments — a field growing in mindshare as databases diversify.

    Seventy percent of companies now use more than one database in their stack. Many organizations now use a blend of on-premises and cloud databases. Styles are diverging, too, as non-relational databases like NoSQL gain favor, and new open-source databases emerge to fit niche circumstances. All of this means database operators may require an increased emphasis on database DevOps to handle additional complexity.

    The 2021 State of Database DevOps report, conducted by Redgate, surveyed 3,200 database professionals on the general state of their DevOps adoption. The survey also inquired about database change automation and deployment and underlying database technologies. Now, five years in, the study continues to reveal shifts in practices across industries.

    Below, we’ll review the report’s main takeaways to see how companies implement database DevOps in practice.

    DevOps Adoption Increases Across Projects

    To meet pressing digital innovation needs, DevOps has surged in importance across all industries — 74% of organizations have adopted some form of DevOps, and 26% of respondents reported applying DevOps practices across all projects — a significant uptick from previous years. The report acknowledges a slight increase in DevOps adoption among larger company sizes.

    So, which DevOps practices are companies employing the most? Well, in terms of DevOps for application development, the majority of organizations (50% or above) have adopted version control, automated builds, monitoring, automated unit tests, static code analysis and semi-automated deployments in production. Only 41% of organizations surveyed have fully automated deployments to production. These methods can be viewed as a basic benchmark for modern DevOps.

    The study demonstrated that in most organizations, DevOps is a communal undertaking. More respondents reported adopting a community of practice, rather than relying on individual decision makers, to disseminate DevOps strategies and direct planning. According to the report, a community of practice is a group of people that “makes decisions, provides recommendations, or is consulted.”

    Lack of Skills Tops DevOps Obstacles

    The pressure is on engineers to produce to meet digital transformation demands. However, organizations experiencing a developer drought may not possess the talent to introduce DevOps practices. It’s no surprise that the common obstacle hindering DevOps adoption was a “lack of appropriate skills in the team.”

    Other cultural barriers to DevOps adoption include disruption of existing workflows and a lack of alignment between teams. Interestingly, a lack of executive support was cited as the lowest hurdle to DevOps adoption, indicating universal consensus on DevOps goals. If we consider what technical barriers prevent improved software delivery and operations, 41% noted insufficient automation in release processes and 39% expressed hindrances due to legacy code.

    Continuous Delivery For Databases Increases

    Database DevOps appears to be growing in production use too. For the purpose of this study, Database DevOps is defined as “automating the delivery of database changes.” According to the report, 51% of respondents are already automating some parts of their database deployment process.

    Though growing year after year, database DevOps is nowhere near ubiquitous. Compared to DevOps in application development, database DevOps has lower overall use. For example, only 36% are implementing automated builds. And, only 22% adopt continuous delivery (CD) for all databases. Nevertheless, this could quickly change, though, as 80% of respondents are working to implement CD for database changes or have already implemented it.

    Setting up a CD process for database changes could increase overall software delivery speed. The top two drivers for adopting database automation are increasing the speed of database changes and freeing up time for other work. The third top reason, minimizing application downtime, rose in rank this year — which makes sense, as the desire to create more reliable systems is an ever-present concern.

    However, to reap these rewards, organizations must overcome some significant challenges, like synchronizing application and database changes and maintaining high speeds for database automation, among other concerns.

    Organizations Adopt Hybridized Database Approaches

    Companies increasingly use a mix of different database technologies and styles. We also see an increase in cloud and hybrid database estates.

    In terms of database styles, relational databases still make up the majority of market share — 98% of respondents have at least one relational database. However, non-relational databases, like NoSQL, are gaining popularity. Forty-seven percent of database professionals have a NoSQL database in their enterprise. Interestingly, only 2% use non-relational databases exclusively, indicating it’s a niche use case.

    Enterprises are now using more databases than ever before. Seventy percent of database professionals reported having more than one database in their stack. This figure rose nearly 10% since last year. Database types are multiplying too — 45% have more than one type of database. In terms of database management systems, Microsoft SQL Server holds the most use, at an 82% usage rate. Others like MySQL, Oracle, PostgreSQL, and MongoDB trail behind.

    Enterprises are also increasing their use of cloud and hybrid database deployment models. Eighty percent of respondents host a cloud database, and 65% report a mix of cloud and on-premises estate.

    One possible explanation for the diversified database effect is an increase in public cloud provider usage, combined with the rise of multicloud strategies. Database options are increasing as well — many free, open-source databases can cater to unique project needs.

    This “polygot persistence” could also simply be a natural occurrence within large enterprises supporting many application nuances. As Martin Fowler describes, “… any decent sized enterprise will have a variety of different data storage technologies for different kinds of data.”

    Quality Database Ops = Quality Software Performance

    In summary, non-relational databases are gaining steam, cloud and hybrid database models are becoming more common, and large organizations are increasingly using multiple database types. The landscape is forecasted to diversify further, meaning database DevOps is stepping up to manage this complexity.

    The report indicates a correlation between quality database Ops and overall software quality, noting that organizations that release more frequently typically deploy database changes in much less time. To achieve these goals, “… the next area for database DevOps is in the data engineering and analytics space,” writes Pramod Sadalage, director, data & DevOps, ThoughtWorks.

    While it’s important to take a vendor-endorsed study with a grain of salt, it’s safe to say we will likely see more interest in database DevOps, especially as DevOps themes continue to permeate through the entire spectrum of software development and delivery.

    For more information, you can download The State of Database DevOps report behind an email gate here.