Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)
Dell Technologies has quietly launched a training and education initiative to significantly expand the number of IT professionals that have DevOps skills.
Brad Maltz, senior director for DevOps and developer relations ecosystems for Dell, said the next era of IT will require IT professionals that understand how to automate the management of application development and deployment. Given its long infrastructure history, Dell is especially interested in helping IT teams to use a mix of graphical tools and command line interfaces (CLIs) to programmatically deploy software at much faster rates and at scale, he added.
Dell has already hired more than 200 specialists to provide that training as part of a DevOps Transformation Services consulting services engagement, said Maltz. The primary area of focus will be working with IT organizations that are embracing platform engineering to centralize the management of DevOps and GitOps workflows, he noted. Many of those teams are now relying on declarative automation tools that ultimately make DevOps more accessible to a wider range of IT professionals, added Maltz.
The number of IT professionals that employ graphical tools based on IT management frameworks such as ITIL still far outnumber the number of IT professionals that have DevOps skills. The ongoing debate between these two camps essentially comes down to the need to be able to consistently manage IT at scale versus being able to adroitly keep pace with the needs of organizations that are increasingly dependent on software. In the digital transformation era, it’s become clear that more of the organizations that hire IT professionals are placing a higher value on agility, said Maltz.
Bridging the divide between ITIL and DevOps to address that requirement is largely a psychological challenge that requires IT teams to address a range of cultural issues in addition to mastering different types of tooling, he added.
Regardless of whether organizations continue to manage IT infrastructure internally or rely more heavily on managed services provided by Dell, the need to continuously deliver secure software faster has become a business issue. The trouble is the level of DevOps expertise within organizations tends to vary widely. A recent survey published by the Continuous Delivery (CD) Foundation found only 22% of respondents are using both continuous integration (CI) and continuous delivery (CD) to automate all building, testing and deployment of code in a production environment. Most organizations that have adopted DevOps make extensive use of CI to build applications, but CD has proven to be more challenging to master. In fact, the CD Foundation survey found only 7% of respondents described themselves as DevOps engineers or specialists.
The bulk of the managed services that Dell provides are focused on managing infrastructure rather than deploying software, but the company does have a managed DevOps offering based on the application development and deployment platform that an internal Dell DevOps team built to construct and deploy applications. Naturally, Dell has a vested interest in accelerating the rate at which software can be deployed on IT infrastructure it provides and manages.
It may be a while before the overall size of the DevOps community expands, but the training initiatives such as the one launched by Dell will help. However, it may become a lot easier to find and retain that talent in the months and years ahead. In the meantime, organizations would be well-advised to invest in upskilling their own IT teams. After all, it’s less expensive to train someone that already works within an IT organization than it is to compete against thousands of other organizations willing to pay top dollar for the best DevOps expertise money can buy.
At its Build 2023 conference this week, Microsoft announced general availability of an Azure Deployment Environment service that provides developers with a portal through which they can employ infrastructure-as-a-code (IaC) templates based on Terraform or Azure Resource Management files.
In addition, Microsoft has enhanced Microsoft Dev Box, a workstation environment for developers that run Azure, to include a portal through which they can manage multiple environments.
Scheduled to be generally available in July, Microsoft Dev Box is also previewing a capability that makes it possible to customize environments using configuration-as-code files stored in a Git repository. Microsoft is also previewing tighter integration between Dev Box and its Visual Studio integrated development environment (IDE) and Windows to further customize developers’ environments.
Scott Guthrie, executive vice president of the cloud and artificial intelligence (AI) group at Microsoft, told conference attendees these and other tools for building cloud-native applications, along with GitHub Actions, make it simpler than ever to build applications in an Azure cloud service that includes an integrated continuous integration/continuous delivery (CI/CD) platform.
Microsoft is also previewing GitHub Advanced Security for Azure DevOps that natively embeds automated security checks using scanning tools into the Azure DevOps platform.
Other additions to the Azure environment include tools for centralizing the management of application programming interfaces (APIs) using an Azure API Center service that is available in preview.
The Azure Event Grid has also been updated to enable it to use HTTP to pull delivery of discrete events in addition to pushing them.
Finally, Microsoft is also previewing generative artificial intelligence (AI) capabilities to make it simpler to manage cloud costs by, for example, using prompts to summarize invoices or breakdown costs.
Microsoft, along with its GitHub subsidiary, is at the forefront of applying generative AI to application development using tools like GitHub Copilot that developers can use to write better code faster. That copilot capability is now being disseminated across the portfolio of applications and services that Microsoft provides.
It’s not yet clear what the downstream impact of such tools are just yet, but the overall quality of the applications being built should improve as, for example, fewer mistakes are made that can lead to vulnerabilities.
Obviously, the way application software is built, constructed and secured is about to change in the generative AI era. Thanks to its alliance with OpenAI, Microsoft has a significant head start on rivals building generative AI platforms that can similarly be applied to improve developer productivity.
In the meantime, DevOps teams should assume the volume of code simultaneously moving through pipelines is about to dramatically increase as developers become more productive. There will inevitably be more opportunities to apply multiple forms of AI to reduce any DevOps bottlenecks that might arise, but it’s clear most of the immediate benefits are being provided to developers.
One way or another, however, the overall rate at which applications are built and deployed is about to change drastically. The issue is understanding how those advances will impact downstream DevOps workflows that were constructed for a different era of application development.
This week at its Red Hat Summit event, Red Hat introduced an expanded set of capabilities to Red Hat Insights that simplify management of Red Hat Enterprise Linux (RHEL).
Managing Linux across the hybrid cloud is complex, and it’s made even more so because of current macroeconomic conditions and the challenge of finding and retaining highly skilled DevOps and IT professionals, said Gunnar Hellekson, vice president and general manager, Red Hat Enterprise Linux, Red Hat.
The enhanced Insights capabilities add a layer of abstraction to help reduce enterprise Linux complexity across the hybrid cloud and make RHEL management easier by lowering skill level requirements for managing Linux server estates, the company said.
Using a single UI, entry-level systems administrators and IT support teams can access Red Hat Insights’ predictive analytics to identify potential bugs, misconfigurations or security vulnerabilities and remediate them. Systems administrators of all skill levels can detect, assess and push fixes for potential problems without a deep understanding of Linux management systems like Red Hat Satellite Server and without having to interact with the command line, the company said.
These enhancements help organizations maintain services and systems even in uncertain macroeconomic times when hiring is constrained. Red Hat research also claimed that these enhanced capabilities enabled IT organizations to find IT issues up to 90% faster and remediate them nearly 66% faster across the hybrid cloud.
A Red Hat Insights image builder service also helps lower the skills level needed to manage Linux estates by enabling IT teams to build standardized, optimized operating system images, including the creation of their own ‘golden images’ that adhere to organization-specific needs for security and compliance. These images can be deployed across public clouds, hybrid clouds, data centers and the edge using the same console.
As the economic downturn continues, organizations are looking to do more with less and identify ways to make the most of the resources, talent and skillsets they already have. Red Hat is making the case for adding a layer of abstraction to allow less-highly-skilled IT and DevOps team members to contribute to managing Linux estates across hybrid, multi-cloud and edge deployments.
“The demands of managing Linux across the hybrid cloud are stretching the ability of resource-constrained IT departments to keep pace,” said Hellekson in a statement. “CIOs need to be able to extend existing skillsets and lower the skill barriers for overseeing Linux estates, which is exactly what the management capabilities of Red Hat Insights are designed to do.”
Red Hat today unveiled a technical preview of Ansible Lightspeed with IBM Watson Code Assistant, a generative artificial intelligence (AI) service specifically trained on domain-specific data collected via the Ansible automation platform.
At the same time, Red Hat is also making available an event-driven capability within version 2.4 of the Ansible platform that enables runbooks and other tasks to automatically execute when received from a third-party tool or application.
Both capabilities were announced at the Red Hat Summit and are part of an ongoing Red Hat effort to democratize IT automation.
First previewed within the Project Wisdom initiative launched by Red Hat and IBM Research last year, Ansible Lightspeed IBM Watson Code Assistant is based on a large language model (LLM) that was developed using Ansible data. Watson Code Assistant, currently in technical preview, is a service that automatically generates suggestions to improve code as developers create an application. IBM promised to make IBM Watson Code Assistant available later this year.
Red Hat is now applying that generative AI capability to the YAML code that DevOps teams need to create Ansible playbooks. IT teams can create a simple natural language prompt that will translate their domain expertise into YAML code for creating or editing Ansible Playbooks. Red Hat and IBM, however, will not automatically use the code from an organization to fine-tune their AI model without express consent.
Rich Henshall, director of product management for Red Hat Ansible, said as IT environments become more complex, it’s clear many are hitting a ceiling in terms of what can be automated using legacy custom scripts. Ansible Lightspeed IBM Watson Code Assistant, along with an event-driven capability, will make it much simpler to automate the management of IT environments at scale using a declarative framework, he added.
Ansible, for example, can now receive events from third-party tools, determine next steps and consistently initiate an automated response using Ansible playbooks. Ansible can also consume events from third-party monitoring, observability and IT analytics tools from providers such as Cisco ThousandEyes, CyberArk, Dynatrace, F5, IBM Instana, IBM Turbonomic, Palo Alto Networks and Zabbix with more to come, noted Henshall. Red Hat is also making available supplementary content for Red Hat OpenShift, Red Hat Insights, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform and ServiceNow, in addition to enabling IT teams to create their own custom integrations.
At this juncture, it’s not clear why more IT processes have not been automated using Ansible or other frameworks. Many DevOps teams continue to rely on custom scripts that typically don’t scale and are generally poorly documented. Any time the individual who wrote those scripts leaves an organization, it’s incredibly difficult—if not impossible—for the original scripts to be maintained, noted Henshall.
One way or another, it’s clear that a more scalable approach to IT automation is required, especially with the rise of more complex cloud-native application environments. It’s no longer economically viable for most organizations to rely on legacy processes that require a small army of IT professionals to maintain.
Modern development environments, in which rapid continuous delivery is facilitated by automated continuous integration/continuous development (CI/CD) pipelines, require thorough and automated testing in development prior to integration. This article explains how test-driven development (TDD) and acceptance test-driven development (ATDD) methodologies reduce CI/CD lead times.
Lack of test automation in development impacts CI/CD lead time for subsequent integration, delivery and deployment stages of a CI/CD pipeline in the following ways:
Although CI/CD pipelines help catch issues and ensure code quality through automated builds and tests, having a strong foundation of test automation during the development stage helps minimize the potential for issues in the pipeline and ensures a smoother CI/CD process.
There are several reasons why many development teams do not automate tests prior to integration, despite the benefits of doing so.
The amount of test automation that should be done in development prior to the integration stage of a CI/CD pipeline depends on several factors, including the complexity of the application, team resources and the specific project requirements. However, here are some general guidelines to help you determine the right balance:
Striking the right balance between automated and manual testing is important. Automate tests that are repetitive, time-consuming or prone to human error. Manual testing should be reserved for exploratory testing, usability testing and other areas where human intuition and judgment are necessary.
Test-driven development (TDD) and acceptance test-driven development (ATDD) are development methodologies that emphasize creating automated tests before writing the actual code, helping teams ensure code quality and catch issues early prior to integration.
Test-driven development (TDD) is a software development methodology that emphasizes writing tests before writing the actual code. The primary goal of TDD is to create clean, maintainable and bug-free code by catching issues early in the development process. TDD follows a short iterative cycle consisting of three main steps: Write a test, make the test pass and refactor the code. The developer starts by writing a test for a specific feature or function, which initially fails. Next, the developer writes the minimal amount of code necessary to make the test pass. Finally, the developer refactors the code to improve its structure and readability while ensuring that all tests still pass.
Acceptance test-driven development (ATDD) is an extension of TDD that focuses on defining and validating the high-level requirements and acceptance criteria for a feature or functionality before development begins. ATDD involves collaboration among developers, testers, and business stakeholders to ensure that the application meets the needs of its users. The process begins with writing acceptance tests, which are derived from the user’s perspective and based on the acceptance criteria. These tests are written in a human-readable language to facilitate clear communication among all team members.
Once the acceptance tests are defined, the development team follows the TDD process to implement the feature, using the acceptance tests as a guide. As the team develops the code, they also create unit tests and other lower-level tests to validate the implementation. The acceptance tests are executed as part of the development process and serve as a measure of progress. When all acceptance tests pass, it indicates that the feature or functionality meets the defined acceptance criteria, and the development task is considered complete. By using ATDD, teams can improve collaboration, ensure that the application meets user expectations and reduce the risk of miscommunication or misunderstanding requirements.
Hurdles to overcome when transitioning to TDD and ATDD include:
In the context of TDD and ATDD, the roles of quality assurance (QA), security and operations teams are crucial for ensuring a well-rounded and robust software development process. Their involvement helps to address various aspects of the application, leading to a more reliable and secure product.
A practical roadmap for teams to transition to TDD and ATDD while overcoming the hurdles mentioned earlier could look like this:
Build awareness and buy-in:
Provide training and skill development:
Start small and iterate:
Foster a culture of collaboration:
Set up the necessary tooling and infrastructure:
Monitor progress with metrics:
Continuously improve and adapt:
TDD and ATDD methodologies contribute to reducing CI/CD lead times by automating tests during development. This improves code quality, minimizes defects and promotes collaboration among team members from the outset of the development process. By writing tests before the actual code and focusing on meeting acceptance criteria, teams can catch and resolve issues early, leading to fewer delays and reduced time spent on debugging and fixing errors during the integration, delivery and deployment stages. These practices also help ensure that the application aligns with user expectations and operational requirements, resulting in a smoother and more efficient CI/CD pipeline.
Anyone who is passionate about modern quality engineering processes for building high-quality software to participate is encouraged to join new ShiftSync Community from Tricentis to engage in discussions about this topic and other development methodologies.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Sources say it’s an open source future for OpenAI, and improving AI perf with Mojo. (more…)
GitHub today revealed it added several capabilities to make it easier for developers with disabilities to access its code repository platform. The new capabilities were announced to coincide with Global Accessibility Awareness Day.
Capabilities highlighted include a navigation tool and a search tool that can be used with a screen reader that GitHub provides.
In addition, GitHub’s contribution graph has been updated to remove barriers for keyboard-only and screen reader users.
Finally, GitHub has made site-wide color contrast improvements to the light and dark themes presented by Primer, the GitHub Design System.
Ed Summers, head of accessibility for GitHub, said there are more than 1.3 billion individuals with a wide variety of disabilities that could, with access to the right tools, write code. That community today, however, is underrepresented as a percentage of the larger IT community, he noted.
However, as tools such as GitHub Copilot—a generative artificial intelligence (AI) tool that helps developers write better, more secure code—is paired with GitHub Copilot Voice—a speech interface for GitHub Copilot that is available in technical preview—the GitHub platform becomes more accessible to developers with disabilities, he said.
Generative AI coupled with a speech interface that eliminates the need for a keyboard is lowering the barrier for entry for every potential developer. But for individuals with disabilities, it represents a tremendous advance, noted Summers. Anyone who understands how code should be constructed can build an application, he added.
It’s not clear what impact generative AI will have on the global shortage of developer talent that has persisted for decades but it’s clear the pace at which applications can be developed is about to dramatically increase. In addition, many organizations have embraced working from home in the wake of the COVID-19 pandemic. That shift makes it easier for many individuals with disabilities to become part of the workforce because they no longer need to commute to an office.
Generative AI platforms are fundamentally changing the way humans interact with machines. Instead of requiring a developer to create a level of abstraction to communicate with a machine, it’s now possible for machines to understand the language humans use to communicate with each other. Via a natural language interface, developers will soon be asking generative AI platforms to not only surface suggestions but also debug applications and write the code used to create an application.
At this point, like it or not, the generative AI genie is out of the proverbial bottle. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated.
It may be a while before the impact of generative AI is fully appreciated, and there will certainly be more advances to come that will be applicable to individuals with disabilities. In the meantime, however, many members of a DevOps team may not even realize a member of the team has a disability as the writing of code continues to be increasingly automated.
In the context of cloud operations, the last decade was ruled by DevOps and infrastructure-as-code (IaC). But what is true DevOps? Is it developers running their own operations? Is it operators learning development skills to automate infrastructure provisioning using infrastructure-as-code (IaC) tools like Terraform?
Though DevOps is widely understood to incorporate both of these, software developers are far from fully understanding or operating infrastructure. IaC is purely a DevOps tool. When an organization claims to have automated its infrastructure, they mean automation for the efficiency of the DevOps teams, not automation for developers. Developers are still left waiting on DevOps for days for even small infrastructure changes.
While there were efficiencies gained by the current DevOps model, there is an increasing acknowledgment that we require a better approach to enable developer self-service. This has led to the platform engineering discipline.
The high-level goals of platform engineering are:
A DuploCloud survey of 40+ enterprises that have made substantial investments in platform engineering showed that the prevalent approach is for DevOps teams to build a DevOps platform with IaC as a core underlying technology. They create templates for the organization’s use cases and publish them in a CI/CD pipeline or a self-service catalog.
Here are the top reasons why platforms that build on top of IaC are failing these platform engineering goals:
Many platform teams have tried to work around these problems with point solutions by using a disparate set of jobs for certain aspects of life cycle management, building a thin UI shim on top of cloud accounts for visualization of resources while redirecting to other systems like Datadog for logging, metrics and alerts. But for most of the use cases, the DevOps team is still very much in the operations workflow. This completely defeats the concept of developer self-service as well as continuous compliance goals.
Two recent examples of successful cloud platforms are Amazon Web Services (AWS) in the context of infrastructure-as-a-service (IaaS) and Kubernetes for container orchestration. These are distributed system implementations using higher-level programming languages like Java and Go. You can’t build such complex systems using scripts and jobs.
Building a true DevOps orchestration platform requires a systems design approach. It also takes expert systems engineers and many years to build and mature, like it did for K8s and IaaS in the Public Cloud.
Since platform engineering teams need systems design expertise that is likely outside of the scope of the typical DevOps role, we need the software development function to intervene with distributed systems programmers and many years of investment. Most organizations trying to build in-house platforms don’t have or don’t want to make this type of investment, as it may not be their core business. Megascale tech companies like Facebook, Uber, Netflix, etc. will likely build an in-house solution, as they have the talent and scale, but for the masses, it is likely that the solution will be an off-the-shelf product from an ISV.
The industry has solved IaaS with public cloud, container orchestration with Kubernetes and observability with platforms like Datadog. The platform engineering category is heading in the same direction.
SapientAI emerged from stealth today to automate the writing of test code using a combination of generative artificial intelligence (AI), machine learning algorithms and data science.
Fresh from raising $5 million in seed funding, SapientAI CEO Rishi Singh said the goal is to leverage AI to push application testing further left using an AI Test Coder the company built.
The concept of AI Test Coder is similar to the co-pilot capabilities that GitHub and Google are providing to help developers write better code faster, but also goes beyond generative AI to add deeper analysis and contextual understanding to the tests it automatically generates, noted Singh.
In the longer term, the ability to use AI to create tests will result in the deployment of more secure, higher-quality applications because there will be no need to short-shrift application testing processes in the interest of meeting deadlines, added Singh.
Prior to the rise of AI, building application tests has historically been prone to error. AI Test Coder is designed to be a plug-in for an integrated development environment that will leverage the large language models (LLMs) created by OpenAI and cloud service providers to employ natural language to create tests based on an analysis of the application environment created using a combination of machine learning algorithms and data science techniques, said Singh.
As part of the effort, AI Test Coder instantly inspects all software code to understand the software testing environment, including any incorrect assumptions that stem from errors and bugs, before generating any test code. It then leverages code intelligence to understand existing code and automatically identify and detect patterns to anticipate any new software issue that may arise. AI Test Coder also locates where code changes will have an effect before any test code is created. This eliminates time that would otherwise be wasted on manually debugging code.
Once those steps are completed, Test Coder then generates more accurate tests without relying on the current trial-and-error approach to testing that can slow application development, said Singh.
In effect, AI Test Coder provides each developer with their own assistant dedicated to managing testing tasks that ultimately improve quality assurance, including removing many of the vulnerabilities that create application security issues that have become the bane of software supply chains.
It’s too early to determine precisely what impact AI will have on application development and deployment, but many of the manual tasks that create a sense of drudgery are about to be eliminated. That means the overall pace at which applications can be built and deployed is about to be greatly accelerated. DevOps teams are likely to find themselves pushing more code simultaneously through pipelines than ever.
Naturally, AI will soon be applied to not just writing code and tests faster, but also DevOps workflows. The issue now is determining which of the tasks that today require manual intervention will be eliminated in a way that enables DevOps teams to focus more of their time and effort on more complex challenges.