Category: DevOps Toolbox

Oracle and Allies Launch ARM-Based Cloud Service

Oracle announced today it will make available for the first time ARM-based processors on the Oracle Cloud Infrastructure (OCI) that will also be supported as targets for deploying applications using GitHub, GitLab or Jenkins platforms. The latest Oracle service is based on processors from Ampere Computing.

The company also revealed today it has joined the Continuous Delivery Foundation (CDF), an arm of the Linux Foundation that oversees the development of open source DevOps platforms and tools such as Jenkins and Spinnaker.

Matt Leonard, vice president of compute for OCI, said the OCI Ampere A1 Compute provides an alternative to existing support for the x86-based processors that Oracle provides. That service will be offered at a cost of one cent per core hour with virtual machine sizes ranging from one to 80 OCPUs configured with one to 64 GB of memory per core or as a bare-metal service with 160 core and 1 TB of memory. Pricing per GB of RAM per hour is $0.0015. At that price point, Leonard said Oracle is now offering the lowest price per core of any cloud service provider.

The growing availability of ARM-based processors is starting to have an impact on cloud economics at a time when organizations are significantly increasing the number of applications being built and deployed on public clouds. Many of those organizations, to reduce overall costs, are now actively evaluating ARM-based services, noted Leonard.

To help jumpstart application development on the platform, Oracle is providing organizations with three classes of OCI Ampere A1 Compute services. An Oracle Cloud Free Tier gives developers $300 in free credits for 30 days. The Always Free ARM option gives developers four A1 cores and 24 GB memory, while an ARM Accelerator option provides open source developers, independent software vendors (ISV) partners, customers and universities with ARM-based development projects that need additional resources beyond what the Oracle Cloud Free Tier provides with credits for a 12-month period.

Oracle Linux, Java, MySQL, GraalVM and the Oracle Container Engine for Kubernetes (OKE) service are also all available on the OCI Ampere A1 Compute along with integrations with DevOps platforms from GitHub, GitLab and/or Jenkins. Other vendors providing support for OCI Ampere A1 Compute include SUSE, Datadog, OnSpecta, NGINX and Genymobile.

Oracle has also made available an Oracle Linux Cloud Developer image that enables DevOps teams to install, configure and launch a development environment via an OCI console that includes OCI client tools, utilities and common programming languages such as Java, GraalVM, Python, PHP, Node.js, Go and C/C++.

Finally, each Ampere core is single threaded by design with its own 64 KB L1 I-cache, 64 KB L1 D-cache and a huge one MB L2 D-cache to eliminate any ‘noisy neighbor’ issues that IT organizations might otherwise encounter.

Oracle is clearly betting that ARM-based processors will present the company with an opportunity to gain share at the expense of larger infrastructure-as-a-service (IaaS) rivals such as Amazon Web Services (AWS) and Microsoft Azure. It’s not clear to what degree IT teams might employ a different cloud service just to access ARM-based processors. However, in a world where IT teams now routinely employ multiple clouds, as far as Oracle is concerned, the playing field has never been more level.

May 25, 2021
Rethinking Your Tool Chain in the Cloud
Every IT shop has a “tool chain” – all of the software other than the OS and the application that fills in the gaps and keeps everything running. This includes solutions for monitoring the application, data protection, software development, release management, provisioning and de-provisioning resources and more. These tools build up over time, often evolving from homemade solutions to commercially available software as the company grows.

The cloud presents another inflection point: a new way to manage systems and applications. When a company is considering moving applications to the cloud, IT and application development teams should take the opportunity to reevaluate their tool chain. Are their existing solutions still the right fit in this new cloud-based world, or are there better alternatives available? Here is a process for making that decision, and some important questions to consider when assessing whether a change is in an organization’s best interests.

Step One: Take Inventory of the Current Tool Chain

First, IT teams should perform a full inventory of their current stack to ensure they don’t overlook anything. This audit should include the following areas:
- Provisioning tools (like Ansible or Terraform).
- Identity and access management (IAM) solutions that allow IT to restrict access to servers, resources and software to only relevant users.
- Development tools including IDEs, continuous integration/continuous delivery (CI/CD), source control and more.
- Data protection, high availability, disaster recovery and backup software.
- Monitoring and auditing software (like Datadog).
Step Two: Assess How Well Each Tool is Meeting IT’s Needs

Next, consider how well each product is meeting IT’s needs now, and how those needs will change as workloads move to the cloud. The first and most important question: is the tool and/or version you’re on still supported by the vendor? Changes like this migration to the cloud are rare, so the likelihood that one or more products are no longer supported is higher than one might think. Second, consider the licensing terms of the software and whether they allow for its use in the cloud. For instance, some tools are licensed to a specific physical server and some vendors require their hardware to be owned by the same entity which holds the license. Neither of these scenarios work well in the cloud. Third, will the application be managed in the same way after the migration? The cloud typically removes constraints; it offers nearly unlimited compute and storage, dynamic workloads, new APIs and multiple regions around the world. IT teams may want to take advantage of these capabilities by moving to a cloud-based tool (replacing an on-premises application log reporting solution, for example), or they may want to keep management as consistent as possible to minimize the amount of retraining required for employees.

Don’t forget to consider non-technical arguments. Is the team happy and productive with the current solution, or do they feel forced to use it? Is it meeting the business needs of the organization? Will the change create more work in the short term? These reasons aren’t enough to keep a tool on their own, but they should be part of the discussion.

Step Three: Create a Map of Alternatives

Next, look at the potential alternatives to the tools that may need to be replaced. This can be a good opportunity to consolidate, so start by looking at what other teams in the organization are already using in the destination cloud. The most common decision IT must make here is whether to switch to a cloud-based tool (either one offered by their cloud vendor or a separate SaaS product). These typically offer pay-as-you-go licensing and are usually easier to scale up or down, move workloads around and to expand to other regions. Licensing for legacy vendors varies widely; some now offer consumption-based options to compete with the cloud, while some are sticking with perpetual licenses. These factors may make a cloud-based tool more or less attractive for specific companies based on their needs. Last, consider if making a change would allow the IT team to expand their skill set. For engineers that like learning new things, adding new products could actually increase satisfaction and retention because it gives them a chance to learn something new.

Step Four: Talk to Existing Tool Vendors

Even if a different option seems like the better choice, it always pays to check in with the current vendor and give them a chance to keep your business. It’s possible they may offer more flexible terms to keep you as a customer. The opposite is also possible; some vendors won’t budge on their licensing because they see the cloud as a threat. Even if a vendor conversation doesn’t change what they are willing to offer, it will usually reveal useful insight about how they approach their customers.

Step Five: Create a Plan for Each Tool

Based on the information gathered in the last four steps, it is time to decide what to do with each tool and make a plan for each transition. Where necessary, note products that will need to be replaced soon, ones where more research is required, and compile any other useful information learned during the process so that the larger IT team can access it. Also consider if teams will need training, if internal documentation or playbooks need to be updated, and how new tools will plug into existing authorization/authentication solutions. Each change will also need a migration plan of some kind that details how and when the organization will move from the old product to the new one, what scripts will need to be rewritten and what to do with historical data like log files from the old product.

Cloud-based tools won’t be a fit for everyone, but their flexibility, potential cost savings and ease of scalability shouldn’t be underestimated. While no one is eager to add more work to a cloud migration project, taking the time to go through this process could uncover useful new options that better fit the new cloud model that the business is moving toward.
May 25, 2021
Tiering Cold Data to the Cloud Without Tears
The cloud provides inexpensive storage. But like everything else in the cloud, there are hidden costs. Cloud storage providers (CSPs) charge to put data into and retrieve data from the cloud. They charge for the API calls and they generally charge an egress cost when the data is extracted from the CSP. So, to keep enterprise storage costs low, infrequently accessed data such as snapshots, logs, backups and cold data are best suited for tiering to the cloud.

By tiering data, on-premises storage arrays need to only keep hot data and the most recent logs and snapshots. Typically, 60% to 80% of enterprise data has not been accessed in over a year. By tiering the cold data as well as older log files and snapshots, the capacity of the storage array, mirrored storage array (if mirroring/replication is being used) and backup storage is reduced dramatically. This is why tiering cold data can reduce overall storage costs by as much as 70%.

The many advantages of this approach include:
- Lower acquisition cost. Flash storage, used for fast access to hot data, is expensive. By tiering off infrequently used data, you can purchase a much smaller amount of flash storage, thereby reducing acquisition costs.
- Lower backup and mirroring costs. By continuously tiering cold data, you can reduce your footprint, license costs and storage costs for backups and replication if the cold data is placed in robust storage (such as that provided by the major CSPs).
- Improve performance and lower capacity of your storage array. By running storage at a lower capacity and by moving access to cold data to another storage device or service, you can increase the performance of your storage array and get by with a smaller storage array.
- Enable processing of cold data without burdening the storage array. Processing and feeding your cold data into your AI/ML/BI engines is critical to staying competitive. With cold data in the cloud, you’re also reducing the load on your storage array, thereby extending its life.
Know First, Then Tier

A key challenge of cloud tiering is determining what data to tier. End users should not have to decide which data to archive or handle the management of shared files. This issue may result in IT organizations keeping too much data in expensive, hot storage. One way around this is to automate the process through business policies dictating when and how the process is applied. It’s also helpful to provide transparency of the data by keeping it in the existing namespace. Users can still find the data and access it as if it had never been tiered. Transparency enables IT to tier cold data continuously and systemically across all storage devices in the entire organization without requiring end user assistance.

To help IT make the right decisions, analytics is needed to see just how much data you have, and how much has not been touched in three, six or 12 months. It will be even more compelling if IT can run what-if scenarios to determine how much they will save with cloud tiering. The last thing you want to do is tier data in the dark. Take time to understand your data assets before you invest in the effort.

Here are other considerations, beyond analytics, when embarking on a cloud tiering initiative:
- Transparent, continuous tiering. Tiering should be transparent to the user. If someone can still search for and access data that’s tiered without any change to the experience, IT can realistically tier data systemically across the enterprise. Without transparent tiering, it’s an onerous process to tier data because you will need user permissions.
- Flexible tiering policies. You need the flexibility to exclude certain data sets, include others and select from a large range of data ages. Be aware that many tiering solutions provide an extremely limited set of policies.
- Tiering across multi-vendor storage arrays. Most enterprises have storage arrays from multiple vendors, and each appliance will have different tiering solutions if any. This makes it difficult to roll out a consistent, global tiering policy across the organization. Look for a solution which can encompass all the storage devices you have today and might acquire tomorrow.
- Fast access to tiered data. Accessing data from the cloud will incur a much higher latency than accessing data on the local storage array. But once the cold data has been retrieved you can cache it locally to eliminate future latency and reduce egress costs. If the data set is large, the tiering solution should stream it so that users can access it even while the rest of it is being recalled.
- Native cloud access to tiered data. The tiering solution should allow access to the cold data directly using the cloud storages native access tools. For instance, if you tier cold data to AWS S3, you should be able to access the data directly from AWS using a standard S3 browser like CloudBerry. Unfortunately, most storage array tiering devices store the data in a proprietary format, meaning that you can only access it from the source storage array.
Cloud tiering is a practical and easy step on the path to the cloud. When done right, it gets you into the cloud while reducing storage costs. There are new solutions available that make this approach seamless with no disruption to users and your existing data protection workflows. By running an analysis of data assets across your storage ecosystem, and setting up policies for migration of cold data, you can ensure that data is always living in the best places from both a cost and business perspective.
May 21, 2021
Unqork Extends Automated No-Code SDLC Platform

Unqork today announced it has extended a no-code platform for managing the software development life cycle (SDLC) to include the ability to automatically document changes to application programming interfaces (APIs) and data models in addition to adding a point-in-time rollback capability.

Ken Gavranovic, head of strategy and engineering for Unqork, said Unqork 2021.5 also adds a range of dashboards for monitoring application dependencies, service logs, revisions, logs and metrics.

Finally, Unqork is adding tools to analyze, test and debug configurations along with enabling workflows to subscribe to events and adding a search capability for release candidates.

Unqork is designed to present IT teams with a set of visual tools through which they can manage SDLC processes without having to build and maintain a platform made up of disparate offerings that they need to then integrate, said Gavranovic. The goal is to reduce the overall size of the team required to maintain an application development environment to enable organizations to allocate more time and resources to building applications, he added.

The auto documentation for APIs is being added at a time when IT organizations are starting to employ a larger number of APIs as they transition toward building and deploying microservices-based applications. In addition to making it easier to reuse APIs, Unqork also now makes sure documentation for those APIs is always current, said Gavranovic.

At the same time, Unqork is now providing a means to automatically document changes to data models based on a data store that is integrated in the platform, noted Gavranovic. That capability enables organizations to implicitly add governance capabilities to workflows with a minimal amount of friction.

As organizations look to accelerate the rate at which applications are being developed to enable multiple digital business transformation initiatives, Gavranovic said more of them are starting to realize their existing processes are too complex and cumbersome to achieve that goal. Unqork is based on a data store that is shared by all the modules that make up the platform, which, in addition to enabling reusability, makes Unqork easier to deploy and manage, added Gavranovic.

In addition, Gavranovic also noted that, over time, complex processes only increase the overall burnout rate of the development team. It’s hard enough to hire and retain developers today without requiring them to navigate a complex application development environment, he added. Eventually, the best talent will migrate to organizations that make it as easy for them to write code as possible, said Gavranovic.

It’s not clear to what degree IT organizations will be revisiting the way they build software in the months ahead but as more of them realize how dependent the business now is on the timely delivery of software it’s only a matter of time before pressure to automate as much of the software development process as possible starts to mount. The cost of replacing an application development environment is, of course, not insubstantial. However, continuing to do the same thing with the expectation of producing a different result is, by definition, insanity.

May 20, 2021

What Data Catalogs Don’t Reveal

In today’s data intensive world, data catalogs, as the name implies, are simply catalogs of information (some human-entered and some extracted from metadata in systems). The importance of data in the enterprise has grown, and analytics plays an increasingly important role in generating value. So, while it’s important to collect and store and organize all this data, data catalogs are missing a major piece of the puzzle – they fail to draw accurate relationships between data assets and how they relate to (or are calculated from) each other.

It is like having a catalog that contains a description of all the electronics components in the world, but without an understanding of how to assemble those components into a clock, a radio or even a computer. Or how to diagnose what needs replacing when the device stops working.

Data catalogs cannot answer a whole host of questions along the lines of, “What are the consequences if I make a change to this dataset?” Even if the information in the data catalog is religiously updated, if a change was, in fact, made, users impacted by that change, whether directly and indirectly, have no way of knowing that a change was made. As a result, risk of unidentified issues is high.

The table below summarizes the strengths and weaknesses of traditional data catalog products.

Issue	Data Catalog Strengths	Data Catalog Weaknesses
Data Discovery	Collect metadata information from a wide variety of systems Give users flexibility to manually update information Simplify searching for data assets across a wide variety of systems Make it easy to identify the subject matter expert (SME) related to a dataset	Rely on manually maintained documentation Do not automatically relate data sets to each other Do not understand how data assets are calculated from each other Fail to identify who uses a particular dataset (and how they use it)
Data Governance	Provide a robust framework for managing the availability, usability, integrity and security of the data	Rely on manually maintained user-defined rules for dictating data access Cannot automatically propagate access rules from one dataset to antecedent datasets
Infosec Compliance	Make it easy for users to label personally identifiable information (PII) and other infosec risk data sets	Do not automatically infer the implications of data processing on the movement of PII inside the enterprise
Data Quality	Data catalogs do not actually inspect data – so they do not assist with data quality issues	Complete failure to identify issues and communicate who those issues impact downstream from a data quality issue

Data Discovery

Data catalog solutions provide powerful frameworks for collecting metadata and facilitating documentation of datasets. The collection and aggregation of metadata is automated, and data engineers/analysts are, hopefully (but rarely), compliant in documenting their work so others can leverage it.

Unfortunately, data catalogs fail to identify how datasets and processing rely on each other. There is no way to reliably identify which users access and rely on particular datasets, both directly and indirectly.

Tasked with this objective, a data engineer can manually answer these questions by searching and analyzing the relevant data processing code. This approach uses expensive data engineering resources, takes time and results in reduced analyst and data engineering productivity.

Data Governance

When new data sets are created, the authors must tag the data to control data governance. Not only does this rely on compliance from the author, but the rules must be propagated to determine who can use the newly processed data. How should the rules be propagated? This is a complex issue. For example, the city in an address may be restricted, but aggregates across that city may not be. Defining these rules, in and of themselves, is complex — applying them is even more complex.

Reliance on data engineers and analysts to comply with data governance documentation and systems for enforcement is error-prone and expensive. Sensitive datasets are at high risk of being exposed.

Infosec Compliance

Current solutions to information security compliance rely on:

Users manually documenting if their data processing activities use (or produce) infosec-sensitive data.
Automated identification of infosec-sensitive data by inspecting the actual data and using algorithms to identify how the data needs to be secured.

Data catalogs are used to facilitate user documentation of data processing activities to comply with infosec compliance requirements. This process relies on humans manually maintaining documentation and systems (and performing actual data inspection.) These processes present significant security risks in and of themselves, because both data inspection – which itself presents a risk – and skilled manual work is required.

Data Quality

Data engineering already has processes to identify when there is an issue landing data in a table. Automated systems are available that inspect data values and make intelligent decisions (some even based on machine learning) about whether or not the data is acceptable. These systems exist outside of traditional data catalogs.

The question is, who will that problem impact? You must identify who are the immediate users of the data. You must also identify the indirect users of the poor-quality data.

The current approach to this problem is to have skilled data engineers review all data processing code and manually identify users to warn – this is both expensive and time consuming. Beyond this, the risk is that a user is not identified; what if the error impacts a report that lands on the CFO’s desk and they are not warned?

Data catalog solutions are good at solving the problem for which they were initially designed – finding datasets that are of interest to data engineers and data analysts.

The risk in DevOps is that data catalogs are relied upon for data discovery, data governance, infosec compliance and data quality issues that they fail to address in today’s complex data processing environments.

May 12, 2021

Consider Telemetry When Rearchitecting Applications

Australia and New Zealand have seen a big move to the public cloud, and “… when I talk to customers, I see huge interest in rearchitecting applications rather than doing a lift-and-shift right into public cloud,” said Scott van Kalken, F5 ANZ solution architect for DevOps and modern applications.

That migration to the public cloud has led to massive interest in telemetry. When you go from a monolithic application to a distributed app, there may be 100 or even 1,000 APIs involved. Without telemetry, you don’t know what’s going on with those APIs. But the telemetry needs to be consistent.

F5’s 2021 State of Application Strategy report found 75% of respondents considered telemetry, especially about application security and delivery, important for meeting business outcomes, and 95% of respondents said they are missing insights from their existing monitoring and analytics solutions.

Van Kalken recounted a conversation with a practitioner as an example. This practitioner gets yelled at when problems arise with partner APIs used by the application he manages. So, even though these APIs are not technically his responsibility, he needs visibility into their status. Only by combining internal telemetry with telemetry from these partner APIs can he make sense of what’s happening within the application ecosystem.

And it’s not just a question of whether an API is available or not. Problems can arise when it is responding slowly. “‘Is that normal? Is that normal for this time of day?’ I’ve got to have some sort of telemetry data available to me to make decisions,” van Kalken said.

Enabling Security Automation with Telemetry

An aspect that’s particularly interesting to him is the ability to automate or otherwise take action based on security telemetry data.

A simple example is a security event that reveals some kind of systems abuse. Automation can add a firewall rule to block the associated traffic.

As organizations increasingly adopt event-driven architectures, events and responses occur at the application, network, infrastructure and other levels. “The minute that you automate those event-driven architectures with security, it opens up a world of possibilities.”

It’s not just about automating security. Imagine an API gateway that serves a variety of APIs to customers. Telemetry might reveal that two of the APIs are starting to slow down; in which case, “I might kick off a workflow that just deploys more of them so that we can handle the load. But if you don’t have that telemetry, you can’t do that,” he said.

In practice, this involves telemetry, analytics and automation.

“When we show [customers] how you can make use of the analytics in a really easy way, their eyes light up,” van Kalken said.

Multi-Cloud Migration

A further consideration is that rearchitecting applications isn’t just about splitting monoliths into smaller pieces and redeploying them as microservices. There’s also a desire to run them across multiple public clouds to reduce risk, and also to deploy applications at the edge where they are closer to their users.

The 2021 State of Application Strategy reported that just over three-quarters of respondents are already using or have plans to use the edge to capture benefits related to application deployment, performance and data availability.

“This is really important, because as an application developer, I want to deploy my application once and I want to say, ‘I want this distributed to all of these places around the globe’,” van Kalken said.

But if the application is deployed in, say, five locations, then from a telemetry perspective, it is five applications, not one.

“I’ve got five sets of telemetry, because I’ve effectively got five different applications deployed out there. It’s the same app, but it’s in five locations, and the consumers are different in every location; the way that they interact is different and the telemetry that I’m getting is going to be different,” he said. “So, I need to actually suck up all of that telemetry, and make it meaningful in some way,” he explained.

Telemetry Enables Clearer Vision

“A distributed cloud architecture is right for these modern applications. We recently bought a company called Volterra, and this is one of the areas that they play in — distributed applications, and specifically, abstracting multi-cloud and multi-cluster Kubernetes application deployments,” he said. This area of focus will be critical in the future, van Kalken said.

He predicts that as organizations move to 5G, applications will become increasingly distributed. If that does happen, it will be extremely important to receive telemetry from all components of the application so it can be analyzed and used to support intelligent decisions, as part of the distributed application deployment.

May 12, 2021
OpsRamp Extends AIOps Reach to Embrace Prometheus

OpsRamp today announced it has added the ability to ingest data captured via the open source Prometheus monitoring software into its namesake IT operations platform infused with machine learning algorithms.

In addition, the company is adding support for customizable dashboards using the Prometheus Query Language to create visualizations of hybrid infrastructure performance. Also, a set of available tools makes it easier for IT teams to discover when workloads have been added to a cloud. These capabilities became available with the Spring 2021 Release of OpsRamp’s IT operations platform infused with artificial intelligence (AIOps).

Finally, OpsRamp has also created an alert definition model that streamlines the alerts generated by the AIOps platform and its integrations for the Microsoft Azure cloud and the Cisco HyperFlex platform deployed in on-premises IT environments.

Deepak Jannu, senior director of product marketing for OpsRamp, said that as IT becomes more complex to manage, many IT teams are starting to realize that managing these environments without help from an AIOps platform won’t be possible. As such, much of the fear and loathing that was initially associated with AIOps is giving way to acceptance as IT teams start to realize these platforms augment their capabilities versus replace them, added Jannu.

Much of the complexity is being driven by the rise of cloud-native, microservices-based applications that tend to have lots of dependencies. OpsRamp has primarily been encountering Prometheus in these environments. However, Prometheus can also collect data from legacy applications, as well, so over time, the expectation is the open source monitoring tool will be employed more widely.

As AIOps platforms become more widely employed, there is no doubt job roles within IT organizations will change. As lower-level tasks become automated, IT teams should have more time to spend resolving complex issues once the AIOps platform has identified a baseline for how applications should optimally behave. In many cases, the need for personnel to perform rote tasks that no one in IT wants to actually do will be eliminated altogether.

DevOps teams that are committed to automating IT tasks will naturally be among the first to employ an AIOps platform to augment the capabilities of site reliability engineers (SREs) that are often challenging to hire and retain.

In general, Jannu noted that the observability that many IT organizations now demand can’t be achieved by simply collecting more metrics. IT organizations will need to invest in an AIOps platform capable of making sense of all the instrumentation data being collected. The challenge now will be defining the right division of labor between man and machine in an IT environment, Jannu added.

At this juncture, AIOps, to one degree or another, is inevitable. In many cases, machine learning algorithms are already being employed to optimize applications or a specific class of hardware. The decision IT teams will ultimately need to make is determining whether to replace their existing IT operations platform or wait for the provider of their existing platform to get around to adding AI capabilities.

May 11, 2021
ServiceNow Acquires Lightstep to Gain Observability Platform

ServiceNow today announced it has acquired Lightstep as part of an effort to add an observability platform to its IT operations portfolio. Terms of the deal, expected to close this quarter, were not disclosed.

The Change Intelligence observability platform from Lightstep is based on a time-series database that is capable of processing a trillion events each day. It was built by the same team that created Monarch, the globally distributed in-memory time series database system that Google employs to monitor its applications and systems. Lightstep team members also co-created both the OpenTracing and OpenTelemetry open source projects for collecting traces, metrics and logs.

Pablo Stern, senior vice president and general manager for IT workflow products at ServiceNow, said ServiceNow will employ Lightstep to enable IT organizations to observe both monolithic and microservices-based applications as IT operations and DevOps teams continue to collaborate more closely with one another. The goal is to provide a single pane of glass through which IT teams can observe their entire IT environment via the Now platform that ServiceNow makes available via a software-as-service (SaaS) delivery model.

That approach will enable DevOps teams to programmatically manage IT environments alongside IT administrators employing graphical tools, noted Stern.

Lightstep CEO Ben Sigelman added that, in time, the number of users of that observability will only increase as more application stakeholders start to track software development projects that are at the heart of various digital business transformation initiatives.

Thanks mainly to the rise of microservices that are now being deployed across highly distributed computing environments, it’s become more difficult to discover the root cause of an IT issue. Microservices-based applications have lots of dependencies that, while serving to make the overall application more resilient, conspire to make it difficult to determine which specific microservices might be adversely impacting the performance of an application.

The time-series database that lies underneath the Change Intelligence observability platform makes it feasible to go beyond logs and metrics to process distributed traces that provide more insight into application behavior. Ultimately, the goal is to enable organizations to innovate faster with a higher degree of confidence in the IT environment, said Sigelman.

As far as the march to observability in the enterprise is concerned, it’s still early days. While observability may be a core tenet of any DevOps practice, most IT organizations rely on myriad tools that surface data the teams then attempt to correlate. Whenever there is a major issue, those IT teams typically convene in a “war room” to compare reports generated by the various IT monitoring tools employed. Observability platforms that collect and process distributed traces used to instrument an application using a consistent set of specifications, frameworks and libraries. The OpenTracing and OpenTelemetry tools are significant because they provide an open source approach to instrumenting applications that reduces the total cost of observability. In comparison, many legacy application performance management (APM) platforms employ proprietary agents to instrument applications.

Naturally, each IT organizations will be moving down the path toward an observability platform that provides greater application context. At this point, however, it’s not clear to what degree observability platforms will need to be installed before making the transition to microservices versus being a capability that only gets appreciated once the trouble with microservices finally becomes apparent.

May 10, 2021
Survey Illuminates Long Dark Road to Hybrid Cloud

While hybrid cloud computing will play a critical role in enabling digital business transformation it turns out most IT organizations lack the skills and technologies required to achieve it.

A survey of 108 IT executives conducted by the research firm Pulse on behalf of CloudBolt Software , a provider of a cloud management platform, finds 94% believe hybrid cloud computing will be critical to digital transformation. A total of 92% also noted that empowering end users through self-service IT is a digital transformation accelerator, while 99% said successful automation depends on streamlining integrations.

However, 78% of respondents said they lack the visibility necessary to optimize cloud deployments. More than half (56%) believe that their self-service offerings fail to meet user needs and more than three-quarters (76%) still rely on some form of custom coding when it comes to integration.

In terms of the capabilities respondents are looking to acquire, 80% noted they want real-time insight into cost optimization and security remediation. A total of 71% want self-service to be easy for end users, while 62% want an approach to integration that doesn’t require deep domain knowledge or special expertise.

CloudBolt CEO Jeff Kukowski said the biggest immediate obstacle when it comes to hybrid cloud computing is the lack of visibility IT organizations have into their existing IT environments. IT teams are not going to be able to unify the management of distributed IT environments without it, much less achieve anything approach observability, noted Kukowski.

Pressure on IT organizations to accelerate application development while simultaneously reducing costs rose as the economic downturn brought on by the COVID-19 pandemic began. The challenge is that each new platform adopted by an organization tends to increase the total cost of IT as organization acquire tools and hire specialists to manage those platforms. It’s not likely all those platforms will be managed via a single console, but Kukowski said it’s clear IT organizations are moving toward a more federated approach that will enable them to centralize the management of IT.

In the meantime, Kukowski noted organization are collectively wasting billions of dollars on IT platforms that are employed inefficiently. Most of that waste stems from either overprovisioning of IT infrastructure, or simply because someone didn’t think to turn off a virtual machine left running on a cloud service.

It may be a while before IT organizations are able to strike that perfect balance between flexibility and control that currently remains elusive. However, organizations that focus on improving a few small things today will find themselves in a much better position to manage what is rapidly becoming an over extended enterprise tomorrow. Right now, Kukowski estimated only about a quarter of IT organizations have the sophistication required to manage a hybrid cloud computing environment.

There’s no doubt the management of IT will become more automated in the months and years ahead. The challenge organizations face today is determining what tasks either can or soon will be automated. Once that’s determined, it becomes a lot easier to identify what DevOps skills those IT teams will require tomorrow.

May 6, 2021
DevOps Practices Gap Assessment

Using DevOps to continuously deliver software has proven value for organizations that are on a path towards digital transformation and continuous improvement. To get the maximum value from DevOps, it is important to use recommended best practices, inclusive of people, processes and technologies. A gap assessment is a great way to efficiently evaluate an organization’s practices for DevOps, and determine a strategy for improvement. Gap analysis provides valuable inputs for formulating a strategy and roadmap to improve a topic including DevOps.

My SlideShare, Gap Survey, Assessment and Analysis For DevOps, explains how my Nine Pillars Gap Assessment Tool greatly speeds up the gap survey, gap assessment and gap analysis process for DevOps.

It is important to clarify a few definitions for this blog, because I have found that different people and organizations use these terms differently. In this blog, and in my consulting work, I use the following definitions.

• Gap survey is a discovery tool used to collect information about the current state of practices for a topic.
• Gap assessment is a process to determine differences (gaps) between current practices and recommended practices for a topic. Gap assessment is also used to refer to the entire collection of activities in this list.
• Gap assessment workshop is a meeting in which the gap assessment is validated prior to gap analysis.
• Gap analysis is the process of determining priorities for reducing gaps found during a gap assessment.
• Gap assessment tool is software that helps perform gap surveys, gap assessments and gap analysis.

For those that want to dig deeper on this topic, additional definitions and contextual information related to the way I do gap assessments are clarified in my book Engineering DevOps.

I define a seven-step process for gap assessments. The steps are as follows:

1. Pick a topic to analyze. The same processes and tools can be applied equally well to almost any topic.
2. Determine recommended practices for the topic. Something that is worth doing a gap assessment for, such as DevOps, will normally be fairly complex with dozens, if not hundreds of practices. Categorize the practices into groupings (I call the groups “pillars”) to make the list of practices more understandable and meaningful. To do this will require some research and work. For example, I have a database of practices and pillars for DevOps, continuous testing, DevSecOps and SRE and some other topics such as DevOps.
3. Enter the practices from step two into a gap assessment tool. The tool can be a simple survey tool you create, a commercial survey tool or something custom-made.
4. Determine who needs to be included in the gap survey.
5. Collect data using the gap survey.
6. Perform a gap assessment.
7. Perform a gap analysis.

For the remainder of this blog, I explain the above seven steps for DevOps.

Step one: Pick a Topic to Analyze
In this blog, the topic is DevOps.

Step two: Determine Recommended Practices for DevOps
For an example DevOps assessment, I used the practices under the nine practices categories described in a blog that I co-authored called Nine Pillars of DevOps Best Practices. The nine practice category pillars are:

• Leadership
• Collaborative culture
• Design for DevOps
• Continuous integration
• Continuous testing
• Continuous monitoring
• Elastic infrastructure
• Continuous delivery
• Continuous security

Step three: Enter the Practices from Step Two into a Gap Assessment Tool
A gap assessment tool, pre-loaded with a sample of DevOps practices, in a file called DevOps Assessment, can be downloaded for free from one of the resource pages found on my website Engineering DevOps. You can edit the practices categories and add/delete practices from each category if you prefer to make changes.

Step four: Determine Who Needs to be Included in the Gap Survey
For the gap analysis to be comprehensive, people in roles that are affected by DevOps practices need to be surveyed, or at least represented, to ensure their perspectives are included. The following are example roles that typically are included.

• Business leaders – because they influence culture and budgets for DevOps.
• Developers – because they need to design in accordance with DevOps practices.
• Project owners – because they influence product work priorities.
• QA testers – because they need to test in accordance with DevOps practices.
• Ops – because DevOps practices affect operations.
• Security – because DevOps affects security practices.

The survey can be conducted for an individual application, a group of applications, or all the applications in the enterprise. However, it is important that the gap assessment and gap analysis be performed on the organization segment that is being targeted for improvement.

Step five: Collect Data Using the Gap Survey
A gap survey should allow each surveyed person to enter an importance level score, a practice level score and comments for each practice. All this information is essential for the gap assessment and gap analysis. In the gap survey included within my gap assessment tool, survey respondents are asked to score the importance of each practice as one of 0=not relevant, 1=not important, 2=nice to Have, 3=important, 4=very important or 5=critical. Practice level choices are 0=not sure, 1=rarely, if ever, 2= sometimes, 3=most of the time, 4=always or 5=We are really good at this. Comments that are relevant to qualify or explain the scores entered for each practice should also be entered, especially if there is any doubt or ambiguity.

Step six: Perform a Gap Assessment
The gap assessment process requires all the practices scores collected from the surveys to be collected and assembled to calculate an aggregate set of scores. Gap scores are calculated using a formula that weights each practice level score with the corresponding importance level score. A visual representation helps to identify practices areas and individual practices that have the highest, most important gaps.

No matter how professionally written the practices and score definitions are, it is not unusual for some people to misunderstand and to enter scores that they would not have otherwise. For this reason, it is important to ensure that the data collected is validated before conducting the gap analysis. The preferred approach is to conduct a gap assessment workshop with key representatives from each role that participated in the survey. During the workshop, the scores for each practice that have a high deviation between survey responses are discussed and, if necessary, adjusted.

Step seven: Perform a Gap Analysis
The gap analysis process involves extracting the high gap practices and tagging and ranking each of them against solution categories determined by a consultant or topic expert.

The results of the gap analysis indicate where solution strategies and implementation roadmaps need to be focused to reduce the most important gaps.

What This Means

While it is true you can easily stand up DevOps tools quite quickly, this alone will not realize the benefits of following effective and safe DevOps practices. Those who are serious about DevOps will do well to learn from others, and be serious about using recommended DevOps practices. This blog explained a gap assessment approach, which uses my nine pillars of best practices for DevOps. My free DevOps gap assessment tool facilitates getting a gap analysis for DevOps and priorities to improve DevOps practices for an organization.

May 4, 2021