Skip to content

staging-devopsy.kinsta.cloud

Latest
Topics
Videos/Podcasts
Webinars
Library
Events
- Upcoming Events
- On-Demand Events
Sponsored Content
Related Sites
Media Kit
About
Sponsor
SEARCH BOX

Tag: Platform

Resourcely Adds Free Tier to Cloud Configuration Management Platform

Resourcely has added a free tier option for its platform for safely configuring and deploying cloud resources using a set of templates and guardrails for application developers using either open source Terraform or OpenTofu infrastructure-as-code (IaC) tools.

Company CEO Travis McPeak said the platform makes it possible for DevOps teams to centrally govern the configuration of cloud resources in a way that doesn’t adversely impact application developer productivity.

Using a set of blueprints that are managed via the Resourcely platform, DevOps teams can embed policies into workflows that prevent application developers from misconfiguring cloud infrastructure resources, he said. The platform generates code that is submitted like any other pull request managed by a continuous integration/continuous delivery (CI/CD) platform.
The free tier includes access to a proprietary Knowledge Graph that collects data about cloud services and their properties. DevOps teams can use that data to pre-populate and auto-fill instances and regions to provide developers with self-service capabilities.

Resourcely is also providing early access to a Campaign capability that enables DevOps teams to define configuration standards, identify and track resources in violation, and automatically generate Terraform code that remediates misconfigured infrastructure.

Longer term, Resourcely intends to also add support for YAML files used to configure Kubernetes clusters as well as the other automation framework such as Ansible and Cloud Formation from Amazon Web Services (AWS).

Given the myriad options developers are presented with when configuring cloud resources, it’s simply too easy for them to make a mistake that might, for example, leave a port open through which cybercriminals can export data, noted McPeak. Resourcely enables organizations to ensure that best DevSecOps practices are being followed, he added.

Naturally. Resourcely hopes that organizations using its free tier will eventually migrate to the more robust paid service plans the company already provides. Many of the cloud security issues that organizations encounter can be traced back to lack of understanding of the shared responsibility model that cloud service providers have imposed. While the cloud service provider is responsible for securing their infrastructure, the security of the applications and the code used to configure those platforms is the responsibility of organizations that are now deploying software at scale in the cloud.

Unfortunately, many application developers still tend to assume that cloud service providers are automatically securing configuration code on their behalf, and in the absence of any effort to secure that code, many organizations are providing cybercriminals with a relatively simple method to compromise their software supply chains. As such, DevOps teams are under more pressure than ever to improve cloud security hygiene, noted McPeak. The challenge is making sure that DevOps teams, for want of a few simple policies that were not being enforced, don’t make it too easy for cybercriminals to achieve their aims, he added.

It’s not clear how often cloud services are being compromised because of misconfigurations versus, for example, cybercriminals simply stealing the credentials of a developer. Regardless of how a cloud computing environment is compromised, it’s apparent that as more workloads are deployed in the cloud there are more attempts with each passing day being made to create a breach.

October 21, 2024
Buildkite Launches Integrated Platform to Manage DevOps Workflows at Scale

Buildkite Pty Ltd today launched a Scale-Out Delivery Platform that combines multiple previous DevOps tools into a single integrated platform.

In addition to the continuous integration/continuous delivery (CI/CD) platform the company developed, the Scale-Out Delivery Platform includes Buildkite Test Engine, Buildkite Package Registries and Buildkite Mobile Delivery Cloud.

Buildkite CEO Keith Pitt said the iteration of the DevOps platform will make it simpler for organizations to implement best practices by eliminating the need to separately install and configure additional tools.

The Buildkite platform is designed to enable DevOps teams to take advantage of a concurrency capability that enables them to run 100 times as many agents and workflows as other platforms, said Pitt. At the core of that proprietary platform is a cloud-based service that enables DevOps teams to self-host customizable open source agents.

That approach makes it possible to efficiently process thousands of daily commits across massive codebases as they see fit, while also ensuring proprietary code and secrets are not made accessible to third parties, including Buildkite.

Organizations such as Airbnb, Block, Canva, Cruise, Culture Amp, Elastic, Lyft, PagerDuty, Pinterest, PlanetScale, Rippling, Shopify, Slack, Tinder, Twilio, Uber and Wayfair all opt to pay for that capability rather than rely on an open source platform because it enables them to build, deploy and update applications at scale, he added. In fact, organizations that rely on antiquated platforms for building software are essentially telling their application developers that they don’t view application development as a key business differentiator, noted Pitt.

Successful organizations are now moving beyond the CI/CD fundamentals to leverage integrated DevOps platforms that dramatically increase both productivity and the quality of the code created, said Pitt, Those organizations, for example, don’t allow flaky tests that waste precious cycles to run at all, he noted.

As the volume of code being generated in the age of generative artificial intelligence (AI) continues to increase, it’s only a matter of time before legacy DevOps platforms will collapse under the strain, said Pitt.

It’s not clear to what degree DevOps teams are currently evaluating the platforms and tools they employ but as they, for example, embrace platform engineering as a methodology for managing DevOps at scale they are naturally considering their options. Organizations that appreciate the degree to which they are dependent on software to remain competitive also have an increased appreciation for the need to ensure that only software of the highest quality gets deployed in production environments. Organizations that are serious about software development are also run by DevOps leaders that regularly “walk the manufacturing floor” to ensure there are no defects and that any changes made to existing applications are done safely, noted Pitt.

No two organizations, of course, have embraced DevOps workflows to the same degree, but just about all are looking to continuously evolve and improve. Of course, achieving that goal might be easier to achieve once AI agents begin to automate tasks in a way that eliminates many of the bottlenecks that today conspire to slow down the pace of application development.

The issue, as always, is making sure that, whatever the next advance, the underlying DevOps platform being depended upon is actually up to the challenge.

October 9, 2024
Don’t Shift Left Without a Platform
Shifting left was a term created by Larry Smith in 2001. It referred to the software engineering lifecycle. At the time, it was normal to build everything, then test everything and then release everything. Larry Smith argued that it is best to bring testing to an earlier point in the project pipeline, hence shifting left.

From here, things started to get interesting. In 2007, DevOps came together as a movement. A clear consequence of DevOps was bringing deployment earlier in the project pipeline, in other words, shifting left. Around 2015, security got into the fray, which gave birth to DevSecOps.

The trend is clear. More and more items were taken out of the hands of specialists, who were turned into enablers (more on this in a moment) and were put into the hands of engineers, who had to rapidly become generalists.

So, Why is This a Problem?

Let’s first consider a traditional approach to software engineering. The developers build the entire iteration. The testers test it and find bugs. There is a lot of back and forth. Then the security team analyzes and the databases team builds queries and stores procedures before the ops team pushes it to production. This has the massive drawback of big batch engineering.

Smaller batches equal better flow, and better flow equals more predictable software delivery. Therefore, we need to break down our work into much smaller increments. However, now, instead of the testing team dealing with one large request, they are inundated with dozens of requests from every application team that builds a different component of the system. This in turn will rapidly become a bottleneck.

The Bottleneck Problem of Specialists

When there is only one person who knows how to complete a task, as an organization scales, that person will become the main constraint on productivity. They are known as single points of failure (SPOF). They need to be in every review, present for every release and are a key part of any design session.

Specialists are crucial to an organization, but their inability to scale themselves is a huge problem, which is driving organizations to focus on generalists for the cost-saving potential (one person can do five things) and the promise of flexibility and therefore a lack of hard constraints. This isn’t a bad strategy, provided you don’t have any specialist requirements. But as the saying goes, you don’t know you need a plumber until your house is flooded.

From Specialist to Enabler

Team Topologies outlines four different types of teams, one of which is the enabling team. This team removes blockers and makes it possible for teams to go live. How they do it can be broken down into a few options: Facilitation, collaboration or ‘X as a service’. In this, we can see the fertile ground-for-a-platform mindset.

Without a platform, we are a group of individuals with specialist knowledge, all seeking to collaborate and, therefore, constrained to the pace of our slowest link. With a platform, we are each building solutions to force multiply our presence in the organization. Our goal is to amplify the impact that our knowledge has. What does this look like?

From Operations to SRE

Rather than an operations team that knows the quirks of deployment and handles it all personally, an SRE team builds solutions that make it possible for teams to deploy safely. They implement guard rails, deploy testing strategies, build self-healing mechanisms and more. All these things make it as if they were in every team at once.

From InfoSec to SecOps

Instead of an army of clipboard-wielding employees, whose job it is to police the activities of the teams, it would be more effective to have a collection of compliance and engineering-minded folks, who are constantly looking for vulnerabilities, consulting in teams, building features and contributing into the shared platform. This, again, greatly increases the team’s reach and cuts down on the number of meetings necessary to ship a product into production.

A Classic Example

Let’s picture an organization that runs Kubernetes. There are security professionals, operations professionals, engineers and more. Rather than working in a silo, demanding meetings of one another to ensure compliance with a framework, they collaborate on creating a pipeline. This pipeline ensures a few things:
- All code has a minimum test coverage (typically 80%)
- All dependencies are free of known CVEs
- The code itself does not have any known vulnerabilities, like an SQL injection attack
- The container file does not contain any CVEs or it is an image from a pre-scanned
Then the application is deployed using a shared helm chart, which has another set of responsibilities:
- Network policies ensure that only the necessary ports are open
- Taints, cordons, affinities and node selectors are defined to ensure this workload runs on the correct server
- The application has a health check that is used for liveness checks
- The application has a HorizontalPodAutoscaler to ensure it can scale to meet
Rather than endless meetings to ensure compliance for every release, each stakeholder team must build confidence in the helm chart. This confidence translates into trust, and that trust unlocks a whole new level of collaboration.

Shifting Left Needs Force Multiplication

If you want to shift your processes left, reducing batch sizes and improving flow — then the challenge you need to maintain is this: How do you keep the rigor of specialists when things are being run by generalists? The answer is a platform.

Platform thinking transforms a thousand meetings into a few hundred lines of code, and if you wish to shift left without sacrificing the insight and wisdom of your specialists, it is the only way to ensure that every team benefits from their knowledge without undue constraints on their ability to deliver software.
August 20, 2024
StarTree Makes Observability Case for Apache Pinot Database

StarTree broadly enhances its real-time database platform by adding query support for metrics, logs and traces.

StarTree today made a case for becoming the foundation upon which observability is gained in IT environments by adding support for metrics, logs and traces to its real-time database platform delivered as a cloud service.

In addition, the company is also making generally available a StarTree ThirdEye service for detecting anomalies and identifying the root cause of application issues. It has also made a free serverless computing tier available on its cloud service.

Viable Alternative to the Legacy Database

Announced at the Real-Time Analytics Summit conference, StarTree Cloud is based on Apache Pinot, an open source columnar database optimized for real-time analytics, originally developed by LinkedIn. StarTree, via a private preview, is signaling that it views Apace Pinot as a viable alternative to the legacy database, upon which observability platforms DevOps teams use to troubleshoot IT environments.

StarTree, to further advance Apache Pinot, is also now adding a vector indexing capability to Apache Pinot along with support for a write application programming interface (API) to facilitate real-time synchronization with extract, transform and load (ELT) pipelines and additional integrations to Grafana and Tableau to improve visualizations.

Chinmay Soman, head of product for StarTree, said the issue with existing databases is they are not designed to scale to the level of performance that modern application environments will require.

Early Days for Achieving Observability

It’s still early days for achieving observability in application environments but many DevOps teams are now moving well past simply monitoring a set of pre-determined metrics. Observability platforms enable DevOps teams to launch queries that make it easier to troubleshoot applications. Not every DevOps team knows what queries to investigate. With the rise of artificial intelligence (AI), it’s becoming more feasible to rely on machine learning algorithms to analyze application environments using metrics, logs and traces stored in an observability platform.

Few DevOps teams will build an observability platform themselves but the capabilities of the underlying databases on they are based are critical, said Soman. DevOps teams should make certain the databases used in these platforms today will be able to scale to meet real-time processing requirements that will only steadily increase as more complex applications are deployed across distributed computing environments, he added.

In the meantime, StarTree is moving toward providing more observability capabilities via the managed database service it provides via the cloud.

It’s only a matter of time before DevOps teams need deeper levels of visibility to manage modern application environments in real time. As organizations become more dependent on software than ever, even the most minor glitch can now have a major financial impact. The challenge DevOps teams face, of course, is finding the funding needed to acquire these capabilities.

In the meantime, DevOps teams should at the very least start assessing which of the many monitoring tools being employed today might one day be rationalized by an observability platform, that in time might lower total costs by streamlining DevOps workflows that today are often more disjointed than anyone cares to admit.

May 8, 2024
Survey Surfaces Substantial Platform Engineering Gains

A global survey of 500 application development and deployment professionals finds that while 43% work for organizations that have had a platform team in place for three to five years, there doesn’t appear to be much consistency regarding where that team reports within an organization.

The survey, conducted by Puppet by Perforce, a provider of an automation framework, found that among organizations that have platform teams, well over half (58%) operate within the context of a larger DevOps or infrastructure management organization, compared to 40% that have their own dedicated leadership.

David Sandilands, a principal solutions architect for Puppet by Perforce, said the survey makes it clear that platform engineering, in general, is not a new concept, but it’s clear more organizations are trying to reduce the level of toil currently associated with building and deploying applications.

Regardless of where platform teams reside in an organization, nearly two-thirds of respondents (65%) said they are important to their organizations and would receive continued funding. The primary purpose for forming these teams is to increase productivity (58%) and automate standardized processes (51%), the survey finds. Top goals for platform engineering teams are to solve problems (30%), enforce security processes (27%) and accelerate transitions to cloud-native computing environments (26%). A full 70% said security was built into their platforms from the start.

That suggests that, in addition to assuming responsibility for traditional DevOps workflows, many platform teams are now also being asked to ensure DevSecOps best practices are followed at a time when application security requirements are becoming more stringent, noted Sandilands.

More than three quarters (76%) also reported that they had deployed two or more self-service portals, with 27% having deployed five or more.

Only 22%, however, said they have deployed applications on Kubernetes in a production environment, with just under half (46%) reporting they have no current plans to deploy a cloud-native computing platform, which are widely seen to be challenging to deploy and maintain.

Overall, the survey makes it clear the primary focus is to ensure platforms continue to evolve as developer requirements continue to evolve, said Sandilands. Platform teams are not necessarily trying to dictate what tools should be used; instead, they are finding ways to automate processes that are increasing the level of friction developers encounter, he added.

In the longer term, advances in artificial intelligence (AI) will make it simpler to achieve that goal using natural language interfaces that DevOps teams can more easily invoke, noted Sandilands.

Each organization will need to decide for itself whether platform engineering makes sense as a methodology for managing DevOps at scale, but as application development and deployment continues to evolve, more nuanced approaches will be required. In general, organizations are trying to strike a balance between empowering developers and the need to streamline backend processes to increase productivity while also simplifying compliance and improving security.

The challenge, of course, is enticing developers to buy into that concept rather than using their expertise to resist platform teams that might, from their perspective, limit their prerogatives in ways that ultimately stifle innovation.

March 27, 2024
Digibee Adds AI Tool to Accelerate Migration to Integration Platform

Digibee is leveraging artificial intelligence (AI) to make it simpler to migrate to its integrated platform-as-a-service (iPaaS) environment by converting code into a JavaScript Object Notation (JSON) format.

A translator tool created by Digibee invokes large language models (LLM) to convert that code into a JSON file that can then run on the Digibee Integration Platform.

Initially, that capability is being made available to facilitate migrations from the Mulesoft integration platform made available by Salesforce, with support for other legacy integration platforms planned for 2024.

Digibee CTO Peter Kreslins Junior said this approach reduced the level of time and effort required to switch platforms by using generative AI to map and refactor legacy integration code. That refactoring process doesn’t completely automate the reverse engineering of workflows, but it does substantially reduce the total cost of switching platforms by, for example, automating data migration and simplifying the transfer of configurations, he noted.

Generative AI also eliminates the need to rely on documentation or maps that either may be limited or non-existent, added Kreslins. Instead, all the integration patterns used can now be automatically discovered by an LLM, he added.

IT teams will still need to vet and test the code created by the translator tool, but overall, the speed at which any migration effort is made should be accelerated, thanks to the rise of generative AI, in 2024. Many organizations are locked into various platforms today simply because the effort required to rework the code that has been created to extend them has required either too much effort on their part or the contracting of expensive consultants. A migration project that might have taken a year or more can now be accomplished in a few days, said Kreslins. The overall goal is to reduce the current level of pain IT teams experience when migrating between platforms, he added.

It’s not clear if the rise of generative AI platforms that make converting code into other formats and languages simpler will drive a wave of migrations, but there are many organizations relying on legacy platforms that are not nearly as easy to use as more modern alternatives. As the time and effort required to make those transitions is reduced, more IT teams will be, at the very least, more open to considering their platform options. The providers of those legacy platforms will naturally have to invest in new capabilities much faster to discourage customers from migrating to another platform.

In the meantime, the amount of code that will need to be integrated is about to exponentially increase as developers rely more on LLMs to generate code. The downstream implications of that increased pace of application development for DevOps teams are profound. It’s not clear to what degree those teams can increase the level of scale at which they today manage application deployments.

The one thing that is clear is there is no going back because, so far as application development and deployment are concerned, the proverbial AI genie is out of the bottle.

December 20, 2023
Opsera Leverages AI to Simplify DevOps Platform Migrations

Opsera is leveraging a generative artificial intelligence (AI) model it built to enable DevOps teams to migrate from one platform to another.

Gregg Holtzrichter, chief marketing officer for Opsera, said the Hummingbird AI model makes it simpler to migrate source code from, for example, a Bitbucket on-premises continuous integration/continuous development (CI/CD) platform to GitHub.

Opsera originally developed Hummingbird as an orchestration engine that integrates DevOps workflows spanning multiple tools and platforms. The primary goal is to make it possible for DevOps teams to unify their workflows, said Holtzrichter.

At the same time, Opsera is now making it possible to leverage a generative AI model to reverse engineer the source code a DevOps team may have developed to extend the capabilities of a specific platform. That new code then makes it possible to significantly reduce the total cost of switching DevOps platforms, said Holtzrichter. As part of such an effort, Opsera is also now providing support for organizations looking to specifically migrate to GitHub Copilot and GitHub Actions.

It’s not clear how many DevOps teams might be looking to switch platforms, but with the rise of generative AI capabilities for writing code, interest in platforms such as GitHub has increased, noted Holtzrichter. In addition, support for platforms such as the on-premises edition of Bitbucket is being sunsetted by Atlassian as part of an effort to drive more customers toward the cloud edition of the CI/CD platform.

Regardless of the underlying platforms involved, generative AI promises to reduce the ability of any vendor to lock customers into a specific platform. As DevOps teams extended DevOps platforms over the past decade, many of them have discovered that the integrations they created and maintained made it all but impossible to switch platforms. With the ability to refactor much of that code based on the insights surfaced by the Opsera platform using generative AI, the time and effort required to switch platforms has been substantially reduced, noted Holtzrichter.

At its core, Opsera makes it possible to apply analytics to DevOps workflows as they become more complex. The number of dependencies that exist between various application modules, for example, makes it exceedingly difficult for DevOps teams to manually identify the impact a delay in one project might have on any number of other downstream projects.

The Opsera platform surfaces that intelligence by scanning all repositories and branches, including commit history and generating alerts and notifications whenever KPI thresholds or compliance requirements are exceeded. DevOps teams can also use the platform to assess their overall DevOps maturity using, for example, the DevOps Research and Assessment (DORA) metrics defined by Google. In addition, many organizations are keen to discover what best practices should be adopted by other application development teams.

Now that AI is being applied to that analytics capability, it’s become possible to extend the reach of the Ospera platform beyond analytics to include source code migration. Each organization will still need to determine whether such an effort is worth the remaining time and effort still required, but at the very least, most DevOps teams can take comfort in the fact they can now keep their options a lot more open.

December 19, 2023
Five Great DevOps Job Opportunities

staging-devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

SimplyHired.com

Selby Jennings
Chicago, Illinois
AWS DevOps Architect
$191,000 to $242,000

Indeed.com

Verizon
Alpharetta, Georgia
DevOps Architect
$159,000 to $201,000

LinkedIn

JetBlue
New York, New York
Senior Principal DevOps Engineer
$134,600 to $210,100

Dice

Northrop Grumman
Dulles, Virginia
Senior Principal DevOps Engineer
$120,900 to $181,300

CareerBuilder.com

Octo
Washington, D.C.
Platform DevOps Cloud Engineer
$110,000 to $170,000

December 4, 2023
Preparing for the Shift to Platform Engineering

Platform engineering is one of the hottest trends in the software delivery world and is expected to represent the next evolution of DevOps. According to Gartner’s 2023 study of top strategic technology, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery by 2026. What used to happen in one day in the developer world now needs to happen in five minutes. With high pressure for organizations to scale operations quickly, especially in a marketplace clamoring for the benefits of artificial intelligence (AI), automation is increasingly important in platform engineering. It reduces manual errors, streamlines the development process and boosts growth and efficiency. As technology advances rapidly, it becomes more critical for organizations to proactively navigate newly emerging trends within platform engineering to remain competitive. The keys to successful platform engineering include cloud-native architecture, AI and machine learning (ML), and evolving tools that use intuitive interfaces and pre-built components to create functional applications.

Platform Engineering is the Natural Progression of DevOps

Platform engineering expands on the practices of DevOps by moving development out of silos and taking a collaborative, bird’s-eye view of an organization’s technology. Platform engineering is built on a foundational infrastructure that enables self-service capabilities so developers can deploy code faster, more reliably and more securely. While platform engineering represents the next wave of change in DevOps, it’s essential for leaders and developers to understand that it isn’t a singular technology or approach. The shift to platform engineering also represents a cultural shift for developers and leaders to address the needs, challenges and rapid pace of modern software development by optimizing processes, accelerating delivery and reducing the complexity of operations. For platform engineering to succeed, it’s critical to approach it from the perspective of a technology-driven culture that creates solutions with a developer’s perspective in mind. By seeking to improve the developer experience first, enterprises can achieve better results.

Trends Shaping the Landscape

Platform engineering itself is an emerging trend, but several essential ingredients shape how it will impact businesses today and in the future. For organizations to reap its benefits, understanding these essential trends will keep them ahead of the pack.

Cloud-native architecture is fundamental to modern platform engineering by designing and building applications for cloud environments. Infrastructure-as-code (IaC) is a key component, enabling the automated provisioning and management of resources. Cloud-native platforms like Kubernetes, Terraform or Docker facilitate container orchestration and scaling, while serverless computing isolates infrastructure management entirely. This trend ensures that platform engineering is agile, scalable and cost-effective, with minimal downtime and easier maintenance.

Automation is pivotal in platform engineering, reducing manual errors and streamlining deployment workflows. It allows developers to build, test and deploy software more efficiently, leading to faster and more reliable product releases. In fact, 43.2% of early adopters of platform engineering saw faster product delivery. Automation tools like Jenkins, Ansible and Terraform are commonly used to automate infrastructure provisioning, configuration management and deployment pipelines. Machine learning (ML) models can perform automated troubleshooting by analyzing historical data to identify patterns and predict potential issues, proactively addressing problems and minimizing downtime. This also helps with performance optimization. By using AI that can analyze application performance data, automation takes platform engineering to the next level by suggesting optimizations to improve resource utilization and user experience. AI-powered tools assist in generating code or script, which is vital to reducing development time and human error. ML algorithms empower the security side of platform engineering, a top priority for any developer. It’s possible to leverage automated security tests during development and maintenance, even detecting and responding to security threats in real-time. When AI handles routine tasks, developers are less likely to feel overwhelmed or experience burnout. This allows developers to focus on more creative tasks, complex problem-solving and innovation, resulting in more efficient operations to help them achieve higher levels of productivity, security and consistency.

Evolving development tools that create functional applications using intuitive interfaces and pre-built components increasingly simplify application development and deployment. Low-code and no-code platforms allow developers to create applications with minimal coding effort, utilizing visual interfaces and pre-built modules. Serverless frameworks like AWS Lambda and Azure Functions enable developers to focus solely on writing code. An increasing reliance on and advancement of application programming interface (API) gateways provides a simplified way to manage and expose APIs, enhancing application integration and connectivity. Containerization platforms are becoming similarly powerful tools. Platforms like Docker and Kubernetes simplify application deployment across diverse environments by offering standardized containers and orchestration. Adopting a strategic suite of these evolving tools requires a clear understanding of product goals, organizational goals and developmental capabilities.

How Leaders can Support Platform Engineering

It’s projected that 30% of organizations are currently evaluating the adoption of platform engineering practices, and its prevalence will only grow. While platform engineering’s trends are empowered by digital transformation and new technologies, it’s vital to remember that platform engineering represents a shift in DevOps culture. To effectively support the transition, leaders must commit to a culture of platform engineering. Simply adopting technology isn’t enough. It needs to be backed by a thorough strategy that allows developers to truly benefit from the tools and structures of platform engineering. What does this look like? Success requires leaders and developers to encourage collaboration and break down silos between operations and development teams. It’s possible to build a bridge between developers and operations by committing to cloud migration, creating a centralized platform and investing in collaborative tools and the strategy to back it up.

To engage in platform engineering requires dedication to a collaborative culture instigated from the top, empowered by overall strategic decisions and operations. This includes continued learning for developers to stay on top of new languages, trends, challenges and priorities, internally and externally. Teams are more successful when they utilize performance metrics to track workflows that help them conduct effective maintenance and improve on a consistent and ongoing basis.

Embracing Change is Embracing Progress

In the fast-paced and continuously improving world of software development, it’s easy to think that each wave of technology will replace the last. Of course, experts know this isn’t true. Iteration is vital to advancing technologies, and it’s important to embrace the next iteration of progress to stay competitive. Automation and platform engineering are tools that enable developers rather than replace them. It’s important to note that platform engineering doesn’t mark the end of DevOps. Rather, it takes the core foundations that DevOps has built to the next level to empower developers with self-service capabilities, foster collaboration, enhance scalability, accelerate deployment and instigate sustained business growth.

October 19, 2023
How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

In a digital world, data is often the differentiator between success and failure. Whether it’s defending against cybersecurity threats, improving application performance, or resolving full-blown outages, data is critical to the modern DevOps team.

In fact, there are plenty of situations where DevOps engineers, SRE teams and other observability-focused departments need to explore reams of data rapidly and flexibly. An SRE trying to figure out the root cause of latency may need to dissect the movement of data across endpoints to find the malfunctioning one. Another engineer may need to compare historical and current performance to find anomalous behavior. Lastly, a DevOps engineer may have to dissect user performance metrics to understand the scope of a slowdown—whether it is global or regional.

Every use case requires dashboards that can provide detailed data through a wide variety of visualizations, including choropleth maps, stack areas, pie charts and bar graphs. Ideally, a dashboard will enable teams to isolate dimensions, apply filters, and dive into data for deeper insights.

Unfortunately, not all dashboards can support this type of fast, flexible analysis. Many still utilize outdated technologies, which lack the flexibility, agility and scalability required to seamlessly explore data. Many were also designed without the urgency that today’s data requires—after all, in previous years, most uses for dashboards (such as internal reporting) were not time-sensitive.

Other dashboards are constrained by templates, which offer a finite array of widgets, tools and drill-down capabilities. These dashboards may feel unwieldy and sluggish, unsuited for a rapidly evolving situation like an application outage.

Requirements

To truly fulfill the promise of self-service, ad-hoc analytics across huge datasets, teams need to work in real-time—and so they need a database capable of timely responses.

When an application goes down, an SRE may not know what to look for and, hence, needs to quickly comb through lots of data. While some dashboards rely on workarounds for faster queries, such as pre-aggregations, precomputing, or rollups, this is not possible in this instance because the SRE simply won’t know what to look for. After all, they can’t necessarily predict what will go wrong, and even if they work on assumptions from previous failures, this current issue could be much different.

Therefore, dashboards must provide plenty of functionalities and visualizations. Users should be able to filter data by time, isolate variables like location, and zoom into specific time intervals with a few clicks. Dashboards should also accommodate diverse data types, including intricate parent-child relationships and nested columns.

Further, dashboards must offer a depth of insight. An online streaming media platform may need to assess user metrics (such as latency or load times) across different devices, operating systems, and regions to fine-tune performance. A cloud provider has to monitor their physical hardware for high temperatures, slowed network switches or devices, and other anomalies.

Because data is now so important to success, many more people across a company, including data scientists, product managers and external users, themselves require data-driven insights. In these situations, dashboards must handle the increased user traffic and query activity, maintaining fast responses even under load. This is especially important considering that a single-user operation (such as a zoom) will require multiple queries on the backend to execute.

A database also must scale seamlessly. If an organization’s environment generates millions of events an hour, that equates to billions of events per day or week—challenging for any database to ingest, store, analyze and query. In fact, many databases cannot successfully provide fast response times while managing large datasets and high query volume. As an example, transactional databases (OLTP) can often query rapidly but cannot execute analytics at scale, while analytical databases (OLAP) can analyze massive volumes of data but not at speed.

Apache Druid for Independent, Ad-hoc Data Exploration

This is where open source Apache Druid comes in. Combining the scale and advanced analytics of an OLAP database with the speed of an OLTP database, Druid offers swift, real-time data exploration.

Upon ingesting data, Druid makes it immediately available for querying and analysis, removing the need to first batch or aggregate the data in some way. In addition, Druid natively integrates with streaming technologies like Apache Kafka and Amazon Kinesis, removing the need for workarounds or connectors.

Druid powers interactive visualizations, providing millisecond response times, enabling more versatile exploration, expanding the range of available dimensions and filters, and even maintaining subsecond speeds in the face of surging user and query volumes.

Druid’s unique architecture also enables easy scaling. By separating key duties among separate node types—data nodes for storage, master nodes for data ingestion and availability, and query nodes for executing queries and returning results via the scatter/gather method—Druid ensures that nodes can be independently scaled based on need. Afterward, Druid also automatically rebalances traffic to ensure consistent performance.

Salesforce: A Druid Success Story

Salesforce pioneered the customer relationship management (CRM) space, serves 150,000 customers worldwide and earns billions in annual revenue.

The Edge Intelligence Team is the division of Salesforce that tackles the massive task of ingesting, processing, filtering, aggregating, and querying the entirety of their log data — anywhere from billions to trillions of lines daily. Each minute, Salesforce ingests 200 million metrics, while each day, Salesforce processes five billion daily events globally. In total, Salesforce accumulates dozens of petabytes of data in their transactional store, five petabytes of logs in their data centers, and almost 200 petabytes in their Hadoop storage.

Salesforce teams use Druid to unlock real-time insights into product performance and user experiences, diving into large datasets instantly. Anyone from engineers to account executives can query a wide variety of dimensions, filters and aggregations to better understand trends, troubleshoot any issues that arise and set strategies for the future.

By using Druid’s compaction abilities, Salesforce also decreased the number of Druid rows by 82%, leading to an overall reduction of their storage footprint by 47% and accelerating their query response times by 30%.

October 19, 2023

1 2 3

staging-devopsy.kinsta.cloud

Where the world meets DevOps

Blog
About
FAQs
Authors

Events
Shop
Patterns
Themes

Twenty Twenty-Five

Designed with WordPress

[wpp_insert]

Modern Software Development and Delivery

1Q1

2Q2

3Q3

4Q4

5Q5

6Q6

How would you best describe your organization's current software delivery environment(s) on mainframe computers? (Select all that apply)(Required)

Primarily manual or legacy delivery processes

Automated CI/CD in limited areas

Hybrid environment with multiple delivery platforms and toolchains

Mostly standardized CI/CD platform

Intelligent, governed software delivery platform

My organization does not utilize mainframes

Not sure / don't know

Which outcomes are most important to your organization's software delivery strategy today? (Select up to three)(Required)

Accelerating software delivery

Automating build, test, and deployment workflows

Integrating security and compliance into delivery

Providing governance and auditability

Supporting AI-assisted software delivery

Improving software supply chain visibility

Standardizing delivery across teams

Supporting both modern and legacy application delivery

Other

Not sure / don't know

What, if anything, is limiting your organization's mainframe software delivery progress? (Select up to three)(Required)

Legacy applications or legacy development processes

Fragmented tools and disconnected workflows

Security, governance, or compliance requirements

Limited automation across the delivery lifecycle

Difficulty integrating AI into software delivery workflows

Skills or resource constraints

Organizational or cultural resistance to change

Limited visibility across the software delivery lifecycle

Budget or investment constraints

Other

Nothing is significantly limiting our progress

Not sure / don't know

In which areas of your mainframe software delivery environment are you currently using AI? (Select all that apply)(Required)

Code generation / developer assistance

Automated testing / test generation

Build and pipeline optimization

Deployment automation

Monitoring and incident response

Security / compliance automation

Other

We are not currently using AI in our environment today

As software delivery responsibilities expand beyond traditional build, test, and deploy, which of the following areas is the most challenging for your organization today with respect to mainframe software delivery? (Select one)(Required)

Integrating security throughout the software delivery lifecycle

Meeting governance, audit, or compliance requirements

Managing software supply chain integrity and visibility

Operationalizing AI across software delivery workflows

Coordinating software delivery across diverse tools, teams, and environments

Other

None of these are significant challenges

Not sure / don't know

Which of the following do you expect is most likely to accelerate your organization's mainframe software delivery progress over the next 12-18 months? (Select one)(Required)

Greater automation across the software delivery lifecycle

Broader use of AI-assisted software delivery workflows

Stronger governance, policy, and compliance automation

Improved software supply chain visibility

Better integration across software delivery tools and platforms

Greater standardization across development teams

Other

No significant changes are planned

Not sure / don't know