staging-devopsy.kinsta.cloud

Tag: devops

  • Data Mobility: Why it’s a Must and How to Achieve It

    Data Mobility: Why it’s a Must and How to Achieve It

    The workplace has undergone a drastic change, and so has the way companies need to move their data. Wondering how those two points are connected? Let’s dig in.

    It’s clear that remote work is here to stay: Just over a quarter (26%) of U.S. employees are currently working remotely, and a staggering 36.2 million Americans will work remotely by 2025. Naturally, this huge influx of remote (and hybrid) workers has necessitated changes in IT infrastructure. It has changed how people work, engage with their corporate databases and applications and what they expect from their corporate technology infrastructure.

    In the past, companies migrated their data every four or five years when they refreshed their storage infrastructure. CIOs and storage experts now think differently about how data centers are built and optimized. Important questions all companies should consider are: Does it make sense to build out storage capacity ahead of the need? Do we really plan to have our databases and applications hosted on-premises over the next five years? One CIO recently told me, “If you aren’t talking about your future in the cloud, you aren’t paying attention.”

    The Need to Migrate Data More Frequently

    The shift toward a more flexible storage architecture, even in a hybrid cloud environment, has created the need to migrate data much more frequently, fueling the demand for data mobility. Robust data mobility is essential in today’s world: Companies need to securely and efficiently move their data to accommodate a strategy that accommodates both on-premises and cloud; even multiples of these. Data mobility constraints can’t dictate the success of an organization’s digital transformation. Moving data to the best location based on an organization’s goals—whether they’re related to cost, manageability, performance or some other factor—is critical.

    Another critical factor to consider regarding data mobility: The data must always be accessible to the end user, regardless of when or where it’s moving. In the age of remote work, people are working based on their own unique schedules, so data must be accessible at all times.

    In fact, data from Microsoft recently revealed that 30% of its employees choose to work at night. Gone are the days when systems could be offline for data migration during “non-working hours.” Remote work demands continuous data access so employees can stay productive and work at their convenience.

    With this in mind, here are three key steps companies can take to achieve data mobility.

    1. Strategy
    When determining where to store data, consider who will need access to the data, what performance level will be required and what the cost will be for each data storage location. Is everything best contained in a single public cloud? Is a multi-cloud strategy or a hybrid cloud storage environment a better approach? We often talk about data mobility and data migration interchangeably, but not all data migration solutions can continuously move data between storage environments.

    Organizations must also determine how they will move their database and applications to the new environment. If this step isn’t addressed during the strategy stage, it can become an afterthought and create problems down the road.

    2. Deployment
    Once an organization has determined which storage environment best fits its needs, it’s time to move the data from its source to its destination. A single, one-time migration involves moving the data from its current location to a new location and stepping away until the next migration. Data mobility solutions are ideal when data needs to move back and forth, requiring data mobility solutions (e.g., moving from on-premises to a multi-cloud environment).

    During this step, organizations must ask the following questions and plan accordingly to minimize any negative impact: How long will this migration take? How secure is the data in transit from source to destination? How disruptive will the transition be to end users? What will the impact on the organization be? How much manual effort is required?

    Sometimes moving data can make applications sluggish, affecting productivity and resulting in workers overloading the help desk with inquiries about what’s happening. Data migration tools can help organizations tackle deployment efficiently, securely and without disruption to end users (again, continuous data access is vital).

    3. Assessment and cutover
    After the data has arrived at its new destination, it’s time for the cutover process (i.e., running the application in its new location and removing the old storage). Cutover is when problems can arise if everything isn’t mapped correctly. Organizations need protections in place to ensure there is no risk to their operations should something go awry during the process. Additionally, minimizing the impact of this transition for end users is vital, so organizations should also be doing whatever they can to reduce downtime.

    Companies can consider leveraging a data mobility tool that can synchronize in the background without impacting users. So, while users are still working on the old site, new infrastructure can be updated simultaneously, requiring just seconds to minutes of cutover. This time can be planned for off hours so that it’s minimally disruptive and doesn’t require notifying employees.

    The technology landscape is constantly changing, and so are storage vendors’ offerings. Companies need solid data mobility to pivot at a moment’s notice to take advantage of newer, better and more cost-effective capabilities. Data mobility gives companies the flexibility to stay competitive, and data migration tools that allow continuous data access keep workers productive.

  • Leveraging Automation for Secure, Cost-Effective Mobile App Delivery

    Leveraging Automation for Secure, Cost-Effective Mobile App Delivery

    As businesses mobilize their customers and employees, they need to develop innovative mobile applications that safeguard user trust. But secure mobile development and manual security testing can be expensive and time-consuming—a bigger challenge in slowing economies. However, the latest innovations in security automation for mobile teams can drive down security costs by half while speeding release time, getting quality mobile apps to market faster to grow revenue faster.

    With mobile dominating all digital time spent online versus the web, businesses cannot compromise the safety of their mobile users and mobile data. Security automation integrated throughout the development process empowers developer, QA and security teams’ efficiency, enabling faster release cycles with built-in security. This approach powers success for digitally enabled businesses that depend on mobile apps to generate revenue, engage customers and tap new markets.

    A single security vulnerability within a mobile app can instantly damage a business’s financial stability, brand reputation and customer trust. Even highly respected brands like UnderArmour and British Airways experienced serious issues after their mobile apps were breached. Business leaders should consider the benefits of adding security automation directly into the development pipeline to drive quality and protect users in a cost-effective manner.

    Faster, Lower Cost Developer Cycles

    Suppose a business creates a 12-week plan to develop, test and release a mobile app software update with a labor budget of $60,000 per week. The development team creates a schedule to write the code in 10 weeks and allocates a two-week period for a manual security pentest. After developers finish writing the code, they must wait for the pentest results to point out any security issues. When testing uncovers security issues, devs, security analysts and QAs must determine which issues need to be remediated. Then the devs can fix the issue, retest to ensure it is properly fixed, then release it. Most of the time, finding any issues delays the release by weeks or months, delaying the expected revenue generation and other business benefits by weeks or months, in turn.

    Instead, businesses can take a cost-effective approach by running security automation continuously through the development life cycle. Dev teams can write code, allow security automation to run during off hours, and have tickets with built-in remediation info sent back to devs the next day. This can remove weeks of testing and remediation delays to ensure teams release on time and on budget. Businesses can cut costs further by establishing common coding, testing and remediation policies in advance and then deploying an automated policy engine to focus on just those security requirements and issues relevant to the policy to operate at the fastest pace possible.

    Integrating automated security testing into development tools enables DevSecOps teams to build, test, fix and release faster with security built in. Not only does this strategy increase the security and quality of the mobile app, it drastically lowers development costs by improving team efficiency.

    Faster, Lower Cost Security Testing Cycles

    Every organization with a mobile app has different security requirements. Those with low-risk mobile apps may outsource their security testing once or twice a year. Those in highly regulated industries like healthcare and financial services may have internal security analysts and test as often as every release and event outsource pen testing quarterly. Whether testing internally or externally, security testing costs can become a major pain point for companies looking to manage their finances efficiently.

    The average cost of a full-scope, two-week outsourced manual pentest costs around $15,000 to $20,000 per test. If an organization tests its mobile app on a quarterly basis, its pentesting costs jump to $60,000 to $80,000 per year. Internal pentesting teams can potentially be cheaper, but organizations still need to pay for analysts’ salaries and the tools and resources they need to conduct tests. Whether a business has an internal security team or relies on outsourced manual pentests, leveraging security automation can significantly reduce costs.

    Businesses with internal security teams can use automation to test builds immediately after devs release code. By continuously assessing code quality from start to finish, security teams can eliminate their testing time and speed up release cycles. Businesses that outsource security testing can use security automation to cut costs by reducing the frequency of full-scope manual pentests. Instead of testing on a biannual or quarterly basis, continuous automation can help reduce the testing frequency to once or twice a year with comparable coverage.

    The most efficient approach possible is the hybrid security automation approach of guided testing that combines daily automated security testing as mobile apps are built with periodic human security analysts for coverage of the parts of the mobile app that automation can’t cover. Guided testing starts below $20,000 per mobile app for unlimited automated security testing integrated into DevSecOps toolchain with four periodic guided tests per year, slashing outsourced quarterly pentesting costs by more than 75%.

    Mobile apps play an important role in helping businesses generate much-needed revenue, but running fast and cutting corners without security built in ultimately places the business and users at risk. So business leaders should take extra steps to keep them secure, and securing mobile apps can be achieved without breaking the bank. By incorporating security automation and innovations like guided testing into the mobile app development process, businesses can maximize the speed of delivery while simultaneously reducing costs and improving quality.

  • Where Does Observability Stand Today, and Where is it Going Next?

    Where Does Observability Stand Today, and Where is it Going Next?

    As we move into the second half of 2023, it’s only natural to engage in the annual ‘state of the industry’ reviews and take a general inventory of the past year while looking out into the immediate future. Observability has gained traction among DevOps and platform engineering teams over the past year and is poised to continue that velocity. But my observation is that the current observability model is broken.

    Too Much Data

    Long story short: Many organizations (and engineers specifically) agree that they are sending too much data to their monitoring tools, resulting in an inability to garner valuable insights because the systems are massively overloaded. Sending this high volume of data also costs too much and requires an inordinate amount of time for analysis, which many organizations don’t have. Put simply–more data doesn’t mean more clarity; in fact, we see the opposite.

    Massive adoption of cloud-native technologies such as Kubernetes that heighten the challenge of efficiently monitoring environments is also accelerating this problem. Beyond increasing costs and complexity, many teams seem to feel that their current approach doesn’t create the desired level of observability for Kubernetes, full stop.

    Another major issue we see on the observability landscape relates to the overall maturity of monitoring processes and teams. In many instances, even when organizations have effective tools, they struggle to interpret the data produced into targeted conclusions that their teams can translate into effective actions. And this is further amplified by the outstanding engineering talent shortage because good people are harder to find and retain than ever before.

    We also see a lot of pervasive challenges related to organizations’ desire to use popular open source monitoring tools-especially if they are attempting to do so in an integrated manner or to support rapidly growing deployments at scale.

    Overwhelmed by Observability Data?

    In summary, if you’re overwhelmed by observability data, if complexity and costs grow out of control and you’re also struggling to make sense of the available information and improve your cloud applications, we think that means that the current observability model really is broken.

    To that end, gaining a deeper perspective into end user opinions on these topics, among others, is the precise goal of the annual DevOps Pulse Report, currently in its sixth year and now in the survey stage. We predict that this year’s research project will underline the aforementioned challenges and help frame how teams might seek to address them.

    What types of findings do we hope to unearth? Last year, respondents revealed that despite a growing perception of DevOps maturity, it was actually taking them more time to respond to emerging problems. Roughly 64% of respondents reported that MTTR during production incidents was over an hour, compared to only 47% the previous year.

    Last year’s report also highlighted the reality that monitoring complex environments remains difficult with over 52% of respondents citing Kubernetes, microservices and serverless as their primary challenges in gaining improved observability.

    Both of these previous findings also seemed to underline the growing issues of data overload and spiraling complexity. We’ve also seen previous evidence of a lack of maturity around distributed tracing—a critical observability practice aimed directly at increasing visibility into complex architectures—despite the fact that 75% of respondents listed it as a priority in last year’s research.

    Is your team struggling to make sense of its available monitoring data, or do so in a way that aligns with your current processes or budgets? Hopefully, the 2023 DevOps Pulse Report, set to publish early this year, will put some numbers behind these assertions and help us all understand how to move forward.

    If you want to participate, you can do so here.

  • Three Trends That Will Transform DevOps in 2023

    Three Trends That Will Transform DevOps in 2023

    The onset of COVID-19 accelerated digital transformation-related initiatives like cloud migration, microservices, observability and CI/CD, giving long-delayed projects the budget and staffing they needed to build momentum. After almost three years of increased investment and accelerating progress, CTOs and software development teams are navigating how to enable the improvements in time-to-market and customer experience that they set out to achieve. In 2023, we’ll see enterprise DevOps mature and expand its business impact by placing a central focus on quality, delivering improved time-to-market, customer satisfaction and productivity.

    DevOps Matures as a DX Enabler

    The past three years have seen massive growth in cloud computing, observability and pipeline automation as enterprises raced to deliver new, high-quality digital experiences to consumers. Cloud spending alone increased from $270 billion in 2020 to 397.5 billion in 2022, according to Gartner. With new technologies in place, the focus shifts to processes that deliver business agility and efficiency.

    Under increased pressure to deliver faster, businesses need to revisit, evolve—and, in some cases, undo—decades of risk and change management practices built around brick-and-mortar experiences and long software release cycles.

    As digital engagement increases, so will expectations for software quality and scalability. We’ll need to invest in new tools and practices—from testing to observability and beyond—to ensure that we’re meeting or exceeding those expectations.

    The economic uncertainty will compel us to look for ways to improve productivity and decrease waste. Quality practices are central to both. We’ll continue to look for ways to shift left and catch defects early in our pipeline when they are the least expensive to fix. We’ll want to optimize our build, integration, testing and deployment processes to identify bottlenecks and causes of rework that drain the productivity of so many teams daily.

    DevOps, with its emphasis on automation and collaboration, will connect the dots between operational goals, technological investments and improved software. Established frameworks like DORA give software development organizations concrete ways to measure the efficiency and effectiveness of delivery pipelines. The key metrics outlined by DORA—deployment frequency, lead time for changes, change failure rate and time to restore service—enable teams to align on where they can improve development practices and quality practices.

    DevOps Will be Measured by Business Success

    The value of digital transformation and DevOps will also extend even further across the enterprise as CTOs evaluate DevOps practices with structured assessments such as DORA. With several years of accelerated digital transformation under our belts, the software development industry can build out a broader definition of DevOps performance over time. The DORA framework sets the stage for understanding how engineering practices impact the customer experience and, in turn, company growth. In one example, Liberty Mutual, a company prides itself on exemplary customer service, is already seeing how DevOps performance connects to business success.

    Architect Justin Robinson notes in their roundtable on data-driven decision-making and DORA metrics: “For organizations, this is improved organizational performance. If we want to keep investing in great tech and serving our customers as best as possible, [we need] behaviors and practices that we can adopt to continuously get better and hit on every front here. Happier engineers, a better culture and a better bottom line for our company.”

    In 2023, more enterprises will harness DORA to measure DevOps performance and its impact on company growth.

    Customers Will Reap DevOps Benefits

    We’re already seeing how metrics like deployment frequency can support an organization’s ability to attract and retain users. According to the 2022 Testing in DevOps Report, teams that increased deployments by 50%-100% were two-and-a-half times more likely to report high user satisfaction. The ability to innovate—without introducing defects to users—is the next frontier in DevOps success. Quality engineering will serve as a safeguard for DevOps teams so they can deliver new features faster and with confidence.

    The Testing in DevOps Report also illustrated the growing connection between digital transformation, DevOps and quality engineering. A full 88% of teams with highly automated pipelines and mature DevOps practices reported high test coverage, which in turn is connected to high customer satisfaction. Test coverage, a core metric for quality assurance teams, is becoming the connection between code and customers throughout development. Quality teams will expand their testing strategies to cover more aspects of the customer experience, including non-functional attributes like performance and accessibility. Automation, particularly automated testing embedded in CI/CD pipelines, will allow teams to adopt these additional quality factors without slowing deployments.

    Connecting Engineering Transformation to Business Success

    2023 is shaping up to be an unpredictable year in technology. But with the momentum propelling digital transformation to an enterprise-level priority, engineering leaders have the data and the tools to shift focus from DevOps processes to transformative outcomes. DevOps maturity, measured and improved with DORA, will help organizations extract more value from their digital transformations by supporting greater innovation and better customer experiences. Quality engineering, augmented with new testing strategies that measure more of the customer experience, will play a critical role in this evolution by enabling development teams to move faster with confidence.

  • Automation Challenges Holding DevOps Back

    Automation Challenges Holding DevOps Back

    A survey of 500 IT and engineering leaders in the U.S. published today finds organizations build an average of 6.7 internal applications per year themselves in addition to investing in 7.7 separate platforms to build applications.

    Commissioned by Internal, a provider of platforms for building application development environments, the survey also found organizations are spending 40% of their time building and maintaining these internal tools and workflows. While more than three-quarters (76%) of respondents said that this work is necessary, it distracts from building applications that drive business revenue. A total of 82% said a lack of true flexibility and customization is the top reason so much time is spent on building and managing internal tools and workflows.

    A full 94% of respondents said they want a more collaborative platform that enables technical and non-technical users to build internal tools/workflows faster together. The survey also found 91% of respondents believe a platform that encompasses application development and workflow automation would save development time.

    Internal CEO Arisa Amano said one thing that holds IT teams back is that managing the platforms used to build applications requires too much programming expertise. Internal is making a case for a graphical approach to building application development environments so IT administrators can easily construct them using a drag-and-drop interface on behalf of an application development team. That approach makes it possible to allocate more programming resources to building applications, she said.

    More organizations than ever are revisiting how applications are built as the number of projects increases despite an uncertain global economy, noted Amano. Organizations of all sizes are reluctant to hire additional full-time engineers even as digital transformation initiatives demand a growing number of custom applications, she added. The only way to effectively address those conflicting goals is to adopt a platform that automates much of the workflow associated with building an application development environment, said Amano.

    It’s not clear how willing DevOps teams are to replace the platforms they use today to build those environments, but many organizations are still relying on bespoke platforms they built and continue to maintain. Many of those platforms are made up of a collection of tools that DevOps teams cobbled together over the years. However, many of the software engineers that initially built those integrations may have since moved on and the documentation left behind is likely to be scant, noted Amano.

    Most DevOps teams in 2023 will continue investing in higher automation levels to accelerate application development. The quandary they all face is whether to continue to invest in the platforms they have already or migrate to a platform that makes it simpler to build and maintain an application environment using automated workflows built and maintained by a vendor. Regardless of what they choose, it’s certain the days when organizations could afford to throw engineers at every integration challenge have come to an end.

  • Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Data streams are all the rage. Once a niche element of data engineering, streaming data is the new normal—more than 80% of Fortune 100 companies have adopted Apache Kafka, the most common streaming platform, and every major cloud provider (AWS, Google Cloud Platform and Microsoft Azure) has launched its own streaming service.

    This shift is fueled by the growing demand to deliver events quickly, reliably and at scale to support a wide range of internal and external applications. From always-on, customer-facing applications to mission-critical operations, a growing number of use cases depend on subsecond updates—further upping the ante for real-time data processing and dissemination.

    There’s no doubt 2023 will be a big year for streaming technology.

    Meanwhile, as streaming moves toward ubiquity, we’re seeing another shift in the way businesses use the data: Analyzing events as they’re created for real-time insights. With the right tools in place, people can instantly compare what’s happening now to what happened in the past, triage problems as they occur and make time-sensitive decisions in the moment.

    With the proliferation of data streams, a new set of use cases and requirements for real-time analytics has emerged. To unlock the power of streaming in 2023, data teams will need to look beyond the traditional batch-oriented stack and adopt a streaming-native approach to the analytics architecture.

    The Streaming Paradigm Shift

    Data has always been at the center of how businesses operate. Until recently, we lived in a very batch-dominant world of business and data. The goal of data infrastructure was to identify data at a fixed point in time and store it for eventual use. The evolution from daily batch operations on mainframes to the internet-driven always-on world replaces fixed, static “data at rest” with fast-moving data in motion, as information flows freely between applications and data systems within and between organizations.

    Data at rest still exists, and batch systems continue to serve a variety of reporting purposes. But reality isn’t fixed or static, and to meet the demands for seamless, authentic data experiences, the systems we build must be designed for data in motion.

    Hence, the rise of streaming technology—and with it, a new mindset for how we think about data. As streaming increases, streaming data platforms increasingly act as an organization’s central nervous system, connecting the different functions and driving mission-critical operations. Evolving technologies are purpose-built to support data in motion systems: Stream processors and event databases.

    Apache Druid fits into the purpose-built category; as a real-time analytics database, it is designed to enable users to query every event as it joins the data stream and do it at a tremendous scale while enabling subsecond queries on a mix of stream and batch data.

    Thousands of businesses are already using streaming processors like Kafka and Amazon Kinesis with Druid to build cutting-edge systems that make terabytes to petabytes of streaming data accessible to applications and humans in milliseconds. Some of those businesses—including Reddit, Citrix and Expedia—were highlighted at Current 2022, the annual streaming event organized by Confluent.

    The next evolution of data intelligence is here, and it’s based on the ability to react to events as they are happening. But on the growth chart of streaming adoption, we are only at the beginning of an upward curve—a critical inflection point where streaming (and the technology that is built for it) starts to become the bedrock of everyone’s data architecture.

    So, when it comes to enabling scalable, subsecond analytics on streaming data, many developers and data innovators are wondering, ‘How?’

    We’re Streaming! Now What?

    At Current, we spoke with hundreds of Kafka users who were turning their attention to building out the next phase of their data streaming pipelines and embedding them more deeply into their organizations.

    While streaming adoption is becoming more widespread, most companies still have just one or two use cases they’re solving with a streaming platform. At Current, many people spoke about how Kafka effectively set their data in motion. But when it came time to analyze or put those streams to work in a user-facing application, their “data in motion” turned into “data in waiting” because their analytics system was designed for batch data, not streaming.

    To solve this problem, a new type of database is needed—and that’s why Druid was created. Turning billions or trillions of events into streams that can then be immediately queried by thousands of users simultaneously is where stream processors like Kafka, in combination with Druid, have unlocked a new set of capabilities that are enabling new developer-built analytics applications.

    For example, Reddit generates tens of gigabytes of event data per hour from advertisements on its platform. To let advertisers understand their impact and decide how to target their spending, Reddit needs to enable interactive queries across the last six months of data—hundreds of billions of raw events to slice and dice. But they also needed to empower advertisers to see user groups and sizes in real-time, adjusting based on interests (sports, art, movies, technology…) and locations to find how many Reddit users fit their target. To achieve this, they built a Druid-powered application that ingests data from Kafka and enables Reddit’s advertising partners to make real-time decisions and get the best possible ROI on their campaigns.

    Stream-to-Batch Vs. Stream-to-Stream

    Reddit chose Druid as the database layer of their application in large part because of its close integration with Kafka, as Druid was designed to ingest and analyze streaming data. This sets Druid apart from virtually every other analytics database, the rest of which were built for batch ingestion.

    Batch ingestion works exactly the way it sounds—it takes chunks of stream data, dumps them into a file, processes the file, then loads it into the database. Using a batch-based system to analyze streams introduces a bottleneck to your streaming pipeline. By contrast, Druid provides a connector-free integration with the top streaming platforms and handles the latency, consistency and scale requirements for high-performance stream analytics cost-effectively.

    In addition to out-of-the-box integrations, Druid has built-in indexing services that provide event-by-event ingestion. This means streaming data is ingested into memory and made immediately available for use. No waiting for events to be processed through batch processes before they can be queried. This capability, coupled with exactly-once semantics, guarantees data is always fresh and consistent. If there’s a failure during stream ingestion, Druid will automatically continue to ingest every event exactly once (including events that enter the stream during the outage) to prevent any duplicates or data loss.

    However, perhaps the most powerful reason to use Druid for streaming data analytics is its near-infinite scalability. For even the largest, most complex ingestion jobs, Druid can easily scale into the tens of millions of events per second. With variable ingestion patterns, Druid avoids resource lag (as well as overprovisioning) by enabling dynamic scaling. It combines the query and ingestion performance of shared-nothing architecture with a cloud data warehouse’s flexibility and non-stop reliability. This means you can add computing power and scale out without downtime or rebalancing, which is handled automatically.

    In essence, while using a cloud data warehouse to serve both batch-oriented and real-time use cases might sound efficient, doing so defeats the purpose of a streaming pipeline.

    “The weakest link in the chain sort of dominates everything,” Databricks CEO Ali Ghodsi told the audience at Current. “If you have one step of your pipeline that is batch-processed and really slow, then it doesn’t matter how fast you are in the rest of the pipeline.”

    That’s why when Confluent’s data engineering team needed to build a real-time observability platform, they chose to use Kafka and Druid together. Today, the platform ingests over five million events per second from Kafka and handles hundreds of concurrent queries from a large team of analysts.

    “We do a lot of things today that we didn’t imagine doing three or four years ago,” Matt Armstrong, who leads the observability and data platform team at Confluent, said in a Druid Summit 2022 interview. “From a [product] perspective, you could think of Druid as being able to power a great product [Confluent Cloud] with a great support experience—but perhaps more importantly—an unconstrained future.”

    It’s Time to Think Big

    We’re already moving to this entirely new way of thinking about moving, analyzing, and sharing data. Streaming technology has opened the door for a plethora of new use cases and products to come to fruition. We believe organizations that view streaming data as a force multiplier and implement a streaming-native approach to analytics will gain a competitive advantage in 2023 and beyond.

    Druid is purpose-built for these new streaming use cases, but it’s also built for a new type of application—one that draws characteristics from both the analytics and transactional database worlds. Industry-leading teams have built mission-critical applications that enable operational visibility, customer-facing analytics, drill-down exploration and real-time decision-making using streaming data and Druid. This trend is now starting to take hold and the next wave of analytics applications is already being built.

  • Don’t Hire for Product Expertise

    Don’t Hire for Product Expertise

    We all know the jokes about talent recruitment that “requires a century of experience in AI/ML.” With the number of layoffs, no doubt this type of approach will get worse. But that is not generally a tech problem; it is an HR problem–the problem of having people hire in tech that aren’t active in or knowledgeable about tech.

    The real concern that is very much a tech problem is hiring for product and/or technology. I will say outright: Stop doing that except in very special circumstances. Hire for the ability to find information. One of the other things we all joke about is increasingly convoluted interviews where the hiring company wants you to solve their problems in a fraction of the time it would actually take them to do it. Again, if you’re a hiring company doing this, knock it off.

    IT is an ever-evolving field, more so once Agile and DevOps are taken into account. Indeed, we would not be far off if we renamed the field Continuous Evolution, because tomorrow is guaranteed to hold different solutions, no matter what. The idea that requiring experience in Grafana or mastery of Python is an indicator of long-term success is downright farcical. There are positions where this is tolerable–if a position is for maintenance of a traditional app that is not going anywhere and is not getting massive updates, then a list of what tools are desired for the role is easy to build and will be lasting. But I would argue that hiring specifically for that application’s needs is not in the best interest of the organization as a whole, because that person is not portable and adaptable. For modern development, particularly of new products that are just now being architected, the needs are going to change in the process, so claiming you want X years of experience in Spring (when it may use a different framework by deployment time) is not productive.

    As we move into belt-tightening yet again, employees who can do the job and know how to look up new information will be worth their weight in gold. Having a person who can work on existing applications and drive innovation in future applications all in one body is important. Giving them a chance to exercise that drive to learn is also important, but is a separate issue. None of the areas I cover–applications, operations and security–are standing still or even coming close to standing still. The need for people who can go find out what is needed is huge.

    And while the internet is a horrendous mix of good and bad, the ability to find information—and even solutions to problems—is undeniably better than it ever has been. Having people skilled at using that information resource is the key to moving forward.

    So, find people that can solve problems given a modern context. Hiring for technology will solve today’s problems, but problems are not restricted to today, so solve ongoing problems. Gate for basic tech knowledge and skills, then evaluate for the ability to find answers. A far better use of everyone’s time during interviews would be—using email as an example—“Hosted Exchange just went down. Walk me through your troubleshooting steps. You have a couple minutes.” Or—using development as an example—“We just discovered that an OSS library we use has unacceptable vulnerabilities. Walk me through how you would find instances of it and what steps you would take to find replacements. Compliance is breathing down our neck, you have a couple of minutes to show us.” You get to see how they approach likely problems, they get to ask about the environment. Far better than “Show us all the steps to get BGP routing configured,” or “Develop an entire B-Tree while we watch,” questions. Certainly better than any question that ends with “And don’t use the internet.” Are you kidding? Can we talk about the field and what year it is?

    As I’ve said off and on over the years, you want the best coworkers you can get or the best employees you can get. Either way, hiring for expertise with a given tech product is not the best route to that goal. You rock—so get people with you who do, too. It’s not going to get less complex, so have peeps that can keep up and carry a load, no matter what tomorrow’s technology looks like.

  • Digital Experience and the Future of Observability

    Digital Experience and the Future of Observability

    Over the past two years, there’s been significant hype around observability in the IT operations and DevOps world. As shorthand for providing better insights, observability has seen a rise in lockstep with distributed architectures, cloud-native applications and the increasingly complex IT environments associated with them.

    As we round out 2022 and move into the new year, there’s evidence observability is gaining mainstream adoption by IT operations and has begun to take off, with the market reaching a better understanding of what observability is and why it matters. This broader adoption offers a moment to reflect on the impact of observability so far and how it will continue to evolve in the year ahead.

    What is Observability?

    Though there’s ongoing debate in the industry about what observability actually means and what an observability platform should include, most people in the industry will argue observability is the evolution of monitoring and promises to offer more complete answers than standalone monitoring. Gartner, for example, first asserted in 2020 observability is “the evolution of monitoring.” But the two aren’t mutually exclusive, and observability and monitoring will continue to coexist and fill distinct needs for different teams based on what information they need about their environments and how they need to act on it.

    So far, observability has had the most considerable impact on application performance and digital experience monitoring.

    The days of monolithic applications are over. Instead, modern applications are highly distributed, are built natively in the cloud using container technology, are organized around microservices and leverage databases in multiple cloud environments. These increasingly distributed software stacks have introduced enormous flexibility for developers but huge complexity for operations teams, making it harder to deliver an optimal experience to the end user. The DevOps teams tasked with building and managing custom web applications are under pressure, and the result is, as one recent article so concisely put it, “Devs don’t want to do ops.”

    Observability has provided a welcome respite for DevOps teams who need intelligent visibility into their environments to determine how applications are performing. This has made application performance a key focus for many observability providers.

    An Intolerance of Bad Digital Experience

    Ensuring external customer-facing applications are performing optimally is the key driver behind this. Digital businesses with external applications core to the company’s success need to ensure those applications perform optimally—customers are fickle. They’re intolerant of bad digital experiences, and alternatives are plentiful. Any e-commerce vendor knows no website equals no sales.

    Traditional application performance monitoring (APM) tools look at telemetry to flag application issues or outages needing to be addressed. But understanding the health of your applications from this narrow standpoint isn’t the same as understanding how the applications perform from a user perspective. In other words, if you aren’t considering how applications are performing (or not!) for customers, employees and other users, you’re not seeing the forest for the trees. Observability helps solve this by elevating APM into a more holistic digital experience approach by adding visibility into how the end user engages with your application.

    By providing a view into how an application is performing with the context of what’s happening throughout the entire stack—including networks, databases, systems, and more—and how issues are impacting the user experience, DevOps teams can shed light on previous blind spots.

    Elevating the importance of digital experience through observability enables teams to understand and respond to issues more efficiently and provides critical data to help DevOps teams further improve their apps beyond just debugging and issue remediation. By understanding how users engage with and experience the application, you can gather critical data to improve or simplify the digital user experience. In our e-commerce example above, this could mean fewer clicks or an easier checkout experience (better digital experience = more sales).

    Cloud migration efforts, coupled with ongoing hybrid and remote work and understaffed tech teams, make prioritizing what matters most even more important. Focusing on the experience of the end user—but with end-to-end visibility across the environment—allows stretched DevOps teams “do more with less.”

    2023 Will be the Year of Digital Experience Observability

    An observability approach prioritizing digital experience is the first step toward building a complete customer-centric environment. This eliminates the potential risks of missing issues directly impacting application performance and creates an enhanced, complete experience for every user. This will make 2023 the year of digital experience observability.

  • Optimizing the Developer Experience in 2023

    Optimizing the Developer Experience in 2023

    2022 was a banner year for innovation, but not only in the traditional “churning out new features” way. Many organizations started thinking about their processes and how to amplify them to maximize efficiency and employee retention. And when they started digging, companies realized that the developer experience was a major pain point that could be rectified–and a ripple effect of positive gains would result.

    Companies started looking more closely at how the tools and processes developers use support and enhance their work. Great tools and thoughtful processes boost developer happiness and satisfaction. And when those levels rise, guess what else increased? Productivity, efficiency and employee retention rates.

    We saw this trend of developer support take hold last year, but with more work to be done across many companies, we’ll see it continue in 2023 and beyond, impacting just about every facet of tech in new ways. Many companies have realized what does work by trying what doesn’t work and are making big changes from there.

    Shifting Results

    A major player in the changing developer experience has been the shift left approach that’s taken hold in recent years. About 80% of organizations have said they are adopting this idea of moving more of the responsibility for security and compliance to the front of the pipeline to avoid a compliance scramble at the point of delivery. The result, however, is a compliance process that ends up being more complex and less efficient, putting additional burdens on developers.

    Involving more people creates more opportunities for security breaches and slows down the entire software delivery life cycle. It also shifts security responsibilities to developers and requires them to do significantly more than is listed in their job description. In fact, 60% of those shifting left said their development team is bearing the burden of the change. Rather than focusing on writing code, developers have been called on to understand regulatory policies, build regulatory controls into their applications or pipelines, inspect massive log files generated by more and more diagnostic tools for security info and spend more time in meetings to discuss audits. That’s not a recipe for fulfilled developers; it’s the path to a loss of innovation.

    Stop Paying the Compliance Tax

    When developers–or anyone on your team–are tied up in compliance tasks, they aren’t spending that time in profit-earning efforts. This “compliance tax” costs companies an estimated 15% each year–that’s 15% of their time spent managing compliance instead of innovating or providing services that add value. That’s not to say that compliance isn’t important–it’s essential, in fact–but limiting the burden of compliance improves the overall experience of developers.

    When developers aren’t innovating, they’re not happy. They want to be writing great code, not translating regulatory governance policies. In today’s culture of quiet quitting, companies simply can’t afford to turn over compliance tasks to developers. It’s a waste of their talent and a surefire way to lose great developers and the chance to attract new ones.

    Shift Left, but Do it Right

    The good news is the compliance tax is largely avoidable. Instead of shifting responsibilities left, shift compliance everywhere. No one person or group, including developers, should be in charge of these tasks–that approach just isn’t comprehensive enough to ensure security and compliance throughout the software delivery life cycle.

    Instead, security and compliance should be moved away from a point-in-time part of the cycle to a continuous process built into every step of the life cycle. CI/CD reduces the burden on any one person or group and is proven to be more efficient at catching anything that is non-compliant. When the tools are doing the heavy lifting, the people aren’t making as many mistakes and non-compliant code isn’t slipping through.

    Baking in compliance allows developers to spend more time doing what they do best: Creating amazing features. This, more than anything, will boost the developer experience. Developers want to create and, when they experience the right environment and the right resources, innovation tends to happen.

    Beyond the Developer Experience

    Focusing on the developer experience pays off in more ways than just creating happy employees (though that alone is reason enough to make a change). Innovative developers are the backbone of leading tech companies. When they’re churning out new features that are getting public recognition and are touting a culture with a positive developer experience, companies attract more talented developers and the innovation loop continues.

    So, if you’re looking for more innovation in 2023, start where you’ve always started: Your developers. Set them up for success by automating what can be automated and baking in the mundane – yet essential – tasks to truly optimize the developer experience. Then sit back and watch what incredible capabilities your developers can generate.

  • Complexity is Still With Us

    Complexity is Still With Us

    We have come a long way and had a lot of improvements over the years, but we still struggle with the ability to simplify things faster than we create complexity. It isn’t enough that each new wave of Really Important Bits™ comes with its own issues and layers and complexity; we also haven’t been cleaning up our previous Really Important Bits™ projects. We haven’t cleaned up the successful ones and we haven’t cleaned up the failures.

    It is time to start. Honestly, we are looking at an extended stretch of ‘Do More With Less,’ and we need to get our houses in order. Want to do more with less? Start by standardizing. Yes, that one project that used Really Important Bits™ (or its companion product Really Cool Tech™) was neat and all, but keeping a technology and its layers and pieces laying around for a single project is not great management. In fact, it isn’t management at all; it is letting things happen. I’d call it chaos, but then we’d have to have a whole conversation about Dungeons & Dragons and, frankly, it is happening in the forum of the overall enterprise architecture so it’s not actually chaos, it is simply excess technical debt.

    DevOps was never meant to be a destination, but on this cross-country road trip, our IT car will get filled with garbage if we don’t take the time to clear out the Really Important Things™ of last year or the last management team or the latest merger. We cannot continue to layer more on without relieving pressure somewhere. AI has the potential to offer some pressure relief in some areas, but we were already behind, so for a variety of reasons, it isn’t the silver bullet we need. Those reasons are simple–it is less secure to leave 50 different ways to do the same thing laying around. AI is resolving some problems and freeing up time, but you’re still wasting time and money on redundant or unused Really Important Things™.

    Some things simply cannot be cleaned up without a massive investment. That’s fine—I hope after years of using DevOps methodologies you have them on a list of things that you’d like to upgrade/replace, but it just isn’t feasible to do so. Focus on the things that you can simplify. The famous low-hanging fruits are fine, just like simplifying things in the early days of DevOps meant that each piece of low-hanging fruit freed up more time to focus on other issues, the same is true now that the low-hanging fruit is generated from DevOps. Or DevOops, in some cases.

    How did I get here, you ask? I was writing a simple overview of Git, CI/CD, DevOps, DevSecOps, GitOps, etc., etc. for a newer technologist, and realized that in between those important parts are a whole lot of tools and technologies we thought were important, but which are no longer terribly useful or necessary. I did her the favor of not including the by-the-wayside technology spaces and product markets. I removed the complexity, you could say.

    And keep rocking it. No matter how far along you are in your IT career, you can look back at all of the things you have learned. Apply that learning to simplifying and improving. Reduce overhead by reducing redundancy and look forward to the next Really Important Thing™ you’ll get to learn about. Keep improving—both your knowledge and the environment. And we’ll keep cheering you on.