Tag: reliability

  • Best of 2022: Measuring Technical Debt

    Best of 2022: Measuring Technical Debt

    As we close out 2022, we at staging-devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    What is enterprise technical debt? Technical debt slows down organizations and hampers their ability to deliver. Studies show that technical debt can triple the cost of support for technical products and services. Beyond this, technical debt can cripple a company’s ability to make the rapid changes needed to compete in today’s marketplace. Those all describe the effects of technical debt, but doesn’t get to the heart of its definition. Therein lies the problem.

    Technical debt is a term that many in technology are familiar with but few seem to understand. Fewer still have a clear way to measure it. Are security vulnerabilities debt? What about teams or applications that have not been migrated to standard communications platforms or shared processes? And, if all of this is debt, how do you measure it?

    Technical debt is often too narrowly scoped, focusing primarily on software defects and other code-related ‘debt’. However, the technical debt that large enterprises acquire goes far beyond code and has much more serious consequences. To truly address issues related to technical debt, we need to understand it not just in terms of software defects and backlog, but also security debt, process debt and organizational debt. Once we understand the scope, we can leverage financial tools as the basis for our measurement. 

    History of Technical Debt

    The term technical debt was originally coined as a metaphor by Ward Cunningham, one of the original authors of the Agile Manifesto. He introduced the concept to justify the work his team was doing to refactor a financial application, saying, “If we failed to make our program align with what we then understood to be the proper way to think about our financial objects, then we were going to continue to stumble on that disagreement which is like paying interest on a loan.” In Introduction to the Technical Debt Concept, the Agile Alliance noted that, “ … [I]n 1992, at the OOPSLA conference, Ward provided additional details (slightly paraphrased here based on feedback from Ward): ‘Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with refactoring. The danger occurs when the debt is not repaid.’”

    So, the original definition provides a metaphor that compares the impact on companies’ ability to deliver code based on the need to refactor to the financial debt incurred on a loan. While this concept has gained wide acceptance, there remains no single agreed-upon definition nor a single way to quantify the concept.

    This concept of debt is, however, very useful as it recognizes there is a cost involved with code defects; defects impede the ability to deliver. However, in modern enterprise environments, this concept needs to be expanded to include other elements which impact the ability to deliver value to the customer, including security-related debt and even organizational debt. There are many items that can (and should) be understood as part of technical debt including:

    • Known bugs that go unfixed 
    • Problems associated with poor code quality or poor design
    • Lack of code test coverage including tests to ensure quality, security and performance
    • Code or artifacts that aren’t cleaned up when no longer in use
    • Obsolete technology that is expensive, difficult to manage and unable to meet the demands of digital transformation
    • Incomplete or outdated documentation or missing comments
    • Known security issues that need to be remediated
    • Diversified tool sets that are not aligned with organizational standards
    • Technology organizational constructs that are not aligned with organizational standards

    It is important to note that, just as with financial debt, not all technical debt should be repaid. There is a concept of “good debt” or, to further the financial metaphor, money borrowed because the opportunity for investment earnings is greater than the rate of cost incurred by the debt. If, for example, one can borrow at 3% and earn at 10%, then it makes good financial sense to accrue debt. Similarly, if the money a firm can make by introducing a new feature outweighs the cost of resolving some debt then the team should prioritize feature work. This cost calculation will often change based on the product life cycle stage, as well. During early product development, the opportunity offered by new feature development often far outweighs the potential cost associated with the risk of compliance issues. However, as a product matures and gains more customers, the risk increases and the benefit of new features may decline. This is important because we do not want to imply that all technical debt should necessarily be fixed. There are many times when the right business decision is to invest in other areas rather than in the remediation of technical debt. However, by applying economic calculations to decisions around technical debt, we can make a more informed and calculated decision based on data. 

    Risk as Technical Debt

    Of the different forms of technical debt, security and organizational debt are the ones most often overlooked and excluded in the definition. These are also the ones that often have the largest impact. It is important to recognize that security vulnerabilities that remain unmitigated are technical debt just as much as unfixed software defects. The question becomes more interesting when we look at emerging vulnerabilities or low-priority vulnerabilities.

    While most will agree that known, unaddressed vulnerabilities are a type of technical debt, it is questionable if a newly discovered vulnerability is also technical debt. The key here is whether the security risk needs to be addressed and, for that answer, we can look at an organization’s service level agreements (SLAs) for vulnerability management. If an organization sets an SLA that requires all high-level vulnerabilities be addressed within one day, then we can say that high vulnerabilities older than that day are debt. This is not to say that vulnerabilities that do not exceed the SLA do not need to be addressed; only that vulnerabilities within the SLA represent new work and only become debt when they have exceeded the SLA. 

    Organizational Debt

    Another type of technical debt often excluded from standard analyses is organizational debt and toolchain debt. These types of debt are often seen in mergers and acquisitions where the organizations are not fully integrated. If one portion of an organization does not adhere to the standard processes or organizational structure for the broader organization, this also can impact the ability of the company to deliver value to customers and represents another form of technical debt. This same paradigm can often be seen in toolchain disparity within companies. If one part of a company is using Slack to communicate while another uses Microsoft Teams, the lack of standardization can cause friction that impedes the organization’s ability to deliver. This is not to say that companies must all use the same toolset or that perfect organizational structure parity is the only way to operate—but understand that there is an ongoing cost for companies that do not standardize. Where standards have been set, they should improve efficiency. When these standards are not met, there is a loss of that efficiency; another form of technical debt.  

    If, for example, a company’s standard chat platform is Slack, that company has chosen to be standardized because they believe standard communication between all teams improves the flow of value to the customer and reduces cost through consolidated license purchasing. If that company purchases another company that uses Teams, that toolchain disparity should be considered technical debt until the platforms are unified. Alternatively, if a company is a portfolio company, there may be limited or no value in standardizing on a chat platform. It may be completely appropriate for that company to decide that the purchased company is free to use whatever chat tool they choose, as that difference will have no impact on the ability of either part of the company to deliver value.

    Mik Kersten discusses this type of debt in his book From Project to Product where he introduces the concept of “accidental complexity” which he says “includes all of the heterogeneity in the tool stack that does not improve the flow of business value. Tools inherited as a result of mergers and acquisitions, or independent selection of similarly functioned tools due to a lack of centralized governance, fall into this category.” He goes on to say that “[r]educing the accidental complexity of tools should be an ongoing effort, as this is a form of value stream debt.” 

    It is important to note that, while many mergers and acquisitions can bring with them significant technical debt, discontinuity in organizational structures or toolchains do not necessarily equate to technical debt. Whether this type of organizational misalignment is construed as technical debt depends, to some extent, on the company merger and acquisition strategy. For companies where there are standards around organizational structure and tooling where some of the benefits of the merger or acquisition are based on the assumption of integration, unintegrated processes and tools should be considered technical debt if they are not remediated within the timeframe projected by the integration business case. If, however, a company is a portfolio company and takes a federated approach, then it is completely acceptable to leave processes and tools unintegrated and these should not be considered technical debt. Organizational technical debt only occurs when tools and processes do not meet standards that align with organizational strategy. 

    By looking across the enterprise, we recognize that there are many forms of technical debt including software debt, organizational debt and risk-related debt. It is important to consider all of these forms of debt. Given this broader understanding of technical debt, we can re-define technical debt as: 

    Unmet demand for technical resources which reduces the ability to deliver value to customers or increases risk. 

    Calculating Enterprise Technical Debt

    With this expanded view of technical debt and the understanding that it can include software defects, organizational complexity and toolchain heterogeneity, we can begin to look at ways to measure technical debt. 

    Quantification of each type of technical debt presents unique challenges. However, the term technical debt itself provides a useful starting point for how to measure all of these. The term technical debt implies a cost and each of the types of debt we have discussed can be quantified in monetary terms—giving us an overall debt cost for a large organization. 

    Interest Rate of Technical Debt

    From a financial perspective, what we are looking at when we are calculating the cost of technical debt is the interest rate on that debt. That is, how much does the technical debt cost an organization over a given period of time? Ward Cunningham discussed this concept in his video on the debt metaphor when he says, “With borrowed money, you can do something sooner than you might otherwise, but until you pay back that money you will pay interest.” Organizational complexity may create delays in developing new products and these delays can be calculated as lost revenue. Unmitigated risks may cost the software due to security breaches and the associated impact on that brand. As time goes on, these costs continue to grow. 

    This is useful because it is also possible to calculate the cost to mitigate the debt. For example, the cost to mitigate a software defect is simply the cost to fix that defect. It is important to note that the cost of mitigation may change over time. The cost to fix a software defect goes up the farther away from creation we get, as familiarity with the worked segment decreases. So we must calculate this cost based on a specified point in time. If we understand the cost of mitigation, we can then look at the cost of debt and weigh that against the cost to mitigate that debt. That helps us to make informed business decisions about the return on investment for remediating technical debt. 

    Software Debt

    While this economic approach to measuring technical debt gives us a common framework for approaching debt, different types of debt still require different measurement approaches. The cost of software-related debt can include defects, unneeded complexity and lack of documentation as well as code that is no longer used but which remains in an application. All of these increase the cost of code maintenance and changes and that may also cause incidents and even outages. We can therefore calculate the cost for this type of debt relatively easily by looking at the cost of support activities related to the issue and adding that to the impact on productivity for the development of new code. The cost of the debt can also be easily calculated by looking at the time and resources required to fix the problem. 

    For example, at one large educational company, there was a bug in the code that caused load spikes. Once or twice a week, the entire team needed to jump on an incident bridge to remediate the load issues by restarting application servers in sequence. Because the application was old, the location of the defect was unknown and difficult to troubleshoot. However, it was easy enough to estimate that the cost to fix the issue would not exceed a couple of weeks worth of effort. As an estimate, let’s say this work would cost $10,000 to mitigate. Comparatively, due to the issue, the entire team, incident management and operations center took one to two hours per week remediating the impact. Let’s estimate this cost at $2,000/week. Based on this high-level analysis, it is clear that the ROI for fixing this defect would be realized five weeks after fixing the bug. 

    Risk Debt

    The cost of risk-related debt is somewhat more complicated to calculate but we can, again, use dollar values as a basis for our calculation. These sorts of debts include security risks as well as compliance-related risks. One of the reasons it is difficult to determine the cost of risk-related debt is because risks do not always become reality. That is, just because there is a risk of a security breach does not mean there will be a security breach. The very fact that it is a risk implies that there is a chance of something happening; it is not a certainty. However, risk analysis provides a very useful tool for calculating this sort of value in the form of annual loss expectancy (ALE). Annual loss expectancy is a means of measuring the expected financial loss during the period of a year. It is calculated by multiplying the expected loss by the probability that it will happen in a given year, or:

    ALE = SLE * ARO

    where SLE is the single loss expectancy, or the amount of loss if the risk occurs ,and ARO is the annualized rate of occurrence, or percent chance that the event will occur in a year. So, if the expected loss due to a given breach would be $100,000 and that type of breach is likely to occur once every five years—giving it a 20% chance of happening in any one year—the ALE for that risk would be $100,000 x 20% = $20,000. Because this number estimates the potential loss due to a risk, it serves as a valuable estimate of the interest for technical debt in the form of that risk.

    Similar to software defect-related debt, the value of the risk-related debt can be calculated as the cost to remediate the risk. This might include activities such as patching outdated systems, software changes to prevent hacking or even process changes to remediate compliance-related risks. For each of these cases, we can look at the cost of resources to remediate the issue and weigh this against the ongoing interest, expressed as ALE, to determine if the investment should be made to remediate the risk. 

    Organizational Technical Debt

    Organizational technical debt is the debt caused by organizational structure, processes and tools that do not align with the standards of the organization and which create delays in the delivery of value to customers. This is frequently seen in mergers and acquisitions where two separate organizations are brought together. Until and unless these organizational structures and tools are aligned, as in our Slack versus Teams example, there may be misalignment which makes it more difficult for the company to deliver as a whole. One way to think about this type of debt is the lack of alignment within tools and ways of working that impedes the flow of value. The “accidental complexity” described by Mik Kersten is a great example of organizational debt. 

    Organizational debt can be measured in terms of the cost of delays in delivering value to the customer. If, for example, handoffs between teams are taking a long time because they are using different development platforms, this delay can be measured and then quantified by the cost of delay. With this type of debt, it is also possible to look at the cost to overall productivity. If the productivity of the organization is negatively impacted because of a proliferation of communication systems that hampers the flow of information, then the cost of the impact to productivity should be calculated; this represents a lost opportunity and another form of technical debt. 

    As with development-related debt and risk-related debt, the cost of the debt itself can be estimated by the cost to remediate the organizational debt. This provides a useful mechanism for determining if the investment should be made to remediate the debt.

    Conclusion

    For large enterprises, the scope of technical debt is expansive. There are many types of technical debt and different methods to calculate the cost of the debt as well as the interest rate on that debt. Software-related debt must be calculated by the impact to the customer or impact to productivity of the people working on the software. Risk-related debt can be calculated by looking at the ALE of that debt. Organizational debt can be calculated by looking at delays in the delivery of value to the customer.

    All this information comes with a lot of uncertainty and a lot of calculations to consider. What is certain is that there are tremendous costs associated with technical debt. The interest on that debt is being repaid every day even if organizations are not aware of it. While it may not make sense or even be possible for enterprises to truly know the full extent of their debt or the amount of interest they are paying, by looking at the cost to remediate and the cost of the interest, we can at least quantify the impact and make calculated decisions based on return on investment for the work necessary to remediate technical debt in the enterprise.

  • The Future of Cloud Strategy: Multi-Cloud

    The Future of Cloud Strategy: Multi-Cloud

    As Gartner’s Milind Govekar recently said, “There is no business strategy without a cloud strategy.” But in 2021, some of the world’s biggest brands suffered disruptions in their services due to outages from their cloud service providers, including Google, Slack, Venmo, Disney+, Facebook/Meta, Tinder, iRobot, The Washington Post and Sony.

    While the outages do not negate the overall benefits of moving to the cloud in the first place, they do make IT leaders think twice about putting all their eggs in one cloud provider’s basket. In this article, we will be exploring the multi-cloud strategy, its advantages and how to prepare for your multi-cloud future.

    Let’s Be Clear About Cloud

    The adoption of a cloud strategy was evidenced in a 2021 O’Reilly survey in which 48% of participants reported a plan to migrate more than half their applications to the cloud while 20% planned to migrate all their applications to the cloud.

    Cloud migration is a multi-factored decision made by business leaders to migrate electronic assets from local servers to the cloud. Cloud storage is rapidly overrunning on-premises data warehouses because of its many positive business impacts including scalability, speed, collaboration, continuity, security and rapid ROI. Although it is simple to select one provider, we’re seeing more and more organizations opt for multiple providers to avoid the negative impacts of outages.

    Multi-Cloud Strategy: The Dominant Choice

    Cloud revenue continues to reach new heights. Gartner predicted cloud revenues globally will reach $474 billion in 2022. That number represents a significant increase from $313 billion in 2020. 

    The challenge among many IT leaders is continually mapping the use of the cloud in relation to their business goals. This alignment is critical to the success of overall objectives as cloud solutions need to match business requirements, whether related to security, compliance or other issues. 

    Cloud-based solutions are highly sought after by businesses and IT leaders. In the quest for the “perfect” solution, multi-cloud options are highly appealing. Benefits of a muti-cloud system include: 

    • Avoiding vendor/provider lock-in
    • Allowing for scaling as business grows
    • Increasing security to thwart attacks
    • Harnessing unique capabilities of different providers

    Multi-cloud and hybrid cloud are often used interchangeably, but they actually refer to slightly different approaches. 

    Hybrid, strictly speaking, is the use of cloud infrastructure in conjunction with bare metal (aka traditional) data centers. As such, it has been an approach favored by enterprises migrating to the cloud in phases.

    Multi-cloud, as the name suggests, involves the use of more than one cloud provider to provision IT. Many would argue it’s also the future of operating as a cloud-first entity.

    Fast-Tracking Digital Transformation

    Digital transformation can be a long haul, however, businesses adopting multi-cloud solutions are navigating the digital services path more quickly.

    Why does the cloud present itself as the best solution for businesses?

    • Rapid advancement of cloud solutions
    • Apps are becoming more readily suited for development and deployment
    • Skepticism toward the cloud is now waning; businesses are porting databases and mission-critical apps to the cloud
    • Cloud is key to delivering AI services, an area of increased growth

    Flexibility meets Functionality

    Flexibility is undoubtedly a major attraction in relation to cloud choices. However, making the correct decisions is imperative to meeting strategic business objectives. The cloud proves beneficial to leaders as, in combination with data engineering, they can procure a comprehensive view of relevant data.

    The first critical component to selecting the correct cloud solution is market research. The necessity of this step is two-fold. Yes, it takes time to conduct research, but in the end, diligence can end up saving both time and money.

    Of course, businesses and providers want their relationships to be symbiotic. App migration may go well and all proper dynamics function at the onset, but what happens when a switch needs to be made? It can prove to be both burdensome and costly. With this reason in mind, it’s important to continually weigh both short-term advantages and longer-term flexibility. 

    Another essential component when assessing options is ensuring seamless interoperability between systems. Simply defined, interoperability is the ability of computerized systems to connect and communicate with one another without regard to their manufacturer or industry. 

    In a modern economy, IT leaders are called on to be increasingly conscientious about cloud services to alleviate any issues with future data portability.

    Edge Computing on the Rise

    Edge computing is a subset of a distributed computing topology in which information processing is located close to or at the location where that information is produced or consumed.

    With the trend of edge computing on the rise, the lines between edge and cloud computing continue to become more obscure. Cloud vendors need to use an edge strategy, and some already employ their own specific suite of services aimed at making data processing seamless.

    Data processing and transition should not be an arduous task. This factor alone influences leaders when making decisions on their cloud journey. Digital maturity is a competitive advantage for which all businesses should strive. 

    Adapting to Multi-Cloud

    A more complex cloud environment can be challenging, but the task is manageable. By dedicating the time and resources necessary, businesses can adapt to muti-cloud or hybrid systems with considerable ease.

    Whether through necessity or design, a multi-cloud or hybrid system has numerous benefits in relation to compliance, security, efficiency, scale, and performance. The multi-cloud strategy is here to stay.

  • Dependability is Crucial for Real-Time Experiences

    Dependability is Crucial for Real-Time Experiences

    Millions of people around the world experienced a major disruption after an October 2021 outage saw Facebook, WhatsApp and Instagram go offline. A few days prior, the popular workplace collaboration platform, Slack, also went down for some users. And in the last few weeks of 2021, Amazon’s AWS caused considerable global chaos after significant networking issues in its busiest data center, US East 1, disrupting streaming services such as Disney+ and Netflix.

    Downtime can be costly, causing organizations to lose $5,600 per minute and up to $300,000 per hour, on average, according to Gartner. But it’s not only financial and reputational loss that businesses need to worry about. Outages also seriously downgrade—and in the case of real-time experiences, completely disrupt—the customer experience.

    At a time when consumers are demanding more live and interactive digital experiences, organizations can ill afford disruption or downtime, as customers quickly move to other services that ‘just work’.

    It’s not just the capacity within a real-time system that matters, but also globally distributed failover capabilities when something does go wrong. While organizations could try to build their own failover mechanisms, it’s extremely difficult to failover so services continue seamlessly, regardless of the status of its underlying infrastructure. Simple failover systems not only take longer to resume but typically suffer data loss during the time that a region is disrupted. Businesses need to know that even if, say, AWS suffers an outage, they won’t be affected.

    This is why it’s vital that organizations get it right when looking to develop their own real-time infrastructure. Managing tens of thousands if not hundreds of thousands of concurrent connections while synchronizing data in real-time means that if anything does go wrong it’s immediately obvious to users. If tech behemoths like Amazon and Facebook can’t get it right, what chance do organizations with far fewer resources have when building and managing their own real-time infrastructure?

    The Steps to Success With Real-Time Infrastructure

    Regardless of whether it’s for synchronized multi-user collaboration, audience engagement or instant updates, developers have a hard enough time developing even the most basic of real-time infrastructure, let alone making sure it’s dependable. There are four key elements to achieve dependability: Predictable performance, data integrity, reliability of service and high scalability.

    • Predictable performance: When we think of performance we usually imagine more computing power or speed. But when it comes to real-time, it’s not just about minimizing latency and bandwidth, it’s about having a predictable system. If developers have global latencies and bandwidth within certain operating boundaries, it provides certainty in uncertain operating conditions. This makes it easier to design, build and scale features around this certainty.
    • Data integrity: As we all know, the internet can be an unreliable network. Users abruptly disconnect and reconnect all the time, resulting in data that gets lost, is delivered in the wrong sequence or that is sent multiple times. As a result, organizations must ensure their infrastructure can guarantee high integrity without sacrificing anything else.
    • Reliability of service: The live nature of delivering updates in real-time requires total reliability. Outages can cause distrust and annoyance from users. Organizations could find their reputation taking a hit or customers quickly jumping ship if reliability proves to be a recurring issue. To prevent this, organizations should develop infrastructure that’s fault-tolerant and redundant at both a global and regional level so they can deliver a great experience even if there are multiple failures.
    • High scalability: More often than not, delivering an experience in real-time can result in unpredictable traffic; for example, a huge number of people wanting sports updates for a World Cup game. If thousands of devices connect over a short period of time but capacity can’t scale up quickly enough, then an outage will occur. To avoid this, developers must use infrastructure that can be elastic and meet changing levels of demand. Otherwise, the user experience is seriously affected and real-time capabilities crash when the network is swamped with connections it can’t cope with.

    While organizations often build their own infrastructure with the best of intentions, it’s easy for them to run into trouble. Most lack the internal expertise, resources and time; plus, going it alone often results in a huge technical debt. It’s also a massive undertaking, due to the amount of ongoing complexity and cost involved.

    Third-Party Partnerships Enable Dependability

    Unless organizations have the resources, finances and expertise of the likes of Amazon or Google, then building your own real-time infrastructure is just setting yourself up for failure. Instead, they should save on development cycles and focus on generating revenue more quickly by offloading their real-time infrastructure needs to a third party. This will mean organizations can design without limitations and focus on solving the challenges that really matter to them while putting in place real-time infrastructure that is as dependable and future-proof as possible.

  • Unreliable Server Scare | Information Batteries | ARM IPO PDQ

    Unreliable Server Scare | Information Batteries | ARM IPO PDQ

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: We worry about chips failing randomly, we ponder a new way of thinking about workload shifting, and we grok Arm’s IPO.

    1. Smaller Features Lead to Random Failures

    First up this week: Could cosmic rays be causing increasing numbers of errors in cloud servers? As chips get faster and more densely packed, the worry is that undetected bit-flips could cause serious problems.

    Analysis: How can you mitigate the problem?

    As workloads grow in number and complexity, this is becoming a real issue—not merely theoretical. (See also: DevSecOps worries such as Rowhammer.) Think about where the critical data are and how to detect random errors. Even consider running important workflows in lockstep pairs.

    John Markoff: Tiny Chips, Big Headaches

    Imagine for a moment that the millions of computer chips inside … the largest data centers in the world had rare, almost undetectable flaws. … As the tiny switches in computer chips have shrunk to the width of a few atoms, the reliability of chips has become another worry for the people who run the biggest networks. … There is growing evidence that the problem is worsening with each new generation of chips.
    …
    Both Facebook and Google have published studies describing computer hardware failures whose causes have not been easy to identify. … Facebook researchers noted that some processors would pass manufacturers’ tests but then began exhibiting failures when they were in the field. … Smaller transistors, three-dimensional chips and new designs that create errors only in certain cases all contributed to the problem … according to Google.
    …
    Intel executives said they were familiar with the Google and Facebook research papers and were working with both companies to develop new methods for detecting and correcting hardware errors. [But] computer engineers are divided over how to respond to the challenge.


    Have we hit a wall? So asks Thutmose V:

    For more than 50 years, semiconductor manufacturers have been trying to keep up with Moore’s Law, which … has effectively served as the target for semiconductor engineers to push them forward. … The size of some parts of the transistors is now 10 nanometers or less. 10 nanometers is about equal to the width of 100 silicon atoms—100 atoms is not very many.
    …
    When things are that small, does quantum randomness start to be noticeable? Normally if the odds of something failing is a trillion to one, you don’t need to worry. But if you have 10 billion transistors operating at a clock speed of 3 billion times a second, a one in a trillion chance of failure is nowhere near good enough. … It is a miracle these things work as well as they do.


    We’ve known about the problem for years, but only now is it becoming reality, says dsgrntlxmply:

    We observed certain network failures to be caused by corruption of an on-chip routing table memory in a SoC that lacked parity protection for these tables. The failure rate was low, but was higher in units that were deployed at higher altitudes. … The problem was eventually mitigated by software table sweeps, and in newer SoC generations, by parity in critical tables.

    These Single Event Upsets are said to be caused by muons generated by high energy cosmic ray interactions in the upper atmosphere. Third semester undergrad physics.


    2. Mind-Bending Speculative-Compute Research

    Many of us are familiar with delaying a workload until a time when servers are idle—or when energy is cheap. But some fascinating research asks: What if we could predict a future workload and perform it in advance?

    Analysis: “Information batteries” can be cheaper than Li-ion

    The researchers argue that pre-caching even parts of a workload’s output when energy costs are low—or even negative—can save far more money than traditional grid-scale storage such as batteries or pumped hydro. For example, speculatively transcoding video versus generating it on the fly, or predicting future machine learning training workloads.

    Caitlin Dawson: A New Way to Store Sustainable Energy

    Renewable energy has an intermittency problem. … What if surplus renewable energy could be stored as computation? … That’s the thinking behind “information batteries,” a new system proposed by … Barath Raghavan, an assistant professor in computer science at the USC Viterbi School of Engineering … and Jennifer Switzer, a Ph.D. student from UC San Diego.
    …
    When renewable energy is available in excess, it is used to speculatively perform computations. … The stored computed results can then be used later when green energy is less plentiful. … For instance, every day, YouTube data centers transcode more than 700,000 hours of videos to different resolutions. Many of these computations are predictable and can be performed at a time when there is excess green energy.
    …
    [But] computations that are completed in advance do not need to match exactly with the computations completed at a later time. … The challenge, said the researchers, is determining what computation to perform, where and when, and how … to efficiently retrieve the results later. … It is only possible in some workloads and in some contexts. But Raghavan believes with improved prediction and integration into large systems, the technology points towards a promising future alternative.


    This isn’t just batch computing with a fancy name. cognizant_ape explains:

    On AWS we do a lot of batch processing when the data centers are low on traffic. This is just the standard SPOT instances which you can trigger once the cost goes below a certain amount and then end when it goes about a certain amount.

    The big issue is being effective with I/O and setup time. The article seems to be getting ambitious to the point of a distributed compiler with a flexible optimization metric.


    Cut to the chase. It hinges on prediction, says bradley13:

    Predicting loads is easy, but predicting the computations that will make up those loads? Not so much. … Imagine: Most employees start work in an hour — let’s predict which database queries they will generate. Your success rate better be high.


    3. Confirmed: Arm will IPO, Not be Sold to Nvidia

    SoftBank and Nvidia have bowed to the inevitable. Arm, designers of ubiquitous chip cores, will IPO sometime in the next 12 months.

    Analysis: The only sane choice, but ecosystem risk remains

    As I’ve said before, it seemed everyone was against the idea of Nvidia owning Arm. It put the kibosh on Arm’s open licensing stance, dooming the ecosystem to scraps from the table as Nvidia grabbed first dibs on everything. Now regulators must turn their attention to the very real risk that Nvidia will still acquire Arm anyway—via the stock market.

    Richard Waters, Arash Massoudi, James Fontanella-Khan and Antoni Slodkowski: $66bn sale of chip group Arm to Nvidia collapses

    The deal, which would have been the largest ever in the chip sector, was set to give … Nvidia control of a company that makes technology at the heart of most of the world’s mobile devices. A handful of Big Tech companies that rely on Arm’s chip designs, including Qualcomm and Microsoft, had objected to the purchase [arguing] that Nvidia would get an unfair advantage by having first rights to Arm’s technology.
    …
    Jensen Huang, Nvidia’s chief executive, hoped to use Arm’s processor designs to cement his company’s growing role in data centres. … Nvidia abandoned its pursuit of Arm at a board meeting on Monday, said a person familiar with the discussion.
    …
    The deal’s failure brought an immediate management shake-up at Arm, with chief executive Simon Segars replaced by Rene Haas, head of the company’s intellectual property unit. … In the UK, where politicians view Arm as a strategic national asset, attention is set to shift to whether the company will be listed in London [but] Haas said … no decisions had been made on where Arm would be listed.


    Reminding us why this matters to DevOps, here’s Timothy Prickett Morgan:

    Now there is no question that Arm Holdings will be going public. … The plan is to do so before the end of [its] fiscal year in March 2023. … Arm Holdings will be returning to its IP licensing roots, having spun off some uncore business back into SoftBank, and seeking out new markets as it has been able to do with hyperscalers [and] cloud builders.
    …
    When Intel was worried about AMD Epyc chips, what it really needed to worry about was boosting price/performance to keep Arm CPUs out of the hyperscaler and cloud builder datacenters. [But AWS] has Graviton Arm server chips on a pretty regular schedule, and you can bet Masayoshi’s last yen that Graviton4 will debut at re:Invent 2022 this year.


    But what happens next? Pun it up, HereAndGone:

    If Softbank is going to take ARM public, Nvidia will find it much cheaper simply to acquire a controlling interest. … Looks like a duck, walks like a duck, but cheaper than a duck—while ducking regulatory issues. This plan is not quacked up to solve what it pretends to solve and will likely prove most fowl.


    The Moral of the Story: Fall seven times and stand up the eighth


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Nicole Geri (via Unsplash; leveled and cropped)

  • Defining Availability, Maintainability and Reliability in SRE

    Defining Availability, Maintainability and Reliability in SRE

    In the world of reliability engineering, you’ll frequently encounter the three “-ability” words: Availability, maintainability and reliability. They sound similar and have similar meanings. In fact, these words may seem so similar that it can be tempting to use them interchangeably.

    That would be a mistake. Availability, maintainability and reliability all have distinct—if related—meanings, and they each play different roles in reliability operations.

    Definitions of Availability, Maintainability and Reliability

    Let’s start by succinctly defining each of the terms.

    Availability, Defined

    Availability is the extent to which an IT resource is ready to perform a task when requested.

    Thus, an application or server that is responding to requests is available (even if it takes longer to respond than desired or otherwise operates suboptimally, in which case it may have a reliability issue but not an availability issue).

    Availability is usually measured as a percentage. A resource that has 99% availability, for example, is one that is up and responding 99% of the time.

    Maintainability, Defined

    Maintainability is a measure of how quickly and easily a resource can be fixed when something goes wrong.

    If a buggy application release can be quickly fixed by rolling back to a stable version, the application would have a high degree of maintainability. On the other hand, if you have a server that needs to be rebuilt manually after it fails, it’s not very maintainable.

    Reliability, Defined

    Reliability is the extent to which a resource functions as required upon request (as opposed to simply being available).

    For example, if your users require an application to process each transaction within one second, and it does this 99%of the time while also maintaining a 99% availability level, that would be a relatively reliable application. In contrast, an application that responds to almost all requests but that suffers from high latency or error rates would not be very reliable, although it might be highly available.

    Differences Between Availability, Maintainability and Reliability

    The simplest way to spell out the differences between availability, maintainability and reliability is to highlight what’s unique about each concept.

    Availability

    Unlike maintainability and reliability, availability is an essentially binary metric, in the sense that a system is either available or it’s not. Although availability status can change over time, there is no such thing as varying degrees of availability.

    Availability also stands out because it is often the most important metric in defining SLAs and SLOs. Contracts typically specify that resources will achieve a certain level of availability, defined in percentage terms. They may sometimes also specify metrics like response times, which are a reflection of reliability, but availability is more likely to be the most important metric within service agreements.

    Maintainability

    Maintainability is unique in that it’s a pretty subjective concept. A system that one SRE considers easy to maintain could seem difficult to maintain to another SRE. The methodologies that engineers leverage to optimize maintainability could vary, too; for instance, someone from a DevOps background may be more likely to focus on optimizing maintainability within the software delivery chain than someone trained in classic site reliability engineering, which focuses on automating IT operations through code more than on the software delivery process.

    That said, automation tools help virtually every team maximize maintainability, no matter their preferences or background.

    Reliability

    Reliability stands out because it reflects how well a system performs. Here again, this is a somewhat subjective metric, because performance requirements may vary from one user to another. Nonetheless, reliability is the most effective means of measuring whether a system meets the performance levels it needs to, even if those requirements change over time.

    Why do availability, maintainability and reliability matter?

    Because availability, maintainability and reliability each measure different aspects of a system’s status, putting them together is a useful means of gaining insight into the overall reliability of a system.

    In order to be reliable, a system requires both availability and maintainability. A system can’t be reliable if it’s not available. It’s also unlikely to be highly reliable if it takes a long time to fix issues due to low maintainability.

    At the same time, by focusing on availability, maintainability and reliability individually, you can drill down into specific issues within the IT resources you manage. For instance, you might have systems that are high-performing when they’re available, but that have low reliability rates because of availability issues. In that case, you’ll know that an investment in increased availability is likely to yield the greatest reward for increasing overall reliability.

    The Bottom Line

    The bottom line: While availability, maintainability and reliability are all reflections of the quality of an IT resource, they measure quality in different ways. Tracking each category separately is important for ensuring that you know where your weakest links exist within overall system performance and health.

    At the same time, however, it’s important to compare and correlate availability, maintainability and reliability data so that you can achieve continuous insight into a resource’s status. When you can track each item separately but also combine them together to gain a complete picture of your system, you are in the best position to optimize reliability operations.

  • The Key Benefits of Observability for SREs

    The Key Benefits of Observability for SREs

    In today’s technology landscape, organizations strive to champion innovative ideas, techniques and technologies to achieve success and outshine their competitors. For this reason, site reliability engineering (SRE) has become one of the fastest-growing enterprise roles and a set of organizational practices for fast and reliable software delivery. 

    SREs use various tools and practices at their disposal to manage services at scale, such as observability. Observability is the ability to infer a system’s internal state(s). It provides actionable insights into when errors occur within a system and, more importantly, why they occur. For SREs, this actionable data can be important to providing secure and reliable applications. 

    To gain more insights into the benefits of observability for SREs, I asked SKILup Day participants and DevOps Institute Ambassadors to weigh in. Here’s what they shared: 

    Sponsor, Priya Satheesh, CEO, Instana 

    “As their name implies, site reliability engineers are tasked with keeping the applications, architectures and websites up and available. Observability solutions put alerts in front of them at the first indications of trouble, with proper context and information so they can take action before issues become incidents and start affecting their customers. In addition, AIOps and machine learning help them to predict events and standardize responses for common occurrences to reduce response and repair times.”

    Helen Beal, chief ambassador, DevOps Institute 

    “Embracing observability supports SREs’ goals by:

    • Reducing the toil associated with incident management—particularly around cause analysis—improving uptime and MTTR.
    • Providing a platform for inspecting and adapting according to SLOs and ultimately improving teams’ ability to meet them.
    • Offering a potential solution to improve when SLOs are not met and error budgets are overspent.
    • Relieving team cognitive load when dealing with vast amounts of data–reducing burnout.
    • Releasing humans and teams from toil while improving productivity, innovation and the flow and delivery of value.
    • Supporting multifunctional, autonomous teams and the ‘We build it, we own it,’ DevOps mantra.
    • Completing the value stream cycle by providing insights around value outcomes that can be fed back into the innovation phase.”

    Mark Peters, technical lead, Novetta

    “True system observability means any process or event within any operational or development phase can be monitored by SREs. Systems with built-in observability provide SREs, as subject matter experts, the opportunity to examine events from different perspectives. The more observable the system, the more different places can be examined to help improve the overall flow. SREs can then create the feedback and design experiments to observe from different perspectives. If the event is showing functional difficulties and the only monitoring point is the input/output of the function and there is no process observability, SREs have to go back to the beginning to find events. If they can observe the entire process chain from commit to deployment, one can find the different areas. Think about the A/B cycle: Limited deployments are planned, and the load shifted gradually to ensure the new system can handle the increased functions. If the SRE cannot observe the function, then they cannot help improve reliability.”

    Ryan Sheldrake, field CTO, Lacework 

    “If you can ‘see’ or observe the complex and ever-changing environments that SREs are tasked with managing, creating and perhaps destroying, you can manage the associated risk posture associated with those environments and corresponding workload.“

    Vishnu Vasudevan, head of product engineering and development, Opsera

    “The top benefit for site reliability engineering (SRE) teams is the opportunity to work collaboratively with the business stakeholders and technology stakeholders to set goals and continuously improve. Based on these goals, SRE’s can create an approval process, which will help them understand where they are with respect to their own planning, security policies, quality metrics and operations practices. The ability to measure every release and verify the project helps improve speed to market and other factors once a robust observability practice is achieved. With DevOps orchestration across all the DevOps tools, applications being released and the different teams involved, SRE teams can then turn observability practices into a competitive advantage.”

    Sushant Mehta, senior manager, application development, Diyar United Company

    “Using observability, SREs can achieve a number of objectives in their day-to-day work, like identifying the root cause of production issues, faster resolution of issues and striving for self-healing infrastructure setup with no-code.”

    Tiffany Jachja, engineering manager, Vox Media

    “Every team that delivers software should be taking responsibility for their contributions to a software service. For SREs, having observability into an application means having the systems in place to help developers gain insights into how software applications function. Ultimately, it’s about building better software.”

    Parveen Arora, co-founder and director, VVnT SeQuor

    “SREs teams can leverage observability to get the following benefits:

    • To keep SLOs intact by detecting customer-affecting issues faster and rolling back before an issue affects the SLOs.
    • To have real-time health updates and transparency of information with regard to a service’s status. 
    • To create better workflows for debugging, optimizing workflows and resolving issues rapidly.
    • To simplify root cause analysis and investigation of hypotheses.”

    Jose Adan Ortiz, solutions engineer, Akamai Technologies

    “SREs are mainly concerned with the reliability of systems and usually monitor practices of toil, SLO, error, budget and stability to maintain systems working as expected.

    You could ask, ‘How can an SRE achieve such an integrated view of all systems components?’ The answer is observability. Without observability tools, it was impossible for an SRE to provide an effective response, fast RCA and analysis of data to improve SLOs.”

    Supratip Banerjee, solutions architect, Principal Global Services

    “It can provide several advantages, including:

    • Detecting customer-impacting errors faster and reverting before SLOs are broken.
    • Fostering transparency and delivering real-time updates on the status of a service. This allows SREs to be more productive and saves a lot of time.
    • Developing better methods for debugging, optimizing and quickly addressing issues.
    • Allows for feedback loops, which are crucial in the SRE role.
    • To avoid downtime, the relevance of observability rises in real-world production systems. There should be proper notification processes in place.”

    Maciek Jarosz, DevOps and process expert

    “The benefits are quite great, I’d say. Unless one is living under a rock with one local system, then it’s rather inevitable that there will be dependencies here and there. And if you’re an SRE, then I’d say that you’d rather know a bit more about dependencies from other branches of your system—or other systems altogether—than less.

    Learn more about observability and similar topics by registering for an upcoming SKILup Day. 

  • How Real-Time Debugging Improves Reliability

    How Real-Time Debugging Improves Reliability

    When designing and building software, service reliability is always at the top of the list of critical focus areas for development teams. Every team that builds software typically has, either directly or indirectly, service level agreements with their customers. These are, essentially, agreed-upon metrics or performance criteria that teams use to measure and ensure the reliability of a software system.

    Organizations may measure reliability in different ways, looking at metrics like service availability, mean time to failure or mean time to repair. Regardless of how an organization chooses to measure reliability, you can often assess the health of an organization based on how reliable their applications are. Since we know that reliability issues happen regardless of how well we prepare for them, having tools to solve those issues quickly and efficiently becomes critical to maintaining smooth operations.

    The Health of Your Organization

    Most software development organizations today have key performance indicators (KPIs) which they measure to understand how well they’re doing and to be able to quickly and easily measure the health of their application development processes. If teams are finding that they are not meeting KPI’s, it’s often a sign that something fundamental needs to change or that new and improved processes need to be put in place. Teams that are constantly measuring what they’re doing well and where they are falling short with a data-driven approach are typically the teams that come out ahead.

    In short, a team’s ability to measure and hit KPI’s and service level agreements with their customers directly influences the overall success and health of that team. If an organization isn’t producing applications that are reliable and available when their customers need them, it’s unlikely that company will have customers for very long. When reliability issues do occur, it’s critical that developers can immediately go to the source of the issue in their code to ensure that they can solve those customer issues as quickly and efficiently as possible. One such approach is to give developers the ability to do real-time debugging directly in the environment where those issues occur, without needing to create costly reproduction environments or having to figure out how to reproduce issues locally that are, many times, data dependent.

    Reliability is Shifting Left

    Historically, software reliability has fallen on the shoulders of operations or production support teams who manage the deployments and runtime of software after it’s been deployed to production (and after developers have thrown the code over the metaphorical wall). Things have changed in the recent years, in that many of today’s development teams are responsible for their applications from development all the way through to running and supporting those applications in production. Because of this, developers have more incentive than ever to ensure that their software has reliability built in, or at least the ability to quickly debug reliability issues when things go wrong.

    Real-time debuggers are one such category of tools that development teams are choosing to package with their applications in order to solve those hard-to-address customer issues faster. Real-time debuggers are solutions which allow development teams to gather code-level debugging information such as snapshots of local variables, stack traces, tracing information and profiling data. All of this data (which traditionally relies on developers having the right logs in their code and, if not, deploying new debug builds) becomes instantly available, on-demand whenever an issue arises. This faster approach to identification of issues can drastically improve the reliability of applications by allowing developers to solve issues in a fraction of the time it would normally take.

    Teams Focusing on Reliability Move Faster

    Many development teams have shifted to an agile approach to delivering software; the goal being to respond quickly to business needs and to get new, valuable features to their customers as quickly as possible. Even though this is a helpful step, unplanned events can and will happen, which need to be planned for as much as possible ahead of time. In a recent Forbes article by Rookout CEO Shahar Fogel, he discusses how businesses should consider these software bugs as mini outages. Any time teams are dealing with software bugs or other unplanned issues, it costs the business revenue and can potentially leave customers frustrated. Focusing on metrics like mean time to repair (and giving developers the tools they need to reduce the mean time to repair) can give teams the ability to better predict how long unplanned issues will take to remediate once they occur.

    In the book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, the authors discuss the notion that, by giving developers tools to fix problems when they occur, teams create an environment where developers accept responsibility for global outcomes such as quality, stability and reliability. Investments in technology are also an investment in people. So, it’s no wonder that by investing in technologies that can help improve reliability, teams will be able to move faster. This means happier customers, happier employees and a thriving business.

    By focusing on service reliability, development teams are not only able to move more quickly, but are also directly contributing to the overall health and success of their organization. We’re seeing the onus of reliability shifted from operations teams to development teams, which adds newfound importance on ensuring development teams are building reliability into their product. Real-time debugging tools are one potential solution to ensuring that developers have the ability to quickly address unplanned customer issues which can directly impact the perceived reliability of an application.

  • The 3 Secrets to Overcoming the Agility-Stability Paradox

    The 3 Secrets to Overcoming the Agility-Stability Paradox

    Regardless of whether an organization subscribes to a bimodal IT approach, one thing is clear: It’s more important, and more difficult, than ever to fight the agility-stability paradox. Today’s businesses must build on a delicate balance of being compliant, plan-driven and keeping the lights on while also seeking ways to become more fluid and agile with high customer touch. The barriers standing between these two key initiatives are numerous and complex; whether it’s mismatched technology or a lack of mutual understanding, the agility-stability paradox remains.

    In a lot of ways, this paradox comes down to a difference between traditional and new forms of IT. Traditional IT can be described as predictable–it’s compliant and plan-driven, with conventional processes and long-term cycles. But as the next generation of IT starts to take hold, today’s businesses must find ways to also fold in a new IT that’s dynamic, fluid, agile and continuous, with short cycles and high customer touchpoints. In this way, traditional IT and new IT are polar opposites, and many companies struggle to find a balance between the two. Competing attitudes, too much change at once and tooling challenges all stand in the way of harmony between the reliability of the traditional approach and the need for agility in the new way of doing things.

    Balance is Possible

    While there is no magic tool or wand that can solve for the incongruities so many organizations face, balance is possible. Enterprises don’t have to choose reliability, customer experience, uptime and performance over agility, release velocity, new markets and competitive response. The real secret to overcoming the agility/stability paradox lies in establishing who, what and how. What this means is that with the right people, processes and tools in place, any company can achieve the balance between agility and stability needed for success.

    Who? People

    People serve as the “who” of finding balance between reliability and agility. This is the chance to integrate your culture into solving the agility/stability paradox. Start by identifying how each member of the team needs to adapt and shift to accommodate changes. The real trick to getting the people aspect of things right is communication. Define a communication path and be really clear about why your organization needs to transform, how it will be transformed and when everyone can start expecting to see some real adjustments take shape. Orchestrate a response mechanism across your teams and enable social operations collaboration to facilitate execution. Be sure to provide ongoing updates and encourage feedback.

    What? Process

    Now that you’ve established who needs to play a role in overcoming the agility/stability paradox, it’s time to talk about what exactly needs to change. Process is absolutely crucial here. There are a few keys to establishing the right process for balancing traditional and new IT models. The first is to identify, and not dictate. Dictating change from the top down is never going to be effective. Instead, work together to determine areas of weakness and highlight them. Use your observations as a guide, not a mandate.

    From a more tactical perspective, be sure to streamline workflows wherever you can. Define instead of inheriting urgencies and alerts. Don’t be afraid to rely on automation wherever possible, either. By automating processes that help out on the people side of things, you can free up your team to spend less time on tedious, manual tasks and focus more on innovation.

    How? Tools

    Tools are your best friend when it comes down to actually making all this change happen. Considering the vast number of tools in the market, choosing a toolset may appear daunting at first. But if you start with strong integration across the operational modes already in place, then you’re already in good shape. In particular, to bring new, agile IT into the fold, you’ll want to incorporate IT service management (ITSM) solutions and establish a DevOps tool chain.

    Don’t Boil the Ocean

    The most important thing to remember is to start small—you don’t have to boil the ocean. Build a culture where traditional and new IT environments are encouraged to work closely together, where you can build and run software that addresses both agility and reliability to respond to the pressures on the business. Start by defining what is is right for your team and organization and acknowledge the existing culture and people. Don’t be afraid to experiment and create isolated learning environments, either. Just make sure to maintain a clear connection to customer value and use this as your North Star. Finally, don’t forget to celebrate. Celebrate your failures and learn quickly from them.

    About the Author / Manish Kalra

    manishManish Kalra is director of Product Marketing at PagerDuty. He is responsible for driving product marketing strategy, product positioning, launches and collateral development for the PagerDuty incident management platform. Prior to PagerDuty, he served as Director of Product Marketing at companies like Sumo Logic and Splunk.