Tag: software engineering

  • Why Developers Should Care About Resilience

    Why Developers Should Care About Resilience

    Recently, a friend reminded me of a joke we used to have when we were both developers at a huge software corporation (we won’t mention names, but back when printers were a thing, you probably owned one of theirs). We didn’t develop printers. We developed performance testing and monitoring tools.

    We were the dev team, which was completely separate from the QA team and from the Ops team (yes, I’m that old – we didn’t even call it DevOps back then). Support was part of a completely different organization and “business” was something that happened in an altogether different galaxy. That is, until a customer had an issue.

    When an important customer had an issue, we definitely felt it at R&D. Managers called hourly to check on status, and we would have intense conversations with the support team who would then, in turn, have intense discussions with the customer. If we, the software engineers developing the product, ended up on the same call with both support and the customer, it was a sure sign that things were bad. You knew that millions of dollars were on the line, and, consequently, that pressure was high. But most of all: you knew that we would do anything we could to solve the problem, because we needed those millions of dollars to be paid, so that we could keep getting paid. And so, we kept the coffee flowing.

    This is where the joke part comes in. We used to have a system for managing support tickets, collecting information, tracking time and assigning it to the relevant software or support engineer, so everyone would know who was in charge. Today, you probably use Zendesk, ServiceNow or one of their alternatives. We used a home grown system, of course.

    Our joke was this: the purpose of the system was not to track and measure and propagate the handling of customer issues. The purpose of the system was to indicate someone else is now responsible for solving the problem. Or, if you would rather paraphrase one of my favorite Douglas Adams jokes, the issue is now under a Somebody Else’s Problem field.

    Because solving customer issues is difficult and frustrating, and being pressured by business could ruin a software engineer’s day. This is because when you’re a happy code monkey, and all you want to worry about is writing beautiful code and releasing cool features and drinking some coffee, solving a customer issue gets in your way.

    You end up spending too much time sifting through endless logs, crossing your fingers and trying to reproduce an issue that only happens at 2 a.m. on a Friday, usually when a blue zebra is passing by. Spending your time on a heated support call is less fun than, say, drinking coffee quietly with your friends.

    As R&D managers, we understand that our developers want nothing to do with solving customer issues. We want to make sure they are motivated and happy and efficient, and that they are free to challenge themselves by learning the latest buzz-worthy technologies and thinking up ways to introduce those into our product in a way that will make our customers happy.

    Additionally, as R&D managers, we are far more likely to understand the business impact of having too many support tickets or having support tickets that take too long to resolve. We know that the loss on such cases is twofold:

    1. The customer that faced an issue and is waiting for a solution may become frustrated, and the company may lose their business.
    2. Our engineers are busy solving that customer issue instead of delivering new value to the business.

    Therefore, as R&D managers, it is our responsibility to make sure that our engineers have the tools to handle these issues as efficiently as possible when they come up. It is our responsibility to make sure they are motivated, and that they understand the business impact of solving a customer issue as quickly as possible. And whenever possible, it is our job to make sure that they make our application as reliable and stable as possible to begin with.

    That ensures that we have test coverage, and the right knowledge, practices and tools to ensure that we have as few customer issues as possible to begin with.

    Happily, changes in dev culture usually means that “support” and “business” and even “customers” are not as far away as they used to be. When developing a SaaS platform, your developers often have an almost intimate, real-time ability to track what users are doing and how it impacts their experience. The culture of DevOps has done a lot for breaking the silos that used to separate the dev team from the support team, and to make sure that all members of the team engage and understand how the code they write and the issues they resolve help our customers.

    I hope you already found the way to motivate your team to solve such issues, and that they have enough time to drink coffee. If you and your team are still struggling, we suggest bringing in a live debugging tool that can help them solve customer issues even faster, by making sure they have all the data they need at their fingertips, and that they can instantly reproduce any issue in a live, remote environment. Live debugging tools give your engineers a developer-friendly experience, and also gives them the power to reduce issue resolution time, too. Now, all you need to worry about is that your engineers have enough coffee.

  • Bootstrapping DevOps for the Internet Computer

    Bootstrapping DevOps for the Internet Computer

    All startups face unique sets of challenges and issues at each stage of their existence. The DFINITY Foundation, a Swiss-based not-for-profit with the remit to extend the functionality of the internet, is no exception. Like any new organization, we have had to evolve and adapt our DevOps and other practices in response to different obstacles.

    After learning to clearly communicate our vision for an open development platform where software runs directly on the internet, we created a road map to translate this vision into a decentralized architecture with distinct features that could be built and shipped on a periodic cadence. Consisting of five milestones, the road map culminates in a public launch in Q4 of 2020—and we haven’t missed a single deadline despite the disruptions of a global pandemic.

    Incorporating a focus on operational responsibility on top of our focus to ship features without losing speed was a major challenge this year ahead of the release of our developer network, called Tungsten. Sustainably bootstrapping operational responsibility for this release, as well as future releases, was imperative.

    We’ve all heard horror stories about operational teams that run through a checklist in a perfunctory fashion without identifying the root cause of a problem. Even worse, they are quick to give up and are resistant to launching changes, increasing the complexity of their workload.

    Pushing Forward With DevOps

    We were determined that this wasn’t going to happen to DevOps at DFINITY, so we embedded these core cultural touchstones into the organization.

    1: Create SLOs and SLIs for cross-functional clarity

    At the end of 2019, well before there was a service to operate, we started thinking about service level objectives (SLOs) and service level indicators (SLIs). These are critical for providing a common, quantifiable vernacular for R&D and Operations teams to measure how a service should be performing.

    An SLI is a clearly defined indicator for one aspect of a service’s performance, ideally focusing on user-visible service behaviors. The SLO sets the target for the indicator, as well as a window of time over which it is measured.

    To communicate the SLIs and SLOs, training materials were presented across the organization, including speaking with various engineering teams, discussing best practices at management meetings and giving an overview at an organizationwide meeting. In addition, the leadership of R&D teams signed off on the approach.

    To provide clear operational expectations as well as a forcing function, we established an early version of the SLOs as a high-priority OKR.

    As an additional consideration, it is crucial that your SLOs reflect aspects of your service that are visible to the user. As Charity Majors, CTO of Honeycomb.io, noted, “Nines don’t matter if your users aren’t happy.” SLOs are not a one-and-done deal; they need to be updated as the service evolves and as your ability to capture the UX improves.

    Having SLOs in place gave staffers with operational responsibility clear expectations to fulfill. If the service is performing within the SLOs (and if the SLOs are representative of user happiness), then things are running smoothly.

    #2: Engineers developing the software should also help run the software in production

    To bootstrap the team with operational and support capabilities, we needed to ensure that we didn’t create a siloed team lacking engineering skills. Operations in this context is a software engineering problem, which meant we needed to involve developers.

    The same engineers developing the software should have skin in the game in terms of how the software is running in production. If there are problems, these engineers are in the best position to diagnose them and propose and implement the necessary fixes. Getting organizational buy-in for this took some time.

    We emphasized the need for senior engineers to be able to operate the software being built. We didn’t want to fall into the trap of throwing a handful of junior engineers into the deep end without proper supervision or training, so we also established an expectation that all engineers should be able to operate the software irrespective of seniority.

    We built two teams of between five and 10 developers in each of our office locations in Palo Alto, California; San Francisco; and Zurich, which were initially staffed with volunteers across the organization. Each week, one team member at each site managed primary operational responsibility for the service instead of their regular project work, and they knew that the entire staff would help if there were any problems. We plan to rotate new people into these ad hoc teams every six to nine months.

    #3: Create optimally sized, communicative teams

    Operational team size is also important. If the team is too small, people are on-call too frequently. If the team is too large, they will be on-call so infrequently that they will either forget how to operate the production stack or their knowledge will become outdated.

    We share status updates through a daily log and informal shared communication channels such as Slack. There is also a weekly meeting to relay responsibility from one person to another.

    This has helped us identify capability gaps. With team members based across the world, we’ve come to acknowledge that if a problem requires a team’s expertise outside of its working hours, then rolling back first and investigating later is absolutely fine.

    The DevOps team decided early on that we weren’t going to start with 24/7 support, instead providing support during business hours in Europe and the U.S., where the majority of our early-access customers are located.

    We’ve been transparent to the wider organization throughout this process, calling out what is and what is not working well, and how we’re working to improve our processes.

    Conclusion

    We were fortunate that unprecedented global events like COVID-19 didn’t significantly affect our operations. That said, shaping our processes in-person and being physically present for people when they first go on-call would have been valuable, and we definitely felt the impact of losing that. But we’ve successfully bootstrapped our DevOps teams nevertheless, and we look forward to continuing to evolve as a team.