Tag: engineering

  • The Curious Connection Between Cloud Repatriation and SRE Ops

    The Curious Connection Between Cloud Repatriation and SRE Ops

    I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

    To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

    Cloud Repatriation Is Real

    Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

    F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

    I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

    For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

    And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

    Digging Into the Data

    On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

    Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

    Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

    Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

    Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

    Meaningful Relationship or Curious Coincidence?

    There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

    I’m going to argue that it’s a meaningful relationship.

    The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

    The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

    The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

  • How to Manage and Engage Remote Engineering Teams

    How to Manage and Engage Remote Engineering Teams

    As engineering teams look to remain remote or hybrid, executives need to fundamentally change how these teams are managed. After tackling the tactical aspects of a distributed workforce—like home office equipment, a comfortable space where employees can get work done and reliable meeting, calendar and communication tools—more nuanced issues come into play.

    Before the COVID-19 pandemic, when a minority of engineering team members worked remotely, those team members knew that they needed to make a concerted effort to connect and to be part of the group. When the majority are remote, however, the expectations of offsite employees have shifted dramatically. For organizations that had built their business with an in-person team, this brings up entirely new questions of communication, culture, loyalty, problem-solving and retention to the forefront.

    Fundamentally Shifting Engineering Team Structure

    Inspiring change requires fundamental shifts in the ways engineering teams work. Our own engineering teams have experimented with some changes that have proven effective, including:

    • A globally revised team structure that embraces a culture of autonomy and breaks larger engineering teams into smaller groups. Smaller groups help engineers feel more engaged, valued and empowered; it keeps meetings short and focused and allows for more parallel streams of work.
    • Overhauling our design process to get a champion or ‘anchor’ from the dev team involved in key features early on. This anchor is responsible for translating product and design requests into technical specs and is the de facto technical lead of the feature. Having this role rotate between team members for each feature means each member gets a chance to grow their skills as a technical lead.
    • Assigning dedicated designers to smaller groups to empower them to make small adjustments mid-flight. This would allow standups to operate as mini working sessions to iron out technical details that would normally be resolved through ‘shoulder taps’ or in-person quick questions.
    • Placing a strong emphasis on sprint retros as a way to make real changes to the way work gets done. When used in conjunction with an initiative to empower devs, sprint retros became a powerful tool to get dev feedback on improving the process.
    • Keeping teams accountable by encouraging transparency and clear communication as a tradeoff for their autonomy. We encouraged note-taking and regular updates, as well as clear goal-setting from the teams at the beginning of each sprint. By holding teams accountable to the goals that they set, it became easy to see when things were and were not working.
    • Experimenting with a four-day workweek to see if we can drive both greater happiness and productivity. This was perhaps the boldest of our experiments; we demonstrated trust in our team and set clear expectations around outcomes rather than hours. This has motivated our team and raised their morale in crucial ways at a time when switching jobs was both tempting and easy.

    Hacks That Worked in Person

    What are some of the things that worked in the office? This list will be different for each organization; for us, it was group brainstorms, in-person one-on-ones, summer Fridays and ongoing vibe checks. Here are some of the ways we hacked these to make them work for remote teams:

    • Group brainstorms: Digital whiteboards like Miro or Mural have been great. Digital sticky notes, too. This has enabled distributed engineering teams to contribute either all at once or asynchronously.
    • Summer Fridays: Give people the afternoon off even if it isn’t technically “leaving the office early.” Or try having everyone shut off notifications on Friday (or Monday morning) so they can get some undisturbed work done before shutting off for the weekend. Monthly mental health days off for the whole company can be a great idea too.
    • In-person one-on-ones: With Zoom, something is definitely lacking. Try spending half the one-on-one meeting chatting casually about things other than work.
    • Vibe check: Open office hours dedicated to work culture; surveys from time to time to gather quick informal feedback or think about other ways to elicit ongoing feedback from every employee, including engineering teams.

    Generate Culture and Well-Being

    What keeps most employees at companies is social bonding, inspiring projects and the possibility for growth. A question a lot of executives at previously onsite organizations are asking themselves is, “How do you fulfill the need for social bonding when most employees, including your engineering team, are remote?” In other words, how do you bond socially and do so remotely? Two goals we have identified are getting team members talking outside of work while avoiding Zoom fatigue issues and ongoing opportunities to share personal successes with colleagues.

    Some of the ways we have built culture are:

    • Having (non-mandatory) Friday lunch and games like trivia, Jeopardy!, games or online games
    • Different ongoing events like photography challenges, health challenges and even an at-home Iron Chef-style cookoffs.
    • Having leaders share how they spent their weekends and time off. We believe in demonstrating a healthy work-life balance from the top down and encouraging our team to take the time they need to unplug and recharge.
    • Figuring out how (and how not) to surface urgent issues via Slack; working on a system to avoid pinging people after hours for non-urgent issues.

    Above All, Manage With Empathy

    These last two years have been exhausting for most people. Hands down, the best way to manage and engage the people you work with—no matter if they are in engineering or marketing—is to be empathetic to the difficulties that they may be going through outside of work. Executive teams can make all the tactical changes to their teams they want, but without managers that really care about their teams, you will inevitably fail to keep the best people around. Empathy-driven leaders lift their team members up and drive the kind of culture people want to be part of.

  • Report Finds Software Engineering Productivity Gains

    Report Finds Software Engineering Productivity Gains

    An analysis of the time 23,000 software engineers spent working in 2021 that was published this week by Jellyfish, a provider of an engineering management platform (EMP), finds cycle times on average were down to 12.6 days, representing a 12.3% decline over 2020. The report, however, also finds commits were down 4% year-over-year, while the number of issues resolved was up 5% year over year.

    Jellyfish CEO Andrew Lau said the data suggests that software engineers are opting to commit less code in favor of resolving issues faster. Overall, coding days per engineer were down 8% to less than 1.6 days per week in 2021, the report found.

    Engineering teams on average in 2021 allocated 35% of their time to growth and innovation, while 19% was allocated to infrastructure work, up nearly 6% from 2020. Time spent on support work (24%), meanwhile, was down nearly 5% year over year. On average, unplanned work accounted for 22% of the time software engineers spent working in 2021, a 3% increase over the prior year. In general, Lau said the report suggests that engineering teams need to factor more time into schedules for unplanned work as application environments become more complex.

    The report also finds engineering organizations using EMPs tend to allocate similar time to “keeping the lights on” and customer support activities as other organizations but spend 29% more time on innovation work and 48% less time on unplanned work. In addition, the report notes teams that devote 50% or more resources to growth and innovation work deliver software at least three times faster than those who commit 25% or less.

    Virtual collaboration in the last year has also increased more than 20% as teams continue to embrace remote and hybrid work models, according to the report. Overall, organizations that allow software engineers to work from home remain productive but it’s not quite clear if managers have completely adjusted to the realities of hybrid work, noted Lau. The report also noted that pull request reviews per engineer are up 30% on average to 1.7 reviews per week. Pull review comments per engineer were up 23% year over year.

    In addition to the report, Jellyfish conducted a survey of 128 engineering leaders that finds more than 79% expect to grow their teams in 2022, with 57% expecting more than 10% headcount growth. Improving operations, tooling and processes (36%) is their highest priority in 2022, followed by hiring and retaining talent (18%), improving infrastructure (17%) and managing technical debt (10%). The biggest overall challenge is making sure everyone is focused on the highest priority work (45%), the survey finds.

    However, only 27% of the leaders currently track productivity metrics, despite more than two-thirds (67%) describing them as being very important or critical to track. Among teams that do track metrics, more than half (53%) do so manually, the survey finds.

    While the productivity of software engineers will naturally vary by organization, it’s apparent the weight of application environments is starting to impact how much time can be devoted to building and deploying new applications. The challenge, of course, is demand for new applications in the age of digital transformation has never been higher, which in part explains why so many organizations are struggling to find and retain software engineers.

  • Prioritizing Scalability, Reliability and Security in Engineering

    Prioritizing Scalability, Reliability and Security in Engineering

    As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

    This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

    The Old Way of Doing Things

    You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

    After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

    Best Practices for Today

    How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

    1. Evangelize the Narrative That the Systems you Work on Matter

    At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

    2. Set Explicit Expectations about Uptime

    When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

    3. Don’t Sweep Problems Under the Rug

    After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

    Here are some questions you can use:

    1. Who wrote the code?
    2. How was it tested?
    3. Should our automated testing have caught this?
    4. Who reviewed this code?
    5. Were they the right people to review the code?
    6. Why wasn’t it caught in the staging environment?
    7. Who released it and who went through the release validation checklist?

    Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

    4. Define Individual Owners for Every Piece of the System

    If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

    5. Invest in Documentation, Runbooks and Logs

    The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

    6. Prioritize the Delivery Pipeline Like it’s a Key Feature

    You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

    7. Get Business Buy-In by Using Terms They Understand

    Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

    Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.


    This article was based on a talk given by the author at Conf42: SRE 2021. 

  • Cuba Embraces DevOps, Despite Infrastructure Challenges

    Cuba Embraces DevOps, Despite Infrastructure Challenges

    A few weeks ago I had the pleasure of speaking at the second DevOpsDays event in Cuba alongside some great technologists from across the United States, Europe and Latin America. The conference was held in a beautiful hotel in Varadero overlooking a marina right by the beach. Varadero is approximately two hours by car from Havana.

    The first thing I noticed is that Cuba is full of anachronisms. This became evident as soon as I walked out of Havana Airport and realized that the whole “they have some old cars” thing was real. It was fascinating to see that every single car was pre-1950s, some perfectly maintained with great paint jobs and others in worse condition yet still capable of doing their core job as a mode of transport.

    Talking technology in Cuba was eye-opening, to say the least. Do not expect to have 3G or 4G data on your cellphone. Do not expect to have WiFi in most hotels (you can purchase prepaid internet credit, but the speed will be slower than what you are used to) and definitely do not expect to have WiFi in a private home. These are luxuries.

    But here’s the surprising part: There are companies in Cuba with fully automated CI/CD pipelines achieving 10 to 12 deployments per day and solving real problems, such as the team working under Anesto del Toro at Datys. Cuba even has its very own Docker Captain, who gave a superb demonstration on the value of Docker in front of the DevOpsDays audience.

    Now let me remind you, these guys are not working with stable internet connections and they manage to get the job done on a daily basis.

    Why am I Rambling on About Cuba?

    Engineering is about solving problems. No matter how big or small, it’s about looking at a problem and finding the solution. Now, this is part of Cuban life. If something is broken, they don’t throw it in the trash and buy a new one; they fix it. Their 80-year-old car breaks down two hours outside of town? They get out of the car to fix it and—take my word for this—their friends, family and neighbors will come outside to fix the problem, too. This creates a mindset of solving the problem at hand without fail. This is easily demonstrated when you talk to Cubans about software development: They won’t say they are a Python engineer, a Java engineer or any other choice of language—they are software engineers who understand how development works as a whole. They will use the language required to get the job done to the highest standard.

    Like everywhere else in the world, there were huge differences in levels of DevOps adoption between the various organizations in the country. In contrast to the companies mentioned earlier, there also are organizations that have only just heard of these methodologies or are now considering setting up their first CI pipelines. Like many of us, they are still figuring out how to solve their technical debt and modernize their IT Infrastructure, but the key point is that the stakeholders, C-level executives, junior engineers and everyone else in between want to learn. They are open-minded to the principles of DevOps. Every person in the room at DevOpsDays Cuba was focused and listening to what we had to say. This excited me as a speaker and everyone else in attendance. We could see people who really did want to change and transform the cultures within their technology teams and overall businesses to start adopting these modern and proven practices.

    The journey ahead is a long one. The infrastructure challenges remain in place and there is much to be learned, but it was great to see such positive energy and eagerness to adapt. I want to thank everyone who attended DevOpsDays Cuba for making it such an amazing experience. An even bigger thanks to the organizers Enrique Carbonell, Rudy Gevaert, Yadier Perdemo, Manuel Oliver, Mike Rosado, Manuel Morejon and anyone I’ve missed. The event couldn’t have been better and it was a genuine pleasure to visit Cuba, I hope to return soon and reconnect with as many of you as possible!

    About the Author / Conor Delanbanque

    conordelanbanque

    Conor Delanbanque has been recruiting and leading recruitment teams in the DevOps space for the past few years. As well as being exclusively responsible for growing some of the most innovative DevOps & SRE teams in the US and Europe as an external consultant and recruiting partner, Conor also founded the Future of DevOps Thought Leaders Debate, sponsors DockerNYC Meetup on a regular basis and co-hosts ServerlessNYC. You can tweet him at @ConorDevOps, find him on LinkedIn or at events, expos and writing articles on staging-devopsy.kinsta.cloud.