Category: DevOps Toolbox

In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Kevin Henrikson about about succeeding with Continuous Delivery and mobile development. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

Kevin is partner director of engineering at Microsoft where he manages the Outlook teams.

 

 

Edith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

 

Paul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

 

---

[soundcloud url=”https://api.soundcloud.com/tracks/235452076?secret_token=s-a8Rxo” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

---

Paul: All right, Kevin. So, what’s your favorite thing about Continuous Delivery?

Kevin: Good question. So, Acompli started kind of the approach, when we first started was just a couple people. And our deal was like, “Hey, you’re trying to get this thing to work, send emails.” So it was like, keep checking it in, get closer and closer to kind of the definition of it was working.

And then as we started to experiment farther out, we said, you know, “What’s the right sprint cadence?” And we’d come from kind of traditional enterprise companies, where, you know you ship every six weeks or every four weeks and we’re like, oh, we can go faster.

People are doing this agile thing. It’s like two week sprints, and we tried two weeks and realized that most of the work came in on Thursday afternoon or Friday kind of thing, ’cause the end of the second week and so what if we just do one week then most of that work will come in, you know, the first Friday. And sure enough, it worked, right? So 80% of the work still comes in on the last day or kind of completes on the last day or gets to the point of ready. And so, what we liked is that it gave us a much smaller kind of chunk of deliverable, right.

So, like the train keeps moving but the size of what’s on the actual boxcars is smaller and so if you have a problem, it’s easier to roll back, easier to fix.

Or if you have a problem, you can just patch it quickly ’cause you’re used to moving quickly.

Edith: Wow. Okay.

Kevin: So unlike in a traditional world where you ship something every six weeks, you’re like oh, we made a mistake, we need to go patch this. Like, the patch felt like an exception and you’re like, your muscle memory for patching was not there, right? So you’re like do we do it now? Is it worth it?

And then it’s like well, if you’re five weeks in and you discover that thing, it’s like well, the six-week thing is out there where with, if you’re shipping every week, you’re like oh let’s patch it. We shipped yesterday? We just click the button again, and it builds and it goes. So you’ve built the muscle memory to move quickly, right. And so I think, what I like the most, kind of going back to the specific question, is just the fact that it lets you get good things done quickly, but also correct mistakes quickly.

Edith: Wow. So this is a good time for you to introduce yourself.

Kevin: Cool. So, Kevin Henrikson. I currently manage the Outlook, kind of non-Windows teams at Microsoft. So Outlook for iOS, Android, and Mac. Previously, I was VP of Engineering at Acompli, which was a mobile startup, focused on iOS and Android.

Edith: Yeah so you had a lot of good thoughts right then. One of the things I wanted you to talk more about is why do you think 70-90% of the work is always on the last day?

Kevin: Yeah, I think people are kind of deadline driven. I think generally they want to work until the last minute and I think software development ends up being this thing where you’re never done. And so the only way to define done is to actually draw a line and say this is done and then still some things miss that line. And so the tighter you give the window for things to get done, the more often they’ll hit that window because you’ve kind of broke things into smaller chunks.

And also, because software’s kind of this long-lasting thing that never is kind of done done that you end up with, if the chunks are small, people are able to estimate and think is that going to take me a day? Very rarely does somebody think something’s going to take them a day that takes months.

But very often, like if you think something will take months, it may take many, many months. Because it’s hard to guess, there’s so many variables, so many things are going to change in the environment, turn what you’re doing before you get there.

So, I think the short answer is if you start with a smaller window and more discrete small tasks, those tasks are easier to estimate, they’re easier to consume, they’re easier if you get disturbed, like oh I’m going to go down and get a soda, or somebody asks me a random question ’cause some fire comes up, I can kind of make it up because I was only expecting to spend half my day working on that anyways.

So, I think that kind of works out in our favor.

Paul: So, there’s an implication with what you said there, that the tasks sort of grew because it was two weeks,, rather than–

Kevin: Yes.

Paul: So, I guess the phrasing is, when it was two weeks, was it that a one-week task had grown to two weeks? Or was it that things got cut when it got shrunk down to one week, and you ended up with technical debt, or something along those lines?

Kevin: I think it was a couple combinations. I think if you feel like you have two weeks to work on a task, you’ll spend more time polishing or potentially doing things that aren’t on the critical path to getting it done.

And I think that’s natural, right? Like, maybe it is you’re paying off too much technical debt, right? In the start-up world, you have to have that lever. And even just in general software development, like, you have to have a lever. We’re not launching the space shuttle here. There’s room for some wiggle room and imperfection.

And I think breaking it into small pieces lets people focus on, “Hey, I’m only gonna do this” and “I have this much time to do it.” Where you say, “Hey, it’s got two weeks,” you may be deep into the second week and saying, “You know, I just don’t like my approach. I’m gonna change it.” And you basically end up throwing away a week’s of work, whereas, if you’re thinking that “Hey, the deadline’s coming in a week,” you’re gonna make that kind of epiphany, like, “Oh, I made a mistake,” or “I’ve taken the wrong path” much quicker.

So, I think it allows you to break your decision-making and optimize on getting it done, because the window is kind of quickly approaching, being a one-week spread versus two.

Edith: This might sound like Seven-Minute Abs, but why not two-day sprints?

Kevin: Yeah, so I think in two days we– or one-day sprints, right, how do you continue to make it faster? So, for us, there was sort of one artificial gate, which was the fact that you really can’t ship quicker than about a week with Apple.

And even then, it sometimes slips by, because, you know, four or five days, and then during a peak season, if there’s lots of apps, like a Christmas holiday, or around a platform update, you’ll see a spike in the review times. So, Apple kind of, and we run iOS app first before we port it to Android. So that ended up driving us towards a little bit to a week. Cause it was like, if you shipped on Monday, the app would get in the store some time end of that week. And that was a natural cadence to start the next one.

So, it’s really hard without pushing the emergency release button, which Apple only lets you do once or twice a year, to ship more than once a week. And then, we kind of let the rest of the team fall behind that. If you’re only shipping the app once a week, let’s keep the new chunks of new work, or big bug fixes to that path on our service.

The way our architecture is, and we can talk about that more, we spend as much, you know, 77% of the logic of the app is down deep in the service, because that we can update all the time. So, when we push a release and don’t wanna fix something, we fix from the service the same day.

Edith: Interesting.

Kevin: So, that allows us to have a much more agile approach, because the service is constantly being updated, but the app is kind of gated in one-week chunks.

Paul:

So, what you’re saying is that the biggest impediment to your shipping frequently is Apple.

Kevin: The platform, yeah. The platform, basically, Apple’s– And specifically, it’s the fact that Apple has a review process, which is great. That Apple review process generally keeps the bar of apps better and puts some good discipline inside of the wider range of development, people that are coming to the Apple platform.

But, it also, when you wanna move quickly, you can’t. The platform itself kind of enforces this four to seven-day lag in shipping.

Paul: So, you’re now running teams that are shipping on non-Apple platforms. Did you guys make any changes to your release cadence as a result of that?

Kevin: Yeah, so I think on Android, for example, we still kind of keep to the one-week junk of we plan a sprint. Build for a week. Release to our beta group or dog freed users, and then release to production. But Android gives you this advantage where you can do percent-based rollout.

So, with Android, we end up releasing usually two or three times a week. So, you’ll release kind of The Release on Monday, at like 1-2%, kind of walk that out three, five, 10%. Usually at five or 10%, you can pick some new signal of crash or behavior that you wanna adjust, make a hot fix, push another release, start at 10%, 20%. Oh, you see another one you wanna tweak at 20%. And finally, from 20 or 30, you end up going all the way forward.

Paul: Do those Android users help you on the other platforms, as well? Are they finding crashes that also exist on the other platforms?

Kevin: Today we’re 100% native on both, so there’s no shared code between iOS and Android from the mobile app perspective. Obviously from the service, it’s 100% shared code. So, what we’ve actually done in the past, is when we launch a new feature that we think is either controversial, or has a high chance of risk, for example, when we launched iMap support.

We launched generic iMap support on Android first, knowing that we could patch it quickly, and adjust if things went wrong, simply because there was no way to test the millions of random iMap servers in the world and get any kind of focus. When we talked to Gmail or Yahoo, or– Single point.

Paul: That’s really interesting, actually, the idea that because the rollout, or the sort of safety features are only supported on Android, that people will end up building features there first.

Edith: How deliberate did you have to be about putting stuff in the service, versus in the apps? Was this the lesson that you learned?

Kevin: So, we had always thought of the service as the place where we always wanted to do as much of the logic that could break us– You just don’t wanna do it twice. You don’t wanna implement active sync twice. You don’t wanna implement an iMap stat twice.

And it also allowed us to pick a protocol between our cloud service and the mobile app. It was optimized for mobile. So, think of the API, the shape of the API looks closer to the screen of the app. There’s a request for “Get my inbox,” and then make a change. Where, if you think of iMap, or the mirror out of email protocols, that we talk today, they don’t have that very linear type of approach of “Give me this screen. Give me that screen.”

They’re thought of in a way where you’re going to sync all the mail down to your desktop client, and then manipulate it locally, and then sync changes back up in some either connected or disconnected state. So, the service was basically set to put complex logic, but also to allow us to get leverage, ’cause everything we could build in the service was one less thing we’d have to do twice on the client, because we picked a path that we didn’t support. We didn’t basically have a lot of shared clip or clients, essentially none, just the API.

Paul: And it’s the same service that sits behind all the different clients?

Kevin: Yes, so we have a single cloud service that basically powers both the iPhone client and the Android clients.

Edith: It’s interesting, so at TripIt we had a lot of similar issues. Like, we were at one time one of the biggest, a lot of our users are Gmail.

Kevin: Okay, awesome.

Edith: People would sync their Gmail with us, and their calendars. Did you find there was a difference between the you or the app experience that Andoid and iOS wanted? How much did you bifurcate that?

Kevin:

Sure, so, a lot of people ask is like, “Hey, did you launch a feature first on Android or iOS, “or do you always have to make them “come out identical?” My answer to that is pretty simple. Most normal people carry one phone. They’re either an Android user or an iPhone user.

So, they don’t know that you released it first on Android or iPhone, because they don’t carry both phones. So, we think of that, from a feature point of view, that when they’re done, then we ship it. Now, if we’re trying to line a press event around something, we’ll wanna make sure they’re both there before we do the press event. One may come out a day or two before the other. But, generally speaking, we don’t try to, you know, legislate you have to do it this way, ’cause it works on iPhone.

In fact, we actually took a different approach, where we said, We wrote it one way on iPhone and then we went to Android. It wasn’t just like, “Oh go port all the screens over.” It was like, “What would this screen look like for an Android user?”

So, we want our apps on the mobile device, and most mobile users wanna look at an app and feel like it fits in with the rest of their phone. So, the way that an Android navigation works, the way that people use the back button, or not, is very different on iPhone and Android. The fact that there’s widgets, and the way the widgets are exposed on Android, and the way that those people want to integrate deeper into the platform.

Those kind of things we let go with the platform. Our view is write it in native code, ’cause it allows us to take advantage, and move quickly to the platform-specific features. But it also lets us tailor the experience, and the UI and UX to whatever that user base wants.

Paul: It’s interesting that’s, that you talk about it that way, because in the old days, kind of ten years ago, when people were making multi-platform applications, there was a tendency of people to try to make it all match the same, the Firefox is an example here, or Chrome one of those tools that people use on all platforms, was designed to look the same on every machine, even though people didn’t have that machine.

I think one of the nice things that’s happened with the mobile ecosystem is people have thrown away that concept, and they’ve made it feel actually native for each of the platforms.

Kevin: Yeah, and I think that also goes to feature set, right? Like, you would think that the traditional app is like, “Oh, if it works this way on Mac, “or on Windows, or on the web, you would expect kind of 100% feature parity between all those platforms. And today on mobile, we go and select where you actually use it. There’s features that exist in mobile only.

Because mobile has all this additional context of your location and where you are, and what you’re doing. Are you running? Are you driving? There’s other sensors in the devices today that allow you to build specific features that are richer. So, we always thought of a commonality.

So, rather than sending more emails, send better email, right. Don’t say, “I’m running late.” Say, “I’m running late and I’m here,” and send a quick tap of a button to put the map and show you that, “Oh, I’m 20 minutes away,” or, “I’m on the train,” or, “I’m at the train station,” so that people can understand you don’t have to type out how many minutes it’s gonna take. You don’t know. Or, you walk in, or you drive in–

Edith: Plus, people lie all the time. Like, “I’m on my way” has a pretty loose meaning of anywhere from “I might leave my house in 30 minutes” to–

Kevin: And depending on the context, right? What that person’s thinking, so yeah, good point.

For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.