Category: Doin’ DevOps

  • Test Better, Test Faster, Test Smarter

    Test Better, Test Faster, Test Smarter

    How do you balance the need to “go fast” with the need to test everything and deliver high-quality software?

    With applications the driving force in today’s economy, the quality and release cadence of your software are critical to your business and your bottom line. You want to get software updated in the hands of your end users as quickly as possible. However, even the fastest release cycle wouldn’t make a difference if your product ends up being buggy or malfunctioning, resulting in a bad user experience or service interruption.

    I want to take a look at one of the most important (if not the most important) aspects of the software delivery process: testing! As software teams become more mature in their implementation of DevOps and continuous delivery to streamline their software delivery, testing takes a central role as a prerequisite to these practices.

    You need to have testing on your mind from the very beginning, and plan your testing as part of the initial stages of designing your software—mapping your test procedures, infrastructure configuration and acceptance criteria. As you design and develop your app, you also need to think about how you’re going to test it. The more testing is on your mind, the more you’ll have a solid understanding of what you need to test, what’s the definition of “done,” and the faster, more streamlined, your overall process will be.

    Let us review four questions every development or QA team should consider when planning their testing, as they try to balance quality with speed—managing and accelerating their testing, while ensuring a reasonable time-to-market.

    How Much Testing is Enough?

    The age-old question: How much testing do I really need to do? Unfortunately, there is no concrete answer. But with a bit of initial legwork, you can get a good feel for how much testing is necessary.

    The amount of testing that is necessary is influenced largely by the industry you’re in. If you are testing mission-critical aircraft software, let’s hope it’s very thoroughly tested! On the other hand, if you’re building a game app, the amount of necessary testing may be lower. The truth is when you run tests, the success rate is usually not 100 percent. Therefore, the key is to define what, for your specific piece of software, can reasonably be considered “done.” Instead of arbitrarily deciding at some point that your release is ready to ship, make the decision before the fact: What are we going to accept? What level of pass/fail are we willing to roll out with? What are we comfortable with based on what we have done in the past?

    The amount of testing required can sometimes be a fluid thing. A good way to measure your comfort level with your testing procedures and the amount of testing you’ve done, is to ask yourself, “If it was now 11 o’clock at night, and I needed to ship this code to my end users, am I sufficiently comfortable rolling this out, or am I not sure how it’s going to behave in the wild?” If you’re leaning toward the latter, than it’s probably an indication that you should have done some more testing. That’s a gut feeling and depending on the industry, the point at which you’ll fire off a release and not have any worries can vary greatly. And, obviously, for some regulated industries, you’ll need to prove compliance and ensure you’ve recorded that all tests were successful before you can even start thinking about releasing anything.

    What Kind of Testing Should You Invest in?

    Decide what kinds of things you really care about and what you need to verify and test for. What are the requirements that we expect from the application, and from the different stakeholders in this project?

    Every piece of software needs to be tested differently. Because of this, you have to plan what the testing will be. You can’t just say, “I’m going to unit test the heck out of my code, each and every time, no matter what.” Sometimes, unit testing isn’t the most important test you need to be running.

    The investment has to be in automation. No matter what type of tests you decide are most functional and relevant to your project—whether you’re doing a ton of unit testing or a bunch of functional stress tests—it needs to be automated.

    Particularly, as you open up your services to the world, you need to put into place the relevant performance and reliability tests so that if suddenly your load increases by 200 percent, you have a plan for how to handle that. Are you going to autoscale? Can you autoscale? And finally, do you even need to auto-scale?

    All of those questions need to be answered as part of your development cycle. Automating your tests will let you not only create tests for these scenarios, but it will let you run those tests over and over. By automating your tests, you can establish a baseline and see where you deviate.

    It can also be helpful to think of testing in a different way. Don’t look at testing as a tool to validate technical requirements, but as a way to make your customers happy. You want to make sure your tests are directly contributing to your customer’s overall experience. So start with the highest-level tests that look at things such as overall business functionality—and make sure the software works at the business level.

    How Can You Make Testing Faster?

    Automate, automate, automate! That might seem obvious, but try to think outside the box. Find things you thought couldn’t be automated and wrangle them into shape. Remember that every minute that one of your developers is sitting there running a test that could be automated, it is costing you time (and money) that he or she could have spent on more productive things. In the long run, not automating can cost much more than making the commitment to doing it right.

    Other simple things you can look at to increase the speed of your testing is to make sure your development environments are mirror images of your production environments, and that you have fidelity of configuration throughout the lower and higher environment.

    It is also helpful to make sure you’re defining user requirements that are thin vertical slices, instead of trying to test huge cumbersome pieces of functionality. The ideal situation is if you can run only tests against components that have changed. Componentization of both your application and your testing procedures is critical because you do not want to have to rebuild and re-test everything all the time. Remember, also, that if you make incremental changes to your code and frequently release them to your users (Agile, anyone?), you will get just enough feedback to improve quickly and reliably. Be cognizant of not biting off more than you can chew.

    Segregating the tests to several phases is a good way to accelerate your CI and test cycle—but make sure that both your application, build and test batteries are designed for that. With this approach, you are essentially running multiple phases of CI, starting with the fastest cycles that require the smallest build(s), with minimal dependencies, and move on to the outer/cascading cycles of tests. Following a submit, run a minimal build, so that you do not need to wait—sometimes many hours—to run an extensive set of all your tests with every single commit to see if it passes. Start with unit tests to minimize the build and provide fast feedback, then in various junctions incorporate additional tests, such as integration tests, installers and usability and performance tests.

    Another way to streamline your testing process is to take a look at the entire scope of your build and test cycle and go after the most inefficient stage. After automating your manual processes, look at the bottlenecks and what takes the most time and figure out how you can optimize that specific process. Often the answer is parallelization, but other times it is hardware. Iterate on your process and figure out where you can gain the most ground!

    Lastly, parallelization and distribution of the build and tests provide a huge time saver, so consider running tests across a cluster of CPUs to cut down the run time.

    Where Do You Start if You Want to Make Sure All Your Builds Are Green?

    Ask yourself, What are you trying to achieve? Is it a process where you can release code more quickly? Or where you can address bugs more readily? Then make sure you’re intimately aware of what you have in place. You need to know every step, task, process and tiny bit of your testing procedures and supporting infrastructure. Draw your entire end-to-end workflow on a whiteboard and find the bottlenecks that are slowing it down.

    Once you’ve defined your objectives and are aware of your current state, you can put the relevant tools in place—from development environment to your testing frameworks all the way to post-dev tools such as your analytics tools—design the right testing pipelines and hire the relevant people with the relevant skills. Don’t insist on making all the mistakes yourself (tongue firmly in cheek), but find folks who have already made the mistakes and learned from them—and hire them!

    Also, give your team a great system that gives feedback fast. Do frequent code reviews so there is accountability and a constant learning process.

    Consider the following best practices for improving your testing processes:

    1. Be sure to plan what your testing approach will be ahead of time if possible. When you’re designing your software, designing how you’re going to test it is important.
    2. The amount of testing you need to do is dependent on what industry you’re in and your specific use case.
    3. Consider what level of test pass/fail you’re comfortable with, based on what you’ve done in the past.
    4. Consider pre-commit or pre-flight builds to ensure your tests are more likely to pass once checked to Mainline
    5. Define your requirements for each test cycle
    6. Segregate your tests to “fast” and “slow” and decide which battery of tests you need to run every time, and which processes you only need to run at certain junctions.
    7. If you hire well and bring new skills to your team, your process will get better and faster.
    8. Consider what problems your customers are facing and perform an analysis. Use whatever tools are best to test your software like a customer would to achieve the highest end user satisfaction.
    9. Automate manual steps that you take for granted to accelerate cycle times and improve productivity. Speed comes second.
    10. Iterate on the testing process, and not just on the code itself. You should always be looking for new ways to test.
    11. Test your test processes. Hold regular meetings with your team to reassess if your testing strategy is working for you.
    12. Enact code reviews so that there is accountability with the rest of the team.

    As you map, design and optimize your tests, remember that your testing policies don’t need to be set in stone, and that you can—and should—adjust and tweak as you go.

    Happy testing!

    About the Author/Anders Wallgren

    Anders HeadshotAnders Wallgren is chief technology officer at Electric Cloud. Anders has more than 25 years’ experience designing and building commercial software. Prior to joining Electric Cloud, he held executive positions at Aceva, Archistra, and Impresse and management positions at Macromedia (MACR), Common Ground Software, and Verity (VRTY), where he played critical technical leadership roles in delivering awardwinning technologies such as Macromedia’s Director 7. Anders holds a B.SC from MIT. He invites you to download the community edition of ElectricFlow to build your pipelines and deploy any application, to any environment for free.

  • Puppet Labs Launches Annual State of DevOps Survey

    Puppet Labs Launches Annual State of DevOps Survey

    Puppet Labs has announced its 5th annual State of DevOps Survey. In the past five years, we’ve surveyed more than 20,000 technical professionals around the world, spanning all industries and companies of all sizes. We’ve captured a tremendous amount of data about how IT performance, culture, technical practices and organizational performance intersect.

    What sets our survey apart from the many other surveys out there is that it’s science-based. Yes, there is a science to creating surveys and analyzing data, and there’s also an art to translating all of that amazing data into something that’s useful to the industry at large.

    This year, we’ve partnered with DevOps Research and Assessment (DORA), founded by our friends Gene Kim, Jez Humble and Nicole Forsgren. If you’ve been following the DevOps conversation for a while, their names should be familiar. Together, our goal is to help advance the DevOps community and help technology organizations improve their practices through rigorous scientific research.

    Through our analysis, we’ve been able to quantify IT performance. We know that high-performing IT teams deploy code 30 times more frequently, with 60 times fewer failures, and they recover 168 times faster. We also know the factors that predict IT performance—technical practices such as continuous delivery, cultural norms and lean management—which in turn predict organizational performance.

    This year’s survey seeks to dive deeper into each of these areas, while also shedding light on new areas of research, including:

    • ROI of IT performance. What can you expect from your investment in DevOps? Our goal is to understand how throughput, stability and quality affect the bottom line.

    • Security and compliance. What role do security and compliance play in a DevOps environment? Some people see security as a roadblock to DevOps, but we want to investigate if it has to be that way.

    • Quality. So you’re deploying code faster than ever. What’s the quality of that code?

    • Containers. How do containers affect IT performance? Are they making IT’s job easier?

    • Lean management. Last year we found that lean management practices such as limiting work in process and creating and maintaining visual displays showing key quality and productivity metrics predict IT performance. This year’s survey looks at additional lean management practices to help us better understand what it really means for a technology organization to go lean.

    The survey seeks answers to questions we get all the time and we believe they reflect the broader concerns of the DevOps community. Your experience matters and we hope you’ll share it by taking the 2016 DevOps Survey. It should only take about 20 minutes, and when you complete the survey, you’ll be entered to win some great prizes (see the official rules), including :

    Please take the survey and share it with your friends and peers. Together, we can make IT better for everyone.

    About the Author/Alanna Brown

    unnamedAlanna Brown heads up DevOps research and go-to-marketing strategies at Puppet Labs. She also works closely with Puppet Labs product owners and engineers to develop a roadmap and support for Puppet on Windows.

  • The DevOps Equation: Agility + Empathy = Quality

    The DevOps Equation: Agility + Empathy = Quality

    I had the chance to catch up with Jeff Sussna (@JeffSussna), founder and principal of Ingineering.IT, ahead of his keynote address on continuous design, scheduled for DevOps Days Atlanta, April 26-27.  He discussed the importance of designing for service, responding to the unexpected, and the importance of building empathy across teams.  

    Derek Weeks (@weekstweets): Today I’m really happy that we have Jeff Sussna joining us for the latest in this series. Jeff, why don’t you introduce yourself?

    Sussna: Sure. Thanks for having me. I’m an independent consultant. I’ve been around, since rocks were young or the 80s, whichever is older. My particular background is that I have built systems and led organizations across the entire development, QA and operations spectrum. So, I’m a little unique in that respect.

    My current focus is on helping IT organizations and digital businesses really bring together their ops and design thinking so they can not only go faster and not only be more resilient, but provide more customer value at the same time.

    Weeks: Let’s talk about that a little. How is continuous design—a subject you have written a lot about—coming into play? How can it help us be more resilient.

    Sussna: Just to explain a little about what we mean by continuous design. It’s basically bringing together the entire organization and a unifying design in operations. It’s taking what we are doing in DevOps and expanding it to include product, design and support. It lets the whole organization really have an agile conversation with a market and with its customers.

    The issue that we face is that we live in a service world now. The point is no longer delivering products like cars coming off an assembly line. It’s much more about responding to our customers and having relationships with them. This is happening in a world that is more and more complex while less and less is in your control. We can’t figure everything out in advance. We have to have the ability to respond to the unexpected. We have to be able to do that at every level.


    “This is happening in a world that is more and more complex, while less and less is in your control.”


    I will give you a couple of examples of things that are perfect illustrations of this:

    I recently wished a friend of mine happy birthday. The reason I did that was that because there was a notice on my calendar that said it was her birthday. It was wrong. It wasn’t her birthday. I had no idea where that notice came from. Did it come from Skype? Did it come from LinkedIn? Did it come from email? I had no idea.

    We face that in IT, too. Systems are becoming more complex. We develop microservices. We use public and private clouds at multiple levels. We’re no longer in a situation where we can control things. We have to be able to respond and respond at all levels.


    When we make design and operations unified and continuous, we are able to say, “Where did we think we were going? Where would we need to go? and what do we do next?”


    What I talk about in my book, in my workshops, and all my work is that this needs to happen at every level. It’s not just a matter of designing for failure inside the data center or inside the infrastructure. We also have to do it at other levels.

    Here is another example:

    I was trying to get paid recently by a large enterprise client in Europe. They were trying to wire money internationally and my bank wasn’t really set up for it and they were having problems. They just kind of gave up. I kept getting these messages saying, “Sorry, we tried paying you and it didn’t work.” That was the end of it.

    I started to get frustrated and my client within the enterprise started to get frustrated to the point where they sent a scathing email to our AP department which they CC’d me on, saying, “Why can’t we take the lead to get his guy paid?”

    That’s an approach that is based on resilience in the sense of assuming that something can go wrong and figuring out what he can do to fix it when it can go wrong. As opposed to the traditional approach where we’ve been trained to push certain buttons and take certain actions. If those actions don’t work the way they are supposed to, well, I guess that is someone else’s problem.

     

    https://www.youtube.com/watch?v=ISSMHmwIMtA

     

    Weeks: This brings to light some recent conversations I was having with Paula Thrasher at CSRA. She was talking about developing systems and how to make them more resilient. She is using pictures as part of her conversations with people to help visual their work. What work steps are there? What work is flowing? What is communicating with what?

    Are you taking similar approaches with continuous design in how you work with clients to pursue it?

    Sussna: Well, I try to introduce them to the principles and I also try to introduce them to a set of practices which actually comes out of Mark Burgess‘s work on promise theory. It is really a conceptual model for how we think about success and failure and how do we think about service. For example, Amazon helps you get everything you need in your life from shaving cream to batteries to snow tires to books as quickly and easily as possible. In order to keep that promise they make a whole variety of other promises. Some of which filter all the way down to a promise to be able to go over and make a change to production as quickly as possible.

    Once you take that mindset. The interesting thing about a promise and the reason we use that word is because it is very intuitive. We all know what a promise is. We all make promises to each other every day. On the other hand, when we think about it in terms of service, IT, or whatever it may seem a little odd. The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think. If I promised to pay my vendor. What do i do if I’m not succeeding in paying my vendor?


    The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think.


    It’s sort of flipping our mind. Again, we are doing this in DevOps world where you start thinking about things like mean time to repair (MTTR) instead of mean time between failure (MTBF). We shift from how do we create a situation to how do we create a situation from where nothing will ever break—which becomes less and less possible. When then need to shift to a situation where we can respond more and more quickly.

    We go to executives and say, “We don’t want to worry about how often stuff breaks anymore,” and they freak out. We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead, let’s be able to fix them so quickly that maybe no one notices. My focus is on really helping people shift their mindset from thinking about ourselves to thinking about who we are helping. And from thinking about what we are doing to thinking about how well it succeeds and how well it overcomes problems.


    We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead let’s be able to fix them so quickly that maybe no one notices.


    Weeks: You talked about delivering on promises as quickly as possible. How does continuous design account for doing things as quickly as possible but making sure the quality is there at the same time?

    Sussna: In two ways—one way is that when you do things in small increments you are able to find problems more quickly. This is classic; it comes out of lean. It comes out of integration. It’s pretty basic stuff for our community now that it’s much easier to find and fix a bug five minutes after you change a code than five weeks after you change the code.

    When you put things like Linux, agile, DevOps, and design all together, they all focus on incremental feedback loops where you discover the gap between what you think you’re accomplishing and what you’re actually accomplishing very quickly. Secondly, when you take this all to the way to the marketplace and you make it part of the relationship with your customer, you can reduce a tremendous amount of waste.

    The way to think about continuous design is: it’s all about steering. Think about a boat and you’re trying to sail across a bay. You can’t just point the boat at the other side and say go because the winds are changing and you’re going to be able to respond to things that you can’t predict. If you sail for 20 minutes then discover you had pointed in the wrong direction, you’re going to be way off-course. You’re then going to generate a tremendous amount of energy getting back on course.

    What continuous design says is just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.


    What continuous design is really just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.


    Weeks: I know you’re going to be keynoting DevOps Days Atlanta coming up in April. Chris Corriere, one of the guys on the organizing committee, tells me that you’re leading a full day workshop on continuous design before that. Maybe you could tell some of the people that are attending DevOps Days Atlanta why they should be there a day earlier for your workshop and what things you’re going to address during that time.

    Sussna: To some degree what I am going to address is everything we have been talking about for the last 10 minutes or so—relative to DevOps. I think the key thing is, and we need to think about it. We need to think about more than just going fast. One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.


     One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.


    You know what we are talking about is essentially developers and operations people providing better service to each other. As a result, they can deliver better service to the business.

    Where we talk about microservices. You know that word service is in there for a reason. If all you do is you kind of shatter your organization and shatter your architecture into a bunch of little pieces, you still have to figure out how those pieces come back together into something coherent. The answer to that is taking a really deep service-oriented approach. That is very much what my workshop is about. What does that word even mean and how do we do it?

    Weeks: You mentioned thinking about interactions as delivering better service to one another within the organization. How does empathy come into your thinking around DevOps and Continuous Design?

    Sussna: I’d like to say agility plus empathy equals quality. What I mean by that, in particularly empathy, I think it’s important to take a minute and clarify. When we talk about empathy we are not talking about wallowing in someone else’s pain. We’re not even necessarily talking about things on an emotion or feeling level. It’s very simply the ability to see things as if from an other’s perspective. I will give you a prime example of where that comes into play and how it’s important and how it helps.


    Agility + Empathy = Quality


    When I work with clients, I tend to look at their process using their organization from the product beginning to the support end. Often times I see support getting the short end of the stick. I see places where they get phone calls from customers with features that they don’t even know exist yet. They sometimes get talked down to by development or product or even operations. They say things like, “Well you’re just supposed to support the stuff. You’re the young people. You’re the less experienced people.”

    When we have the ability to see things from their perspective, we can think about how support is actually experiencing our delivery process. If we don’t think about them when we go through the process of delivering new features, they will be lost. Them being lost will show when customers call for help. That will significantly affect customer satisfaction and then even beyond.

    The reason we need to empathize with support is so we can empathize with the customer. Often times when we build things and we don’t think about the notion that our customers might have trouble with it. On the one hand we have the illusion that if we design it and we build it really well, they won’t have trouble. We can’t perfectly predict who they are, what they need or what they will encounter. There’s an inevitability to the idea that they are going to need some amount of help.

    Weeks: I think that is probably the best description of empathy within a DevOps environment that I have heard anyone talk about. Jeff it has been a great conversation. Thank you very much. I really appreciate it and we look forward to seeing you at DevOps Days Atlanta coming up.

    Sussna: Thanks. It has been great talking to you and I am really looking forward to the conference. It will be fun.

    If you want to learn more about continuous design from Jeff, be sure to join us at DevOps Days Atlanta. I will also be at the event giving an Ignite Talk, titled, “Real Heroes Draw Pictures,” focusing on lessons learned from the 31 DevOps and continuous delivery reference architectures I assembled.

  • To Be Continuous Podcast: The Clash in Open Source

    To Be Continuous Podcast: The Clash in Open Source

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul talk about the clash in OSS between developers looking to get paid and software companies looking for things for free.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/244279037?secret_token=s-oeBCP” params=”auto_play=false&hide_related=true&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Edith: So Paul, what do you like about open source?

    Paul: So I guess that depends on whether we’re talking about writing open source or using open sources as part of a company. Writing open source, I like just having all my code out there.

    And I’ll do that as often as possible. Anytime I start a little bit of code, a little library, I’m looking to open-source it and have a readme and have it out there for other people to use. I had a very long discussion, which I eventually won, about the …

    Edith: I like the way you think of all discussions as win or loss.

    Paul: Well, yeah, yeah. It was more on the argumentative side of discussion. But with the other co-authors of the PHP compiler I wrote, which was originally GPL. And I argued very strongly that I actually don’t care if people give back, I just want people using my software. So to me, open source is a way of having people using my software.

    Edith: That’s really interesting. So, I liked what you said at the beginning about, like, you liked open source ’cause it meant getting stuff out sooner. ‘Cause I’m a big fan of that.

    Like, I always say you should just publish stuff. Like, my co-founder John said something really good at our team meeting last week. He’s like, if it’s 90 percent done and you don’t publish it, it’s actually 0 percent done.

    Paul: It’s 0 percent done, yeah.

    Edith: Yeah, and that’s a hard lesson, ’cause you always try for perfection lately.

    Paul: Right, right, right. Well, I think there’s also some really hard downsides about that. So when you publish something before it’s ready, before you really —

    Edith: Well, ready.

    Paul: Well, there’s ready ready, but let’s say ready is, let’s say you publish it before you clarify what the software is to do, or the values or the mission of the software.

    And then you get contributors who are saying, ‘Oh, this is great, but I really want it to do this other thing,’ and you’re like, ‘Well, that’s not exactly what I was going for here.’

    Edith: Well, I’d say most open source projects are so lucky as to get contributors at all.

    Paul: True, true, true. So I’ll give you an example. I wrote the V8 bindings for Closure. So a binding that allows you to write JavaScript and run it in Closure. And I did that for a purely selfish thing, for another, for a nascent pipeline that I was building called Stefan.

    And then people came along and they wanted to do other things with it. They’re like, ‘Okay, let’s add a binding for this other thing.’ I was like—

    Edith: Well, that’s kind of like a pivot.

    Paul: It is, it is, but I really was not interested in it.

    I’d rather someone took it over, took over the project and actually ran with it where they wanted to go, and I can stick on the old version, or I can see if the new version solves it for me, but I wasn’t interested in becoming a maintainer in this thing, I was interested in solving my itch.

    And I just happened to open-source it, because I felt it might be useful for other people.

    Edith: So Paul, I hate to say this, but I agree with you.

    Paul: No, not again.

    Edith: Not again.  No, I mean, I think what you said is exactly right, that you said, I did this for this reason, and I want somebody else to take it over now.

    I think where open source runs into issues where people are like, it’s only this purpose and you can’t do anything with it besides my original thing.

    Paul: So you think that’s a good thing, or you think that’s a problem? Or it causes problems in the community?

    Edith: I think open source is so misused for so many different purposes that it’s hard to even say. Like, so, in Europe, people say open source is good because it means free software.

    Paul: Libre or gratuit?

    Edith: I’m sorry?

    Paul: Free as in beer or free as in, what’s the other side of that?

    Edith: There’s no such thing as a free lunch, is that what you’re talking about?

    Paul: No no no, like people say open source software, it’s like free-as-in-beer software, as in software that has no cost vs. software that has no price tag vs. software that is, they use the French word libre for—

    Edith: Oh yeah, that’s what I was talking about. So I remember I went to Europe in, I think it was like 2004 or 2005, and they just had this big push for open source, ’cause they thought it meant free.

    Paul: Oh, this is like the German government and that sort of thing.

    Edith:  So literally like, so I went to this customer and they’re like, ‘So it doesn’t cost you any more to give us an extra copy.’ So why shouldn’t just everything be free?

    Paul: Oh, right, yeah, well.

    Edith: And so I think open source has been misused by many businesses who think of it as just like, hey, we get all these developers working for free.

    Paul: So I think that the main advantage to business of open source is not that it’s free, but it’s that it’s low-risk.

    Edith: Low-risk?

    Paul: Lower risk. So it removes—

    Edith: Ah, I would argue—

    Paul: Well, so, what I mean is that it removes a certain element of risk from it. So there’s, Joss Belsky wrote a blog post about how you shouldn’t use, I think it was a database that you didn’t have the source code to.

    Edith: Well, so I’ll argue the reverse, and I think open source is sometimes more risky. ‘Cause I talk to people now who are like, nobody’s maintained this open source project that I depend on.

    Paul: Right, right, right. But that vs. a closed-source version.

    Edith: Well, so the opposite risk is that you end up that you are responsible for this open source project, and you’re like, well, actually, I just wanted to use this thing. I didn’t want to do the maintaining.

    Paul: So absolutely, you don’t wanna end up the only person that, well, it’s actually fine to end up the only person which is using this thing.

    Edith: Well, no, it’s not—

    Paul: It’s not ideal.

    Edith: It’s that many people are using it, but nobody is paying money, and then you’re stuck.

    Paul: Because no one is maintaining it.

    Edith: No one is maintaining it, and you depend on it.

    Paul: Right, so, but imagine that no one maintained it and all that you could get was like a binary block. That’s the real problem. The company goes out of business. You have random database version seven, the binary, that only runs on 32-bit, that you’re maintaining, like, companies are literally maintaining COBOL—

    Fortran things from 30 years ago. It sits in a machine, you can’t touch it, you can’t fix it, you can’t apply security updates to it.

    Edith: Yeah, that’s fair. So Tim Chou, who is, he’s an awesome guy, he was early Oracle, he said that people don’t buy, when they buy software, they’re not buying it what it is today.

    Paul: Right.

    Edith: ‘Cause basically, anything that you’re buying today, you could just hire 20 engineers that could build it. What they’re buying—

    Paul: Yeah, you could just copy it.

    Edith: Yeah, what they’re buying is the promise of stability, that it will continue to be maintained. That there is a vision. And that’s actually, I went over to a big customer who won’t let us use their name yet, but they said they love LaunchDarkly because we thought every day about feature flags.

    Paul: Right, right, exactly. This is why people buy SAS, because, because people think all day about CI, about QA as a service, about queues, or whatever it is from various Heavybit companies that I just shilled for, they’re, you can probably figure out yourself which ones they are by looking at the homepage.

    Edith: Well, and Paul, by the way, thank you very much for wearing your LaunchDarkly t-shirt today. It looks very handsome.

    Paul: The, I definitely agree that the benefit of the thing is that you use it every day. But you can do that with open source as well. You can have open source software that, so, for example, our front end is open-sourced.

    Edith: Oh, I’m absolutely a fan of open source. Like, we use open source every day. Our client libraries are open source. I’m not saying that it’s, I don’t think it’s a religious thing vs. open source vs. not, I just think that many times people assume open source is something that it’s not.

    Paul: Yeah, fair enough, fair enough.

    Edith: I mean that if you have open source, that just means that developers spring out of the ground willing and happy to update your software every day.

    Paul: A mixed blessing, definitely. If it even happens.

    Edith: Like, I remember, so I remember when Sun, so I used to work on a, I used to be at Epicentric, and we had a portal server, and we were killing Sun. Absolutely killing Sun. And this was in 2003, so what they said is, ‘Oh, we’re gonna make our portal server open source.’

    Paul: Right.

    Edith: And so somehow this whole thing, we’re like, everything will be open source, and like, developers will spring out of the ground, and it’s like, well, no, developers, they don’t wake up every day like, ‘I’m gonna work on your thing.’

    Paul: Right, right. They will if it’s sexy or exciting. There’s lots of developers who are showing up at Mozilla just to contribute. And in fact, they’ve fought through incredibly large hurdles to contribute to Mozilla’s software.

    Edith: Well, that’s fascinating, ’cause I see so often the reverse, where people, so what do you think about Mozilla made people want to contribute?

    Paul: I think the fact that it was an open-source-first company, and that it was a nonprofit, and it was all about the free and the open web and that sort of thing. Also, people had had very good experiences with Firefox for, like, a decade at this point by the time I was there. Well, maybe seven years, but, you know.

    Edith: It’s fascinating, ’cause I went to dinner last night with the CEO of MySQL, and I don’t know if I should say this, but he said—

    Paul: This is Marten Mickos?

    Edith: Yep. So he gave a talk—

    Paul: He has a new startup?

    Edith: Called HackerOne?

    Paul: Oh, he joined HackerOne.

    Edith: Yeah, he’s the new CEO. So the dinner was kind of like a debutante ball for him.

    Paul: Nice, nice.

    Edith: So what he said was that somebody had asked him flat out how much of MySQL was actually from contributors. He said 1 percent.

    Paul: Seems about right.

    Edith: He said the open source was more just a branding thing. Like it wasn’t actually like a productivity thing. It was more just a branding thing.

    Paul: So you have this in Mozilla as well, that any time that there’s a really avid contributor, that Mozilla would hire them.

    Edith: Yeah, that’s what he said.

    Paul: Right, and if Mozilla didn’t hire them, very often they stopped being a contributor. Like, it was a real sort of a double-edged sword.

    So someone who did it for six months, they were doing it for the love of the game. They weren’t doing it to be hired at, at least I think for the most part, they weren’t doing it to be hired at Mozilla. And then six months in, someone’s like, well, you know, you’re doing a really great job, wouldn’t you love to work on this full-time? They’re like, yeah, that’s so exciting. They’re flown out for an interview. And then it’s like, you know, actually, I don’t think so.

    Edith: Wait, they didn’t wanna work for Mozilla, or Mozilla didn’t wanna hire them?

    Paul: Mozilla didn’t wanna hire them. I mean, this isn’t a particular case. But you know, some subset of people, Mozilla didn’t wanna hire. And then what happens? Well, you kind of don’t wanna keep contributing to the company that just told you to f*ck off.

    Edith: Yeah, I mean, it’s so true. I mean, I think there’s a very human thing about volunteering.

    Paul: Right, right.

    Edith: I volunteer, I was the moderator for a long time for the Lean Startup Circle, and somebody new came on, and they’re like, yeah, I’d love to raise some money so that we can get paid, and I’m like, they cannot afford to pay me.

    Like, I don’t do this to get paid, ’cause—

    Paul: Yeah, it’s like, it’s an altruism that people are trying to do.

    Edith: Yeah, I was contributing like 10 to 15 hours a week, and at my going rate, it’s not, it’s like, I’m not doing this to get paid, I’m doing it ’cause I like giving back.

    Paul: So there’s this thing that we were talking about earlier, about the current discussion around people getting paid in open source.

    Edith: Yeah, and I think a lot of what has fueled open source is the whole 20 percent time. Like, that Google will give their engineers 20 percent for open source, and I know that other big companies, like Twitter and so on, do that too. So these people are, in essence, getting paid.

    Paul: Right, right, there’s a, in most companies, in most startups at least, there’s an unspoken rule that you’re expected to contribute to the libraries that are in your application, and that sort of thing. If you’re spending time upstreaming paths that you’ve deployed, that’s work.

    Edith: Yeah, so I think we’re agreeing, which, I thought we agreed that we were not gonna agree. It’s funny, ’cause I’m Twitter friends with a guy I’ve never met, who’s, he started Jango, and right now what he tweets a lot about is—

    Paul: This is Jacob Kaplan-Moss?

    Edith: Yeah.

    Paul: Okay.

    Edith: That people are working their nights and weekends, and contributing to open source, and then just, people are kind of, for lack of a better word, jerks. People are like—

    Paul: People are jerks.

    Edith: Yeah, like—

    Paul: Even when you’re paid, people are jerks.

    Edith: But it’s more aggravating.

    Paul: Right, right, because you’re doing it for free, or whatever.

    Edith: Yeah, and so people will, like, they’ll be like, ‘I have this pull request and it’s been open for two hours. Why haven’t you reviewed it?’

    Paul: Right, right. I mean, we get the exact same thing. We get customers who email us, or more often it’s the people who get, who are using the free version, who send us a support request and then, 30 minutes later they’re on Twitter going, I can’t believe Circle hasn’t solved my problem yet. It’s been 30 minutes.

    Edith: What do you think drives that?

    Paul: There’s a pattern across the population in general, that people are jerks.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of Circle CI, and Edith Harvaugh of LaunchDarkly.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • To Be Continuous Podcast: How Do You Deal with Your Data?

    To Be Continuous Podcast: How Do You Deal with Your Data?

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. Edith Harbaugh and Paul Biggar talk with Peter van Hardenberg about about continuous delivery in the context of organizational structure, databases and pricing.

    Peter is a co-founder of the Heroku Postgres product team. For more than three years he has been responsible for designing, developing and operating the largest cluster of Postgres databases in the world. He knows how to get databases running, keep them safe and ticking fast.

    This podcast is brought to you by Heavybit, a program that helps developer startups take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/241491956?secret_token=s-oZ2T7″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Paul: So Peter, it’s traditional to ask you, what do you like best about continuous delivery?

    Peter: Well, I’m increasingly old and I’ve done a lot of jobs and so I remember delivering Gameboy cartridges to the market and a Gameboy cartridge is a physical device that goes into a Gameboy, hence the name, and that process involves physical manufacturing and you get one shot and so there’s a tremendous amount of basically wasted and un-fun energy that goes into making sure you never make any mistakes.

    My favorite thing about continuous delivery is the ability to make mistakes and move on with them.

    Paul: Nice.

    Edith: This would be a great time for you to introduce yourself.

    Peter: Yeah, hi, my name is Peter van Hardenberg. I’m a Heroku old-timer, I’ve been with Heroku about six years now and I co-founded the Heroku Postgres team and have built a lot of products since then.

    Edith: So it’s funny you’re talking about the Gameboy because Paul and I were just talking about that.

    Peter: About Gameboys.

    Edith: Because I said one of the things that had fueled the rise of Zynga was continuous delivery.

    Paul: That’s the first thing we found about continuous delivery that was actually bad.

    Peter: So I guess they’re probably still continuous delivering somewhere. In the great startup in the sky. They’re still around, aren’t they?

    Paul: They’re still around, they’re still public and,

    Peter: Farmville will never go away, I’m sure.

    Paul: I think they’ve moved into other lesser things.

    Peter: Developer tools.

    Paul: Imagine.

    Peter: Gamified developers. Well actually, it’s how all developer tools are becoming now.

    Edith: Github is gamified. How many chickens do you have, can you make a nice pattern out of them.

    Paul: New watches.

    Peter: It’s a bit like a game except if you’re losing the game, you lose your job.

    Paul: Or you just can’t get new jobs.

    Peter: Well, that’s actually probably more accurate, yeah. Yeah it’d be a weird company if they let you go because you didn’t have a long enough commit streak.

    Paul: On open source projects that weren’t your job.

    Peter: I like that, actually, let’s make that a game.

    Paul: We were talking in the last episode, as well, about open source funding models, or the anger over lack of open source funding models.

    Peter: That’s a tough problem to solve.

    Paul: That would be an interesting one. You get fired if you don’t have enough contributions.

    Peter: Yeah, just put it on your quarterly review, current Github streak.

    Paul: Let’s put this suggestion on Hacker News and see if people agree with it. I can pretty much picture how this is going to go.

    Peter: I try not to read the comments.

    Edith: So what do you think is liberating about the ability to make mistakes?

    Peter: Well, I mean, more important than the ability to make mistakes is the ability to try things, right, I mean we, I’m so accustomed now to being able to see in great detail because of a SaaS model, sort of the reality of what my product is in the market and then vice-versa, I can turn that around and say, well, basically live and in real time, what’s happening if we put something out there, do people care?

    Does it help with adoption? Is it suddenly throwing a bunch of errors? And if there’s a problem, you can roll back very quickly. You know we run a database product and so availability is really, really important but only for a very small core of the service. People are generally pretty understanding as long as your, sort of, problems and outages are brief.

    Small and, sort of, blast radius and quickly remediated and I think that they prefer to get features vs. stability, sort of, at the periphery. At the core, they need stability over features, so, you know if you’re a stripe, right, you need to keep the payment processing online but if some ancillary piece of your API has a problem that’s probably not as big of a deal.

    Paul: The dash part is less important—

    Peter: Right.

    Paul: than the payment going through.

    Peter: Well, and responsiveness and progress are so important, and I think because we’re in such a changing market, the ability to drive forward is at least as important as the ability to keep things stable but you can’t do both in the same place, basically, right?

    I think continuous delivery is cool because it’s separated, sort of, a slow-moving core from a fast-moving periphery.

    Paul: So when you’re creating a software product, you really want every single component of it to be continuously deliverable, which means you want to be able to migrate from the thing that you are doing now to the new thing.

    And so it’s interesting, or there’s an interesting thing about databases because databases, I think, were fundamentally not built for this model.

    Peter: They’re heavy.

    Paul: Right. We had a problem recently where we deleted an index.

    Peter: Ah, yeah.

    Paul: And, whoops, that index was in use. And we thought it wasn’t in use but it was in use.

    Peter: You should’ve been using Heroku Postgres, it would’ve told you.

    Paul: Oh so, that’s interesting. So the thing that I wanted was to be able to, you know, partially delete that or let’s run all the queries without that and then after an hour, I can actually delete it. What does Heroku Postgres offer you?

    Peter: Well so we have a number of diagnostic tools where you can see exactly how often an index is being used. In addition, next time you’re trying to delete an index, wrap that in a transaction, so say, begin and then drop the index, then do whatever and then say, rollback, and that way, you can actually test as though you didn’t have the index because Postgres MVCC model means that all that stuff is totally transactional.

    Paul: So you’re, when you say do all the stuff in between, what do you mean?

    Peter: Whatever queries you want it to run. Also, and we’re getting into minutia and then some baseball here but you can actually disable indexes specifically, sort of at a session level as well.

    Paul: Oh, interesting. This is all Postgres or Heroku?

    Peter: Yeah, that part’s just Postgres I mean, we built the tooling around it so that you can do these things, kind of, the easy way but, I’ll tell ya, if you read that 3,000-page Postgres manual, you’ll find some real gems.

    Paul: Well so part of the problem might be using a thing that’s not a real database?

    Peter: Oh, you’re not using a real database?

    Paul: We’re not—

    Peter: Oh, that’s right, I forgot.

    Paul: I mean we are using a real database, we’re using many databases but that particular one was not in a real database.

    Peter: Well you know, even baby databases grow up to be real databases someday.

    Paul: Yeah, well, fingers crossed that that particular database that starts with an M and that we shall not discuss.

    Peter: Oh, we don’t like to speak ill of anything.

    Paul: No, no not at all, especially Mongo.

    Edith: No, you said the word!

    Peter: You said the M word! So coming back to, sort of continuous delivery of every part of the stack, yeah, it’s interesting, I mean we’ve had some interesting challenges with databases in particular and I actually gave a talk at Heavybit a while ago about, kind of, how we treat databases less as part of a service in sort of the classical sense but more like a factory.

    So basically what happens is when people request databases from us, we stamp one out and then, you know, we try and keep it pretty low entropy from that point on so we have a lot of tooling and support services around the database but the actual database itself, once it’s given to a customer, we mostly try and keep our hands off it.

    Now if there’s security issues or performance issues or whatever, we’ll go in and we’ll kind of warm it up and do some work on it. But even still we mostly prefer to actually replace the database. What we do is we create a replica, bring it up to speed and then we transition the load over to it. And that’s kind of a form of continuous delivery for data services, which has been surprisingly powerful as an abstraction.

    Edith: So you’re basically, you’re treating them as cattle, not pets.

    Peter:

    Oh absolutely, it’s all about the cattle and model and when you’re farming a very large crew of livestock, you can’t really afford to give each one their own personality.

    Paul: This reminds me a lot of, I think, what started out as an IMVU way of doing data migrations, I’m sorry, of doing schema migrations without locking a whole table. For our people at home, I’ll describe this.

    So when you want to add, let’s say you want to add a new value and it has a default to a particular table in your schema. There’s a possibility of that if it’s a particularly large table, for it to be locked while that happens. At least this used to be the way, I think modern databases are much better at this now.

    Peter: I believe for some default values, it’s now the case that you don’t need to do a table lock but that might be pushed until 9.6 or Postgres, at least but it’s a common and challenging operation.

    Paul: And so what IMVU started doing was instead of adding to a schema, they said that for every table, that schema is immutable and instead they created a new table and then they migrated the data from that table so every time you want to read from the user, you’re reading from user table 38 at the moment and then you start to read from user table 39. If the user isn’t in user table 39,

    Peter: You fall back to 38.

    Paul: fall back to 38, find it, migrate it, write into 39 and then do the thing. Then you have a background process.

    Peter: If you have any listeners who are listening from the world of the Internet and would like to try implementing this technique for themselves, Postgres has something called a schema search path so you can actually, basically make each schema its own version and then you have a search path so if the table wasn’t in the newest version, it would use the last version’s search path.

    Paul: Oh, wow.

    Peter: You could set a version on your code then basically, as you make new versions of the tables and the new schema, you can kind of move them forward. This actually sounds like it would probably work. Now it’s really easy to say that from sitting around a table here in beautiful San Francisco but you have any listeners out there who do give this a try, have them send me a note.

    Paul: I was not expecting this to be a session about continuous delivery databases and the intricacies of modern versions of Postgres.

    Peter: I have to say, it’s one of the hardest things is basically, how do you deal with your data? I mean, data is sort of fundamentally heavy, I mean I think about it as having mass, which is to say, if you’ve got a few terabytes of data on a disk somewhere, just getting it out of that server and into another server is gonna take hours.

    Paul: Right.

    Edith: Right, just querying the data, loading it all into memory, that’s gonna take time and so this is really a challenge for continuous delivery and people have tried all kinds of schema versioning and replication strategies and sharding strategies but, you know, the old question, okay you deploy a new version of your code and it needs the new column but the new column’s not there yet. Okay so you move the data to the new column and then you deploy your code but then the old code doesn’t have it.

    Paul: Right, right.

    Peter:

    And these are hard problems and no one’s really done a great job of answering them. So far, sort of the state of the art is, you know, you continuously deploy the new version of your code that supports both schemas then you continuously migrate your database and then you deploy yet another version.

    Now without continuous deploy this is even harder—

    Paul: This has to be tough on a Gameboy cartridge.

    Peter: Oh yeah, let me tell you, I think we had 4k of RAM or something like that so at least it was fast. But yeah I think data and continuous delivery are both challenging interactions for sure.

    Paul: We end up at this problem in the testing world when large customers, in particular, large enterprises use the old way of testing, which is you actually use a dump of your production database as part of your test suite and one of the major challenges we get when we’re looking at older companies who are coming to the new world, is, you know, your server’s over here, my server’s over there and I want to get a 5 Gig database dump onto your server to run my tests, how do I do that?

    Peter: Right and it’s trying to, like, move a bathtub through a straw. So how do you do that? You just wait, right?

    Paul: We wait until they stop doing that and then they can become our customers.

    Edith: So do you turn customers away then?

    Paul: We don’t turn them away but we don’t really support that use case, they’re going to have a miserable experience.

    Peter: I mean there’s data migration again, right, just coming back to the physics problems. There are things that are getting better so,

    Paul: The speed of light is improving?

    Peter: the speed of light has gone up 13 percent in the last year, did you know that?

    Paul: I don’t know if you’re joking, I presume you’re not.

    Peter: It absolutely has not, yeah. They call it a constant for a reason. I suppose it could.

    Paul: I thought there might be something about, like,

    Peter: Dark matter, no but it was very convincing the way I said it, wasn’t it?

    Edith: Like this is Hindenburg’s Principle?

    Peter: It’s uncertain. So anyway, the logical replication scene is getting better. Amazon’s just launched a new service, it’s three letters long.

    Paul: Oh really?

    Peter: Yeah, uh, got it, hang on. DMS, Data Migration Service. And so through black magic and probably by not working very well, it will move all your data from one database to another in sort of a streaming online way which is really cool if it works.

    Paul: Okay.

    Peter: But, you know, moving from Oracle to MySQL

    Paul: So you can talk to this service as if it’s MySQL and it will pull it from—

    Peter: The way it works is you basically set up a migration job and if you think about it, it basically queries one database and writes to another and it’s a little bit of a hack job in the sense that you have to run, like, a desktop client if you’re moving from a local machine. It’s all very complicated and kind of baroque but,

    Paul: So it’s an Amazon service? It isn’t you’ve used Amazon before, I gather.

    Peter: But yeah, no, it’s what everybody’s always wanted which is to move their data from one place to another without having to think about it and, of course, it’s never gonna work.

    Paul: Okay.

    Peter: But, you know, it’ll help, it can help, things could be less terrible. I think ultimately that’s what got me into databases was the understanding that no matter how impossible it would be to make things good, at least we could make them a little bit less bad.

    Paul: Right.

    Paul: The soul-crushing misery of writing software could at least be slightly less crushy.

    Edith: I feel like writing software is one of the greatest joys in life because you get to create something from nothing.

    Peter: So you don’t work in databases.

    Paul: I’m not sure you work in software.

    Edith: So what are the biggest changes you’ve seen in continuous delivery?

    Peter: Oh, ubiquity, certainly. It’s rare to meet someone these days that, maybe I’m just lucky enough, I live in San Francisco.

    Edith: You hang out at Heavybit.

    Peter: And I hang out at Heavybit with a bunch of other crazy people but I guess I would say growth then, right, I mean we’re seeing large enterprises practicing continuous delivery—we’re seeing small shops practicing continuous delivery, it’s not, sort of, a bleeding-edge concept that only, sort of, a small cadre of, like, sort of, avant-garde developers are doing. It’s really, I think it’s kind of the mainstream.

    Listen to full podcast here to hear Peter talk about continuous delivery in the context of organizational structure, databases and pricing.

     

  • To Be Continuous: Empathy in Software Development

    To Be Continuous: Empathy in Software Development

    Thanks for listening to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Sam Stokes about empathy in software development, how mini-services are the new microservices, and constantly updating software in a constantly updating world. Sam is currently a freelance software consultant and previously was the co-founder of Rapportive.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/237727552?secret_token=s-k6OKe” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=flase” width=”100%” height=’166′ iframe=”true” /]

    Paul: Alright, Sam, so what’s your favorite thing about continuous delivery?

    Sam: I like to think about it terms of shortening feedback loops. A lot of things that startups do can basically be looked at as doing something and then waiting to get some feedback on it, and then deciding whether to do more of that thing or to do a different thing.

    And continuous delivery is, people talk about sort of reducing the batch size and also about shipping stuff faster and clearly having some kind of automation. And continuous delivery is going to help you with both of those things.

    Paul: So is this specifically on the, sort of on the journey of a small startup towards product market fit? Is that kind of the sense you’re thinking of it as?

    Sam: I see it as applying to teams of different sizes. So for sure in a small startup you’ve got you need to iterate fast, you’ve got all these product ideas. But I think even if you’re a small team in a big company as I was recently working on, and still have a lot of the same pressures you have, product managers who want to try new ideas, you have customers who are sending in feedback all the time—some of which you want to listen to, some of which you want to tune out.

    It’s sort of, in some ways, a lot like being in a startup. At least, ideally it would be. And a lot of the problems you can run into are basically about not being able to close those loops quickly enough.

    Edith: Sam you’re someone who knows a lot about loops, so now might be a good time to introduce yourself.

    Sam: Sure, my name is Sam Stokes. Until recently I was working as a staff software engineer at LinkedIn. And right now I am, I guess a freelance software consultant, thereabout.

    Paul: I feel there’s a little bit of history missing there.

    Sam: Alright. Going back in time a little bit. So I was a co-founder of a startup called Rapportive. That was a plugin for Gmail that when somebody emailed you, would show you a social profile of that person in the right-hand side next to their email.

    So that you could get some insight into who you were talking to and put some more humanity into your communications with people—that’s sort of the way we saw it.

    Paul: That’s still the only plugin to Gmail that I use.

    Edith: I use it, too, and it’s for exactly that reason, ’cause I do a lot of customer development, and it helps so much to do, to empathize. To see that this—

    Paul: Right, right, yeah is a real person.

    Edith: You know, to see their face and you feel a connection.

    Paul: It’s interesting just with kind of customer communication. One of the tools that we use is Intercom. And once I started using Intercom it was the same sort of feeling I got the first time I used Rapportive, that like I feel that there’s a human at the other end of this and I feel I know who that human is and it just changes how you communicate with the people.

    Sam: It’s really motivating to hear and to see other examples as well. We really felt like email and sort of a lot of business communication in general can be sort of cold and formulaic.

    Yeah, just bringing that human element, even just seeing a face. I haven’t used Intercom but I’m guessing they have some similar ideas.

    Paul: Similar, when you send out messages or reply to someone it does the person’s face there.

    Sam: Right, we had a sort of plugin API in the, it was a sort of private beta version of our product and one of the things people used it for, so this would let you plug into the sidebar, people would plug it into their customer database and so they could see something like, they’d get an email from a customer it shows them how long they’ve been a customer, how much they’ve spent in their store—

    Paul: Right, right.

    Sam: What version of the app they’re running, that sort of thing.

    Edith: Was your original idea to do that sort of empathy or was that something that just you iterated into?

    Sam: The empathy was where we started. There were lots of ideas about taking all the information from all these different sources, you know public sources like, truly public sources like Google, quasi public sources like a Facebook profile if you’re friends with them on Facebook, and then private sources like stuff in your company database.

    All stuff that you would be able to search for if you could be bothered to go and do the searching but doesn’t necessarily bring itself to you while you’re talking to that person. And yeah, it was very much, what should you know when you’re talking to this person in order to make them feel like they’re interacting with a real person and an empathetic one.

    Paul: So I remember when you launched Rapportive it was like, or it felt to me like it was, an instant hit and maybe that was just the overnight success that comes after years of trying. I had a doco and what was the kind of software delivery process of that?

    Sam: Hmm, that’s a great question. I guess there’s two parts to that really. So how did it get there and what was our process that allowed us to survive it? So I guess how did we get there, it was a certain amount of luck in that process as there probably always is. We built our prototype and it was a pretty minimal prototype.

    You know, it had the functionality that was like this awful demo website up. The only reason there was a website up was we were trying to raise a seed round at the time, we were applying to OIC. Y Combinator requires that you have a demo site that’s public. They don’t like it to be password-protected. And so we had this thing up where you could install it and we had like 20 users. One of our friends decided that he was going to email a friend of his at one of the blogs, I think it was ReadWriteWeb and say, you know, hey you should check this out.

    And his friend checked it out and it turned out that one of the demographics that really liked the product that we built was journalists because they get a lot of emails from people they don’t know and they want to sort of know an extra nugget. I guess that was one of our takeaways was like, if your startup is really useful for journalists, that’s a great way to get some early press.

    Edith: That’s what worked for TripIt because journalists travel a lot. So TripIt just got all this free press basically ’cause journalists used it, loved it, and blogged about it. Unfortunately, journalists do not do continuous delivery so much.

    Sam: So how do we survive that? ‘Cause what happened in that 24-hour segment was that we went from 20 users up to 20,000.

    Paul: Wow.

    Sam: So I guess one part of it was our CEO had made the original platform choice. His name’s Rahul Vohra, he has a computer science degree, he’s an awesome technologist, but his focus is not generally on writing the software. He’d been writing the prototype, and he was like, ‘I need the quickest thing that I can get to get this market, so I’ll put it on Heroku.’

    Which turned out to be a life-saving decision because when the 1,000 X traffic spike arrived we just added some more dynos. So that was how we didn’t burn down. What did we do after that? I guess in terms of software process we were pretty young as a company, so it was pretty much hack away, read all the tweets. But there was a lot of, what you might call customer development more than software development at the time. So we pretty quickly decided that we were going to try and respond to every single tweet and every single email and so on to like try and live this idea of a good customer experience.

    And I think that helped to sort of amplify the viral effect of that press spike because people were sort of retweeting the things we were saying and getting a generally good feeling about us.

    And so a lot of those tweets were like, ‘Oh I wish I did this,’ or ‘I wish I did that,’ or ‘I wish you could see Twitter profiles in there.’ So it was a little bit of sort of customer support-driven development. Which obviously doesn’t scale but it can be a pretty good way to build some goodwill early on.

    And I think a lot of our users were sort of quite passionate about using us, ’cause they felt like they got that personalized attention.

    Paul: So did the, when you scaled the dynos did it just work?

    Sam: It worked perfectly until we hit the rate limit for the API that we were getting all our data from.

    Paul: The LinkedIn API?

    Sam: At the time we were using a service called RapLeaf.

    Paul: Oh yeah, yeah, I remember them.

    Sam: It wasn’t their main product, but they provided an API which … So basically the thing that Rapportive does, which is take an email address and produce social profiles about that email address. At the time that was entirely just UI wrapped around RapLeaf. We ended up building sort of our own version of that.

    And so we’d been using their API on a sort of demo basis. Because again, 20 users. Yeah, we immediately had to learn about things like request queues, so that our app wasn’t just falling over waiting for their API, and then frantically Biz Dev-ing to try and increase our rate limit, while at the same time figuring out the best ways to get around the rate limit without upsetting the Biz Dev deal. Yeah, we didn’t really have much of a sort of continuous delivery process at the time.

    But I guess not as many people were talking about it—this was 2010. I think the first blog post had come out from Iron View and so on, but I think it wasn’t as, you know, there weren’t full startups supporting it. What we did have was the Heroku deployment model, which was basically like within five seconds of writing the code you can deploy it to production.

    Which if you squint at it, kind of is continuous delivery because you’re saying, all of the bits of your deployment that you have are automated. We didn’t really have many tests at the time, so the tests we had were automated ’cause there weren’t any. Our deployment was absolutely, you do it get push, you don’t have to like move any servers around or anything like that.

    Paul: Seems like the testing model around, or the continuous delivery model around a Gmail plugin is probably going to be a fairly difficult process anyway.

    Sam: We never really had a good answer to, you know, what happens when Gmail changes their UI. The surprising part was it didn’t happen that often. We thought it would be breaking every week and we’d need all kinds of canary accounts and end-to-end testing to monitor that kind of thing. In the end, it happened rarely enough and we noticed quickly enough that we could usually just fix it on the spot. So even when we got better at testing and that sort of cycle of things.

    Paul: We had a problem or I guess solution with GitHub. So GitHub used to be super flaky around two, two and a half years ago, when they were still on Rackspace. And whenever something would go wrong with GitHub we would find out immediately. All of our, all the developments would start failing and so before they knew that there was a problem, we knew there was a problem.

    And we would often like, declare on Twitter, oh we think GitHub’s having a bit of an issue and two minutes later their status would tweet us and then we would retweet their tweet.

    Edith: So they were using you for their free testing.

    Paul: Well, our customers at least.

    Edith: No I mean you were acting as their tester and telling them that—

    Paul: Well, I mean it was very, you know, automated on our but we didn’t have necessarily automated fixes at the start. It kind of happened every, you know, two or three months and they would swear this is the last time and—

    Edith: It’s always the last time.

    Paul: And in the end it happened like six or eight times and by that time we had built a thing that allowed, you know, certain fixes, but it wouldn’t be quite, you know, fully automated process that you would expect if it was happening all the time ’cause it always seemed like it was gonna be the last time.

    Sam: And it’s probably always happening for a slightly different reason or in a slightly different way. It’s actually, it’s an interesting question to me in the subject of continuous delivery in general. People always talk about every time you commit you run your tests, every time you commit, you deploy.

    But that’s actually only one of the kinds of events that you’re dealing with in a software life cycle, right, when something’s actually running. I mean, sure you’re changing the code but also the world’s changing around you. And I feel like you always end up needing a little bit of both—you need something that’s gonna trigger when you change it, but you also need sort of an eye on everything you depend on. And, of course, even figuring out what all those things are can be interesting.

    Paul:

    So I wrote a blog post about this on the Heavybit Blog, I think a while back, and it was about third-party APIs you know, how do you test third-party APIs. And the conclusion that I came to was that, so if you run the tests as part of your test suite, then if something breaks, your test suite will be broken, but also some things will just like randomly break in production.

    And then of course you can’t deploy because your test suite is broken. So you’re in kind of the worst of both worlds. And so what I considered is, is that you need something which, you know, is just permanently monitoring. Like whatever API things you do constantly hits your personal accounts or your test accounts for those things, and then stub it out in your test suite.

    And that way you kind of get that best of both worlds, your test suite doesn’t fall over and you get an actual monitoring event when the third-party service goes down in production.

    Sam: I can tie this to an experience from LinkedIn, actually. So LinkedIn has the model where there’s a staging environment and a production environment. And the staging environment is—I don’t know if it’s technically called a staging environment because it’s not exactly production-like, but what they actually call it is early integration. Because it’s the environment in which teams integrate with other teams.

    And so the idea is you deploy your stuff into EI, early integration, as soon as you can so that if any other team depends on you or vice versa, you can find out there before the thing hits prod. But what you’ve effectively got in EI is this constantly changing environment full of dependencies and you have exactly this problem and different teams which using different solutions to, well, let’s write some integration tests which actually hit the dependencies and then we’ll run them in our test suite when we deploy to EI or when in the build step which—

    Paul: I can’t tell as you’re describing this whether this is a good thing or bad thing.

    Edith: Yeah.

    Sam: I think there wasn’t a clear conclusion, but my conclusion is that it doesn’t work for the same reason that you’re saying.

    You need to know that if your test suite fails, it fails because of something you did. And then you need something else that can fail because of something someone else did.

    But if you’re ever unsure which of things it is, then the signal is lost for both systems.

    Be sure to listen to the full episode. While you’re there check out Heavybit’s library, home to great educational talks from other developer company founders and industry leaders.

  • Version your infrastructure

    Version your infrastructure

    Infrastructure is code, and like any code, you need to version it. Notice how that works:  If it’s infrastructure, then it’s a fundamental part of the system; nothing’s going to work without it, which makes it valuable. So valuable that in full-stack deployments, it is part of the application. And if it’s code, then it should be under version control.

    (more…)

  • 5 Ways Continuous Integration Reduces Risk in Development

    5 Ways Continuous Integration Reduces Risk in Development

    ***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

    There’s a lot of talk about DevOps and Continuous Integration in the tech space right now, but sometimes the implementation of those methods can be tricky.

    An effective Continuous Integration strategy requires DevOps teams to manage risks in software development with a range of practices.

    Here are five processes you can use to make Continuous Integration work for you.

    Automated and Self-Testable Builds

    DevOps professionals can use separate machines for software integration to eliminate the tight coupling between the build process and Integrated Development Environment (IDE) that leads to last-minute integration concerns.

    Everything needed to build the software should be contained in a separate version control repository.

    Dedicated Continuous Integration servers and open source build automation tools can run these builds through continuous testing processes and deploy the product in the required environments.

    Automated deployment further eliminates the bottleneck of waiting for other team members to deploy testable versions of the latest software.

    Synchronize Version Control Repository

    Continuous integration is only possible as long as Dev, Ops, QA and the wider IT team is tightly integrated, collaborative and supportive of each other’s efforts.

    From a tooling perspective, effective collaboration requires a separate-but-unified master database updating changes in real-time.

    All database artifacts — including scripts, procedures and other assets should be placed in this version control repository.

    DevOps teams can enable Continuous Database Integration by using automated database integration, local database sandbox and version control repository to share database assets, as well as empowering developers with capabilities to modify the database and making DBAs a part of the Dev team.

    Build a Stronger Foundation and Expand Test Coverage

    Limiting test cases to individual assets executed at every check-in to the version control repository ensures the continuously integrated builds are defect-free in combined form.

    Tests should be repeatable and performed continuously in virtualized systems replicating unavailable testing environments.

    Continuous Integration methodologies allow you to determine how much of the code was actually tested without much manual intervention.

    Although some code coverage tools can assess the percentage of test coverage by package or class, Continuous Integration systems ensure this coverage is updated in real-time.

    For instance, DevOps teams can run these tools as part of the Continuous Integration server build script every time the version control repository is updated with a code change.

    With this information, untested code would never get through to the next stage and the latest test will be conducted across all areas as required.

    Enhance Project Visibility

    With automated build testing and deployment using a centralized code repository, build feedback mechanisms update Devs and Ops with the latest test results and the necessary progress information.

    In cases when the same level of automation for complex builds seems impractical, DevOps teams using manual procedures must establish a feedback loop to bring everyone on the same page with real-time communication and notification of accurate performance and results.

    These capabilities ensure the same high standard of reporting is maintained for manual testing practices as available for automated build testing systems.

    Follow Consistent Dev Standards

    Quality code forms from the ground up, with adherence to common code and architectural standards established for an organization’s specific requirements.

    Automated inspection tools enable Devs working on short and quick builds to maintain their focus on quality from the very foundation of the software itself.

    DevOps teams should assess every integration build for potential defects that surface after integrating several iterative builds but should be taken care of far earlier during the development stages.

    In Conclusion

    The right toolset is key to Continuous Integration, as is the cultural change that facilitates this DevOps practice.

    Continuous Testing presents several areas of improvement for DevOps teams to develop iterative builds in individual, isolated branches and then merge the latest builds in a unified, single and well-synchronized Master branch.

    The idea is to ensure all risks are mitigated beforehand so DevOps professionals are not bogged down with endless needs to rework the product — and that gives them time to focus on true Continuous Integration.

  • Top 3 DevOps Practices for Operational Stability

    Top 3 DevOps Practices for Operational Stability

    Operational Stability is one of the main concerns in organizations. Especially some larger organization struggle to get a more stable IT landscape. Too often it’s political discussions that are preventing them from doing something about it. Organizations that have DevOps teams in place can just smile because they are enabled to be in control of their entire delivery chain from development to production. Read about what DevOps teams do in order to be real DevOps and why they’re smiling all day long.

    (more…)

  • Moving from Infrastructure Automation to True DevOps

    Moving from Infrastructure Automation to True DevOps

    Deciding on two separate teams, or an integrated end-to-end team

    While a lot has been said about “the right way” to do DevOps, what we’re seeing in practice are two different approaches being tried to enable development and operations to work together as a team. On the one hand, there’s the approach to create an end-to-end, integrated development team, which you might call ‘Devops’. On the other, there’s ‘DevOps’, featuring two development teams – one working on application code, with another developing a platform using Infrastructure as Code techniques in a virtualized environment.

    DevOps: providing a platform

    Perhaps the most obvious way to go is to have a platform team to provide a runtime environment that “just works”, and a separate application team that develops the business logic to run on the platform.

    For some organizations, particularly where one of the teams is outsourced, this separation is going to make more sense. You might buy an off-the-shelf product, build your own platform in-house (which seems to be what many organizations are looking to do with Docker right now – warning: it’s harder than it looks!) … or sign up with a PaaS vendor.

    Whatever the situation, the platform should be versioned, just like any other piece of software.

    This approach can be ideal for developers because they are free to work on code and focus on the business logic, and they really don’t need to worry about how it runs in production. It’s up to the operations team to make sure that the platform is stable and that it delivers what the developers need.

    Communication is still key

    Even with a physical or organizational separation, the importance of effective dialogue between the two teams cannot be overstated. This won’t work if the relationship is adversarial. There has to be a genuine awareness of common goals. If the operations team doesn’t understand where the development team is going, and what they need from the platform, then there’s going to be trouble. The idea that one platform is going to cater for every development need is often unrealistic, so versioning and the ability to run multiple platform versions side-by-side, or cater for exceptional situations where a particular application might need its own stack, is very important.

    There has to be a clearly defined boundary, which should be close to the application code and configuration layer. The application handles the business logic, the platform handles the deployment, logging, monitoring, failover and auto-scaling.

    The development team can focus on getting those new features and improvements into the live product, while operations handles the actual roll-out and manages traffic. For this to work, the platform must enable you to quickly determine who is responsible for any problem that crops up. It’s worth investing time in the setup to ensure that errors are explicit, so you don’t waste time tracking them down.

    Arguments between the two teams are bad for progress and morale.

    Devops: A full stack development team

    For some businesses, it’s going to make more sense to put together integrated development teams that work directly on delivering the full stack from a shared code base, from virtual machine to application code.  This allows you to design and optimize each system in a way that takes the application’s specific perspective and requirement into account, and can be an especially good fit if your applications differ from each other greatly in terms of the operating systems, middleware, etc., that they run on.

    This approach can only work if the overall group is small (Amazon’s “two pizza” teams are a good example) and has sufficient knowledge and experience to see the big picture of the entire system.

    It’s not possible for everyone to understand every part of the full stack in detail, but there are areas of overlap where greater collaboration will increase the level of shared knowledge and allow the team to build a better overall system.

    Pulling together

    The shared foundation of the Devops approach builds mutual respect and enables more free-flowing communication, but it must be based on shared goals and a vision of what the overall purpose of the system is. This integrated approach can give you much more flexibility, but it also means holding the whole team responsible for keeping everything running. Shared code equals shared responsibility.

    The right approach

    You need to take a baseline, and then put metrics in place to track your progress. You have clear goals in mind and a system to measure them. So how do you find the right approach to achieve those goals?

    It would be wrong to say that one approach is better than the other. What you need to do is assess what will work in your organization. Both are means to achieving similar goals, but finding the right approach is about understanding what your company is looking to achieve. You might choose common goals, like reduced expenditure, or reduced error, or you might choose something more specific to your industry. Establish your goals at the outset, and make sure that they’re measurable.

    The right approach is going to be the one that gets results for you — that’s why setting up a baseline and metrics is so important. You may find that a hybrid of the two approaches is going to be the right fit, but you aren’t going to know for sure until you have some hard data to back up your expectations.

    Be sure to join staging-devopsy.kinsta.cloud editor-in-chief Alan Shimel on Thursday, May 21st  as he hosts the webinar, “Ship Faster without Breaking Everything” to further your smarts on Continuous Delivery.

    About the Author/Andrew Phillips

    Andrew_Phillips_XebiaLabs_cropAndrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.