staging-devopsy.kinsta.cloud

Category: Features

  • Transition to a new human ecology? We still need managers

    Transition to a new human ecology? We still need managers

    Recently I replied to an article “DevOps Cuts Out The Middleman” that excoriated Middle Managers as a cancer. I commented that I disagreed.  Nicely, I was invited to make further comment as the loyal opposition. I guess that makes me a sanctioned Troll. Hopefully I am informed enough to make a useful comment. FYI, I have developed software since the time of using x86 assembly gotoxy to move the cursor location on an amber screen. I consider MVC wonderful, but trivial. I have seen many, many projects, much good, much bad, some serious ugly and I have patiently watched like a Troll does. Worse than any Troll, I recently published a book on Philosophy, so asking me a question is generally contraindicated. Sew, you say Middle Mangers are an evil waste of time. May I politely disagree… and go down hill from there. The entire vision is deeply flawed and note they were described as “cancer”. How emotive. From strength comes weakness. From weakness comes strength. Know your weaknesses so they don’t bite you in the byte.

    In a perfect world, no, we would have no use for Middle Mangers. The philosophy behind that critique of them was based on that. It is not a perfect world though and managers provide critical, useful leadership.

    The description of a DevOps team in the article was the description of a machine. Why not? Machines are notoriously efficient and who is to argue with that? Except that we are humans and on deep levels trying to imitate a machine is something to be very cautious about. In the long run, it is a disaster. Machines are different. Machines are static, humans are not. Humans are far more random, so the system itself has built in risks. Face it, we’re developers and we are often best at imitating machines, which has its own liabilities, but we are not machines and even machines have problems. Human systems have problems sooner than machines, especially when acting like machines and trying to provide the performance of one. Managers are the repairman and lubricants of human based machines. They do the Configuration and Release Management for the team. They are the long term employees that know what is going on and how to handle the unexpected. Their job is providing continuity that only they can. They have the breadth of view that we lose at our keyboards. They smooth things over when a shock to the company comes like loss of a critical system or employee. Someone has to take responsibility. Who do you want it to be… Oh, stake holders. Yah right, they never did before they were called stake holders and still usually don’t. Another thing middle management can do (and I have seen this repeatedly) is when upper management makes a mistake (and they do), it is only the middle managers that can compensate. I won’t mention training and entry level, but obviously there is no consideration of that no matter how critical it is in the real world.

    I will just refer to a Middle Manager as a manager, because in this day and age, you have management that are financial planners and you have management that produce a product, whether you call them managers, middle managers, project leaders or what not. That kind of manager provides many functions, hopefully. Think of it this way. Their importance is illustrated by the disaster that ensues when they fail. We all dread that. Leadership is a great skill and of great value.

    The article assumed that everyone was proficient at their task. No, that is not the case. Skills and knowledge vary. I used the predecessor to .Net, the C++ Builder, but I am still learning C# now (await). Always a manager’s job is to get the job done with the resources they have. You may not have a team made of master developers. I have always appreciated a good manager, defined as one who gives me direction. A great manager is one that makes me more than I am. A good manager can make very good use of a weak employee and if the employee has the potential, a good manager can make them a strong strong, valuable asset. I know, Brian did that for me.

    Developers are not known for our people skills. Those are the skills of a manager. Managers must be psychologists. Developers are smart and most know how to cooperate, but don’t tell me ego problems don’t arise. You can’t even get in the same room with some of the Prima Dona egos I have worked with. In any decent team, you likely have one Aspergers and likely a couple people with a problem dealing with it. Is it possible that anyone is immature? So, do you have someone who refuses to go to meetings? By the arguments of the article, meetings are very like Middle Managers. Then there are team members who need to catch the 4PM bus in Mumbai as well as the one who may not even speak English. Software development is like science. People think it isn’t personal, but it is. Someone has to manage the personal. Managers are a lubricant and without it, the gears of the machine are very likely to rub and grind. They can make a team succeed that could not operate without them.

    A manager is a brain. Most team members are necessarily working on the details. Even the designer is swamped. The Scrum Master, besides being a Middle Manager anyway, has varying degrees of authority. An enforcer is needed. Someone has to tell people what the standards are … and enforce them. Someone has to decide how exceptions will be handled consistently. It provides its own efficiency. Chaos is never efficient. The manager provides the focus and order. That is their function. They must provide a focus that no one else on a team can provide. Systems help, but like there are no perfect laws, there are no perfect systems. Who enforces code reviews? Don’t tell me they just happen. I’ve been on great teams that were not under pressure and sorry, they were lax. How many times have I heard “we didn’t need to go down that rabbit hole”. A manager is like a judge that interjects the essential human element to a non-human system. A good project manager makes sure that the project is delivered. That is the preview of all team members, but just as we all have our special strengths, A good project manager will be the one that keeps the project focused and is going to deliver it come what may.  We all know just how common a skill among developers it is to be able to interface with senior, non-technical management. Sure, anyone can do it or would you rather have someone with the skills.

    OK, I’m a troll, so in respect for the thorough job I always do, it’s time for the mandatory ad homin attack. Lemme get to trashing the author of this article, Tony Bradly. Definite ignoramus. So he authored a few … well maybe a few dozen books. That means nothing! Decades of Security, Communications, Compliance and Systems Administrations leadership. Dum Dum Dum! Consults on systems for a bunch of Fortune … err … Fortune Rarefied companies. So what! Founded the Bradley Strategy Group. Anyone could do that. TV and Radio. A Celeb! Celebs always have opinions. So where does he get off thinking he needs to speak up? Actually, that’s not so funny. The guy is genius in a group of some of the brightest. Have you ever met anyone like him? I haven’t and I’ve worked on many teams and I remember a few truly incredible people, but not in that class. The article makes the assumption that the members of the team are all extremely competent, but that the manager is necessarily not. I don’t get it… or maybe I do. I reviewed the article to make sure I knew what it said. Odd, it said little. It did say middle managers were lazy, useless and a cancer. How very emotive. Ah yes, it looks like ideology. I recognise that flavor. That is a philosophy descended from Ayn Rand. I hear it is popular in Silicon Valley and much of IT culture. This is a model of a team of perfect individuals in a perfect world operating in a specific time frame. It is mostly illusionary, risky in the short run and self destructive in the long run by its static nature. The real flaw of the concept underlying the article is philosophically very deep. I know where this originates. Just remember the end of Ayn Rand’s story. They kept their ideological purity, but it resulted in the fall of darkness. This concept is based on a static illusion and like the story, there is no room for imperfection. I bet Tony has little need for managers or leadership, but he is one. Sorry, but most humans are imperfect and it works better to work with that than live an illusion that ends up fraught with danger.

    About the author

    MB1ccMichael Breeden is a software engineer with over 15 years of demonstrated success providing a wide range of software solutions including the complete software application life cycle, relational database development, business objects and business support. He can be contacted at a1swdeveloper@yahoo.com or visit his website at http://a1swdeveloper.com/

  • Developing and enabling a DevOps culture in your team

    Developing and enabling a DevOps culture in your team

    We’ve all been there. You stare at an application that was built over the last four or five years. You look at the code, which is ugly, full of all sorts of workarounds and hacks built up over time, and you say to yourself, “If we just rewrite this from scratch, it will fix a whole bunch of edge cases and be easier to maintain.” But most of the time, starting over is the wrong approach.

    That’s even more true when you look at creating a team to work in a more DevOps-y manner. People think of some platonic ideal of a DevOps team and say, “If we just hire people with these skills, everything will be perfect.” They look at the current state of affairs between developers and IT and say, “There’s no way we can get from here to DevOps with the people we have. Let’s start over.”

    The people with the skills and experience you need to create a DevOps team are probably already in your organization. In our 2014 State of DevOps report, 57 percent of  of our 9,200+ survey respondents said they were already using DevOps practices, compared with 16 percent who said they were part of a DevOps department. It’s pretty clear that you don’t have to be in a named DevOps department to be committed to using these practices. Since the term “DevOps” is pretty recent, but the practices are not, it’s also safe to assume that these named departments weren’t formed by a super class of engineers, but by people who were practicing DevOps before it was a thing.

    Starting over is a clean-slate approach

    People want to replace the team for the same reason that they want to replace all of their code: “Our application is warty.” All working code has bugs, workarounds and messy encounters with reality. Sure, some of your workarounds can disappear with a new design, but it’s not likely all of them will. Most of us feel a strong temptation from time to time to chuck it all, and start from scratch as if nothing were already in place — design and build a new, clean pristine system, and move forward from there.

    Bringing in a new team to support DevOps is also a clean-slate approach, and it’s one that guarantees you will lose all the accumulated domain knowledge you have built up over the years: knowledge of the code base, of the business needs that drove development of this code base, and of how to work with others across the company. All this knowledge is not only valuable, it’s critical. A team of new people, whatever their technical skills, don’t have that knowledge, and can’t obtain it instantly.

    The problems you’re facing are not all due to particular individuals on your teams; often, they originate in your organizational culture. That’s what you need to change. However the odds are good that you already have at least some of the people you need to create a functional DevOps environment. After all, your team has already had some successes, otherwise you wouldn’t still be in business, and it’s likely they’ve had to work collaboratively across the dev/ops divide to achieve these. Our 2014 State of DevOps survey showed that the longer a company has been using DevOps practices, the better its IT performance. It’s likely that you have individuals and teams succeed due to practices that fall under the DevOps umbrella, even if they weren’t using that term then.

     

    Cultural changes (especially for management)

    There are cultural changes involved in bringing DevOps to an organization, and sometimes these can seem insurmountable. These changes can be harder on management than they are on staff, because they represent values that are often quite different from how management teams traditionally work, or expect their teams to work. Taking a DevOps approach requires you to be comfortable with working in the open, with a much higher degree of transparency. Managers need to enable and support different kinds of information flow, across to peers instead of, or at least in addition to, the traditional flow of information from the top of the hierarchy down, and back up through the chain of command. Managers and technical leads also need to become much more comfortable with failure on their teams, viewing it as a sign of growth and a way to learn.

    Start simple

    How do you start building a DevOps culture? I recommend that you take a simple, easy-to-solve pain point and automate it away. Your existing staff are the ones who know what that pain point is, and you can’t automate a process without understanding what it’s meant to do.

    Automating away one pain point will start the ball rolling, help staff feel more successful in their roles, and allow them to reclaim time to make further improvements. Done right, this is a self-sustaining process.

    Create good interfaces between teams, and introduce reliable measurement so your team can improve day to day. That will help everyone feel greater trust in new tools and processes.

    A window into what’s happening, and when

    Get used to working transparently by putting in place a single and accessible ticketing system for tracking work and prioritizing it. The rest of the company should be able to see exactly what you’re doing, particularly the consumers of your services. People coming from a service desk background may have to adjust to providing more frequent updates and more details around issues, and for these updates to be visible to a much larger group than they are accustomed to.

    Use your ticket system to drive buy-in across the organization. It will make prioritization of work highly visible. Anyone can go look at the ticketing system (however it is implemented), and say, “Ah, it looks like they planned to deliver this in the next two weeks but have pushed it out by a week due to an unexpected security issue.” The trust engendered by this visibility is critical when you’re trying to instill new values.

    Across the silos

    Get comfortable with horizontal connections in your company, with one team deciding that they need help from another, and asking for it directly. Don’t make information travel up and down a hierarchy — it’s a time-waster. Plus, remember the telephone game? The information that comes out the far end of the chain doesn’t bear much resemblance to what went in.

    Encouraging people to work more laterally can be a really difficult change for a lot of managers. It doesn’t necessarily mean that they’re micromanagers; it just means they are used to information flowing up before it flows back down to another part of the organization.

    Your DevOps efforts will probably fail unless your entire management team buys into the required changes, and executives recognize that they’re going to have to change the way they operate, as well.

    A single source of truth

    What makes large-scale system administration possible is automation and a single source of truth. That source of truth is the description of the desired state of the system.

    The infrastructure-as-code movement really calls for a declarative approach: “This is the state a given machine should be in. It should have these services running, these packages installed, and configuration files that look like this.” Once you have a simple text-based description of that information, you can then check it into version control, you can create version tags, and you can cheaply maintain release discipline. At that point, you can actually start applying software engineering principles to managing infrastructure, another key DevOps principle.

    What wakes you up at night?

    We need to ask what causes manual work — especially work that ends up having to be done in the middle of the night — and how we can prevent it. What actually happens that makes us wake people at 3:00 AM to log onto a box and fix something? How do we prevent the change at its source? How do we test whether a given change will cause someone to be woken up at 3:00 AM to fix something?

    For an operations team, these questions really matter. If you start thinking about everything framed by, “How will this behave at 3:00 AM? How will I understand what’s actually being changed?” and (better yet), “How can I stop my team from being woken up at all?” you have a pretty strong incentive to start writing effective tests.

    Cultural barriers

    The technical aspects of DevOps are not beyond anybody’s skills. It’s the cultural aspects that are going to make people uncomfortable up and down the line. The road to DevOps will be much smoother if you work at it incrementally to overcome these cultural discomforts, rather than throwing out all the institutional knowledge that you have, and bringing in people who are culturally comfortable with DevOps but know nothing about your business.

    Just as DevOps is about incremental improvement of your infrastructure, you should be incrementally improving your people. It’s going to keep them happier, you’re going to make them more valuable employees, and it sends a really strong signal to the rest of the organization that you’re the kind of company that grows people according to organizational needs, and doesn’t just deal with change by hiring from the outside.

    If you’d like to dive more into the data behind the suggestions in this article, or are looking for more concrete recommendations for managers and individual contributors, check out the 2014 State of DevOps Report

     

    About the author

    nigelNigel Kersten, CIO (@nigelkersten) came to Puppet Labs from Google HQ in Mountain View, where he was responsible for the design and implementation of one of the largest Puppet deployments in the world. He’s been a sysadmin for Linux and Mac deployments since before Mac OS became a UNIX, and was voted one of the 25 Most Influential People in the Mac Technical Community in 2006.

     

  • Driving Product Innovation, a Q&A With Arcweb’s Chris Cera

    Driving Product Innovation, a Q&A With Arcweb’s Chris Cera

    Founded in 2011, Philadelphia, PA-based Arcweb is an enterprise software development firm that specializes in mobile and cloud computing. Arcweb’s team is experienced in product management, agile software development, and lean startup methodologies, and helps clients to manage the product lifecycle, including design, architecture, development, deployment, and support.

    We recently had a conversation with Chris Cera, the CTO at Arcweb about balancing security and usability, building new products, and the evolution of development practices. Here is our conversation:

    staging-devopsy.kinsta.cloud: Can you tell us a little bit about Arcweb?

    Cera: We specialize in the product design and development of enterprise software. The majority of the work that we do, from a vertical perspective, is in financial technology solutions. We service three micro verticals within financial technology. One is bank operations, which people sometimes call retail bank operations. The second is investment management software, so we do a lot of broker–dealer and broker and financial advisor software. And the third is insurance, such as claims management, sales enablement on the insurance side.

    That’s the majority of our business; the other side is general enterprise software. That runs the gamut of the business, including operations, marketing, and sales. From a technology perspective, about a third of our business is mobile application deployment – a lot of iOS enterprise-focused applications. We also do traditional web application software development.

    staging-devopsy.kinsta.cloud: Well, since you’re there, let’s start there. I mean, what do you see as the key to obtaining that right balance among usability, security, and compliance?

    Cera: Naturally, we encounter a lot of security-oriented scenarios, and we’re always dealing with challenges there. I find that design and security are almost diametrically opposite. If you want something that’s really usable, that usually means it’s also kind of insecure. In the banking space, this is a huge challenge for innovators and disruptors; you’re trying to build something that is easy to use but it also needs to have bank-level security. Usability and security are always on opposite paths, so we’re always toeing the line of those two worlds to produce products that meet those needs.

    DevOps: When releasing a new app, or product, what separates successful innovation from ideas that fall flat?

    Cera: It ultimately depends on the end users. They’re number one, and number two is the customer. We were recently working on a mobile application project that got killed. The risk, security, and compliance people killed it because innovation often scares people.

    I think a lot of it comes down to education, because if a big company doesn’t do it, then a small company is going do that innovation. That’s a challenge for any large company: it wants to innovate in its space but it often is not really able to.

    staging-devopsy.kinsta.cloud: There always seems to be a struggle between innovation and the status quo, so how do you bring security people in early in the design process?

    Cera: It usually takes someone who is much more visionary at the customer site to push something like this through. That usually means the executives of the business need to really endorse what’s happening. Ultimately, it’s a very high-level business decision to make innovation work in a large company. The products I see getting killed are those in which product managers have a budget.

    They’ll try a test market and the feedback on the product is great. Next, they want to get it through product governance, or whatever that company calls the division that has to approve any new product development. That’s where the product gets killed – that because someone who works in risk and compliance will give 100 reasons why it will cause legal or fraud risk.

    staging-devopsy.kinsta.cloud: How have these challenges changed as development methodologies changed from those of 10 years ago to today’s agile and continuous integration/deployment organizations?

    Cera: I would say people are getting smarter and smarter about how to launch software projects. I think if you looked at how many software projects failed 15 years ago versus how many fail now, I’m positive you’ll see that we’re learning how to increase the success and minimize the risk of projects. Obviously, there are always outliers, and failed projects, but we are getting better.

    Also, agile software development has been around for 15-plus years, and a big part of that was focusing on tighter iterations and continuous integrations or continuous deployment. I think that’s largely believed to be a good thing, but a lot of the big enterprises still have not adopted it. If you have a large multinational workforce, trying to get them to do one or two week iterations is nearly impossible. There are just too many challenges with it.

    From a security perspective, it also means that the security people who might have been invoked once a quarter to do reviews, penetration testing, etcetera, now need to be involved much more frequently. That means much more focus on automated testing tools. I mean that there has been tons of innovation on automated testing tools in just normal development cycles.

    Good security just requires the investment and for the business to understand that this is how you have to invest time and energy into a project. Security tests would just be part of the building process. So, if you talk to agile project managers, everything is focused on iterations. The security test just becomes a part of the deployment process. Then, of course, you’ll always want to do some manual test. I mean that even though we tried to automate as much as possible, some tests are too expensive to automate, so you just need to have a trusted party run a manual test plan. We do that with every product we build.

    staging-devopsy.kinsta.cloud: How do you like to start a product? Do you like to build a minimum viable product and just try it, see how people kick it around and use it?

    Cera: For one of our customers, we literally went into coffee shops and just asked people to test out some apps for us. It wasn’t real data or anything; it was a simulation. We were trying to get a sense of whether the design made sense to the customers. Do they understand it? Would they actually use it? This is something that we specialize in because we’re oftentimes trying to prove that a business is viable rather than just assume that the product is viable and then just go build it. That is a huge assumption, by the way, and I think modern-day entrepreneurs know better than to assume straight off the bat that this product or this vision is going to work.

    So, in my opinion, your first level of testing should always be a paper-based prototype thing to see if the business is viable. Then, you should create a vaporware mobile app.
    Put it in front of you: you can actually click it; you can swipe; you can move it around; there’s no coding involved. I can have one of our better designers whip that up in like three days, whereas it might take me three months to have an engineering team build it. Then, test its viability. If it makes sense, move on until you’ve validated the business model and you’ve proven demand for the product. Now, you can have more confidence investing in the process.

  • DevOps cuts out the unnecessary middle-man

    DevOps cuts out the unnecessary middle-man

    How many IT people does it take to produce an application? Actually, let me rephrase that. How many IT people does it take to produce an application effectively and efficiently? For many organizations, the answer to the second question is less than the answer to the first because the traditional IT hierarchy includes layers of middle-manager decision makers that weigh everything down.

    One of the things that DevOps helps companies accomplish is to eliminate unnecessary bureaucratic obstacles so the people trying to get things done can just do what they know how to do. The traditional IT model reminds me a little of putting a police officer in the middle of an intersection to direct traffic when there are already stoplights or stop signs present. Every licensed driver already knows the proper protocols and procedures for driving collaboratively and sharing the road so that everyone gets safely through the intersection. The police officer just disrupts an otherwise smooth process, and wastes resources that could be put to better use elsewhere—like maybe pulling over that car going 75mph in a 45mph zone, but I digress.

    For some reason, it is a fairly normal, natural evolution for a business to add layers of middle-management decision makers as it grows. Sometimes those roles are a function of wanting to promote people when there is really nowhere to promote them to, and the role makes sense at the time. In fact, the initial creation of many such roles is actually intended to help the business operate more efficiently. An upper manager starts to get too much on his plate, and delegates power to intermediaries so that decisions can be made at a lower level, and the upper manager’s lack of availability doesn’t become a roadblock to progress.

    Over time, though, middle managers become a cancer on the organization. The need for the role typically fades away, and you’re left with people in unnecessary roles going out of their way to justify their own existence by asserting themselves and making sure everything goes through them. When big companies like Microsoft or General Motors decide to pare down the company, middle managers are generally first on the chopping block because nobody can even explain why they’re there in the first place.

    That’s where DevOps comes in. By its very nature, DevOps enables teams to function more autonomously and take responsibility for their own work. It accomplishes the same goal that originally justified creating middle-manager roles—it moves decision-making power to a lower level so that scarce upper managers don’t become obstacles for productivity. DevOps does it by bringing decision-making power down to the trenches where those involved understand what they’re doing and why they’re doing it, though, rather than adding an extra layer of red tape that ultimately ends up adding, rather than removing obstacles.

    Like drivers approaching an intersection, the people doing the actual work have a firm grasp on what’s supposed to happen. They understand how things work, and what needs to happen to get things done. The traditional middle-manager hierarchy is like the unnecessary police officer “directing” traffic, while a DevOps approach treats the drivers like sentient adults, and assumes they all know what a stop sign or a green light mean without being told.

    In the end, it may add to our employment challenges. The US is already experiencing a shift away from manufacturing jobs as a result of technology and automation making workers obsolete. If we also decide collectively that middle managers are obsolete, we end up with more people without a job. That is unfortunate, and it is a problem that will need to be solved, but businesses can’t solve it by creating unnecessary roles just to give people something to do.

    For younger organizations that have embraced DevOps concepts since inception, DevOps should help prevent the cancer of middle managers. If teams and individuals are given responsibility for themselves, there is no need to create an additional babysitter role to police them. For established organizations that already have middle managers, eliminating those roles creates a natural vacuum where DevOps concepts can flourish if allowed.

    Regardless of how organizations get there, the result of embracing DevOps is that the middle-man is removed from the equation so development occurs more effectively, and more efficiently with fewer people.

  • Q & A: Puppet Labs CIO Nigel Kersten

    Q & A: Puppet Labs CIO Nigel Kersten

    We recently spoke with Nigel Kersten, CIO at Puppet Labs, regarding high-performance IT and trends he sees in continuous integration in his role at Puppet Labs. Nigel joined Puppet Labs from Google HQ, where he was “Puppet Master” and responsible for the design and implementation of one of the world’s largest Puppet deployments. Kersten is also an experienced Linux and Mac deployment sysadmin.

    In case you aren’t familiar, Puppet Labs provides IT automation system configuration and management software, which is used by start-ups to very large enterprises. Here is our discussion with Nigel, where we talk the recent Puppet Labs DevOps report and high-performance organizations.

    staging-devopsy.kinsta.cloud: In the 2014 State of DevOps Report, there seems to be a relationship between high-performance IT and high-performance companies. Can you see the causation and what correlation is there? Is it high-performance companies driving high-performance IT, or is it high-performance IT driving the high-performance company?
    Nigel: At the moment, it’s a correlation in terms of the amount of data we have. What the data actually show us from the survey is that adopting DevOps practices leads to a higher performing IT organization. This means you’re releasing code more frequently, your recovery time for failures is much, much faster, and all those sorts of things that we know go into what makes a great IT organization.
    We’ve actually been able to correlate that those organizations with higher performing IT teams are outperforming their competitors in the market, and they’re exceeding their own corporate goals. Now, at the moment, we’re just being a little bit tentative in saying that that’s a correlation rather than causative.
    To be clear, statistically we have a really high degree of confidence that it’s actually a causal relationship, not just a correlation. But we’re not quite ready to make that claim yet.

    staging-devopsy.kinsta.cloud: We wonder how you’re seeing continuous integration change the relationships among security teams, application teams, operations, and QA. What are you seeing, broadly, when it comes to how these relationships are changing from the way they were several years ago?
    Nigel: I think that a couple of things are changing those relationships. One has been the slow death, I would say, in some ways, of waterfall approaches in favor of agile. That’s not to instantly label agile as good and waterfall as bad, because I actually think there are some development environments and problem spaces where a really strict waterfall-based approach is the right way to do things.
    But we’re definitely seeing a mass of organizations that are moving to a more agile setup. I think what this has meant is that the demands on those continuous integration systems and QA teams have changed quite a lot. Developers expect much more self-service, and they expect to be able to operate with a much tighter feedback loop. They don’t want to have to commit code and wait till the next day – until a 12-hour set of tests has finished running – before they find out whether their code is any good.
    Prior to this, I think what we saw for a couple of years was developers starting a sort of shadow QA on their own using Vagrant, using local virtualization. But definitely in the last year or two, we’re seeing a trend where QA teams are being forced to provide faster feedback and more self-service to the development organization.

    staging-devopsy.kinsta.cloud: That must give heartburn to those who are used to having strict, steady, sequential processes in place.
    Nigel: It’s interesting, isn’t it? Some of this is a little speculative, because a lot of security folks tend to be very forward-thinking people, and I don’t think that’s necessarily true of everyone in the security industry, just like every industry. But I think it’s forcing people to look at their current security practices and work out what is just security theater and what actually improves the security of the product, or whatever it is that they’re shipping.
    So how can we automate those things that are actually improving security so that security doesn’t become the bottleneck in these fast feedback loops. agree that it can definitely give security people heartburn, especially people who are really used to rigorous code change approvals and the sort of segregation of duties that you put it in one of your pieces.
    What I see is a common compromise here, and I actually think this is a really effective way to do it. For certain environments, you really want to maintain what goes into production and have a strict change approval process. But it should be a fast and sort of appropriately light process.
    We’re seeing people giving developers the ability to run tests, change the testing environment, and have a significant degree of autonomy over the whole stack in pre-production continuous integration. People are focused on giving developers the ability to raise classes of machines, modify the tests that are running on them, changing all of those things so that they get a really fast feedback loop. But then when it comes to the actual merge to production, that’s something that your existing type change control process can take care of.
    This means that we see a lot of people segregating their infrastructure into dev, test, and prod, as far as continuous integration goes.
    And the developers do a lot of dev work locally on VMs. When it comes to test, that’s often on a shared infrastructure that, using something like Puppet, can be configured to be exactly the same as the production system. And that’s good enough for the developers. They’re getting their tight feedback loop. They’re testing code in pretty much the same way as it’s going to be tested in the official continuous integration pipeline, and yet you can still manage tight change control process around the merge from test to production.

    staging-devopsy.kinsta.cloud: And if it works as planned like that, you would think you’d get improved code outcomes because the developers are working much closer to any defects.
    Nigel: Absolutely. And I think that’s the whole promise that this sort of system is meant to provide. I think it does fall down if you don’t have a very tightly disciplined development organization, and it’s just open season on the production infrastructure. In that case, it actually becomes counterproductive, because the feedback loops become longer and longer. Then, if the test fails, you’re never quite sure if it’s due to someone else messing with the system that’s telling you whether it passed or not.

    staging-devopsy.kinsta.cloud: It sounds like they are self-correcting problems over time, because security often uncovers “hidden” or latent problems that you don’t see until something bad happens. But if it were a full onslaught on production, then I would imagine you would have other issues with availability and performance and related quality issues would mount up very quickly.
    Nigel: Absolutely. I think this is one of the reasons why people drive a lot of this work to the cloud, and I’m speaking from experience here, because the sys ops team I manage actually manages the infrastructure for the continuous integration system here at Puppet Labs. We have a pretty complicated testing matrix, and clearly security is a really big concern for us, because Puppet in many ways has the key to the kingdom when it’s deployed on someone’s infrastructure.
    So it’s really hard for a small company to provide enough internal hardware infrastructure to allow for that sort of elastic peak consumption. I think this is one of the big drivers toward people, at the very least, using the cloud to smooth out their internal hardware requirements so that, at times of peak capacity, they can essentially burst out to the cloud.

    staging-devopsy.kinsta.cloud: Organizations that move this route, I assume, need to get to know their own processes very well before they move from what they were doing before – more manually driven processes – to continuous integration?
    Nigel: I think what we tend to see is that it ends up being on a project-by-project basis. If you’re a company that sells on-premise software or software as a service, you’ve got a reasonable amount of investment there, and it’s rare that companies can take the time to just pause the delivery of software. Usually, they need to keep making money; they need to keep shipping releases.
    So, what we tend to see is that people start experimenting with green field environments. This, according to my gut, is often enabled by people moving to more service-oriented architectures, where rather than shipping large, monolithic applications, they’re shipping components that have well-defined boundaries with each other.
    As part of that refactoring goes on, as they build a new component, they might do the testing and continuous integration for that component in a fundamentally different manner, and sort of step through their code base that way.
    Eventually, you have to deal with the brown field deployment. But, ideally, by that point you’ve gotten a whole bunch of huge wins from the green field continuous integration systems that make that sort of return on investment obvious.

    staging-devopsy.kinsta.cloud: I would imagine that knowledge gained from those groups is then shared with small workgroups that experiment and then there’s a lot of knowledge sharing and expansion organically from within.
    Nigel: Yes. And with people being people, this is often where turf wars strike up. The have been a number of times when we’ve sort of walked into organizations and talked to the QA team and the developers and the operations people – and dev and ops have started working together and they can ship something much more quickly by doing things in that way – and yet somehow the QA team feels slighted that it’s somewhat bypassed by this whole process. That can be a rather complicated conversation to be in the middle of.

    staging-devopsy.kinsta.cloud: Do you see companies make common missteps when they move toward continuous integration deployment?
    Nigel: One of the more conceptual mistakes I see people make when they’re considering it is that they confuse continuous deployment with making lots and lots of public releases, and they’re not necessarily the same thing. I often try to tell people that continuous deployment is really about enabling the ability to choose to ship something. It should be a shippable unit that shouldn’t necessarily be shipped, if that distinction makes sense.
    A lot of companies will start stressing out. They’ll wonder about their existing processes and educating their channel resellers. They’ll believe they simply can’t afford the cost of a release every two weeks, or every week. But that’s not necessarily the whole point of continuous delivery. By having artifacts that are shippable, you enable testing internally, you enable easier experimentation, and you still can choose to ship your product only once every six months if you want. But the fact that you might have 20 releases in between those public releases is important. They could all be shipped and could be a bug fix for a certain customer, or enable you to do certain types of development in the future. These are really huge wins, even if end users never actually see those specific releases.

  • Rackspace DevOps automation service broadly available

    Rackspace DevOps automation service broadly available

    Rackspace is the #1 player in managed cloud. While they recently announced they would be withdrawing from the pure IaaS market, they have now double-downed their bet on managed cloud with the general availability of the Rackspace DevOps automation service around managed cloud.

    Rackspace had previously announced their DevOps automation service a few months back, but that was a limited release. Now DevOps automation is one of the two choices Rackspace customers will have in setting up a managed cloud service.

    With this general release Rackspace is offering DevOps automation on both Linux based clouds utilizing Chef among other tools and on Windows using native Windows tools. In addition Rackspace offers enhanced environment stack support. According to the release:

    Rackspace now supports the following environment stacks, which can be deployed in hours on the Rackspace Hybrid Cloud: Chef, Rails, Node.js, PHP, Tomcat and Python. Customers can use recommended stack architectures and will receive updates, security patches and the latest versions of environment stacks, which are constantly maintained and tested by Rackspace DevOps engineers.

    These stacks are not containers per say a la’ Docker, but are self-contained packages that Rackspace can automate installation of and get customers up and running even faster.

    I spoke to Prashanth Chandrasekar, GM of Rackspace’s DevOps Business Segment about this announcement today. Prashanth and his team have been offering DevOps automation to customers since last year.  Today’s announcement represents DevOps going mainstream as one of the two preferred methods of deployment for Rackspace customers. I also spoke to Chris Jackson, the CTO of the DevOps team at Rackspace. Chris wrote a nice blog on the SysOp approach versus the DevOps approach.  It explains much of the thought process behind the Rackspace offering.

    There is also a good, if long Youtube from Rackspace on their DevOps automation service.

    Managed cloud DevOps automation was not the only thing Rackspace announced today. Just automating your cloud infrastructure by using a DevOps managed cloud service does not mean you are a DevOps organization.  Frankly that is just tools being used by a 3rd party. How do you instill a DevOps culture, mindset in your organization? How do you get your enterprise to actually become a DevOps organization?

    Rackspace is offering a full blown professional services offering called “DevOps Advisory Services”. They will come in and evaluate and analyze your needs and present organization. Based upon this they will make recommendations to help you. These include:

    • DevOps Workshops: Rackspace is working with customers to perform a range of services including application assessment, transformation planning and knowledge transfer, plus enablement around key areas like Configuration Management Tools, Provisioning and Deployments, Continuous Integration, Continuous Delivery (CI/CD) and Monitoring.
    • DevOps Strategy Planning: Rackspace DevOps specialists work with customers to understand their requirements and help create a custom project plan for their business needs and timeline.
    • DevOps Professional Services: Rackspace provides access to DevOps professional services from writing infrastructure code to designing structure and building migration plans for applications. This helps customers automate, deploy and test their infrastructure for their main applications, or a set of proof of concepts that the customer helps build.

    With today’s news Rackspace continues to demonstrate how important DevOps is to the go forward planning. Some may question how a managed services model would embrace DevOps, but Rackspace is showing the way. Branching out to DevOps pro services will allow them to compete head to head with IBM/SoftLayer offering cloud hosted and managed services wrapped with pro services.

    What I always find most compelling about Rackspace’s DevOps services though is that they eat their own dog food. Their DevOps services were born from their own internal use before they brought them to customers. It takes a DevOps organization to offer DevOps services.

  • The DevOps Diet (Getting Lean)

    The DevOps Diet (Getting Lean)

    IT has gotten fat – fat silos, fat processes, fat procedures.  Adopting DevOps is a great opportunity to look at some of our unhealthy habits and identify where we can eliminate unproductive waste from our process diets.  DevOps will ultimately teach IT how to exercise more with less effort, trim the fat and get lean.

    According to lean principles, the seven main areas of waste are:

    • Defects –  variations from requirements that result in interruptions and re-work
    • Overproduction – delivering something more or before it is required
    • Inventory  –  carrying excess raw materials, work in progress (WIP) or finished goods
    • Over-processing – doing more work than is required
    • Motion – moving people or equipment more than is required
    • Transportation – moving products from one location to another
    • Waiting – doing nothing or moving slowly while waiting on a previous step

    What are the greatest sources of waste in your IT organization?  Are defects being regularly passed downstream? Are ITSM processes overproducing through complexity and bureaucracy?  Are Agile teams  dealing with excessive work in progress and less finished product?  Are unresolved bottlenecks resulting in frequent delays while waiting for something to get done or someone to be available?  Could automation reduce “motion sickness” and make transportation nonstop? These questions will hopefully facilitate dialog between Dev and Ops about analyzing and streamlining existing workflows.

    While waste is fattening, it is not necessarily ineffective.  Projects and other tasks are still getting done and therefore some areas of waste may be difficult to recognize.  To become leaner, start simple by identifying and eliminating one unhealthy habit from your IT diet.   Engage stakeholders from Dev, Ops, customers and suppliers (if appropriate).  Discuss ways to improve the work with healthier habits.  Agree on what to do next or what not to do anymore.

    Every diet should be supported by recipes for success.  The upcoming DevOps Cookbook by Gene Kim, et al. will provide more tangible guidance and recipes for healthy IT habits.  More on that in future blogs.

    Most of all,  identifying and eliminating waste through Lean DevOps is a great opportunity for collaboration.  After all,  it’s always better to diet with a friend.

  • Microsoft strives to embrace open source culture

    Microsoft strives to embrace open source culture

    In many ways Microsoft is going through the corporate equivalent of a mid-life crisis—it has reached a stage where it longs to drop a few pounds, and turn back the clock a few years. After years of tremendous success, it’s now starting to show its age, and it is struggling to adapt to the new tech landscape and compete with younger, leaner, more innovative competitors.

    The shift began under ex-CEO Steve Ballmer. Ballmer’s approach, though, was misguided. Ballmer’s Microsoft suffered from hubris, and ignored the seismic shift in technology until it was too late. When Ballmer finally recognized the existential threat, and tried to turn the ship around, it was done from within the established bloated corporate culture. There was an effort to rearrange the chairs on the deck with a management reorganization, but it simply wasn’t enough.

    Then Satya Nadella took over. Nadella has a more aggressive vision. It’s still not an easy job, nor a goal that can be accomplished overnight with a behemoth the size of Microsoft, but Nadella has dramatically changed the culture, and the vision for where Microsoft is heading.

    Part of the new and improved Microsoft includes embracing communities that Microsoft has long ignored or alienated, and shifting the way things get done at Microsoft so the company can be more agile. It is evident in how Microsoft partnered with Chef and Puppet Labs to integrate popular DevOps tools into the Azure cloud platform, and it can be seen in how Microsoft is adopting open source.

    “Recent articles have shown that Microsoft is “open sourcing” code to their internal teams. The fact that that is advertised as innovation is telling,” declared Michael LaVista, CEO and Founder of Caxy. “That’s what smart organizations have always done. In fact, Microsoft has somehow found itself the underdog in this sense and probably needs to do things like this to attract talent.”

    Leveraging open source concepts internally isn’t quite the same thing, though, as actually making Microsoft code open source. Open source projects like Apache, Project Libre, or Docker share source code with the general public, and anyone can review the code, and submit changes to improve or fix the code. Open source communities collaborate to continuously enhance the code. Microsoft appears to be doing something similar, but without exposing its source code to the general public.

    LaVista suggests that Microsoft should take that next step, though, and work collaboratively with the open source community outside of Microsoft’s walls. If it does so, Microsoft will some issues to address. Will it accept code fixes from a 14-year old on the other side of the world? Will exposing source code for the Windows operating system, Microsoft Office, or other popular Microsoft applications open up a Pandora’s Box of security concerns?

    “They will have to embrace what happens to code when it is open sourced. Once the code is out there, everyone can see the flaws,” explains LaVista. “The benefit of the model is that you have so many more eyes looking at it from more perspectives. If Microsoft is used to running their own software and making their own decisions about what to work on and fix, this could pose problem.”

    Perhaps. LaVista makes good points about concerns Microsoft would face, but in and of themselves they’re not a reason not to adapt and evolve. Yes, exposing Microsoft source code to the general public would probably result in a spike in security concerns, but once the dust settles the collaborative approach of an open source community addressing those issues could result in quicker patch releases, and make the software more secure.

    LaVista recommends that Microsoft use companies like Red Hat, IBM, and HP as examples of how a larger corporation can be successful in an open source culture. Only time will tell. We’ll have to wait and see to what extent Microsoft makes the shift to open source, and we’ll have to wait and see how Nadella’s Microsoft adapts, and whether Microsoft can successfully reinvent itself to remain competitive in this new tech landscape.

  • Q&A with Kevin Behr on DevOps and Enterprise Excellence

    Q&A with Kevin Behr on DevOps and Enterprise Excellence

    When we want to get an inside peek at the trends within major corporations when it comes to continuous integration, continuous delivery, and DevOps, we turn to Kevin Behr, the chief science officer for the CXO and Board Advisory Practice at Praxisflow, where he consults, mentors, and coaches IT organizations and executives on how to increase their business effectiveness. Behr also is the of seven IT management books, including the bestselling “The Visible Ops Handbook”, which he coauthored with Gene Kim and George Spafford. His most recent book, The Phoenix Project, also coauthored with Kim and Spafford, has helped to spark discussion about DevOps globally.

    Today, we sparked a discussion with Behr about how smart enterprises are using DevOps to further differentiate themselves and enterprise excellence.

    staging-devopsy.kinsta.cloud: There seems to be a lot of misconception out there about what DevOps is – and I’m increasingly hearing it referred to as a software methodology. Are you seeing this?
    Behr: Definitely. DevOps is not a software development methodology. I think that is a big misconception out there. It was a big misconception in the in WSJ post, DevOps is great for startups but for enterprises it won’t work yet. The author had that completely wrong: DevOps doesn’t tell us how to develop software. DevOps literally is collaboration between IT operations and development. So DevOps could be about developing software. It could be about fixing bugs. It could be about fixing integration issues. It could be a whole bunch of different things they’re collaborating on. But it doesn’t necessarily denote how you would develop software.

    staging-devopsy.kinsta.cloud: But there’s no prescriptive connotation to the word at all, is there?
    Behr: Exactly. My whole point is that the only prescriptive connotation is “Can’t we all get along?” Can’t we work together to do the work we do? I think sometimes that means we’re not developing software that means sometimes we’re doing a lot of different things. In some cases, software developers are helping sys admins write better scripts, and do better sys admining. And vice-versa: sys admins are helping software guys understand requirements better, and understand the end delivery point, the people that are going to use software or service this, better. That’s not telling us how we must develop.

    DevOps is about asking: what can we do to improve, what can we change, and what does this change mean? We’re talking to CIOs like crazy right now about this. We’re trying to figure out how can they leverage DevOps without causing all kinds of organizational trouble. On one hand, we’ve got people in the enterprise that are happy with an ill-defined DevOps. To be honest with you some are saying it’s all kinds of things because that creates a subterfuge where they don’t have to do things to make it work. “If DevOps is confusing, if it’s impossible, or if it’s stupid, maybe we won’t have to actually do it,” they think.

    It’s kind of pathetic to watch. We are watching them and wondering: “Wow. Do you really think that doing these in Ice Age speed and style deploys and old school change management that takes 30 people in a room, once a week, to decide what changes are going to get made is okay today?

    staging-devopsy.kinsta.cloud: That’s just insane to hear and think that it is going on. Outside of IT, how are business people responding to DevOps and continuous delivery.
    Behr: I’m telling you, the coolest part about it for me is the business people have reacted to the DevOps message. They’ve done so because we’re talking about changing the business payouts. They’re drastically different now. Organizations that can do these deploys, not just because they’re putting out more software, but because the organization as a whole is focusing on delivering more software to the end user. Even the commercial software companies are way behind what a lot of DevOps companies are doing now when it comes to driving innovation and better outcomes.

    When you look at these big software companies that are still releasing software every eight months, they’re really slow and terrible. We’re all suffering from the quality and the slow feedback loops, and the disconnectedness of regular commercial software. These smaller companies, a lot of them are start-ups albeit, but not all of them, and a lot of these banks that have embraced DevOps are drastically changing the old-school cycle time game. They’re getting faster. They’re reducing cycle time, and they’re reducing defects. That adds up when it comes to better applications and producing a lot less waste.

    staging-devopsy.kinsta.cloud: What does this mean when it comes to impact, can the deficits be measured?
    Behr: Yes, because it ends up taking you forever to get anything out. You’re now suffering from that nasty cycle of unplanned work. It’s similar to all of the work that they call failure demand, where the failures in the quality of what you build are creating demand for work. You’ve already got competition for that work, because your company wants to release new value features, but you can’t because you’re cleaning up the old messes. It’s ugly.

    Now, and the other thing that’s interesting too, here, is that when you move to SaaS, you’re moving to more release cycles per year. The average SaaS provider is going through four major platform upgrades a year. They’re doing four releases. Think about that change from a traditional model, where you’re adding three more releases a year, essentially.

    And every release requires maintenance releases, so now you’re going to have patches. A lot of these companies have been putting SaaS into place, but there’s not a whole lot of SasS that’s profitable from a business perspective right now. There are a lot of companies doing it, but there are not a lot of companies that are in the black. What’s interesting is that they’re spending a lot of labor to maintain the software. It’s expensive, and you have to build it before the customers come. That’s why we see so many people come and go out of that space.

    staging-devopsy.kinsta.cloud: What does that mean for business then, when it comes to success and IT’s relationship to business success?
    Behr: I think what’s interesting is that you’re hearing business people’s expectations change. The business always used to say, “Well, I need that thing tomorrow.” And IT would say, “Well, you’re going to get it when you get it.” Now the business is coming back, saying, “No, I need that tomorrow, because my competitor already has it.” Those rushed cycles used to be crying wolf. But today there is legitimacy to the pressure. They’re actually crying wolf because the wolf is there.

    The dinosaurs are going to be in big trouble, because they are competing with organizations that can do 10, 15, 20 or 40 deploys a day.

    Some are deploying with feature switches, as dark code, so all of those deploys that they’re doing may not even be active features that people are using. But they’ll turn on a feature, see if it works, and if it doesn’t they will turn it back off. No customer impact. They test it in production. They can also then roll out code that nobody even knows is there, and also be testing it on actual, live infrastructure. What they’re doing is improving so much faster than everybody else this way.

    These organizations are going to separate themselves from the pack because of how efficient they are becoming. This is how DevOps is one of those approaches that a lot of organizations are using to make it harder for other people to catch up with them. If they’re a super-predator, in other words if they have a breakthrough idea that nobody else does in some sort of technology, why not spend the dollars and couple that technology with a service approach, or a design from an organizational perspective, so that you can continue to out-improve everybody? DevOps does this.

    You’d be insane not to. So now you have a dual threat. You’re not just offering a better product, but you’re a better product that’s actually iterating faster and separating at a rate that’s going to be hard for other people to deal with. This is how the slow moving enterprise is losing right now. It’s content to think that it’s not losing that bad. The thing is, the game is changing so fast in organizations that are able to do this that they’re able to separate so much faster, and outcompete those that can’t.

  • 8 tips on how to best manage distributed teams

    8 tips on how to best manage distributed teams

    More and more organizations are using distributed testing teams yet many are hard pressed to know how to manage teams that may be separated by thousands of miles away or on another continent. Too often, companies think about offshore product development only as a means to cutting costs and accelerating time to market. Such an approach can produce poor results with your projects if you don’t take into account certain issues like cultural barriers, metrics and workflows.

    The smart strategy is to focus on the needs and skills of the team and to implement procedures that will help manage them better. Your standard bottom-line goals — cutting costs and accelerating time to market — will take care of themselves if you can follow these 8 tips for managing distributed teams.

    1. Know Your Team

    It’s important to address people correctly. People respond well when somebody knows how to spell and pronounce their name. To infuse a sense of team spirit, attach a face to the person who is communicating via email, phone or IM. Have a kick-off meeting for a project team, where each person introduces themselves with some basic personal information such as a hobby or interest, to help establish a ‘human-side’ to the offshore team. While meeting in person might not be a possibility, the liberal use of video conferencing can really improve the relationship with remote teams.

    2. Eliminate Language Barriers

    Working with remote teams on different continents/countries requires strong communication. It’s critical for the key contacts between teams to familiarize themselves with the pronunciation and grammatical styles of each other, while also ensuring that your team members obtains International English certifications, such as TOEFL. In addition, when possible, I recommend that the team leaders from the vendor side and client side visit each other’s offices to understand the culture of the companies and to instill a sense of partnership.

    3. Clearly Define Roles and Responsibilities

    To ensure that all members of the project team understand their expected tasks, it is vital to establish clear roles and corresponding responsibilities early on. Create a collaborative, pyramid-style team structure for the offshore team, so that the key people responsible for daily communication with the rest of the project team are identified, along with an obvious reporting hierarchy.

    4. Create Strong Communications

    The distance between remote teams working on the same project can be eliminated by use of communication tools such as IMs, VOIP-based phone calls, online meeting tools and other electronic mediums. However, communication is only as good as the information provided. All relevant documentation and domain knowledge must be continuously shared with all team members.

    5. Track Detailed Onshore/Offshore Planning and Tasks

    To ensure that your teams understand the assigned tasks and is progressing as expected, it is very important to have a milestone-driven WBS (Work Breakdown Structure) that identifies the effort associated with each task, and identifies the timeline of when the tasks need to be completed. This level of detail ensures that everyone is clear on their assignments, and that any delays (and corresponding affects) can be identified as early as possible, so that a mitigation plan can be implemented if need be. Team progress on assignments and activities should be followed up with daily, weekly and monthly reporting, leveraging defined templates where possible.

    6. Establish Process Workflows

    A clearly defined testing activity workflow should be established and reviewed with the offshore team, to ensure that all team activities are synchronized. Documenting a process workflow allows the team to streamline, validate and verify expected activities that could impact the project delivery timelines.

    7. Don’t Forget Metrics

    Offshore team deliverables and activities should be gauged with quantifiable metrics such as timeline delays, actual-to-estimated effort, QA-to-production defect ratios, and so on. Metrics should be established under at least three categories: timelines, productivity, and quality. These metrics should be established at the beginning of the project, and tracked either by release or on a monthly basis.

    8. Establish Roadmap for Refinements

    To ensure that the offshore team continues to increase its value to the project team, it is important to leverage the daily, weekly and monthly reports/metrics to identify areas of improvement/streamlining for testing processes and test strategies. These refinements should be prioritized and scheduled for implementation by the team, and tracked for progress along with other project deliverables.

    In summary

    Establishing and managing distributed teams is a complex challenge. Cultural differences, time zone variations and lack of face-to-face time contribute to the intricate nature of this model. In addition, some companies focus too much on cost-cutting objectives. To successfully manage distributed testing teams, a company needs to focus on people, processes and communications.

    About the Author

    dave-keil-QASymphony CEO_cropDavid Keil is CEO of QASymphony, a leading provider of test management platforms for agile development teams. He previously served as CEO at Digistrive and at Integrated Broadband Services. He holds an MBA from The Wharton School and a BS in Applied Mathematics and Economics from Brown University.