Segregation of duties will change because it must change. It has a tremendous impact on our motivation, time to market and IT Security. It influences many parts of an organization. Most organizations have started with DevOps, Continuous Delivery and Continuous Deployment and it is only natural to think about segregation of duties at one point and how we deal with it today. Today, it costs us a fortune that we’re not willing to pay in the near future. And why should we?
What Developers Think of Segregation of Duties.
As a parent we watch our children grow up. We watch everything our children do so we don’t miss anything. One of the greatest moments in life is when they take their first steps on their own. We feel pride and happiness. It’s our job and our responsibility to protect them and to help them to become great adults, 100%.
As a developer we feel pretty much the same about our applications. When we release a first version of an application into production we feel pride and happiness exactly like a parent with its child. It’s our job and our responsibility to keep that application up and running and also to maintain it and to ensure that our application delivers value at all time, 70%. What?
That number is made up but probably not far from truth. In many large organizations it’s impossible to be 100% responsible and to take full ownership of our own applications even if we are supposed to and really want to. We call it “Segregation of Duties”. IT Security makes sure that no single person can do everything from development to production alone. Segregation of duties increases protection from fraud and errors and is a key concept of internal controls. But it can also do a lot of harm in the organization. Segregation of duties can rob people’s motivation and it can make us frustrated and sometimes angry.
Are you familiar with the Pareto Principle? It says that 20% of the employees do 80% of the work and that the remaining 80% of the employees do only 20% of the work. Imagine you’re one of the 20% that gets work really done. Imagine you’re getting things out and delivering value to the customers, the business and the IT because you’re being trusted and given responsibility, 100%.
I’ve been part of Task Forces solving production problems many times. In one of those Task Forces we were 12 persons from different departments like development, security, operations, infrastructure and change management. The root cause was found immediately. A hotfix was ready right after and change management approved releasing it into production immediately. The time between when the hotfix was ready and when we actually released it into production were long lasting 2 hours and 5 minutes. We hadn’t have the right resource from the Operations team, the one who is allowed to do the job. 125 minutes were spent finding the right person from Operations with access rights to the production systems and teaching that person how to do the deployment. The deployment is a Jenkins job that takes 3 minutes. 125 minutes x 12 persons = 25 hours that we wasted because we separated the duties very strictly. Not all resources can be available at all time and I don’t believe it ever will be possible.
I want developers that have the knowledge about their applications to become even smarter and take the responsibility to run their application in all environments. How fast could everything be. Time to market could be reduced by many times. The motivation would shoot into the sky.
If we want the developer teams to take 100% responsibility over their own work we must put the responsibility exactly there and show them trust, 100%. Developers want to care but they simply can’t.
Ron Westrum’s work on safety culture shows that no process or control can compensate for an environment in which people do not care about customers and organizational outcomes. Instead of creating controls, the solution can be to create a culture in which developers take responsibility for the consequences of their actions and there actually is a simple prescription to enable this.
- You build it, you run it. Teams that build new products and services must take responsibility for the operation and support of those services.
- Turn central IT into a product development organization.
Employees are a company’s most valuable assets and should be trusted to do their jobs. High-trust organizational culture should be favored and not the contradictory command and control culture.
What IT Security Thinks of Segregation of Duties.
Recently I had a chat with one of the people working in IT Security. He loves segregation of duties. “Compliance and stability is the key to get customer’s trust” is his mantra. “Legal compliance demands will always be implemented with as little interference to daily operations as possible and access control is an easy and very efficient way to enforce segregation of duties. But it comes at a cost in flexibility. When enforcing segregation of duties through access control mechanisms, it’s important to follow guidelines for what roles to allow what type of access in order to – not only be compliant – but to protect systems and data and maintain operational stability, integrity and confidentiality”. He is really serious about his job, what is good. Security is challenging and hard work.
Imagine our systems would live in a locked box and really everything would be automated. Everything just works and we wouldn’t need discussions about IT Security, access control and segregation of duties. There might be a way – the DevOps way.
How DevOps Deals With Segregation of Duties.
The drive towards being agile versus implementing stronger access controls, DevOps versus Segregation of Duties – do we have two conflicting initiatives? It depends on the organization and how far it has come automating deliveries and how changes are released into production.
Imagine you commit an emergency hotfix into the common source repository 10 minutes after a very serious bug was reported by customer service. And imagine depending on how fast this hotfix must get out you can adjust the speed your change will get out, skipping maybe some non functional tests. 5 minutes later your hotfix is release ready. IT Business opens the organization’s Production Release Dashboard where it can find your change. And all they have to do from now on is to push a big green button on the screen. 2 minutes later final tests will decide whether your emergency hotfix stays in production or the automated roll back routine will be initiated.
Unfortunately, in many larger organizations this picture is still far from reality. I was once part of a 12 people Task Force where we had an emergency hotfix ready to be released after 10 minutes after the route cause was found. But we didn’t have the big green button. And neither did we have an automated roll back routine in case of failure. What we had was a Jenkins job that nobody knew where that particular job was. Our people from IT Operations maintain hundreds of Jenkins jobs and the only person who was available at that time and who had access to the Jenkins Dashboard in production didn’t know where that job was. Time went by and after more than to hours we got the hotfix out to our customers who were complaining about bad customer service. It took over 120 minutes for 12 people to make an important change. It took 24 man hours to start a Jenkins job, insane.
Conclusion
I see DevOps as a way to loosen up segregation of duties but to keep IT Security and to make organizations stronger and as powerful as they never have been before. Simplicity is the final key to real power. More on http://DevOps.Town
