staging-devopsy.kinsta.cloud

Tag: devops

  • Hacking Your Auditor

    Hacking Your Auditor

    No, not that kind of hacking, give me a little credit…

    Recently I was having a conversation with a consulting friend about DevOps, and he found himself in a bit of a tough situation. The organization he was working with was a pure cloud/DevOps shop, but one that was stepping into having the meet compliance obligations. Needless to say, things weren’t going well with the auditor.

    For those of you who haven’t had to deal with the miasma of audits, one of the strongest requirements, for nearly any regulation/contract/agreement, is strong change management. This is especially true after those Enron guys screwed everything up for the rest of us.

    Change management makes a ton of sense from both an operational and compliance standpoint. I, myself, have blown things up in production but not being smart about my change management. For compliance, it is absolutely essential. If you are dealing with anything with financial or privacy implications, you sure as heck want to get it into a stable, secure, compliant state and ensure that state is kept over time.

    There is a big misconception out there that DevOps-style organizations don’t have change management. That when you push multiple changes per developer per day into production, there is no accountability or control. Now, while you can certainly build things this way, that sure isn’t how I see successful teams doing it. Not even close.

    The problem isn’t that DevOps necessarily skips change management, it’s that it looks different. Much of change management becomes automated, and the fundamental nature of changes themselves is different. But think about how you describe DevOps to your traditional IT-focused friends. Now imagine trying to explain this to an auditor working off set standards that are still figuring out if cloud includes a co-hosted Mac Mini duct taped to a rack.

    Here are some tips on how to both adjust your process, and explain it to the auditors:

    • First, figure out which pieces of your application and operations are in-scope for the regulation. Odds are only financial and privacy related bits are, and you can then apply more-granular controls or logging to these if you need.
    • Make sure your continuous integration tool (if you have one) includes some security checks and logs all updates/changes. If you push right from your code repository to production, get a CI tool.
    • Keep records of all changes as they flow from dev to test to prod (or whatever you use). Again, this should be trivial to automate and dump the records to a secure log repository someplace.
    • Unless your cloud provider tracks all API calls (like CloudTrail does on AWS, although that only works for some services and regions) use an API or web UI proxy like Netflix’s Asgaard. This really is a good security practice anyway, and can actually speed up infrastructure and platform management processes anyway. Keep the logs from this to make the auditor happy.
    • If you SSH into instances, you need to log that activity and dump the logs to a central repository.
    • Make sure you keep revisions of both all code changes and infrastructure automation changes (e.g. your CloudFormation templates, Chef/Puppet scripts/etc.). This is often built-in, but make sure you configure it correctly.
    • Those suggestions now give you logs of all application changes, infrastructure changes, and direct access to instances. These are the things the auditor wants to know exist.
    • Filter them based on your in-scope compliance requirements. Such as payment systems components only.
    • Explain to your auditor that all changes go through an automated approval system that checks the code, tests it (detail the tests), tracks the change, and then moves it into production. That all server logins and infrastructure changes are logged, as is the complete history of the structure of your entire environment. And that you can provide all the documentation, filtered for the pieces within compliance scope.

    This might look like a lot, but odds are very high you are already doing most of this. You keep code revisions, run changes through test suites before they go into production, and track all infrastructure changes. You might not have your own Ops portal/proxy, but you really should for a lot of reasons that have nothing to do with compliance.

    The trick is explaining this to your auditor in their terms. They expect manual checks and records. You need to show them that you still have these checks and balances, but they are automated. You still might need to add some pieces, such as an alert if certain parts of your code are changed (e.g. the file with payment processing or account reconciliation), but these will hopefully be minimal.

    A good DevOps environment actually has a much better compliance infrastructure in place than traditional infrastructure. The automation and template-driven models give us much better control and insight as compared to manually fixing everything, even when we push more day to day changes. The trick is to document this, add a couple pieces if you need it, and then explain it all to someone with… less than refined technical skills.

     

     

  • Don’t let automation be the tail wagging the dog

    Don’t let automation be the tail wagging the dog

    There is quite a bit of talk right now about DevOps and how important automation is.  This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot.  In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market.  This failed with famous stories of robots painting each other and others welding doors shut.  Ultimately this project failed and the robots were scraped.  It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

    Automation dominates the conversation when it comes to DevOps.  If you do not learn how to build cookbooks you are not doing DevOps.  I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there.    Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand.  The best way to understand it is in two key ways, change and performance.

    Let Science set you free

    This was the secret that Toyota used to dominate the automobile industry.  They were able to transform all of their employees into little scientist.  Constantly tinkering to squeeze our inefficiencies in the system.  Every worker was empowered to think how things could be done better.

    At the heart of the agile and DevOps processes is the practical application of the scientific method.  Think back to your 8th grade science fairs.  No not the volcanos but the ones that won best of show.  These had a clear hypothesis that stated what they were going to try and prove.  They were able to understand what they were going to change and measure the difference between the before and after.  They could clearly state what changed and if it was a successful experiment or a failure.

    This thinking is key for companies trying to transform to a DevOps IT organization.  If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

    Tracking your experiments

    Any good scientist knows exactly what he is working on.  He knows every experiment that is running anticipating the outcome.  He knows all the details about each change that has been made to see how they impact the outcome.  This is the same way that IT should be tracking every change happening.

    Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application.  Developers and Operations will be more informed about the cause and effect of their changes.  There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

    Measuring the outcome

    There is no need to track the experiments if there are not being measured.  A scientist is very deliberate in how they measure the outcomes.  Good enough just does not get this done.  This is beyond just monitoring.  Monitoring in a lot of cases is after the fact, in production.  This is about measuring the difference for every experiment regardless of where it is being tests.  This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update.  Following it through all of testing and to production.  Understanding how each experiment impacts the system as a whole.

    Efficiency through automation  

    Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit.  Because every experiment it known the lower priority ones can be identified and automated.  Risk is better assessed.  Processes can start to speed up because there is visibility and accountability.  Applications are no longer classified as high risk or low risk only change is classified that way.  Determining what can be automated just becomes another experiment that is tried and measured for success and and failure.  Automation becomes just another outcome of DevOps to the business and not the route to achieve it.

  • Why Baking Security Into the DevOps Framework is Vital to Its Success

    Why Baking Security Into the DevOps Framework is Vital to Its Success

    I attended my first official DevOps conference in Israel in September 2013.  But I was initially introduced to DevOps by my customers – firewall and network managers who, through their efforts to automate  security policy management, found themselves dealing with a new and like-minded set of IT stakeholders  –  DevOps teams.

    They said DevOps was also all about automation – automation that allowed IT to increase their business contributions by enabling new applications and other IT services to be rolled out faster with greater control and efficiency.  

    Conceptually, it was clear to me that the DevOps value proposition is just as relevant to Security as it is to Operations and Development groups, especially since Security teams apply automation with the same fervor as DevOps.  

    In fact, using automation to accelerate IT has caught hold in the network security world to fuel a new movement and class of solutions, called Security Policy Orchestration.  As the CTO of Tufin, a Security Policy Orchestration company, I could not help but recognize the synergies.  I was (and still am) intrigued by DevOps, so when I heard about the conference, I decided to go.

    And I am so glad I did.

    The alignment between DevOps and Security Policy Orchestration is obvious. They both champion the use of automation to accelerate the pace of business. We Security Policy Orchestration folks are just coming at it from a different area within the organization.  Kindred spirits, right?

    In theory, absolutely, but in reality…not so much.

    The conference as a whole was great.  It completely validated my theories regarding the synergies between Security Policy Orchestration and DevOp.  But what surprised me was that security was a topic that was not being discussed at all at the conference.   Maybe a hat-tip to application security, but network security was not even on the DevOps radar, at least not at this particular conference.

    It needs to be.  For DevOps, to really be able to move the needle, we have to work together.  If history has taught us anything, it’s that when new applications and services are not created or rolled out with security in mind, at some point we will have to go back and retrofit it back in.

    Envision the typical experience:   the development team builds a new application, brings IT Operations in to implement it.  Once that is done, IT Ops prides itself on all the corkscrewing it took to make the application work on the actual infrastructure.  Except for once the app is in production, it doesn’t work.  The troubleshooting begins.  Invariably, issues of network connectivity come up.  Solving them requires app, dev and security managers to break down the cultural and organizational barriers that exist within their organizations to work together to ensure the application is architected correctly into the network.  This starts and the process of change requests between AppDev and IT to get the app up and running. 

    If that process is not automated, it can take months longer than it needs to resolve everything.   Not to mention any company subject to PCI has to consider terms like secure zoning, a concept that is not second nature to DevOps folks.   If PCI is not an issue, there are plenty of other internal and regulatory mandates and security –related considerations that, if for no other reason than common sense, can exponentially reduce the chances of a company ending up in the headlines due to an APT.    

    These are just a couple of many scenarios that can be avoided if development, IT and security work together from the get go.

    The net/net is that Dev and ops have the same goal but they are siloed, which is slowing down the pace of business – inhibiting it.  Collaboration is not some warm and fuzzy team-building thing – it is a fundamental business requirement.

    And that’s why I am so excited to have the opportunity to do this blog.  I think I can add a unique perspective on how DevOps and Security can work together to reach our common goals.

    My next post will explain will dig into what Security Policy Orchestration actually is, so we can begin to tease out the similarities and determine how to build on that to true alignment and partnership. 

    Stay tuned as the drama unfolds! This should be fun….