staging-devopsy.kinsta.cloud

Category: Blogs

  • The Automation Continuum

    The Automation Continuum

    The speed of innovation these days is often influenced by the speed of adoption of new and exciting tools, and specifically open source tools.  Open source projects have been growing at an exponential speed over the course of the past few years; however the process for exploring, testing, configuring, and ultimately integrating these in production often times is still quite complex and time consuming.  In many cases, this process is still done manually.  Like with all DevOps processes by automating this, we are then able to adopt new tooling much more quickly, and with less human error in the process.

    The Open Source Revolution

     

    The friction in this process is due, in large part, to the fact that each of these stages from exploration through POC and testing, integration and then management, many times requires the use of very different tools for each stage, on top of a completely different setup and environment.  This means that every time you want to move from exploration to testing, for example, you have to basically rebuild the entire system.  This can become quite frustrating and, as mentioned above, time consuming.

    What if we could streamline this process  and explore tens and even hundreds of projects quickly, thereby enabling us to seamlessly increase our innovation?

    How this could work – the flow:

    1. Exploration: Explore the product without having to install any software at all.  This could be through a click of a button on Github, and seeing the product in action.  This would enable you to play with it without having to download, configure, or set up any machines.

    2. Testing: When we move to trying it out, we want to use the same model as in the exploration phase.  We’d like to try it out also on a SaaS basis, but this time with more control.  By this I mean testing with real clusters & machines, in a more high performance, and production-like environment.

    3. Development: At this stage, we want to be able to test it with our actual stack, and plug it into our project quickly.  Then, eventually at a later stage download it and run it in our own environment.

    What’s Needed

    With a unified tool for all of these stages, the majority of the friction in the process can be diminished.  One way to achieve this is by running the project as a service or in our environment, with an orchestration engine that can basically be used to take us through all these stages.

    By wrapping the entire project into an uber-launcher [which with Cloudify we call it “the widget”] that any open source project can plug into – like an app store or repository, and any user can start using immediately without necessarily having to know how exactly it runs or works.

    The nice thing is that we can eventually take the software from that app catalog or store, since it’s not packaged as a black box but rather as a blueprint or script. We can then fork it and customize it to our needs.  Because of this, we can actually use the same tools through the entire cycle and break the friction completely.

    So what it all boils down to, is the basic principle of combining the power of SaaS and cloud orchestration to avoid complexity of installation, use a script-based model to make the packaging of software generic – but still customizable, and use a cloud abstraction to move environments without having to change recipe.  This is what we call the automation continuum, and we’ve been using to demonstrate how to take the power of any open source project to any cloud.

  • DevOps Contrarian

    DevOps Contrarian

    I admire and appreciate the ideas of DevOps as a result of living through two decades of working in IT and experiencing the pain of what I would call the “failed state of IT”. The failed state of IT being something inherently broken, something that couldn’t survive on its own, something that eats its budget and provides no value and something where the people working in such a system don’t even see the worth of the system and stop challenging it all together. In order to correct or prevent these “failed states” I often apply my experiences learned through DevOps and Lean processes and values.

    DevOps to me has been about recognizing and pulling yourself out of this “failed state” concept of IT and doing so with a little bit of empathy, a little bit of common sense and a lot of science. I don’t necessarily mean pure science in trying to come up with an underlying law that describes and predicts your system of work, but the foundations of science – The ability to question everything, the ability to collect data, analyze data, measure data and develop analysis thereof. By thinking “DevOps Science” you choose to develop models that describe your work, define your flow, show the value of your system and tailor it to your business needs and to research the best models to achieve your end goal and strive to continually improve those models. DevOps is about putting this science to work, and doing so with collected intelligence & experimentation behind it.

    Where I find myself being a DevOps contrarian of sorts is in the often over simplified representation of DevOps as an applied model. I don’t have any issue whatsoever with the concepts described below by themselves; they create amazing discussions and speak to very core foundations of DevOps, but they’re often only spoken of superficially.

    DevOps is not just about Culture.

    I often see people speak of DevOps as an applied “Cultural Invention”, an innovation found to be useful to a group of people in their behavior. While that is a great description from a 30 foot view as if you’re looking from the outside, it really doesn’t help one understand culture in any sense to help foster it. DevOps culture isn’t about copying what others do and hoping for success, it’s about developing it within realizing that its affected by both forces encouraging change and forces resisting change and finding the balance there of is that is important. You can’t simply force DevOps on an organization and call that culture. The real power of  DevOps as a cultural invention is being able to develop DevOps values that directly apply to your organization so that it can be passed down generation to generation and survive on its own.  Thinking scientifically, we learn from these shared examples of cultural invention and their successes by experimenting with them to see how they fit our organization as there is no universal law of culture but merely models we can research and identify with. It’s great to build upon the success for others, but the only way to validate those successes is to test them yourself!

    Another danger of thinking applied culture is the development of cultural optimums, I parallel this to what Garret Hardin coined “The Tragedy of the commons”, whereby individuals, acting independently and rationally according to each one’s self-interest, behave contrary to the whole group’s long-term best interests by depleting some common resource. I won’t claim to define what your resource is that you’re depleting, but it’s something to keep in your mind as you think of culture because it’s important to understand culture external of yourself as much as it is within. The value of culture isn’t in how you rationalize it for your own goals, but how it operates on its own and if that culture is appropriate for the good of the entire system (business). The strongest metaphor in DevOps for depleting the wrong resource is wasted resources not working towards your common goal. Are you actively engaged on the right tasks? Right automation? Right flows? Are you merely creating more work or are you adding value? DevOps isn’t something you apply methodologically, its something you experiment with methodologically.

    DevOps is not just about automation.

    Automation is the process for operating with minimal or reduced human intervention.  If you’re focused on automating processes are you still improving that process or did you think to yourself “I automated that, now I’m free to work on other stuff”? Automation can tempt you into doing “more work” if not kept under control and that creates a real danger of losing focus, having too much work in progress and actually hurting your overall performance.

    Often times the best value in the systems we call “Systems Automation“ – the Chef, CFEngine, Puppet, Ansible & Saltstack systems  is not necessarily their amazing capability to automate processes but their ability to document processes. Those cookbooks, manifests, scripts and DSL’s you’re using are absolutely the best documentation of your system you can have. They provide a great resource when done in such a way to easily describe the processes that they’re automating so that they’re well enough documented for improving them. You can’t automate that which you don’t understand and the best way to understand them is to make sure the processes that describe them are easily readable, documented, tested, and validated. They should be part of your experiment, not just testing for failure but testing for improvement.

    Through the lens of science, we can look at automation more as standardizing on a process and applying the best resource to that process.  Sometimes it is compute power instead of human power  but don’t fall into the trap that the process is said and done (Automated all the things!) and time to move on! Science is a never ending process by where you build upon your conclusions!

    One conclusion I and many others have come across is that people are the most flexible resource you have! It may be a better value to your business to have the flexibility of a human resource if the process you’re trying to build requires flexibility. Even when planning for automation, requiring human intervention is a perfectly valid result and a justification you can sell to your business that has been proven through experimentation & data. Don’t ride the automation bandwagon when your data shows otherwise!

    DevOps is not just about measuring.

    What are you doing with the data you measure? What value does measuring give your organization? When you have the flu and you take your own temperature, what are you measuring for? Do you have the cultural aptitude in your organization to accept the ramifications of what you measured? Can your organization stop working when it’s “sick”, take the time to heal and start back up after taking corrective or remediation actions?

    Measuring is extremely important, but measuring isn’t what DevOps is.  Measuring is a part of what makes your organization responsive, culturally adept, and accountable for itself. It’s what allows you to convey value, to compare itself against and strive for improvement. If you’re not measuring, what are you improving against? Inversely, If all you’re doing is measuring, what is the value of your measurements? What are they the basis of? I like to think of biological systems when I start thinking of computing systems where there is a cost of energy to do something. Are you utilizing that cost the best way you can by measuring what you are or should you evolve the system to spend that energy elsewhere? Are you collecting the right metrics to even comprehend your systems? Do your metrics allow for capturing exaptation and evolution thereof? When you experiment, measure, rinse & repeat you get patterns; measuring helps identify those patterns. Creativity, experimentation and the process of science allow you to act upon those patterns and create value from those patterns. Some people call that innovation.

    DevOps is not just about sharing.

    There are people who believe that unless you are open source, have a GitHub repo and publish your code, you’re not really sharing and you’re not DevOps. I certainly hope we’re not going down this path and certainly hope we can appeal to the broader ideas of sharing. Sharing is about the joint use of a resource and sharing is knowledge charity. Sharing goes hand in hand with culture in not just expressing your needs and sharing your knowledge, but having the ability to listen, feel empathy and accept what others share as well.  When it comes to collaboration & sharing it’s much easier to accomplish if everyone sees it through the lens of science, not only do you speak to DevOps values more rationally but you can even accept opposing views and ideas much easier because you have evidence to them and aren’t emotionally tied to their success, you’re tied the process that challenges their success as a way to improve them!

    DevOps through the lens of Science.

    The beauty of DevOps through the lens of science is that science is about challenging ideas,  accepting failure, learning from history/prior experiments and having the strength to not only say those successes and  failures are a valid output of your experiments but also come up with a new experiment or new hypotheses and do the process all over again. This helps define your culture, this helps setup what you should automate, this helps you validate what to measure and this develops what you should share.  Through the lens of science you learn through experimentation, testing & verification what works for you, what creates value, what brings you closer to your goal. When honestly done you also learn what doesn’t work for you, where you failed, and how you can use those lessons in failure to continuously improve.  A failed hypothesis after all is still good data!

    Why so much focus on the sciences? Personally, I enjoy astrophysics not because there is direct monetary value in understanding the cosmos around us but because of the philosophical nature of being, the beauty of knowing and the questions it gets you asking as the more you know, the more you wonder. That wonder, is what makes DevOps so great to me. That wonder is what drives continuous improvement. That wonder is what drives the “what next…”. That wonder is what pushes “question everything”. That wonder is always what pushes me to accept and understand the science we do know and trust it because it’s based upon the values that are respected by thousands of scientists all over the world through peer review and consensus. That foundation is not the assumption that we think we know it all, but that we know enough to describe something and we trust that description good is enough that we challenge you to break it.  I would highly recommend that any DevOps practitioner cross pollinate their technology passion with the sciences as there is nothing more exciting than realizing commonalities of biology, physics, astrophysics and the laws of nature within your own thoughts and patterns as they’re applied to DevOps.

    So be a DevOps contrarian; for its not accepting DevOps as the answer, it’s the challenge there of that will make it aspire to bigger and better things. It’s not that Culture, Automation, Measuring and Sharing is wrong by any means, but they’re just scratching the surface of what you need to understand to make DevOps “Work” for your organization.

    I’m not only a DevOps contrarian, but I’m an OS contrarian, a Developer contrarian, a Systems Contrarian. I’ve made the best of running Windows, Oracle Linux, Oracle Database, eBusiness Suite, Weblogic and much more. Not because I’ve personally identified or I am idealistically aligned with any of these tools & systems, but because they solve the needs of the business; they create value, they operate and perform admirably and we have proven their worth. It’s no utopian system by any stretch of the imagination, but I find great reward in the continuous experimentation of making these systems work and improving upon them in a very DevOps way.

    After practicing DevOps & Lean principals and having  over two decades of IT experience, I’ve realized the “failed states of IT” and people who thrive in them are people who stopped critical thinking, stopped challenging the status quo and stopped experimenting.  They stopped having a contrarian.

  • DevOps Dirty Little Secrets

    DevOps Dirty Little Secrets

    By all outward appearances DevOps is gaining real velocity within IT shops worldwide, Duh tell you something you don’t know. I have read much on this topic as I am sure you have, but I never seem to read much with hard data results, other than we are doing great with some very broad  metrics. What are the hard numbers as compared to the legacy IT shop?  So my search begins; where in the world are devops metrics in Fortune 500/Enterprise organizations?

    What got me thinking about this were a couple of recent AppDynamics blogs “Measuring your devops success” and   “Quantifying the value of devops”.  Both blogs do an excellent job at breaking down DevOps to the implementation level. Hard data creates metrics that set a baseline with the legacy system that is the critical measurement component, without a baseline there is nothing to compare or measure against. This approach creates a meaningful way to measure the success of executing a DevOps strategy within an IT shop.  The blogs also point other factors not quite as easy to measure, particularly human elements. Many others have written on measuring DevOps, you need to adopt the approach that makes sense for you and your organization.

    At that point I figured it was time to visit Puppet Labs, when you think DevOps Puppet Labs and their survey’s comes to mind. I utilized the 2013 State of DevOps Report; the 2014 report is not available yet. I believe that in order for any new philosophy to be adopted as a true standardized practice, the C-level in Fortune 500 organizations need to embrace it. In my long experience, typically the first sign of acceptance in traditional Fortune 500 organizations is when the financial vertical embraces it.  As an example, Wall Street traders will pay good money to save two milliseconds on a trade.

    The Puppet Labs report numbers demonstrate real velocity in the adoption of DevOps as an accepted business practice. The first takeaways I had from the report were that organizations ship code 30x times faster with 50% fewer failures. The numbers paint a glowing picture for DevOps. But looking beyond the glowing numbers, I see something else. DevOps is at a critical fork in the road and needs to cross the chasm before it loses its luster and becomes just another application development tool, as opposed to a practice.

    Only 17% of the respondents were Managers or C-level, with only 22% of respondent companies having 10,000 employees or more.  It is essential to look at individual Fortune 500 companies IT shops and measure them with hard data before and after the implementation of DevOps.  I would love to read some in depth case studies of the Fortune 500 with hard numbers and metrics; I did not find anything with the legacy baseline data. A huge benefit of DevOps to the Fortune 500 is it will enable them to fail fast as opposed to continuing an implementation that will inevitably fail anyway and consume a huge amount resource in the process. Anyone who has worked for a Fortune company I am sure has witnessed this.

    In truly understanding what DevOps is and is not, certain questions need real answers; Are Enterprise/Fortune 500 companies really doing the same old thing and just need to write better code utilizing some new application development tools, does that make it DevOps or just continuous improvement? Will some IT execs change the organizations title from sysadmin to Devops Admin to protect their silo and make them seem kinder gentler Ops? Are the DevOps tools available today ready for prime time in the Fortune 500 IT environment? Is Devops about technology and metrics or really about culture? We can speculate, but time will tell us the real answers. I am looking for good metrics on what does DevOps do to a Fortune 500 company Not some so called Unicorn that was born with a DevOps spoon in its mouth, but a good old fashioned Wall Street firm who adopts DevOps. Until then call me the DevOps skeptic.

    If you would like say something; email parker@devops.com and don’t forget to comment here.

  • Hacking Your Auditor

    Hacking Your Auditor

    No, not that kind of hacking, give me a little credit…

    Recently I was having a conversation with a consulting friend about DevOps, and he found himself in a bit of a tough situation. The organization he was working with was a pure cloud/DevOps shop, but one that was stepping into having the meet compliance obligations. Needless to say, things weren’t going well with the auditor.

    For those of you who haven’t had to deal with the miasma of audits, one of the strongest requirements, for nearly any regulation/contract/agreement, is strong change management. This is especially true after those Enron guys screwed everything up for the rest of us.

    Change management makes a ton of sense from both an operational and compliance standpoint. I, myself, have blown things up in production but not being smart about my change management. For compliance, it is absolutely essential. If you are dealing with anything with financial or privacy implications, you sure as heck want to get it into a stable, secure, compliant state and ensure that state is kept over time.

    There is a big misconception out there that DevOps-style organizations don’t have change management. That when you push multiple changes per developer per day into production, there is no accountability or control. Now, while you can certainly build things this way, that sure isn’t how I see successful teams doing it. Not even close.

    The problem isn’t that DevOps necessarily skips change management, it’s that it looks different. Much of change management becomes automated, and the fundamental nature of changes themselves is different. But think about how you describe DevOps to your traditional IT-focused friends. Now imagine trying to explain this to an auditor working off set standards that are still figuring out if cloud includes a co-hosted Mac Mini duct taped to a rack.

    Here are some tips on how to both adjust your process, and explain it to the auditors:

    • First, figure out which pieces of your application and operations are in-scope for the regulation. Odds are only financial and privacy related bits are, and you can then apply more-granular controls or logging to these if you need.
    • Make sure your continuous integration tool (if you have one) includes some security checks and logs all updates/changes. If you push right from your code repository to production, get a CI tool.
    • Keep records of all changes as they flow from dev to test to prod (or whatever you use). Again, this should be trivial to automate and dump the records to a secure log repository someplace.
    • Unless your cloud provider tracks all API calls (like CloudTrail does on AWS, although that only works for some services and regions) use an API or web UI proxy like Netflix’s Asgaard. This really is a good security practice anyway, and can actually speed up infrastructure and platform management processes anyway. Keep the logs from this to make the auditor happy.
    • If you SSH into instances, you need to log that activity and dump the logs to a central repository.
    • Make sure you keep revisions of both all code changes and infrastructure automation changes (e.g. your CloudFormation templates, Chef/Puppet scripts/etc.). This is often built-in, but make sure you configure it correctly.
    • Those suggestions now give you logs of all application changes, infrastructure changes, and direct access to instances. These are the things the auditor wants to know exist.
    • Filter them based on your in-scope compliance requirements. Such as payment systems components only.
    • Explain to your auditor that all changes go through an automated approval system that checks the code, tests it (detail the tests), tracks the change, and then moves it into production. That all server logins and infrastructure changes are logged, as is the complete history of the structure of your entire environment. And that you can provide all the documentation, filtered for the pieces within compliance scope.

    This might look like a lot, but odds are very high you are already doing most of this. You keep code revisions, run changes through test suites before they go into production, and track all infrastructure changes. You might not have your own Ops portal/proxy, but you really should for a lot of reasons that have nothing to do with compliance.

    The trick is explaining this to your auditor in their terms. They expect manual checks and records. You need to show them that you still have these checks and balances, but they are automated. You still might need to add some pieces, such as an alert if certain parts of your code are changed (e.g. the file with payment processing or account reconciliation), but these will hopefully be minimal.

    A good DevOps environment actually has a much better compliance infrastructure in place than traditional infrastructure. The automation and template-driven models give us much better control and insight as compared to manually fixing everything, even when we push more day to day changes. The trick is to document this, add a couple pieces if you need it, and then explain it all to someone with… less than refined technical skills.

     

     

  • Logging Wins for Devops and Security

    Logging Wins for Devops and Security

    Its always great when companies can invest in tools that serve multiple groups and purposes.  Ensuring proper setup and use of logs is one toolset that can serve both DevOps and Security.

    Why logs are important to DevOps

    DevOps use  logs primarily  to understand what’s going on inside an application server, database system or the operating system.  For the most part, logs are a resource that helps DevOps diagnose problems and ensure proper working order of systems.  Sometimes logs provide a tracking mechanism that helps DevOps understand what customers are doing in their application.  DevOps folks are intimately familiar with newer and popular logging aggregation services like Splunk, SumoLogic and Loggly.

    Security Loves Logs Too

    The good news is that security teams love logs too.  The use, review and management of logs is so important to the security function of an organization, its even part of the SANS top 20 critical security controls.  From a security standpoint, logs provide a ton of important data for everything from forensic analysis to audit records.

    Recommendations

    Since logs are so important to both DevOps and Security teams, it pays to implement your logging tools in a way that will serve the needs of both teams. Here a few quick tips that will help you set up logging tools in a way that will create a better world for both DevOps and Security.

     

    Centralized Logging & Log Off Site

    Setup all your systems to log to the same centralized location.  If you have all your logs in one place it’s much easier to setup log reviews and alerts.  Its also a good idea to log to multiple locations for security reasons.  Consider logging to a centralized location within your network and also to a remote location.  For example, have all systems log to a central server and also log to a remote service such as SumoLogic, Loggly or Splunk.  That way you have multiple copies of your logs available for forensic purposes.

    Time Sync

    Make sure all your systems are in tight time synchronization.  There is nothing worse than reading your centralized logs to find that a few systems are not properly time synchronized.  NTP is your friend for setting up time sync.  Be sure to use two or more reliable upstream time servers and configure all your servers to peer with each other.

    Applications and Operating System Logs

    Start with the most common base configuration  so that all systems use syslog to capture and distribute logs.  In the beginning, the logs in syslog will probably only be from the operating system and few processes that  are already configured to use syslog.  However, there are a ton of other processes running that should also be logging to your centralized system.  The DevOps group will want to ensure that all applications, even custom developed code, logs important information.  Make sure that your custom apps log enough useful information that problems can be diagnosed when they occur.

    Setup Alerts & Reports

    Nobody wants to be trolling through tons and tons of logs. To prevent this, one of tne of the first things you should do is to set up a basic set of alerts and reports.  Security teams will want to make sure that logs record and alert when users gain administrative access to a system or an application, but there are lots of other events that should need logs and alerts. Spend some time thinking about events that could mean something is not working correctly and needs attention.  Ask the development team to specifically log non-critical events that should be reviewed and incorporate these into a daily report.

    Be Verbose, But With Care

    Crank up the logging levels on all systems but don’t go too far. You want to be careful you aren’t logging too much.  For example, you probably don’t need to run all your systems in debug mode all the time.  You also need to make sure you aren’t logging sensitive information like user passwords or authentication tokens. With logs, there is such a thing as too much.

    If logs are set up correctly, they will provide nearly endless benefits to both the DevOps and Security teams. Now, go forth and log!

  • Config Management & F***ing Shell Scripts

    Config Management & F***ing Shell Scripts

    Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

    Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

    Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

    Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

    We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

    People either missed this, or chose to ignore it:

    “This is simply brilliant! I’ve dreamt about such tools.”
    – artellectual

    “I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
    – sdegutis

    It provided a perfect platform for many to voice their frustrations with current tools:

    “Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
    – mattjaynes

    “In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
    – s_kilk

    “We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
    – sdegutis

    “Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
    – sp1982

    In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

    Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

    Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

    Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?

  • What “Programmable” Means

    What “Programmable” Means

    One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

    Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

    When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

    The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside. 

    programmabilty-illustratedWhat that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

    Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside. 

    To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership  to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside. 

    So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic  necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

    Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

    Okay, so that’s data path programmability. What about control path programmability?

    Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

    So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

    But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

    SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

    Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

     

  • Chuck Norris doesn’t need DevOps, but have we beat “What is DevOps” to death yet?

    Chuck Norris doesn’t need DevOps, but have we beat “What is DevOps” to death yet?

    Chuck-Norris-Card (1)Some interesting and lots of the usual around the DeOps blog beat last week and over the weekend. An interesting piece by Daniel Bryant on his Tai-Dev Blog on why “Chuck Norris doesn’t do DevOps”. It is actually a video of a presentation he gave at the recent London Java Community event and is the subject of a talk that Bryant has been working on for some time.  You can also see the slides from the show on slideshare here.

    Chuck Norris doesn’t do DevOps because he doesn’t have to. Chuck can do everything and he can do it by himself. He has no need to work with Dev or Ops, he does it all. But for the rest of us, to get something done we have to work as part of a team. As such we need DevOps to help us work better, faster deeper. So until we are all as good as Chuck Norris, we need to keep at it.

    To understand why Chuck Norris doesn’t do DevOps, do we have to also define DevOps?  Hasn’t been there enough good descriptions of what DevOps is?  Usually when you keep seeing people trying to solve the same issue over and over again, it generally means it hasn’t been done satisfactorily to date. Is this the case with a definition of DevOps? You would think so by how “What is DevOps” blog posts we see.

    Vishal Biyani on his blog last week writes exactly that. It is actually part of one of a multi-part series of blog posts that will according to Biyani cover:

    • Getting basics right: What is DevOps anyway? (This post)
    • It’s all about delivering: CI & CD
    • Configuration Chaos :  Managing your infrastructure, as code!
    • Monitor, measure and move: From logs to performance and beyond
    • Putting it all together: Collaborate, communicate and continuously improve
    • Further Reading

    Giving Biyani credit he refers back to Magnus Hedemark at Redhat. The first part explains the basic premise behind DevOps, a brief history and the motive behind the DevOps movement.

    I think this is what we should do more often. Refer back to a well written definition that we can point to with consistency rather than having as many definitions of DevOps as there are stars in the sky. An example is OlinData who back in January wrote a well done “what is DevOps” post.  It is no worse than countless others I have seen.  The issue, how many more people will write their own definition instead of just pointing at OlinData’s or Dev2Ops.com definition or any others.

    As long as we spend so many cycles defining what DevOps is, we are wasting cycles that we could be using on doing DevOps and expanding DevOps knowledge. I think we have beaten the what it is fairly well by now, let’s move on.

    An example of moving on is the “BlueMIx” 100 Days announced by IBM and written about on this IBM blog. I will be writing more about IBM and BlueMix, but suffice to say here, IBM is scheduling 100 workshops around the world to help Developers do DevOps with some pretty cool tools from Big Blue.  It promises to be a great event if you can get to one in your area.

    Another good example of blogs moving DevOps forward is the Ranger4 blog. They had a good post up with the 7 Habits of Highly Effective Continuous Delivery Experts.  Good article that take it to the next level.

    Anyway, that is my report today. For those that notice these things, I will be handling the Cup o’ Joe blog here for a while, as reading blogs is something I do.  You can write me at ashimmy@devops.com if you have something on your blog you want us to mention.

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.

  • Follow Friday, DevOps Twitter Accounts to Follow

    Follow Friday, DevOps Twitter Accounts to Follow

    twitterdevopsAs part of doing my research for this blog I have been looking around for a good list of DevOps people to follow. I have found a small list here and there, but unlike in the infosec community where we had the @securitytwits, there is no analogue to that in DevOps.

    So with a tip of the hat to Bill Brenner of Akamai (and formerly of CSO.com) I am going to start a new tradition here on staging-devopsy.kinsta.cloud.  We will have our own Follow Friday: DevOps Peeps to follow on Twitter.

    I am starting with a small list that I actually found on a Puppet Labs blog. I will add to this list every week and also start a Twitter list of them.  You are already following most of these folks, but you have to start somewhere.

    If you would like to be added to the list or have a suggestion for the list please leave it in a comment or email me at parker@devops.com.

     @patrickdebois: @RealGeneKim: @botchagalupe: @jezhumble; @garethr; @kartar; @KrisBuytaert; @jordansissel; @agentdero; @allspaw; @monkchips: @DEVOPS_BORAT; @lnxchk: @damonedwards;  @lusis; @obfuscurity; @shanley: @adrianco; @scriptrock; @jumpcloud; @sd_architect; @govictorops @SoftLayer @docker; @chef;