One of the big advantages to smaller deploys and continuous integration is that it can make it easier to provide more proactive security. In short, continuous integration (and the associated automated testing it enables) makes it easier to focus your security team on analyzing areas where the security risk is higher.
If the notion of rapid development, rapid deployment, and continuous integration make your security team nervous you may be able to get them engaged by sharing proven ideas from companies who’ve made this approach work well in the real world. For example, I was recently flipping through a presentation from Etsy on SlideShare, and discovered a clear explanation of how continuous integration enables better security.
At right, you see slide 34 from this (very long but clear) deck. this is the first in a series of slides that talks about how you can identify high-risk areas of code, then use those designations to drive automated checks for changes to those areas of code. The power comes in when you automatically alert the security team that someone has just check in code which changes one of the high-risk areas.
This seems like a commonsense way to focus high attention on high risk.
Another positive benefit of this approach is that it forces the conversation upfront about what constitutes a high-risk area code. This is an opportunity to get security, ops, and the business on the same page about where risk is likely to occur. When you incorporate this approach with periodic reviews of where things stand, it allows you to refine your approach to automated analysis based on your experience. In other words, during post-incident review if you discover one of those”Gee I wish I’d known that I had of time” sorts of situations, you can add a check into the automation library that looks for those conditions in the future. This sort of continuous improvement model benefits everyone.
Oh, by the way – one of the scenarios this approach can check for is the introduction of new third-party libraries. In today’s climate of elevated concern over third-party risk (as well as fears of security flaws in open-source components) having that capability can be a very good thing.
What about you? How are you integrating continuous security checks into your continuous integration?
One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.
So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).
So far so good.
But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.
The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.
Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.
The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.
One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :
“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]
Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.
Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process an optimize it by eliminating redundancy. That results in greater deployment velocity.
Recognizing that processes should be comprised of composable tasks offers a higher degree of reuse across operations. The task of provisioning and configuration a web server with the appropriate base software means consistency, aka standardization, across all applications. Processes that take advantage of standardization result in greater infrastructure stability, another benefit often cited by organizations that have adopted a devops approach to operations.
The key here is to recognize the difference between automation and orchestration and how they relate to one another. Automation is about codifying tasks and orchestration is about codifying processes. Orchestration takes advantage of automation by reusing these basic building blocks.
There is a difference, and done right, the two play off one another to result in streamlined, more accurate execution of operational processes that speed up successful application deployments.
All start up business endeavors must go through the phase of crossing the chasm. Most of those business endeavors usually fall prey to the chasm. The chasm is this black hole that everyone claims to understand but no one truly does. We have all seen businesses cross the chasm or fail to cross the chasm and fall into the black hole. But no one seems to understand why and how that crossing happens. The bestselling book by Geoffrey A. Moore titled “Crossing the Chasm,” is all about the heart and soul required to get early stage technology across the chasm, from early adopters to mainstream customers.
There is a big difference between people who are enthusiastic to try leading edge technologies and the rest of the inhabitants, who tend to be much more conservative. The reasons that people spend time trying to figure out how to cross or jump, if you will, the chasm is because that black hole is what stands between them and making their fortune. The early adopters are just a small sample of the market and there is no money in building a business or technology around them. To make real money, you need to cross over into the mainstream.
In Geoff’s model, the innovators are all about technology. You know those people, when a new iphone is released, they spend days standing on line to get latest and greatest new technology. The early adopters are the visionaries that are always on the bleeding edge. The mainstream are the pragmatists, the people who practice risk avoidance. These pragmatists are the key to jumping the black hole, they move the needle.
DevOps first order of business was getting on the curve and to that end DevOps has evangelists. But, understand that what worked in converting these early adopters probably won’t work for the pragmatists. These DevOps evangelists and visionaries want DevOps power tools, as opposed to the pragmatists who simply want to push the button and say “that was easy”.
By all appearances DevOps has been accepted by the early adoption set. Just how does DevOps cross the chasm? Jumping the black hole after all can be quite a daunting task. To do so DevOps has to create what I refer to as the tsunami effect.
The Tsunami effect is where DevOps gains acceptance in multiple industries or markets in what seems like a gold rush, it’s here that fortunes are made. More accurately, it’s where cult communities transform overnight into mainstream communities. Many technologies simply create a single tornado which remains in a silo forever never gaining enough momentum to really emerge. Another common path what I call an earthquake, there is a lot of hype but it tends to fizzle out quickly, if not tended to properly. Right now DevOps has all the appearances of an earthquake, Devops evangelists have created a lot of buzz, but acceptance beyond a set of power tools is still a huge question.
Considering DevOps is in the earthquake phase, as opposed to the tornado phase, DevOps cannot be successful converting one group at a time. The strategy in the tornado phase is convert industries and markets one at a time, via tornados forming within industries or markets. The tactical execution is to get smaller industries or markets committed and therefore the larger industries or markets will then embrace the movement. That is simply crossing the chasm. DevOps has too many cultural implications to convert believers over a long period of time; DevOps requires reorganization, not a subtle shift.
Since I consider DevOps to be in the earthquake phase, crossing the chasm is not an option. DevOps must jump the chasm in a surge of overwhelming momentum, hence creating a tsunami. DevOps has created a very closely knit culture, DevOps needs to push hard to create a tsunami and flood the pragmatic population. But, be advised, DevOps has to be all but universally accepted, there is no almost in bringing together Dev and Ops. But if DevOps wins, it wins big, DevOps will go from being a niche culture to a truly mainstream practice.
So what’s the strategy to create a tsunami? And more importantly, how do you move beyond the earthquake phase? Here’s how I see it according to Geoff, with my DevOps spin: First, DevOps must clearly establish that the mission-critical software development process is broken and is causing management enough pain that it would gamble on unproven methodology that is populated with a bunch of prima donnas. I am not convinced the DevOps cult has clearly established this as fact. Is the development and operations process broken or have just new power tools been created to improve speed and accuracy? Second, DevOps needs to present the total solution, not just tools. Overwhelm the community; each new DevOps tool should leverage the success and technology of the last hence creating a methodology. Even more importantly, because the DevOps tools are closely related, credibility can cross industries and markets and create the momentum required for a tsunami. Once Devops has reached critical mass by providing the total solution in a few industries or markets simultaneously the tsunami is created and DevOps has jumped the chasm.
The tsunami will emerge suddenly with everyone seeming to adopt DevOps all at once. This will result in explosive growth and will pull, willingly or unwillingly, everyone into its path. It’s driven and sustained by the pragmatists. Here, everybody has to get the latest thing all at once; the masses have been converted to DevOps. DevOps needs mass-market success, DevOps needs to become a tsunami and everyone will want to play DevOps.
Now that you know the formula, go forward and create the perfect storm.
If you would like to say something; email parker@devops.com and don’t forget to comment here.
Based on my unscientific poll of friends, one of the least used and most overlooked features of AWS is CloudWatch. Not only can CloudWatch be used to monitor the availability of your AWS services, but it can also be used as anomaly an detection tool. Did I mention that these feature are free?
Since the holy grail of DevOps is to bring the security and the operations teams together to meet the same goals in an automated fashion; Cloud Watch is a perfect DevOps tool. In fact, once it’s implemented for your environment you are one step closer to the exalted state of SecDevOps.
If you aren’t familiar with AWS CloudWatch then head on over to https://aws.amazon.com/cloudwatch/. As a quick introduction, here’s how AWS website describes the service:
“Amazon CloudWatch provides monitoring for AWS cloud resources and the applications customers run on AWS. Developers and system administrators can use it to collect and track metrics, gain insight, and react immediately to keep their applications and businesses running smoothly. Amazon CloudWatch monitors AWS resources such as Amazon EC2 and Amazon RDS DB instances, and can also monitor custom metrics generated by a customer’s applications and services. With Amazon CloudWatch, you gain system-wide visibility into resource utilization, application performance, and operational health.”
Let’s assume that both security and Ops teams need to be aware when weird things are happening with their servers and applications. Events outside the norm (aka weird things) can be either be related to security or operational events; either way they may merit investigation. Here’s a simple CloudWatch monitoring process you can use to give you a heads up on events that could indicate someone is trying to exfiltrate data from your systems.
Opening Scene
Think about what kind of information attackers are after. They want your intellectual property and your customer data because it can be converted to cold, hard cash. Next, consider the architecture of your systems and where that data is stored. Most likely your treasures are stored in databases or Git repositories.
Guess what? Attackers know that so that’s exactly what they are looking for. Take a few minutes and think like an attacker bent on exfiltrating your data. Obviously, the first thing you should do is carefully monitor your egress traffic for suspicious activity. In addition, look for anomalies in traffic flows – they can also help determine if suspicious events are happening. Amazon CloudWatch can help you do both of these things.(Note: This process assumes you haven’t already been breached.)
1. Enable CloudWatch
2. Establish a baseline
Review the historical data on a daily, weekly and monthly basis. Soon you will start to see a trend in your egress bandwidth usage. Start with eyeballing the min, max and average usage over time. Put these in your favorite tracking tool (text file, spreadsheet, abacus, whatever).
3. Configure alerts
When you feel comfortable with your baseline, configure your first alert to send an email if traffic usage goes over the baseline. Consider how “chatty” you want this event to be. For example, alerting over the average baseline will often generate more alerts than alerts over maximum usage. This is a tricky balance and you’ll need to make trade offs between missing problems and the time spent responding to potential false positives.
4. Respond
However you choose to respond is up to you. I suggest you start with automatically opening a ticket in your tracking system so that someone will be sure to review the alert and make notes on their findings.
5. Tune as needed
Make a plan to review the alerts and the data on a regular basis. Architecture and other changes to the system are going to change the baseline and the alerts may need to be reconfigured.
Building and automating security into DevOps is a requirement that everyone should be working toward. It’s a theme I’m going to be returning to in future posts because it’s really important.
AWS Cloudwatch is a free tool that you can help you get started automating security right now, so what are you waiting for?
Does your bank offer the ability to deposit a check by taking its picture with your smartphone? When that feature debuted a few years back, I remember thinking it was interesting and convenient enough that if my bank didn’t offer it soon, I’d be willing to switch to one that did.
I bring this example up for two reasons: one, it’s a microcosm of the larger race to innovation that is happening in every company and industry as we speak; and two, it involves banks, which are good touch-points for a discussion about the unique challenges established enterprises face as they attempt to implement DevOps practices in their IT organizations.
We know that DevOps means incorporating best-practice application delivery methods that improve communication, integration and collaboration between development and operations, but what is its ultimate goal? Quite simply, getting high-quality services to market faster than competitors – at a cost that aligns with both IT budgets and business goals. And as my check-depositing story illustrates, if you’re not first in the market, you at least need to be nimble enough to catch up, or you risk losing customers.
For long-time enterprises, this can be easier said than done. In my previous blog about DevOps culture, I used Etsy as an example of an organization that has had great success with its DevOps transformation, but the e-commerce company has been around for less than a decade. Its young IT architecture is apples to the oranges of established enterprises, whose complex environments often comprise a mix of legacy and modern technologies that have been cobbled together over time.
As a result, the DevOps journey is typically longer and more difficult for such organizations – but that doesn’t mean they can afford not to pursue it.
Improving Deployment Practices at Every Infrastructure Layer
When you look at the IT infrastructure of an established enterprise, you normally find three layers of technology. Let’s call the first “systems of record.” These are the legacy transactional systems (often mainframes) that have been with a company since the beginning. They do their jobs well, but their size and inflexibility make them resistant to the type of Agile development that DevOps typically embraces.
The next layer is “systems of differentiation,” or the various hardware, software and middleware components an organization accumulates as it grows, merges with/acquires companies and enters new markets. While these systems are often more Agile-ready than the systems of record, their decoupled/distributed nature can complicate and slow down development and deployment processes.
Finally, there is the “systems of innovation” layer, which is made up of mobile and Web applications and leverages the latest technologies. Because of its light footprint and flexibility, this layer is the most adaptable to DevOps practices. In fact, as you move up the layers from systems of record to systems of innovation, the holy grail of high-quality, fast and low-cost deployment becomes more and more attainable.
What does this mean from a macro perspective? Basically that established enterprises need to take a pace-layered approach, where application delivery occurs at different paces within the different areas of their architectures. The point, though, is to improve in each layer.
So if an organization pushes out a mainframe update once per year, perhaps it can move to a bi-annual schedule (with support from the right people, processes and technology, of course). On the other hand, a Web team that publishes updates once or twice a week could leverage DevOps tactics and technologies to adopt a daily update cadence. Bottom line: Different deployment paces are fine across layers, as long as improvements are occurring across the board.
Kick-starting the Journey with Service Virtualization
Now that we know why the DevOps journey can be difficult for established enterprises, let’s look at how one such organization, namely Union Bank, overcame these challenges and achieved success.
According to an article on Bank Systems & Technology, Union Bank was looking for ways to compete with top-tier banks, so it began “acquiring other institutions, updating legacy systems and launching new front-end applications.” In order to support these efforts, the bank implemented a service virtualization solution, which simulates constrained or unavailable systems across the software development lifecycle (SDLC). This allowed its developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability.
Here are some of the results Union Bank has achieved since:
Cut testing environment setup time from 42 days to 3 days
Slashed three and a half months from mobile and Web application development time
Reduced the cost of testing cycles by 65 to 85 percent
Obviously, a service virtualization solution is only one piece of the DevOps puzzle, but Union Bank stands as a powerful example of how a long-standing enterprise can leverage such a tool to be more innovative and nimble in the marketplace.
I’m a huge fan of The Phoenix Project and I’m honored to count Gene Kim both as one of my professional inspirations and also as a personal friend. There is, however, a dangerous precipice inside of the Phoenix Project that, if misunderstood, could cause many DevOps practitioners and enterprise IT shops a daunting problem in adopting and progressing in the ways of DevOps.
Roadmap or Roadblock?
One of the main concepts discussed in both Kim’s The Phoenix Project and Goldratt’s The Goal is “the bottleneck” and what what to do about it. The basic premise explained is that all actions to increase throughput before a bottleneck, pile work up at the bottleneck and all actions to increase throughput after a bottleneck create a void of work to address after the bottleneck. The logical conclusion from this idea is that: in order to have a net positive effect on throughput, action must be taken at the, holistically viewed, tightest constraint on throughput within the operational system.
This concept model is both accurate and valuable as the IT shop and the manufacturing floor have a lot in common. The power of this concept model, also happens to be where the aforementioned dangerous precipice is revealed – getting alignment on which bottlenecks to focus on can turn into a multi-month effort in many enterprises which can require consultants, workshops, deep-dives, process studies and other activities that are at once highly valuable and not always likely to be approved by a shop that has not fully bought in to the three ways of DevOps.
One of the lesser understood secrets of corporate america is that even when you have formal authority, permission is required to do almost anything of substance. One “no” from the wrong party can kill even the most promising initiatives. This “veto effect” doesn’t require an absolute “no” either. Much like a parent saying “let me think about it”, the veto effect only requires the absence of a “yes”. This absence can take the form of “maybe”, “let me think about it”, “let’s wait and see”, or even a “run this by the other department heads and see what they think”.
Given that any reasonably sophisticated roadmap takes more than a few weeks to develop, vett and roadshow around a medium to large enterprise, a mature DevOps roadmap suffers from a classic chicken/egg paradox for many organizations. I extend my deepest sympathies to those who have found that a clear and easily understood roadmap can somehow both provide and require the catalyst to consensus and permission to proceed in the ways of DevOps. You can take heart, however, in the fact that neither you, nor DevOps, are alone in this conundrum as this is what can be referred to as “the corporate condition” that we must all bear.
While company may not always provide solace to the miserable, there is however, another way.
Just Follow the Sun
What is this one side road still available to you that has the power to relieve almost any non-alignment traffic jam? Start heading west! Just like the American pioneers, you can start following the sun to the wide open spaces of DevOps land where automation is plentiful and provable performance gains via instrumentation are gold. All you need is one or two comrades with some skills worthy of the road ahead and you can be off!
Just because you can’t get the unix team to automate environment creation, doesn’t mean you can’t head west and automate your build processes. Just because you can’t automate your entire production deployment doesn’t mean you can’t head west and automate your performance testing and benchmarking processes. Just because you can’t get everyone to agree on the tightest constraint, doesn’t mean you can’t head west and alleviate the ones you already know about. To head west on any of these things, you don’t need full concensus, you only need a traveling partner or two.
The Trail Has Already Been Marked
For mid to large size enterprises, a true DevOps transformation happens over the course of three or more years. Allowing a lack of consensus to become a total standstill is not the right answer. The tool at your disposal to overcome the veto affect is public consciousness.
Take UX design principles for example. It was less than a decade ago that design activities were considered a luxury item that produced pretty pictures; and then Apple happened and tipped public consciousness over. How many people are willing to casually veto design now?
When public consciousness to the value of DevOps tips over, it will no longer be you going to the powers that be to get the “OK to proceed”. It will be them coming to you and asking “what is taking so long?”. In that one moment, you’ll be able to fire back at them: “Here’s my roadmapping proposal. I’d love to have you be my business owner!”
Last week was a big one for Microsoft in DevOps. At the Build conference they unveiled several new features and functionality in both Azure and Visual Studio Online to make give them more DevOps chops and DevOps friendly.
Frankly it is to be expected though. In doing my research for staging-devopsy.kinsta.cloud prior to launch, I had a chance to speak to some folks at Microsoft about DevOps. Frankly they were frustrated. They thought they had a great story around DevOps and had not been able to attract the DevOps community to what they had available. Microsoft, much like IBM at one time owned the developer market. In fact the sheer number of developers who still develop in Visual Studio and code for .Net and other Microsoft platforms, is pretty amazing.
Now you don’t just have .net folks on Azure anymore. The platform now supports Node.js. Java. PHP and more. It blends both IaaS and PaaS, allowing users to slide between them almost seamlessly.
In the meantime DBMaestro announced it was bringing DevOps for database to MS SQL server. This is the same teamwork functionality that DBMaestro already delivered to Oracle users.
In addition to these announcements Microsoft also released some metrics that show just how scalable and quick the uptake has been on Azure and Visual Studio Online. Ultimately the answer will be do enough developers think Microsoft gives them what they need to build great applications on a great platform. This will determine whether Microsoft becomes a competitor to Amazon and Open Stack in the public Cloud.
But they are making their play. Whether you love them or hate them, it is never smart to bet against Microsoft. The numbers they announced last week show they can gain critical mass pretty quickly.
There are some good background stories on this at Redmond Mag, eWeek and TechWeek Europe. We will have a full feature story here on staging-devopsy.kinsta.cloud shortly.
On another note, here at staging-devopsy.kinsta.cloud we have put up our DevOps business directory in beta. Over the next few weeks we will pretty it up, but right now we are testing the functionality. We have populated the directory with some companies, but need more. Listing a company is free and if you list your company this week, we will upgrade you to the deluxe listing free ($99 value).
You can add long and short descriptions, logos, files to download, contact info and more. The directory over time will we hope become the source for finding DevOps related companies.
Look for more staging-devopsy.kinsta.cloud features and events over the coming days and weeks.
I often find myself coaching colleagues (and reminding myself) about managing down the number of simultaneous tasks or work assignments. Everyone in IT knows that demand always outstrips the capacity of IT to take on work. IT is in demand by all parts of the organization, often at no direct or hard cost to departments or individual staff, which can balloon workloads and exacerbate the growing backlog of requests.
This capacity limit, or constraint, to do work isn’t the only reason IT continues as the big bottleneck in organizations. I believe the “silent killer” is the wasted cycles, or “overhead”, that comes with taking on too much concurrent work. Said another way, one of the greatest challenges of managing work is managing work-in-progress within IT. This is one of the reasons the Kanban approach brings great benefit to IT organizations struggling to stay above water.
The Context Shift Tax
We frequently pride ourselves on our ability to multitask but there is a cost, or tax, we pay when juggling too many work items. Each time we pause on one task and shift to another, time and effort is required to reset, reload and return to a point we can productively make progress on that work. Think of this as a context shift that functions much like paging between memory and swap space in a computer operating system. If you’re multitasking 5 items, each swap to a different task requires another knowledge context shift. Expand the list to 12 or 15 work items, add the interruptions we all experience, and we’re potentially spending more time during the day making those mental shifts than doing actual work.
A legitimate argument can be made that over-multitasking increases complexity and decreases quality. As our mental bandwidth is split across too many work assignments, focus and concentration pays a price. We also experience a form of memory loss — some details are forgotten over time, even what the original problem was or why we are doing the task.
IT organizations function much in the same way — shifting from project to project, task to task, as priorities change, new needs arise and existing work is delayed. As the list of priorities and projects increase, the organization and its people expend more and more on this context shift tax, wasting energies multitasking, losing focus, extending delivery times and decreasing customer satisfaction.
Kanban – Keeping a Lid on the Multitasking Dragon
Kanban’s secret sauce isn’t just about organizing work differently, it’s about thinking about work differently. Kanban provides a visual image of work in any system, how it flows, and where bottlenecks appear. The true secret sauce is in applying a limit or constraint to work in progress. If you’ve tried kanban, you know what I’m talking about. Much has been written about the profound effects of limiting work in progress.
Using Kanban we can decrease multitasking, both by individuals and the organization. This reduces the context shift tax, freeing up cycles to maintain focus, perform work and complete tasks. I like to think of it as keeping a lid on the multitasking dragon.
Limiting work in progress means work items are completed, in shorter time frames, with fewer context shifts. Imagine spending 5% or less during the day making context shifts, versus a heavy multitasking load that could consume 20% or more overhead when shifting between work. Less or no memory loss is experienced, decreasing complexity and reducing the opportunity for quality to suffer. By limiting multitasking we also free up cycles to focus on measurement and improvement, essential ingredients to any truly successful IT organization.
When I look at how compliance and regulations have affected network security over the years, I’m reminded of what dog breeder standards and regulations have done to dogs over the years. I’ll use this analogy to make the case that networks have become too domesticated to the point that they are not able to adapt and thrive when faced with their natural predators on the Internet. DevOps in my opinion is a way to undomesticate your network so that processes and technologies can deal with the threat in the most natural manner, striking the balance portrayed in complex ecosystems.
This entire thought process began when I was hanging out with my buddy Hugo. He is my one-year-old French Bulldog. We hang out a lot, and there are times when I think he telepathically helps me solve hard technical problems (but that is off topic). Anyway, if you know anything about French Bulldogs, you know they are mutants. They have been bred to the point where their breathing is compromised and they can overheat and die when it is too hot. They have common back and spinal diseases, and I can go on and on, but they essentially need humans at this point to keep them alive. I have to put a life vest on poor Hugo if we are near water because, while all dogs can swim, his head is so heavy that after a short while he will drown. Nonetheless, he is my sidekick in many ways, and we both lack vital characteristics of our ancestors who thrived in much more hostile environments.
While I can go on and on talking about my buddy Hugo, the point I’m trying to make is that while standards like PCI help to raise the bar regarding online security measures, I have to stop and ask: security measures against whom? The level of innovation, change and adaptation that is taking place on the threat side of the equation is out pacing the rate at which security standards can be institutionalized and implemented. So your network may be cute and cuddly like my Hugo, and maybe even win a ribbon or two, but it would also not survive when faced with an advanced threat.
When we trace the domesticated dog back tens of thousands of years, we arrive at the grey wolf. They traveled in packs, they faced threats on a daily basis, they thrived and adapted in a very hostile environment. DevOps takes us back to the time when the wolf finds its strength in the pack, and the pack finds it strength in the wolf when faced with conflict. Survival is very real and practical; there is no level of certification, checklists or deterministic process that must be followed. It is a delicate dance between attacker and defender, and like it or not, we are headed back in that direction.
DevOps brings us the opportunity to constantly represent the changing threat in our everyday lives – such is the case with anything connected to the Internet. It is back to the wild, back to the days where a loss at the organism level meant evolution and resiliency at the species level. Our goals change from securing our networks to evolving our networks as this tempo of change raises the costs to the adversary. DevOps makes your network more expensive to attack!
The pressure to prove one’s coding prowess can make a developer do some very clever and creative things with their source code. Sometimes it’s just a unique approach, but every once in a while this cleverness may shave a few milliseconds off of some performance metrics as well. In the end it’s not worth it. While this may seem like the type of innovation we want from our peers I’ve come to the conclusion it often does more harm than good. A quote that ran across my twitter feed a few weeks ago sums it up nicely: “So if you’re as clever as you can be when you write it, how will you ever debug it?” This pearl of wisdom comes from a book called “The Elements of Programming Style”. It’s a programming book from 1978 that is still entirely relevant. There’s a Wikipedia article you can check out if you’re not ready to take my word for it.
I know I’ve coded something in a unique fashion, and was quite sure I was being brilliant at the time, only to go back later and have absolutely no clue why I wrote the code the way I did. If I added comments I may be able to find my place again but sometimes it takes a while. What about when another developer runs into one of these bright ideas? Will any amount of meditation or contemplation allow them to make sense of what I was doing when I wrote it? Should a new or junior developer be expected to have some out of body experience before being able to work with me? Of course not. I hate it when I do this to myself, never mind when I run across someone else’s flash of genius. I certainly try to keep other devs from going through this type of mind freak as often as I can.
While this exercise can make for a fun parlor game and help extend our knowledge of the language we’re coding in it causes more problems than it solves in a team environment. A wiser developer sticks with a tried and true approach. They’ve been around long enough to know that the code needs to make sense to any garden variety developer that may be working with it in the future (which may in fact be the developer that wrote it in the first place).
I first learned this lesson years ago while tutoring mathematics. I had a developed an algorithm to generate the unit circle. Computer Science majors I was tutoring in math had an easier time using this to quickly generate the circle on scratch paper than arbitrarily memorizing it or understanding the standard proof that shows a bunch of triangles somehow morphing into a circle.
Radian Values of the Unit Circle
The algorithm requires a certain amount of seed data. One must know that a full circle is 2π radians in circumference. This puts the other quarter circle measures at π/2, π, 3π/4 respectively. Only a simple understanding of fractions is required to see this. Next we need to fill the first quadrant, which is only three more values, π/6, π/4, and π/3. From here the algorithm helps us extrapolate the remaining radian values for the circle.
First Quadrant: π/6, π/4, and π/3 (initial seed values, you have to remember these)
Second Quadrant: π – First Quadrant Corollary Value ( e.g., 3π/3 – π/3 = 2π/3, our 2nd quadrant value)
Third Quadrant: π + First Quadrant Corollary Value ( e.g., 3π/3 + π/3 = 4π/3, our 3rd quadrant value)
Fourth Quadrant: 2π – First Quadrant Corollary Value (e.g., 6π/3 – π/3 = 5π/3, our 4th quadrant value)
So, the formula for quadrants two through four can take the form of (n – s), (n + s), and (2n -s), where n = π and ‘s’ is one of the three seed values from the first quadrant. Three simple equations that any CS major should be entirely comfortable with using. There are a few other notes on this proof to get the coordinates, signs, and degree measurements correct. This is not a post about mathematics however, so I’ll leave that as a exercise for the more mathematically inclined to complete on their own time.
Other mathematicians where not as impressed as the CS majors and the student athletes were with this. Watching students that actively despised math struggle with memorizing the unit circle in an effort to complete their core was part of the fun of being good with numbers. It was suggested (by some troll of a jerk) that I might be a math noob myself, some poser that wasn’t any smarter than the fools he was tutoring. Impostor syndrome set in and I I felt I had no choice but to prove my mathematical prowess.
First I went big and reproved what I was doing with a Klien Four-Group. The trolls were quick to point out that this, like my first model, wasn’t a circle at all but just a bunch of points on a plane. Next I came back with a version using a rotational matrix. This did in fact generate a full circle with any value you like on it. The trolls had no more jabs for me but they also fell short of any compliment or encouragement for the work I had done.
As for the students I was helping, none of them quite understood the issue I had encountered. They viewed those trolls as nothing but trolls. Peers that only laughed at their struggle and refused to even try to help them. I ultimately found out my students were concerned that I had gotten sucked into this quagmire. The research I was doing was beyond their understanding and they were still using the first model I developed. It did exactly what they needed it to do. I was informed I should take the rising grades as evidence that my work was of merit and move on to a more productive use of my time and energy.
It was only then I noticed the trolls did have a point. The Klien Four-Group model and my original algorithm weren’t circles at all, just dots with values attached to them. A mathematician would have issue with this where a developer would not, since an object oriented developer should have a deeply rooted appreciation for abstraction. A class called Tree rarely resembles what we know to be a tree. It’s the use case that ends up being important. To a paper company a tree only needs to be represented by the amount of pulp they can expect it to produce. To an apple farmer it probably boils down to the number of apples the tree puts out and maybe its location in an orchard. What we take to be a tree, its size, the shape of its leaves, how it smells, the sound it makes when the wind catches it, are all details beyond the scope of most software we’ll ever write.
This happens to be a huge blocker for most new OOP developers, especially the ones that are good at math. The object model I was using to teach CS students how to do trigonometry ended up being the perfect tool to teach mathematicians how object oriented programming works. The rotational matrix version is important because sometimes non-standard values are needed. It’s useful to show how a method call may use standard values or extend onto another model, like the rotational matrix, when standard values don’t apply.
Mathematicians appreciate the Klien Four-Group model and I’m happy to show it off when I have the chance. More importantly, it’s a reminder to not get too fancy with math they’re putting into software. The math is beautiful more than it’s clever, and I’ve written code I feel the same way about. Like most works of art it stems from a hobby and should be left out of any applied scenario if it’s not practical. If we can get the same result with something that’s simple, something that’s easy to understand, easy to use, and even common, then that’s what we should use every single time because it will be easy to maintain. Wisdom is nothing more than knowledge applied over time. Time and time again I’ve come to the conclusion that maintainability trumps performance. We should be quick to sacrifice any minimal performance gain that makes our code difficult to understand. If a major performance gain is within reach then we’d be wise to abstract any strange code that was used to obtain it. I’d also advise you to comment the living day lights out of it as well, since you may need to remember how it worked yourself someday.
At right, you see slide 34 from this (very long but clear) deck. this is the first in a series of slides that talks about how you can identify high-risk areas of code, then use those designations to drive automated checks for changes to those areas of code. The power comes in when you automatically alert the security team that someone has just check in code which changes one of the high-risk areas.
