There isn’t much that makes IT operations happier than receiving that brand new application neatly boxed, wrapped, tied with a bow, and outfitted with exhaustive instructions for every what-if, what-then, and what now scenario.
Before you give the gift that keeps on giving in all the wrong ways, take these top three must-haves in developer to IT operations project handoffs to heart.
Number One Must-Have: Define Normal and Abnormal Operational Conditions
What does the development team consider normal operational parameters? At what point does the software break? How? If, giving an extreme example, at 20K users per server IT operations should expect ~50-70% CPU load but at 2K users per server they should not see 100% CPU load, they need to know this.
“Development can outline similar behavior/symptom scenarios for infrastructure (CPU, MEM, DISK, NETWORK) and application-specific (USERS, SESSIONS, USE LATENCY, TIME BETWEEN ACTIONS) metrics,” says Ivo Vachkov, whose title is literally DevOps Engineer, Xi-Group Ltd (Bulgaria; www.xi-group.com). This is the kind of information that helps operations to recognize when a software component is in trouble.
This may sound like obvious information for developers to include in an application handoff. “But many developers (if not most) will not be able to precisely define what is the normal operational condition for their software,” says Vachkov, lecturer in networking and programming security, New Bulgarian University (http://nbu.bg/entrance.php).
IT operations needs to know about factors that affect normal operation including whether the software is CPU-bound, i.e., whether its performance is directly related to the processing power of the computer it runs on. Another such factor is whether the application is IO-bound, whether its performance is directly related to the throughput of the IO (MEM, DISK, NET) subsystem.
Development should prep operations about other software behaviors and risks. If the application will degrade gracefully under increased load, they need to know that. If it will come to a screeching halt under load, they need to know that even more. If the software will crash when it slams up against an administrative/technology timeout, operations can deal with it, if they know ahead of time.
“As operations people, we usually don’t expect too much. I would usually be happy to know that at 50k users per server the main process will crash. And this is OK, because I can work around it. I have the tools to prevent a breach of that limit. I would be happy to know that every night there is a batch processing that will hit the DB hard, instead of being awakened by an alarm about my DB being inaccessible,” says Vachkov.
Number Two Must-Have: Share the Known/Collected Technical Debt
In many cases the technical debt is not obvious, i.e., the documentation states that the component needs library X to work properly, but due to some particular reason, the developers used a specific version of the library that is not available in the default package repository of the OS. Someone needs to build it in manually and take additional care to keep the configuration in sync to make the code work. “Technical debt may have many faces, from bad/rigid pieces of code to bad algorithms to unspecified dependencies,” says Vachkov.
Technical debt is also commonly omitted by the development team. “Pride, politics, other considerations are in play here. Many developers will just not keep track of technical debt. But this is crucial for proper operations. I want to know the shortcomings of the piece of code I’ll be dealing with,” says Vachkov.
IT operations wants to know, for example, whether that key algorithm is O(n) instead of O(log n) because this will enable them to correlate metrics and explain what’s going on. These notations are part of Big-O Notation, which is used in mathematical and algorithmic analysis and generally represents a function that describes the upper limit of performance of a particular piece of code in relation to the number of input elements it needs to work with, according to Vachkov.
Cliffhanger!
This concludes part one of “Top 3 Must-Haves in Development/IT Operations Project Handoffs”. Tune in for part two when we take up with technical debt and Big-O Notation, learn more about why the Big-O is so important to operations and to corralling unwieldy software, and discover the third and final must-have in development/IT operations project handoffs. Don’t touch that mouse or touchscreen! Stay tuned to www.staging-devopsy.kinsta.cloud.