Any organization that has been around for more than a few years (and many who haven’t) have a massive system that would cost a fortune to replace. Most midsized enterprises have several of them. Those systems are one of the reasons that enterprises don’t adopt CI/CD at the same rate as startups. Sure, any new project or small things go there, but the elephant in the room just hangs out.
So what happens when you get called upon to move one of these systems into CI/CD?
Lets’ start with the easy case: the rewrite. In that case, you’re almost greenfield from the start and can build CI/CD into your development and test processes as you would for any brand new project.
The harder case is when the org has seen the benefits of CI/CD and decides it wants to apply them to the monster that slows every project it touches to a crawl. These big systems are generally mission-critical, so there’s an added layer there. But this is doable.
Yes, It’s Daunting, But…
First, get a copy of the app building in a separate environment. Containers are good for this, since it’s just a development environment anyway. Replicate the entirety of the system (except unscrubbed real data, whether you use that in testing or not is a per-org question) to this new environment, make sure it builds and passes the existing test environment.
Next, configure Dev/CI/CD tools (for a project of the size we’re talking, I recommend configuring application release automation [ARA] also to help track dependencies), and get the system building and testing under them. This will take time—time to configure the CI/CD tools, time to get the build environment to work reliably under them and time to figure out how to integrate existing test structures into the CI tool. That’s okay, the point is investing now to save later. Note that this is also the time to upgrade your dev tools. If you’re on J2EE 5, for example, it is time to upgrade. It’s been a decade, and you’re working on a copy of the project.
Once that reliably builds and passes tests, the next step is to move those pieces into a more CI/CD-friendly format. Traditional test scripts can be called out in CI, but consciously building the tests within the CI environment is more resilient. A close look at tests, what’s missing, what could be better, what needs to stay as it is and just report back are the biggest parts of this test.
Note that change requests will have to be copied to the new environment’s change request system, too. Some projects are lucky enough to lock out changes to the source application while the new one is brought up to speed, but for most, changes made in the original will need to be replicated. Once you start modifying code, they can no longer be just pulled from the source tree.
Then start fragmenting the app. Apps of that size inevitably have massive functions/methods that do a lot of things. Not a terrible plan in single-processing days, but if you’re moving to CI/CD and upgrading dev tools, you may as well optimize code, too. Breaking a 5,000-line subroutine into a master and four child subroutines, each of 1,000 lines (yeah, it’s never that easy, but you get the idea) might offer room for parallelization, and definitely reduces your change footprint when you have to go in there and modify code.
Fear Pundits Bearing Advice
It is relatively common in DevOps for someone to offer the advice, “Get the newest of X every time.” Don’t follow that advice. Pick a version and hardcode it. It is anathema to hardcode versions of tools/OS/libraries in DevOps, but the point of DevOps is to control change, and anyone offering that advice missed the havoc that their advice would have created with the release of Python 3 or RHOS/CentOS 7.0, both cases where backward compatibility was left in favor of progress. Plan updates; don’t come in to, “The entire build failed last night,” and have to fix it then.
By this point, you’ll have a better understanding of the app, and will know (or at least suspect) where monitoring points will be best placed. Add them in to do active monitoring instead of logging. Your first set of points won’t be perfect, and you’ll want to review them to see what improvements you can make, but the weak points of the system should be monitored at a minimum, and alerting or corrective action automatically take place.
Test, Test and More Test
As soon as things are stable in the new CI/CD environment, bring in beta testers from the user community to help you find issues where it doesn’t behave like the original. The point here is to copy the original first, most likely to enable faster changes in the future, so just ask users to do what they would normally do when interacting with the app, and see what happens. Anywhere there are issues is a good place to add testing (I don’t know any monolithic app that has 100 percent code coverage, and honestly have only heard rumors of new apps that do, so there’s always room for more testing at known weak points).
Before you know it, the system will be moved into CI/CD, and you’ll have a list of things you want to improve under the new environment. At that point, you’re covered, and can move the process to replace your old build process.
Voilá!
It is work to move a large app into CI/CD, and more work to keep from letting it get out of hand. In the new environment, it is easy to want to change everything at once. But don’t make that part of the move; build change requests for after the move to CI/CD is done, to keep project lines clean and evaluate those desired improvements in the light of all the other change requests building up.
— Don Macvittie
