Cancer sucks. But with folks like Sarah, DevOps is helping make a difference in the race to a cure.

Sarah Elkins is not curing cancer herself, but she is employing DevOps practices to help those who are. Sarah supports the technology infrastructure for those who are trying to cure cancer at the National Institutes of Health (NIH).

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Sarah Elkins (@configures) configures technology solutions at the National Cancer Institute (NCI), where she has worked for nine years. NCI is a federal agency, part of the National Institutes of Health. It supports more than 700 websites, from basic HTML to content management systems to complex bioinformatics systems that have multiple tiers and thousands of servers. It uses multiple operating systems, containers and physical and virtual machines (both on-premises and some AWS). Developers range from lone scientists to large teams.

Sarah and I have been following one another on Twitter for several years, but I recently caught the session she delivered in the “DevOps in Government” track at All Day DevOps. Sarah led a discussion about how the National Cancer Institute is automating its builds and deployments—showing it is possible even in a large bureaucracy.

Her talk covered the processes and technologies that enable software to move from source code repositories all the way to production servers at nci.nih.gov, including the use of GitHub, Jenkins, Nexus and other technologies, with a variety of teams involved.

We Need to Talk

Sarah’s cure begins with engagement: development, infrastructure and security all working together, just as teams of specialists work with patients to help them beat cancer.

To help speed development and approvals—a necessary evil that can grind development to a halt if not managed well—they agree that applications drawing on an existing technology catalog are approved as operational and security teams provide guidance to development on what is required for an Authority to Operate (ATO). In other words, they are trying to preapprove as much as possible and communicate earlier in the process.

DevOps Practices are Maturing

Source code is primarily kept on GitHub, including a public repository for non-proprietary source code, and NCI primarily uses Maven or Apache Ant for build scripts. Infrastructure teams provide XML templates to provide consistency.

Most NCI software relies on build dependencies on open source software components. The organization uses artifacts during builds and utilizes Sonatype’s Nexus as its repository manager.

All of this supports the automation of builds and deployments at NCI. For builds, most active projects are either on or migrating to Jenkins. Build artifacts may be .zip, .war/.ear or Docker images.

For application deployments, NCI uses development, quality assurance, stage and production stages. Most teams use some form of automation, from simple (copy content, stop/start container) to more complex scripts. Developers are allowed to perform deployments for robust applications, and some manual orchestration is required—for instance, for database timing or related applications.

As an organization, NCI is moving toward continuous integration (CI), with varying progress among teams.

Building In Security

NCI also now includes security early, often and automatically. The organization has deployed self-serve/on-demand Nessus scans, which allow developers to see, on their own, how they are doing as soon as the application is stable. Security teams run AppScan and Twistlock for Docker image scanning during Jenkins builds, just before deploying and frequently scanning the repository in between. For issues found, development and infrastructure teams work together to remediate security concerns.

At the end of her talk, Sarah offered up three takeaways:

1. DevOps at NCI is a work in progress (isn’t it everywhere?)
2. The wide-ranging needs at NCI require flexibility, communication and teamwork
3. There is no shortage of work

Since it is a work in progress, what opportunities are ahead at NCI?

• More automation for individual applications with Jenkins
  • Developers can perform container restarts on lower tiers
  • Database updates (some projects are using Liquibase already)
  • Some applications are still manual/batch scripted
• More Docker and improving Jenkins/Docker instances
• More orchestration automation and Puppet/Ansible integration
• Security improvements through
  • Scan dependency artifacts before building
  • More integration and automation

Sarah dives into more details in her talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. DevOps in Government is one of the five tracks.

— Derek E. Weeks