My earlier blog, “From Laboratories to DevOps: Factories for Continuous Testing,” discussed the importance of and suggested practices for a fully automated, orchestrated software DevOps laboratory infrastructure, akin to a modern factory staffed by robots churning out products at a dizzy pace. The blog applies to development of software for networks, and the practices increasingly are important as the network becomes more software-based thanks to software-defined networking (SDN),  network functions virtualization infrastructure (NFVI) and cloud-based networking. The edges of mobile networks, even the radio access networks, are becoming more software-dependent due to the need to position computing resources closer to mobile customers to resolve latency, bandwidth and spectrum concerns.

However, the need for highly automated software processes does not stop within the development lab.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

At a recent DevOps Networking event in Santa Clara, California, Google Network Operations Manager Anees Shaikh indicated that Google’s live network has many thousands of nodes and management elements, supported by millions of lines of software-based configuration files that are updated tens of thousands of times per month. The sheer volume and speed of software updates for large networks is not feasible using manual methods. Automation tools for live network software updates are vital.

Typical live network DevOps environments use DevOps continuous tool chains such as Git for software version management, YAML for defining abstract configuration definitions, Jinja2 to translate the configurations into concrete templates that can be consumed by real devices and Ansible for automating the distribution and activation of config files to the network nodes. But what about testing? How can a network operator be confident that the many updates will perform as expected once deployed across the live network? This requires much more than simple syntax and semantics checking of the config file entries.

Recommended Practices for Continuous Testing of Live Network DevOps Environments

For continuous testing of live network DevOps environments to work, here are some best practices to consider:

1. Lab-to-live transition: Prior to deployment of any updates to the live network software updates should be tested in a lab environment using production-equivalent equipment and topologies.
2. What to test? The actual tests to be executed depend on the risk of changes that are being deployed. If a small, low-risk configuration change is all that is being deployed, then a simple configuration check may be sufficient. On the other hand, if a change has the potential to disrupt the network performance significantly, then a much more comprehensive performance test is needed.
3. Test orchestration tools: The test environment prior to deployment will vary depending on the tests that are selected. Test environment orchestration tools are needed to set up network test topologies that match the tests rapidly—and release them just as rapidly so the test resources can be used for the next tests.
4. Elastic on-demand topologies: The ability to scale test resources horizontally and vertically, on-demand, is essential to be able to match the speed of testing required for short duration deployment cycles.
5. Automate live network tests during deployment: To verify software updates do not interrupt or degrade network performance, functional and performance tests between pairs of nodes across the network mesh are required as the updates are deployed in stages across the live network nodes, clusters, centers and regions.
6. Automate live network testing after deployment: After verifying that the deployments are successful, network tests need to be scheduled periodically to ensure latent problems with the new software do not impact the network.

This is just a partial list of suggestions for continuous testing essentials for live network DevOps.