At CloudSlang, we’re passionate about all thing open source. Integrating with other open source projects is a key part of what we’re all about. We focus on integrating with many cloud native technologies, such as CoreOS, Docker, and Consul.

As such, it is important for us to test these integrations automatically as part of our CI. But setting up a complex cluster environment (like CoreOS/Swarm cluster) using popular CI solutions such as Travis or CircleCI for testing can be tricky.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

In this post we’ll outline how we test our content integrations using a combination of DigitalOcean, CircleCI and CloudSlang. The same formula can be used to set up your own complex testing environments.

Recently CloudSlang introduced a testing framework to write tests for CloudSlang content and run them with the Build Tool. The Build Tool’s features include grouping tests into test suites and validating flow results, expected outputs or exceptions.

CloudSlang has a fast growing repository of content which is housed on GitHub. When a flow in the repository is changed it can affect other parts of the content that depend on the changed flow. Therefore, it was crucial for us to come up with a proper CI solution.

After investigating the available CI providers and our configuration needs, we decided to split our test suites in two categories and distribute them between CircleCI and Travis. Both solutions are based on the CloudSlang Build Tool to validate our content: CircleCI runs cluster related test suites (e.g. CoreOS, Docker Swarm) and base content (which does not require SSH access) while Travis handles most of the Docker content and flows that require access to the environment via SSH.

Logical Steps of the Build Process

The second container’s execution phases are presented in the following image:

These steps use bash scripts to execute relevant commands. The scripts and configuration files can be found under the ci-env directory. Droplets are created and managed using DigitalOcean’s REST API. The first step (create_droplets.sh) generates a discovery URL for the CoreOS cluster and updates the cloud-config file with the newly created URL. The  cloud-config file contains regular CoreOS configuration data for the cluster (private networking configuration, starting up services like etcd and fleet) and a write_files section where we prepare a unit file (docker-tcp.socket) for the Swarm use cases (this socket will be used to enable Docker Remote API). Once the cloud-config file is ready we can create the droplets by sending POST requests to DigitalOcean:

  CURL_OUTPUT=$(curl -i -s -X POST https://api.digitalocean.com/v2/droplets \                -H 'Content-Type: application/json' \                -H "Authorization: Bearer ${DO_API_TOKEN}" \                -d "{                  \"name\":\"${COREOS_MACHINE}\",                  \"ssh_keys\":[${DO_DROPLET_SSH_PUBLIC_KEY_ID}],"'                  "region":"ams3",                  "size":"512mb",                  "image":"coreos-stable",                  "backups":false,                  "ipv6":false,                  "private_networking":true,                  "user_data": "'"$(cat ci-env/cloud-config.yaml | sed 's/"/\\"/g')"'"                }')

The request is dynamically generated using global environmental variables defined in the project’s properties page (DO_API_TOKEN, DO_DROPLET_SSH_PUBLIC_KEY_ID), environmental variables defined in the script (COREOS_MACHINE: name of the machine, depends on build number) and cloud-config file. The script also verifies that the requests were accepted by DigitalOcean (status code: 202) storing the droplet IDs in an array:

DISCOVERY_URL: https://discovery.etcd.io/5f30dd91642ae055ff24efd6b2359f3dci-1439-coreos-1 (ID: 6748964) droplet creation request accepted - status code: 202ci-1439-coreos-2 (ID: 6748965) droplet creation request accepted - status code: 202ci-1439-coreos-3 (ID: 6748966) droplet creation request accepted - status code: 202

The second step (which is executed in parallel for both build machines) downloads and prepares the Build Tool. The related commands are read from the circle.yml file.

The third step (wait_for_droplets_and_update_test_inputs.sh) waits for the droplets to become operational, adds the necessary post startup configuration (e.g. activating TCP socket for Docker Swarm) and updates the inputs with the actual information (droplet IP addresses, private key file location). The script is periodically checking the droplet status by sending GET requests to DigitalOcean:

Droplet(6748964) information retrieved successfullyDroplet(6748964) status: newDroplet(6748964) information retrieved successfullyDroplet(6748964) status: newDroplet(6748964) information retrieved successfullyDroplet(6748964) status: newDroplet(6748964) information retrieved successfullyDroplet(6748964) status: activeDroplet(6748964) IPv4 address: 188.166.45.76

Here we combine regular bash scripting (grep, awk) with Python (for complex data processing):

          IP_ADDRESS=$(\          echo "$RESPONSE_BODY_JSON" | python -c \'if True:          import json,sys;          obj = json.load(sys.stdin);          ipv4_container_list = obj["droplet"]["networks"]["v4"];          public_ipv4_container_list = filter(lambda x : x["type"] == "public", ipv4_container_list);          print public_ipv4_container_list[0]["ip_address"] if len(public_ipv4_container_list) > 0 else "";'\          )

Two CoreOS machines will be used as Swarm agents (the remaining one will have the Swarm manager) so we need to enable a Docker socket for these machines. The unit file which defines the socket was created at droplet startup time based on the cloud-config file. At this stage we need to enable the socket with the following command:

  LAST_LINE=$(ssh -i ${SSH_KEY_PATH} \  -o UserKnownHostsFile=/dev/null \  -o StrictHostKeyChecking=no \  core@${DROPLET_IP} \  'sudo systemctl enable docker-tcp.socket \  && sudo systemctl stop docker \  && sudo systemctl start docker-tcp.socket \  && sudo systemctl start docker \  && echo -e "\nSUCCESS"' | tail -n 1)

Now the droplets are ready to be accessed. While the CoreOS cluster is properly configured, the Swarm cluster will be set up using flows since Swarm is based on Docker containers (need to run manager and agent containers on the machines). The last step before starting the Build Tool is updating the test input files with the actual data (e.g. IP addresses of the droplets):

find test -type f -exec sed -i "s/<coreos_host_1>/${DROPLET_IP_ARRAY[0]}/g" {} +

At this point everything is prepared to run the tests. The Build Tool is executed by activating the relevant test suites. The flows in these suites interact with the droplets using SSH calls. The Build Tool displays relevant statistics about the execution – coverage, how many test cases passed and cause of the problem in case of test failures.

02:41:58 [INFO] 97% of the content has tests02:41:58 [INFO] Out of 158 executables, 154 executables have tests02:41:58 [INFO] 02:41:58 [INFO] ------------------------------------------------------------02:41:58 [INFO] BUILD SUCCESS02:41:58 [INFO] ------------------------------------------------------------02:41:58 [INFO] Found 153 slang files under directory: "/home/ubuntu/cloud-slang-content" and all are valid.02:41:58 [INFO] 11 test cases passed02:41:58 [INFO] 145 test cases skipped

Once the tests finish – in the last step – we need to delete the droplets used for this build by sending DELETE requests to DigitalOcean (cleanup_env.sh).

Since the working mechanism of our build system is based on heterogeneous components we need a secure way to store sensitive data (e.g. DigitalOcean API token for authentication). CircleCI provides the possibility to define global environmental variables where we can store this kind of data. These environmental variables are not included in builds which come from external pull requests (pull requests created by users who are not part of CloudSlang organization), otherwise they could be easily obtained. Someone could print them by modifying the circle.yml file inside the pull request.

In this post we’ve show you how CloudSlang developed its own continuous integration mechanism by combining technologies like the Build Tool (the worker that executes tests), GitHub to host the codebase, CircleCI (CI solution to define the skeleton of the build system) and DigitalOcean (IaaS provider to host testing machines).

If you would like to learn more about the build system or just check out the project, visit us at CloudSlang’s website or GitHub page.

And remember – keep the bar green …