This article illustrates how to deploy an application into a Hadoop cluster using GitLab/Jenkins/Spark application.

In this article, we will cover the following:

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

• Auto-triggering Jenkins build when code is committed in Git.
• Pulling the latest code from the repository through Jenkins.
• Building, compiling and packaging the “.jar” using SBT tool.
• Uploading the .jar to the Nexus repository with defined version.
• Pulling the versioned jar and dependent files from Nexus repository to server.
• Deploying the versioned “.jar” file through Spark job into Hadoop cluster.

Getting Started

The following must be performed first:

1. Download and install Java 1.8.
2. Install Jenkins.
3. Integrate GitLab and Jira.
4. Install SBT.
5. Create the Hadoop cluster: Hadoop and Spark.

End to End Steps in Detail

When a developer pushes code to GitLab, Jenkins will trigger a predefined job using webhooks.

The Jenkins job will pull the code from version control using Git; it builds the code and makes the package as .jar file using the build tool SBT. This .jar file can be deployed into a Hadoop cluster with the help of a Spark command.

Once the deployment is completed in the Hadoop cluster, the application will start running in the background.

Initially, we can see the current running application in Hadoop (see Figure 1). It has an application ID, which is in a running state.

• If we click the tracking UI, which is Application Master, it will navigate into the Spark streaming page.

• In Figure 2, we can see batches are running in 40 seconds. For this scenario, we are changing from 40 seconds in the code.
• When a developer makes changes to the code, he will commit in his local machine and test his code, then push to remote repository. I have updated steaming time from 40 to 50 seconds and committed.
• If we can specify the Jira task number in the commit message, the same comment will be updated in the Jira task as below.

In the Jira application, we can see our commit message as comments.

GitLab and Jira integration process:

• Install Git plugin to pull code from GitHub repository and SBT plugin to build and make a package as a .jar file in Jenkins.
• Configure the Jenkins job under source code management and provide the Repository URL along with credentials and branch details.

• Optional: Set webhooks to automatically trigger the Jenkins job when the push happens in GitLab.

• In the “build” section, add the parameters clean compile package in the Actions field, which will build the code and create a .jar file.

• Once the job has been triggered in Jenkins, monitor the Jenkins console for the next step and errors, if any.

Optional: If we have configured SonarQube, it will analyze the code files and provide the result in the SonarQube dashboard. This allows us to know the code quality, code coverage vulnerabilities and bugs to improve efficiency of code.

• Optional: To maintain the versioning, we can upload our *.jar into Nexus/Artifactory.
• Once .jar has been created, we can keep it in a specific folder and trigger a Spark job. This can be performed by triggering a Spark command from Jenkins. Execute the below command to submit spark job:

• Once deployed, we can see the application ID number has been increased.

Timeout in the Spark streaming job has been increased to 50 seconds.

— Nodagala Nagarjuna