Using the following code, we can configure Jenkins Master to integrate Sonar, Nexus and Jira and manage them as a code. Code as configuration is a plugin in Jenkins, which is very popular and simple to use.
First, we need to install plugins from Jenkins Market place.
Manage Jenkins > Manage Plugins > Advance
After installing, got to Manage Jenkins and you will find Configuration as Code settings.
After opening this link, it will give us the Jenkins Configuration as Code screen.
Here, you have two steps:
URL or Path.
Export the Existing configuration.
Provide URL or Path
We have two options for storing Jenkins.yaml file:
Any directory Jenkins is running
Version Control (such as Git)
If we go with option one, then we need to place Jenkins.yaml file in JENKINS_HOME directory, place that directory path and click on “Apply new configuration.”
If we go for version controlling, we need a place for the Jenkins.yaml file and then we need to open the file in a raw format in GitLab, as shown below:
Now, place that link and click on Apply new Configuration.
Export the Existing Configuration
Export the current configuration into Jenkins.yaml, click the Export Configuration button to download the Jenkins.yaml file.
After opening this file, you may see some errors. Keep in mind, this is still in under development so the export function is now working properly.
We can write Jenkins.yaml file in simple YAML format.
Plugins
This is what the YAML format looks like:
Let’s move to next step tool’s installation.
Tool installations
This is what it would look like during the tool installation:
Tools Integration
The next step is the pipeline job configuration.
Setting Up Job with JCasC
Now, let’s bring everything together and see what happens in Jenkins:
After applying the new configuration, it will show the date, time and path of the configuration.
If there is an error in syntax format, it will throw the error as shown below:
Rest API Method to Submit the Configuration
Here, officially we have JCasC. There is a way to apply changes through CURL.
Here we have two CURL commands:
Validation
Applying the new
Validation
Here, I am using Jenkins YAML to store in version controlling tool with public repo:
In addition, Pivotal revealed it will be adding support for open source technologies including Envoy load balancing and Istio service mesh software developed outside the auspices of the Cloud Foundry Foundation (CFF) to its Pivotal Cloud Foundry (PCF) distribution. Envoy and Istio are being employed to add a new routing tier to tier to PCF to give developers more control over network traffic.
Richard Seroter, vice president of product for Pivotal, said Platform Automation for PCF consists of a series of building block modules that will be applied to all the components of PCF as well as eventually the distribution of Kubernetes that Pivotal curates in collaboration with sister company VMware.
Pivotal also announced that Pivotal Application Service (PAS), the runtime for PCF, will soon support Windows Server 2019 along with the ability to configure multiple custom ports per PAS container to make deploying containerized applications in a PaaS environment more efficient from a networking perspective.
Finally, Pivotal is adding more descriptive metadata to the application programming interfaces (API) it exposes across the PAS runtime.
Seroter said Pivotal continues to focus on automating as much of the IT operations pipeline as possible. Rather than allocating IT staff to create and manage these pipelines, Pivotal is making a case for relying on a set of managed services that span instances of Cloud Foundry and Kubernetes running on top of virtual machines deployed on either a public cloud or on-premises. That approach relies on a highly opinionated operational pipeline defined by Pivotal that IT organizations will need to adopt to benefit from a highly consistent and repeatable approach to managing IT operations at scale, he said.
The need for a more structured approach to managing IT operations is only becoming more acute as organizations embrace Kubernetes platforms alongside Kubernetes, Seroter added. In fact, a survey of 501 IT leaders published by the CFF finds almost half of respondents (48%) report using some combination of PaaS, containers and serverless technologies together.
It’s still unclear how much IT organizations will standardize on a particular application development platform. PaaS environments typically have a more robust set of application development tools, but they can consume tens of virtual machines to stand up. Development tools for Kubernetes platforms are still evolving, but the platform provides a more efficient way to consume both virtual machines and bare-metal servers. It is apparent that for some time to come that many IT organizations will find themselves managing multiple platforms. The challenge that creates for DevOps teams is finding a way to manage those platforms consistently using a common framework rather than having to deploy and master separate management. It may take quite a while for IT leaders to rationalize their management platforms, but with each passing day it becomes clearer that goal increasingly is within reach.
A part of that effort, Chef also launched the Chef Enterprise Automation Stack—which combines Chef Infra for managing infrastructure, Chef InSpec for maintaining compliance, Chef Habitat for managing applications, Chef Automate for managing hybrid clouds and Chef Workstation, a starter kit for launching Chef—within a single distribution of Chef software. Chef Infra is the original Chef project around which the company was launched.
Corey Scobie, senior vice president of product and engineering for Chef, said the company previously offered a mix of open source and proprietary software that made engaging with some customers overly complex from a licensing perspective. This approach aligns the entire Chef business model around providing support for open source software, he said.
At the same time, Chef is now open to collaborating more closely with developers who want to contribute code to its various projects. That’s critical because code contributions are now the primary means through which many IT organizations help shape the direction of products as they continue to evolve, Scobie noted.
Of course, many IT organizations today either mandate the use of open source software or have instituted an “open source first” policy that makes it challenging for any proprietary software to be adopted.
Chef expects that by making its software easier to consume via a unified open source transition it will be able to further accelerate adoption of what is described as a “Coded Enterprise” to managing IT. Rather than manually configuring IT assets in isolation, that approach encourages organizations to adopt best DevOps practices that treat all aspects of IT, including infrastructure, as code, said Scobie, adding that capability is more important than ever as IT organizations are increasingly require to manage heterogenous cloud computing environments.
Chef claims that approach has already resulted in more than half of the Fortune 500 having adopted at least one module of its software and, in the fourth quarter of 2018, the highest level of bookings in the company’s history.
While Chef helped pioneer the IT automation category, competition across this sector has intensified. Chef and its primary rivals Puppet and Red Hat are the leading proponents of frameworks for automating IT operations. Although adoption of those frameworks has increased substantially in recent years, use within IT organizations often remains fragmented, often resulting in islands of automation. Chef is clearly hoping that IT organizations will increase adoption of its software stack now that it’s based entirely on open source software. Many organizations face challenges teaching their IT operations teams enough programming skills to automate their processes. In absence of those skills, adoption of IT automation frameworks within the context of a set of well-defined DevOps best practices often remains uneven, at best.
Compuware has updated its Topaz for Total Test suite to enable organizations applying best DevOps practices to the development of mainframe applications to combine unit, functional, integration and regression testing within a single test scenario.
In addition, Compuware is making it easier to test multiple applications simultaneously. The latest update to Topaz for Total Test adds the ability to not only orchestrate a sequence of steps, but also design tests to measure how programs interact with each other and various data sets.
Also added is the ability to automate several testing functions, including the ability to launch functional tests within a browser and execute unit, functional and integration tests without having to manually create their own Job Control Language (JCL). According to Compuware, testers now can design test cases upfront based on program specifications before developing a program. What’s more, Topaz for Total Test has been integrated with the DevOps pipeline management platform developed by XebiaLabs.
Finally, Compuware announced that the Topaz 3270 Emulator now makes it possible complete one-off “green screen” tasks without leaving Topaz. Initiating a CICS transaction or accessing REXX utilities through the Emulator within Topaz, for example, makes it simple for developers to quickly resume Eclipse-based tasks without having to exit its testing environment or toggle between tools.
Compuware CEO Chris O’Malley said the pace at which IT teams in mainframe environments have been adopting DevOps practices has accelerated considerably in the past year. Not only are applications being developed and deployed faster, the quality of those applications has increased because DevOps tools provided by Compuware make it easier to both visualize a complex application development process and then test those applications as they are being developed. Given the often mission-critical nature of mainframe applications, it’s crucial that any increase in development velocity doesn’t have a negative impact on application quality, he noted.
DevOps has become a greater imperative in mainframe environments, O’Malley added, as organizations look to leverage investments in existing mainframe applications rather than try replicating functionality that already exists on a mainframe in a distributed computing environment. That approach helps to reduce sharply the amount of technical debt an organization needs to service when they duplicate functionality across two or more platforms. Mainframe teams that historically have relied on waterfall developments process have been challenged by keeping pace with the rate of change. As applications that are deployed on a distributed system look to invoke a mainframe application function, organizations that have deployed those applications need to keep pace with a much faster rate of development, especially when it comes to application programing interfaces (APIs).
It may take a while longer for DevOps practices on mainframes to become adopted widely. But as the rate of innovation continues to increase, thanks to the rise of various digital business transformation initiatives, it’s apparent that mainframe application development will need to become more agile. The challenge at this point is not so much the availability of DevOps tools for the mainframe, but rather training the teams that build and deploy those applications on when and how to implement DevOps practices.
With serverless, you delegate the responsibility of running your infrastructure to a platform provider as much as possible. This frees your engineers to focus on building what your customers want from you—the features that differentiate your business from your competitors’. For this philosophy to work, however, the platform needs to not only give you the tools to build those features, but also deliver the performance that will keep engineers satisfied.
Yes, services such as AWS Lambda are powerful, but they have a number of well-documented limitations that make them unsuitable for some workloads. These include:
Cold starts and high latency make them unsuitable for real-time applications that require consistent and millisecond-level response times.
The pay-per-invocation model makes them costly in high throughput scenarios.
The event-driven model, coupled with a max execution time or data size restrictions, makes them unsuitable for long-running tasks or heavy data or AI processing.
They are limited to triggers and event sources that are specific to the cloud provider.
Enter Nuclio, an open source serverless platform that is built on top of Kubernetes and comes in both managed platform-as-a-service (PaaS) and self-hosted flavors. In this post, we will compare Nuclio with AWS Lambda, see how Nuclio addresses the above limitations and explore the new use cases Nuclio is unlocking, such as:
High throughput data processing pipelines or ETL.
Serving machine learning models in real time.
Real-time applications such as multiplayer games.
Long-running jobs and services.
Nuclio vs. AWS Lambda
Concurrency
One of the biggest shifts from traditional application development to AWS Lambda is how concurrency is managed. An Express.js web application running in a container/VM can handle multiple requests concurrently. In other words, your application manages its own concurrency. You scale the application by increasing the number of concurrent requests it can handle (scaling up) and then increasing the number of containers (scaling out).
Figure 1: Traditional applications scale up the currency within the application, then scale out the number of nodes running the application.
With AWS Lambda, concurrency is managed by the platform, and a concurrent execution would process only one request at a time—much like an actor in the actor model, which would process one message at a time. The application can scale out by increasing the number of concurrent executions of the function.
Lambda’s concurrency model has its advantages:
It simplifies application development. Concurrency is a constant source of bugs and performance issues, as developers struggle with writing highly concurrent applications.
It allows you to price each invocation individually because you don’t have resource contention from handling concurrent requests. This enables the pay-per-use pricing model and is one of the cornerstones of findev.
Still, Lambda also has some disadvantages:
You can’t make use of idle CPU cycles when a function is waiting on IO. Since most functions are IO-heavy, this results in significant inefficiencies.
It requires more scaling, which means more cold starts. To handle X concurrent requests, you need X concurrent executions, which equates to X cold starts. This makes it nearly impossible to achieve predictable performance.
Nuclio, on the other hand, can scale up as well as out. A function processor (analogous to a container) can host multiple function workers. Each worker is then able to process one message at a time. This system supports concurrency within the same processor, and then autoscales the number of replicas of the processor based on load.
Figure 2: The architecture of a Nuclio function.
Another interesting difference between Lambda and Nuclio is that Nuclio lets you specify both the minimum and maximum number of replicas. And as an aside, Nuclio also supports GPUs.
Figure 3: Nuclio lets you configure both the minimum and maximum number of replicas.
With Lambda, you can configure a reserved concurrency for a function, which (counterintuitively) sets its max concurrent executions. But there is no way to tell the system to always keep a certain number of concurrent executions running at all times. The system always scales to zero when there is no traffic—behavior that is highly undesirable for systems that experience regular spikes in traffic.
For example, my employer, DAZN, is in the sports streaming business. We see huge spikes in traffic all the time, as millions of users flood in seconds before a sporting event starts.
Figure 4: Sports streaming platforms, such as DAZN, experience huge spikes in traffic just before a sporting event kicks off.
Without the ability to configure the minimum number of concurrent executions, these spikes result in large numbers of cold starts. Also, in these cases, Lambda has to scale out the number of concurrent executions quickly to meet the surge in demand. Here, we also run into the 500/minute limit on how quickly Lambda is able to scale out. As such, we are currently not able to use Lambda on the critical path of our system, which has to shoulder these spikes.
With Nuclio, you can programmatically update the minimum replica count prior to the event so that when the spikes come, you don’t have to worry about hitting scaling limits or enduring the performance hit from cold starts. You can also tell Nuclio to scale to zero immediately via API calls, which saves resources.
Figure 5: Adjusting the minimum number of replicas enables you to deal with predictable spikes in traffic.
Timeout
Another major difference between Lambda and Nuclio is how timeout is handled. With Lambda, an invocation can run for up to 15 minutes. While it’s possible for you to use recursive functions or step functions to extend this limit, both approaches have their own problems. As such, Lambda functions are great for event-driven architectures and performing short, ephemeral tasks.
With Nuclio, there is no max execution time. The containers can keep on running, which in turn allows you to go beyond the constraints of the event-driven model. You can now turn a function into a long-running service, which unlocks some interesting use cases such as long-running ETL jobs and training machine learning (ML) models.
Caching State Between Invocations
Both Lambda and Nuclio functions accept an event and context during an invocation.
Figure 6: Both Nuclio and Lambda functions accept an invocation event and context as arguments.
With Lambda, the context object is ephemeral and does not persist between invocations. If you want to persist data between invocations (e.g. static configurations, database connections), then you need to declare them as global variables outside of the function handler.
With Nuclio, the execution context itself is persisted between invocations and can be used to cache state. The context object also includes a built-in logger, which supports structured logging with JSON and four log levels: DEBUG, INFO, WARN and ERROR.
You can configure the default log level for a function and can even override this default log level per invocation—which is very useful for debugging. You also have the ability to export your logs to external services such as Elasticsearch.
Figure 7: With Nuclio, you can select the default log level for every function.Figure 8: You can also override the default log level for an invocation.
In addition, there are hooks for performing context initialization, which is called before the first invocation on the function. This allows you to perform initialization logic before the container is put into active use and removes the dreaded cold start problem that Lambda suffers from.
Data Volumes and Persistent Data Connections
With Lambda, functions only run when they are triggered by events. Furthermore, concurrent executions are garbage-collected when they have been idle for a few minutes. This behavior makes life difficult when you are working with relational database management systems (RDBMS) and other systems that require persistent connections. Indeed, a set of guidelines has been developed to help you avoid the many pitfalls of using Lambda with RDBMS. Furthermore, Lambda doesn’t allow you to use persistent data in standard file system mounts, which forces you to copy to/from object storage before working on large files such as images, logs, ML models, etc.
With Nuclio, however, you can use the context object to maintain persistent connections to databases. Since the context is maintained at the processor level, you get better reuse, as they are shared across invocations. You can also take care of other aspects of data-fetching with data bindings, including batching and caching, which helps improve IO performance of the application. Additionally, you can mount volumes to functions, which is useful for working with ML models or Tensorflow, and you can mount Kubernetes secrets as volumes. Indeed, Nuclio supports all of the volume types that Kubernetes supports.
Runtimes
Both Lambda and Nuclio support popular languages such as Go, Node.js, Python, .Net Core, Java and Ruby. Also, both platforms offer a way for you to customize the execution runtime.
With Lambda, you can create a custom runtime and distribute it through AWS Lambda Layers. This lets you introduce additional language runtimes that are not natively supported by the Lambda platform. Several vendors have published runtimes for PHP, Rust, Erlang and Elixir, to name a few.
With Nuclio, you can run functions on your own Docker image (see Figure 9), which can come from a private image repository. This allows you to tailor the execution environment itself.
You can also support additional language runtimes through a Shell function, which allows you to handle invocation events with any executable binary. See this example for more details.
Figure 9: Nuclio lets you run functions on top of your own Docker image, which can come from a private image repository.
Nuclio ships with 13 triggers, including cron, HTTP, Kafka, Kinesis and RabbitMQ. Since Nuclio is open source, you can also write your own trigger for services that you want to integrate with and leverage other people’s contributions. The HTTP trigger gives you an easy way to integrate Nuclio with other event sources that support HTTP as target, such as SNS. Also, since Nuclio supports the CNCF CloudEvents standard, it can arguably support many more event sources that are CloudEvents-compliant.
Nuclio triggers are all normalized to the same Event object. This removes the need to understand the specific event signature for each event source, which is often confusing when working with Lambda. It also makes it easy to switch a function between different triggers.
Built-In HTTP Trigger
With Lambda, you need to use API Gateway to create an HTTP endpoint for your functions. API Gateway is a feature-rich service, but it’s also complicated and often costs more to run than the Lambda invocations themselves. It also adds another source of latency to your API, which, at the minimum, is around 5ms to 10ms, and it can overhead spike to more than 100ms. This cost and latency overhead make it unsuitable for applications with high throughput or hard real-time requirements.
With Nuclio, every function gets a private HTTP interface by default. To expose the endpoint publicly, you need to specify an HTTP trigger similar to API Gateway. However, unlike API Gateway, this HTTP trigger does not incur extra costs and has minimal latency overhead.
Figure 10: With Nuclio, every function gets a private HTTP interface by default.
Unlocking New Use Cases with Nuclio
As you can see from our comparison, Nuclio differs from Lambda in a number of important areas, such as its concurrency model and that it has no max execution time. This opens the door to a whole range of use cases. Let’s take a moment to look at a few.
High-Throughput Data Processing Pipelines or APIs
For a high-throughput API, the cost of both API Gateway and Lambda is drastically higher than an equivalent application running in containers. This cost discrepancy has lead many to rewrite their applications.
Nuclio’s concurrency model makes more efficient use of available resources, which significantly reduces operational cost when running at scale. Functions have built-in HTTP interfaces so you also don’t need to pay for an expensive API Gateway service.
The ability to mount a volume to your function also lets you read and write data to/from a mounted volume at high speed and enables building stateful applications. Again, you can’t do this with Lambda today, as it doesn’t allow you to attach Amazon Elastic File System (Amazon EFS) volumes to functions.
These features make it economically feasible to run high-throughput and high-performance APIs on Nuclio functions. The same can also be said about high-throughput data processing pipelines that have to process multiple terabytes of data per hour.
APIs with Predictable Spikes in Traffic
Nuclio gives you the ability to configure a minimum number of replicas. This allows systems that experience predictable spikes in traffic to reserve sufficient resources ahead of time and avoid degrading the user experience when the spikes happen. Examples include food ordering services such as Just Eat or Deliveroo or sports streaming services such as DAZN.
At the same time, Nuclio still lets you scale to zero by setting the minimum number of replicas to zero. It gives you the control to optimize for resource usage or latency, depending on the situation.
Real-Time Applications
Nuclio does not suffer from cold starts and is able to deliver consistent and sub-millisecond response times on invocations. This makes it a suitable solution for applications with a hard, real-time requirement such as multiplayer games and real-time bots.
Serving Machine Learning Models in Real Time
To serve ML models in real time, you need to have both strong and predictable API latency and the ability to load and work with ML models that often are large in size (GBs).
Nuclio lets you attach Kubernetes volumes to your functions, and there are no size limits on these volumes. Combined with the performance characteristics, it’s possible to implement these demanding workloads with Nuclio functions.
Nuclio also has native integration with Jupyter, and lets you automatically deploy Jupyter notebooks and ML models as Nuclio functions.
Long-Running Jobs and Services
Nuclio does not impose a max execution timeout on function invocations. This allows you to turn your functions into long-running services and perform long-running ETL jobs.
Nuclio’s init_context() hook lets you create long-running services, such as an application that constantly reads off a Twitter feed (e.g. loads the TwythonStreamer with the context, which is polling an external service, versus becoming triggered by an event).
Conclusion
As we discussed, Nuclio is one of the few open source solutions that are also business-viable. It has key architectural advantages for higher performance or data-driven applications, including:
Nuclio is cloud-agnostic. It can be used in multi-cloud/hybrid-cloud scenarios. You can also run Nuclio on a laptop using plain Docker, which makes debugging easier.
Nuclio’s concurrency model and the ability to mount file volumes lets you build high-performance APIs and high-throughput data processing pipelines.
Nuclio gives you greater control over the concurrency of your application, which makes it viable even for applications with very spiky traffic.
Nuclio’s execution contexts are persisted between invocations, and it doesn’t impose a max execution time on invocations. This allows you to create long-running stateful applications such as ETL jobs.
Nuclio lets you attach Kubernetes volumes to your functions and has no limitations on the size of the volume. This makes it possible for you to work with large ML models that often are GBs in size. Nuclio also has native integration with Jupyter and can automatically deploy your Jupyter notebooks and models as functions.
Nuclio provides a good out-of-the-box developer experience. It has a built-in logger with structured logging support, and you can easily export your logs to an external log aggregation service.
If you are looking for a serverless platform that is not tied to a specific cloud provider, Nuclio is worth a look.
When a developer pushes code to GitLab, Jenkins will trigger a predefined job using webhooks.
The Jenkins job will pull the code from version control using Git; it builds the code and makes the package as .jar file using the build tool SBT. This .jar file can be deployed into a Hadoop cluster with the help of a Spark command.
Once the deployment is completed in the Hadoop cluster, the application will start running in the background.
Initially, we can see the current running application in Hadoop (see Figure 1). It has an application ID, which is in a running state.
Figure 1
If we click the tracking UI, which is Application Master, it will navigate into the Spark streaming page.
Figure 2
In Figure 2, we can see batches are running in 40 seconds. For this scenario, we are changing from 40 seconds in the code.
When a developer makes changes to the code, he will commit in his local machine and test his code, then push to remote repository. I have updated steaming time from 40 to 50 seconds and committed.
If we can specify the Jira task number in the commit message, the same comment will be updated in the Jira task as below.
Figure 3
In the Jira application, we can see our commit message as comments.
Figure 4
GitLab and Jira integration process:
Install Git plugin to pull code from GitHub repository and SBT plugin to build and make a package as a .jar file in Jenkins.
Configure the Jenkins job under source code management and provide the Repository URL along with credentials and branch details.
Figure 5
Optional: Set webhooks to automatically trigger the Jenkins job when the push happens in GitLab.
Figure 6
In the “build” section, add the parameters clean compile package in the Actions field, which will build the code and create a .jar file.
Figure 7
Once the job has been triggered in Jenkins, monitor the Jenkins console for the next step and errors, if any.
Figure 8
Optional: If we have configured SonarQube, it will analyze the code files and provide the result in the SonarQube dashboard. This allows us to know the code quality, code coverage vulnerabilities and bugs to improve efficiency of code.
Optional: To maintain the versioning, we can upload our *.jar into Nexus/Artifactory.
Once .jar has been created, we can keep it in a specific folder and trigger a Spark job. This can be performed by triggering a Spark command from Jenkins. Execute the below command to submit spark job:
Compilation can make or break the performance quality of an application. A developer might spend weeks creating the most elegant, efficient algorithm known to man, only to have it run at a snail’s pace in production because the compiler used to create the application’s executable binary is old, and not optimized to take advantage of the enhanced performance features offered by modern hardware.
When it comes to getting the most out of code, companies in the know figured out a while ago that compilers count. Yet, for the most part, they remain a complex—if not somewhat mystical—set of technologies. I’ll admit it, for the longest time my understanding of compilation and compilers went only so far as running a make or build command. I was woefully ignorant about what was going under the covers.
But that was then, and this is now. Recently, I’ve been taking an interest in IBM’s Automatic Binary Optimizer (ABO) technology. ABO is a technology that optimizes existing COBOL code to run better in mainframe environments, without having to alter source code or modify existing compilations.
ABO intrigues meet for two reasons. First, it spurs my curiosity about mainframe compilation in general. Second, it makes me wonder how it’s possible to refine code that is already compiled and operating in a production environment.
So, I decided to go beyond my comfort zone and find the answers to some of the fundamental questions around this mysterious process. Exactly how does a mainframe compiler work? What’s the difference between a good compiler and a bad one? And finally, how does a low-level optimization technology such as ABO do its work, given that it’s being applied to code already compiled?
The following is the result of my research.
Understanding Front-End and Back-End Compilation
Compilation is the process of taking text-based source code and transforming it into a binary file that is executable against a specific piece of hardware. Mainframe computing compilation is divided into two parts: front-end compilation and back-end compilation. Front-end compilation has absolutely nothing to do with clients or GUIs. Rather, the purpose of front-end compilation is surprisingly similar to that of linting, as applied to interpreted languages. Front-end compilation is the process of making sure that the text-based source code is grammatically correct in terms of syntax, semantics, type usage and function declaration.
Once source code is subjected to front-end compilation, it’s passed on to the back end compiler for further processing.
Back-end compilation is the place where the artifact created by the front end, called the intermediate representation, is made into the binary format specific to the particular operating system and CPU of the targeted hardware. This binary file is what gets run on the targeted system. (See Figure 1).
Figure 1: Compiling COBOL source code transforms it into an opaque binary.
Back-end compilation is where the rubber meets the road in terms of machine-specific performance. A well-designed back end compiler will create binary code that’s optimized to take full advantage of the CPU architecture of the target system, squeezing out every bit of performance benefit possible. A good compiler will create a binary file that is fast to load and fast to run. However, if the underlying source code is inefficient—for example, having code that makes unnecessary trips to the database, thus creating episodic increases in network latency—no amount of back-end compilation is going to improve such runtime inefficiency. However, given that the source code is well-written, a good back-end compiler will improve overall application performance significantly. A compiler that is just so-so can do more harm than good; thus, the importance of compiler technology in the software development life cycle.
Yet, for all the benefit that it provides, there is a potential problem that is inherent in any piece of compiled code. Once source code is compiled into an executable binary, it becomes difficult to maintain—particularly when the source code is no longer available.
Compilation Creates Challenges
Unlike interpreted languages such as Python or JavaScript, in which source code is stored as plain text on a machine and executed at runtime by an interpreter, compiled code exists in a binary format that is opaque to human inspection. It’s machine readable only. Once the code is compiled, it’s “locked down” and opaque.
This creates two challenges. First, because the binary is locked down, it uses only the instruction sets that were available on the targeted system at the time of compilation. Thus, even though hardware that supports more robust instruction sets evolves, older code compiled for older systems will not be able to take advantage of the features of the modern hardware. All the compiled code can do is execute the instruction sets that were available on the older systems. The result is new hardware, yet old performance. The compiled code just doesn’t know how to move any faster.
The second challenge is that while it’s easy to change programming behavior later on by rewriting the source code from which the compiled code is created, there’s no easy way that compiled code can be changed directly. Thus, maintenance developers are left in a lurch should they need to upgrade an application for which the source code is unavailable.
In the world of mainframe computing, this is a big problem. Don’t forget that 80 percent of corporate data still resides in mainframes. There’s a lot of code out there that needs to be upgraded.
More than one company has been in the situation of having to update code that is decades old, yet both the source code and the person who created the code is nowhere to be found. The result is that companies are left with an asset that is not realizing its full value. The code remains static while significant technological advances have taken place.
There are tools that can be used for decompiling the binaries back to the original source code against which changes can be made, but the process is time-consuming and error-prone. Even if the decompiler gets it right and manages to reproduce source code that accurately reflects the original version, you still need to find a developer who can make the required changes. Such developers are in high demand in the mainframe world. Many companies simply prefer to leave well enough alone and allocate their mainframe developers to projects that have the highest return on investment.
Clearly, having a way to transform old, legacy binaries into ones that are optimized to run on modern mainframes, without having to go through a complicated decompilation and recompilation process, will benefit any company that needs to get better code to market, faster and in a cost-effective manner. Fortunately, IBM has provided a way with its ABO technology.
ABO: Solving the CompilationChallenges
ABO is designed to modernize old code without having to go through an arduous compilation/recompilation process, as described above. The way it works is interesting.
ABO works directly on the compiled binary to re-create its program structure as represented in the intermediate representation. The program structure is then passed onto the optimizer, which recompiles a new binary. This new binary takes advantage of the additional instruction sets offered by new hardware. And, it does so in such a way that the original programming logic is not altered in any way. (See Figure 2).
Figure 2: ABO performs optimization on program structures extracted from the legacy binary without affecting program logic.
For example, ABO optimizes standard math operations in COBOL such as SUBTRACT, MULTIPLY, DIVIDE and EXPONENTIATION by creating instructions in machine language that are more elegant and execute faster. No doubt, improving math operations is significant considering that COBOL is primarily used for business-oriented number crunching and correlation (being able to do math faster really counts, no pun intended), yet the important thing to understand is that the essential value of ABO optimization is that it’s taking advantage of the advanced instructions sets offered by the new target system. Remember, running older compiled code on new systems is not going to make that much of a difference in performance. In the PC world, it’s the same as running 32-bit code on 64-bit machines. Sure, in most cases it will work, but the code will never go beyond the efficiency of the low-level instructions supported by 32-bit processors.
Yes, there is some performance boost that can take place without having to recompile due solely to the improved clock speed of the CPU. However clock speeds on new mainframe models are beginning to plateau. As clock speed levels off, so will the time it takes for code to execute on the processor. This fact has not been lost on mainframe designers. New ways of enhancing overall performance, beyond simply moving the bits through the processor faster, are being devised. Sadly, there is no magic in play; for legacy code to realize better performance that’s not dependent solely on clock speed, the code will need to be compiled.
Recompiling from scratch using the source code is viable, but doing so is expensive. And, if that recompilation affects any part of the source code, additional testing needs to take place. Thus, the cost of change is increased due to both the expense of hands-on development and the added testing.
ABO, on the other hand, works with existing binaries. The result is an upgraded binary that’s optimized to the latest mainframe technology. In addition to having state-of-the-art code that’s fast and efficient, a company saves time and money in the refactoring process. It’s pretty amazing when you think about it.
What does such increased efficiency look like? IBM is reporting 24 times the reductions in CPU consumption time. That’s a substantial number.
Now, granted we’re talking about optimizing existing functionality. Refactoring an existing application to add new features requires code rewrites, no question about it. Yet, there are a lot of applications in play today that are doing what’s required. The problem is that they’re performing poorly and slowing down workflows overall. Optimizing existing binaries means not only that applications can have better performance, but that entire workflows become more efficient as more room is made for other applications to run.
Putting It All Together
Making legacy code better is, and has been, a continuous burden for companies since the first days of computing. Updating code is difficult enough when the source code and developer who wrote it are available to do the work. When both are absent and all that’s left behind is an executable binary, refactoring the application becomes a herculean set of tasks. Decompilers will only get you so far.
Fortunately, an optimization technology such as ABO makes things easier. ABO modernizes legacy binaries without the need to have the source code on hand. The result is improved system performance, reduced costs and faster release cycles.
As those of us who have been out on the terrain have learned, the rate of change in computer technology never slows down. It only goes faster. Companies that can adapt to the breakneck speed of technological innovations will prosper. This includes companies that rely upon mainframe technology to get work done. Reducing the time it takes to optimize legacy code is essential to stay competitive in today’s marketplace. For those companies that rely upon mainframe technology to get work done, ABO’s ability to optimize code directly against existing binaries provides a transformational advantage that is difficult to ignore.
Artificial intelligence (AI) is starting to be applied to the writing and update test scripts. Perfecto has unfurled Perfecto Codeless, which employs machine learning algorithms to automate writing of test scripts in a way that allows them to not only run continuously, but also adjust to changes made to the application.
Eran Kinsbruner, chief evangelist for Perfecto, a unit of Perforce, said writing test scripts is still largely a manual process, which means as the number of applications being developed continues to escalate, the testing process becomes highly constrained.
To address that issue, Perfecto has created a codeless approach delivered via a cloud service as part of the company’s Smart Automation platform. If any code within an application gets deleted, moved or changed, the changes will automatically be reflected in the test script. That capability eliminates the need for DevOps teams to wait for a developer to update the test script.
Exactly who in an organization is writing those test scripts varies: Kinsbruner noted that many developers now prefer to write their own test scripts during the development process before their code is reviewed by a peer. That approach eliminates many of the more routine errors, enabling more time to be spent on optimizing code. Other organizations employ dedicated testers who spend most of their time writing test scripts. But many of those organizations are finding that testers cannot keep pace with the current rate at which applications are being developed in the DevOps era.
There’s no doubt machine learning algorithms are about to play a much bigger role in application testing and DevOps at large. AI is not going to replace DevOps teams anytime soon, but many mundane functions such as the writing of test scripts can be automated using machine learning algorithms. Arguably, testing is one of the areas that lends itself most to emerging AI-based platforms and services. After all, if testing becomes a bottleneck, the chances that applications with issues are deployed in a production environment increase substantially. Many of those issues can be fixed easily, but many times at the risk of annoying end users who increasingly are unforgiving when it comes to application experiences. It’s not uncommon for users to discontinue using a mobile app if the first experience isn’t immediately gratifying.
Of course, many DevOps teams already view machine learning algorithms with a certain amount of skepticism. But the rise machine learning algorithms within DevOps is now all but inevitable. IT leaders, however, should realize it may take a while for those algorithms to learn the environment. The return on investment for any AI project is not likely to be immediate because a significant amount of time needs to be invested in training the algorithms. The good news is, those algorithms never take a day off and suddenly don’t resign one day to take a better-paying job elsewhere.
Fresh off raising an additional $20 million in funding, the Blameless platform provides across-the-board visibility into the complex network of application programming interfaces (APIs) and applications distributed across an organization, said CEO Ashar Rizqi. Specific modules within the Blameless platform address incident resolution, reliability insights dashboards, postmortems, service level objectives (SLOs) and automatic error budget calculation.
Blameless takes its name from an SRE principle that no one should be held at fault when complex systems fail. Rather, the assumption should be that inherent complexity of the IT environment is at fault. While DevOps is more of a philosophy surrounding the management of DevOps, SRE is emerging a more prescriptive instance of implementing DevOps that Google is now promoting. The goal is to make sure that whatever the issue, the right people to address it are brought together at the right time, said Rizqi.
The Blameless platform was created by Lyon Wong, who now serves as the COO of Blameless. Previously, Wong was a partner at Lightspeed Venture Partners, founder of S28 Capital and a product manager for Microsoft Windows. Rizqi is a former head of core platform at MuleSoft and SRE manager at Box.
Rizqi noted that while most organizations are trying to embrace DevOps principles, they often lack the tools needed to build the workflow processes required. All too often, organizations rely on little more than a Wiki document to manage their response to an incident. The Blameless platform is designed to provide access to an integrated set of tools for managing everything from the incident itself to the impact that event had on the business, he said.
It’s still too early to say with any certainty how widely SRE practices will be adopted. This far, most organizations that have adopted DevOps have tended to prefer to define a set of processes that are customized to their environment. But there are just as many, if not more, organizations that have found it daunting to get started. Platforms such as Blameless represent an opportunity for those organizations to embrace DevOps using a more prescriptive approach.
Of course, a big part of the reason that DevOps has been embraced unevenly by various departments within the enterprise is the lack of a consistent approach to implementing DevOps at scale. Teams that have embraced DevOps with enterprises have tended to create their own individual processes using a wide variety of tools. Many of those organizations have also found it challenging to get IT operations teams to embrace those processes because they require an approach that can be implemented consistently across multiple projects at scale. SRE provides IT operations teams with a framework that Google has already proven works at scale.
Regardless of the path chosen, the one thing that is for certain is more agile approaches to managing IT operations will be required to keep pace with the rate at which applications are now being developed and deployed.
Keith Kuchler, vice president of engineering for SolarWinds, said live code profiling provides a breakdown of the most frequently called functions and methods in a transaction, including class, method, filename and line number. That capability makes it possible for DevOps teams to identify what line of code might be causing a performance issue without requiring any changes to the code itself, he said.
According to Kuchler, the ability to profile live code extends a distributed tracing capability that SolarWinds previously incorporated into SolarWinds AppOptics. Collectively, the two offerings show how committed SolarWinds is to providing DevOps teams with access to advanced capabilities at a fraction of the cost of rival APM platforms delivered as a software-as-a-service (SaaS) application, said Kuchler. Pricing for SolarWinds AppOptics starts at $7.50 per month. SolarWinds AppOptics is part of what SolarWinds describes as a DevOps Toolkit that also includes Pingdom, a website performance monitoring tool, Papertrail, a log management service, and Loggly, a log analytics service.
The ability to profile live code supports applications written in Java and PHP, with support for additional programming languages forthcoming, said Kuchler. The goal is to reduce the amount of time DevOps teams need to spend guessing where the source of any issue might be after an application is deployed or updated, he added.
As the rate at which applications continue to be deployed and updated, DevOps teams need to be able to pinpoint issues at the code level to enable developers to address that issue as soon as possible. Sharing a bunch of generic alerts typically doesn’t provide a developer with enough actionable intelligence to solve the problem. Not surprisingly, a recent survey of over 300 developers conducted by SolarWinds finds that most of them spend a lot more time troubleshooting existing applications versus writing new code. All that time spent troubleshooting code winds up having an adverse impact on the ability to make the transition to DevOps.
In general, SolarWinds is trying to expand the total APM market by making its tools available as a service that organizations of any size can invoke. Historically, reliance on APM tools has been limited to mission-critical applications mainly due to cost. But over time, the number of applications deemed to be mission-critical has increased. In addition, because of the all interdependencies between applications, the potential impact a single line of code can have across the enterprise IT environment can be substantial. As a result, the need for an affordable approach to APM has become more acute.
Naturally, it will be interesting to see how much pressure there is on APM pricing over the course of the coming year. Competition between providers of APM platforms is already fierce. The challenge will be finding a provider of an APM platform that not only provides all the capabilities required, but also has deep enough pockets to be around for the long haul.
Here, you have two steps:
Also added is the ability to automate several testing functions, including the ability to launch functional tests within a browser and execute unit, functional and integration tests without having to manually create their own Job Control Language (JCL). According to Compuware, testers now can design test cases upfront based on program specifications before developing a program. What’s more, Topaz for Total Test has been integrated with the DevOps pipeline management platform developed by XebiaLabs.
