Tag: data

What is MLOps? DataOps? And Why do They Matter?

Let’s look at three distinct disciplines—DevOps, MLOps and DataOps. In 2011, Marc Andreessen famously proclaimed that software was “eating the world.” A little more than a decade later, it’s all but impossible to argue with his premonition as software has embedded itself into virtually every industry.

But now another shift is underway and AI is gobbling up software as we know it. Our emails can finish our sentences for us, connected cars are helping us stay safer on the road and predictive technology is helping minimize supply chain disruptions caused by COVID-19. In the near future, almost every piece of software we interact with will have intelligence built-in, and as AI gains acceptance and new use cases are rolled out, applications will only continue to become more intelligent. The widespread move toward ML-enabled software has the potential to be as transformative and far-reaching as what Andreessen predicted.

But innovation is rarely easy or simple—and intelligent applications are no exception. While conventional software involves one thing—code—and not much else, intelligent software relies on a complex relationship between three interconnected variables or legs of a three-legged stool.

A Three-Legged Stool: DevOps, MLOps and DataOps

Model: The one or more AI/ML models (e.g., linear regressions, rules, deep neural networks, etc.) trained to recognize patterns in data and make decisions are what makes an application “intelligent.”

Data: Most often, a model is trained on historical data and programmed to emulate it. As a result, an application’s behavior depends heavily on the data—the raw input data, labels and features, as well as the new data to which the model is applied and ground-truth for this data.

Code: Code is the language an application uses to function. When used in the context of intelligent applications, the code pillar might refer to business logic, calls to models, receiving outputs, decision making and calls to other data systems.

A Real-World Example

To see how these pillars play out in real life, consider the example of a perception system for a self-driving car. Such systems use data (provided by offline sources and real-time cameras and sensors), models and code to determine myriad ever-changing variables (for example, if there are obstacles on the road, what lane markings look like, how fast a car can safely travel, etc.) While the following illustration is greatly simplified, it outlines the basic steps needed to build and deploy such a system and indicates which pillar each step belongs to.

Regardless of application type, the legs that intelligent applications stand on are always the same: Model, data and code. And if you try to remove any one of them, the application will topple over.

With the three distinct pillars that make up intelligent applications, three distinct disciplines have emerged to keep each individual pillar functioning as efficiently as possible: DataOps, MLOps and DevOps. The rest of this post examines the details that make each discipline distinct and necessary to the model life cycle.

DevOps

Once upon a time, companies used waterfall processes to develop software. The process often moved so slowly that products didn’t realize their potential before they died or were replaced. In 2000, Agile was introduced, allowing developers to work quickly and iteratively in tight sprints where they were constantly delivering. By 2009, Flickr, for example, was the envy of the industry, routinely performing 10+ deploys per day. Companies seeking to increase their own productivity combined software development with established operations principles—and the term “DevOps” was coined.

As the familiar loop graphic below illustrates, the DevOps cycle goes from the planning stages to the creation, testing/verification, packaging, release, configuration, monitoring and then starts over again with planning. Over the past two decades, a host of unique tools have emerged to support individual steps in the DevOps life cycle—helping developers ensure quality by building in continuous integration and continuous testing from the start and helping speed up time-to-market with workflow automation. The DevOps ecosystem is robust, well-thought-out and always iteratively improving.

MLOps

MLOps helps teams move more quickly and efficiently through the model life cycle—a loop that involves six distinct steps, each of which contains unique considerations.

Since MLOps is related to DevOps, businesses sometimes try to adapt the processes used to write conventional software to execute the unfamiliar task of machine learning operationalization. But the approach doesn’t work, as the two disciplines are actually quite different.

MLOps primarily deals with models, not code—and models are quite different from code. For example, model training and testing look very different from code building and testing. (What does “test case” even mean for a model?) And monitoring model performance is completely different than monitoring traditional software. (DevOps looks at things like CPU utilization, latency and throughput, while MLOps looks at model quality, drift and data quality.)

Furthermore, the people that MLOps engineers are building for are quite different than their DevOps counterparts. As one might expect, the primary users of models are data scientists while DevOps generally serves software developers. Most often, MLOps engineers come from a background in ML and data science or software engineers who picked up ML.

As with DevOps, no single all-inclusive MLOps platform exists. Rather, different tools support different individual steps in the model life cycle and provide some amount of workflow automation to speed up processes. Different tools support specific pieces of the ML life cycle—but the ecosystem is still somewhat fragmented. Ideally, MLOps will evolve to offer more DevOps-like tools that work well together and offer more automation. For companies looking to build an MLOps platform, build a suite of supporting tools with speed, safety and automation in mind.

DataOps

We’ve seen very well-defined life cycles for the other two pillars discussed above; however, when it comes to DataOps, the life cycle is still being defined. But the overall objective of DataOps should sound familiar: DataOps involves processes and tools to ship high-quality data frequently—which requires a combination of data engineering, data quality, data security and data integration.

As with DevOps and MLOps, what the DataOps platform or DataOps engineer is doing is supporting the tools that help perform one of these activities and help build workflows. For example, a workflow may take data from a database, apply some transformation to it and then make it available for a business intelligence (BI) tool. That’s the kind of automation that is involved in DataOps.

As you might imagine, the skillset and the tools required for DataOps are quite different from the ones needed in MLOps and DevOps—so expecting a DevOps engineer to also be an expert in DataOps seems a bit unrealistic. While MLOps requires a much tighter integration with data than DevOps historically has required, MLOps is not the same thing as DataOps. More so than the other pillars that support intelligent applications, DataOps tends to be quite separate and distinct.

Conclusion

From perception systems to enterprise-scale fintech software, every intelligent application depends on three distinct disciplines: DevOps, MLOps and DataOps. And therein lies the complexity of developing intelligent products. Each of these fields is distinct, dealing with a different set of questions and objectives within the ML life cycle and requiring different kinds of people and tools. However, they are all fundamentally united by a common goal: Optimizing quality and speed of iteration of the ML life cycle. The tools, practices and organizations that will rise to the top are the ones that will enable more seamless collaboration between different ML teams, helping them move through the ML life cycle quickly and efficiently.

March 10, 2022
Apollo GraphQL to Provide More Control Over APIs

Apollo GraphQL today previewed an update to its Federation platform for tracking and managing application programming interfaces (APIs) based on the GraphQL query language to give organizations more control over how APIs are reviewed and ultimately approved.

Matt DeBergalis, founder and CTO, Apollo, said the alpha edition of the Federation 2 platform provides a unified view of distributed APIs based on GraphQL so they can be more easily governed.

Federation is based on a unified graph created by Apollo GraphQL to initially provide more visibility into how APIs based on GraphQL are employed within an organization. That capability makes it simpler to increase collaboration among development teams that typically build APIs in isolation from one another, noted DeBergalis.

The latest version of Federation also adds a shared ownership model that empowers teams to evolve unified graphs across smaller teams, he noted.

In general, DeBergalis noted that Federation reduces the chances developers are going to create redundant APIs because they can more easily discover existing ones. They can then either employ those APIs as-is or modify them to suit their requirements as they increasingly compose an application using tools such as Apollo GraphQL Studio versus building every capability from the ground up themselves.

Overall, Apollo GraphQL claims that Federation is currently being downloaded more than one million times a week, with more than 30% of Fortune 500 companies such as Netflix, PayPal, Zillow, Walmart, Peloton and Priceline already using its tools.

The company is also now making Federation and the rest of its tool portfolio available under the Elastic License 2.0 which provides the right to use, copy, distribute, make available and prepare derivative works of the software, but prevents organizations from offering a managed service, circumventing license key functionality or removing/obscuring features protected by license keys or remove or obscure any licensing, copyright or other notices.

GraphQL, originally created by Facebook, is gaining traction as an alternative to REST APIs because it gives developers more control over how data is accessed via an API using a set of query capabilities. That capability is especially appealing when constructing digital business processes that require access by specific classes to large volumes of data. In fact, DeBergalis notes that, in many cases, it’s the existence of a rich set of APIs that enables organizations to construct digital processes that bridge the various isolated processes organizations previously had in place.

Most IT teams will not be replacing REST APIs with GraphQL-based APIs overnight. However, there will soon be enough APIs based on GraphQL to have a significant impact on how applications are built and deployed. Many applications may wind up invoking external services via both REST and GraphQL-based APIs for many years to come.

In the meantime, the rate at which applications are being built should substantially increase as tools such as Federation make development teams more productive. The challenge many DevOps teams will soon face is managing a large volume of deployments and associated application updates as more applications are deployed in production environments.

November 3, 2021
A New DevOps Database Platform is Needed

The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

The following are examples of the capabilities required for such a new DevOps database platform.

• Automated database provisioning
• Distributed repositories support
• Simple declarative builds
• Dynamically scalable
• Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
• Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
• Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
• Many clones per virtual machine (VM)
• Security features including access management, data masking and encryption
• Compatible with and easily integrates with CI/CD platforms
• Compatible with multi-cloud (AWS, Azure, GCS)
• Built-in data security and governance into DevOps data.

I will explain these more completely in a subsequent article.

What This Means

Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

July 8, 2021

What Data Catalogs Don’t Reveal

In today’s data intensive world, data catalogs, as the name implies, are simply catalogs of information (some human-entered and some extracted from metadata in systems). The importance of data in the enterprise has grown, and analytics plays an increasingly important role in generating value. So, while it’s important to collect and store and organize all this data, data catalogs are missing a major piece of the puzzle – they fail to draw accurate relationships between data assets and how they relate to (or are calculated from) each other.

It is like having a catalog that contains a description of all the electronics components in the world, but without an understanding of how to assemble those components into a clock, a radio or even a computer. Or how to diagnose what needs replacing when the device stops working.

Data catalogs cannot answer a whole host of questions along the lines of, “What are the consequences if I make a change to this dataset?” Even if the information in the data catalog is religiously updated, if a change was, in fact, made, users impacted by that change, whether directly and indirectly, have no way of knowing that a change was made. As a result, risk of unidentified issues is high.

The table below summarizes the strengths and weaknesses of traditional data catalog products.

Issue	Data Catalog Strengths	Data Catalog Weaknesses
Data Discovery	Collect metadata information from a wide variety of systems Give users flexibility to manually update information Simplify searching for data assets across a wide variety of systems Make it easy to identify the subject matter expert (SME) related to a dataset	Rely on manually maintained documentation Do not automatically relate data sets to each other Do not understand how data assets are calculated from each other Fail to identify who uses a particular dataset (and how they use it)
Data Governance	Provide a robust framework for managing the availability, usability, integrity and security of the data	Rely on manually maintained user-defined rules for dictating data access Cannot automatically propagate access rules from one dataset to antecedent datasets
Infosec Compliance	Make it easy for users to label personally identifiable information (PII) and other infosec risk data sets	Do not automatically infer the implications of data processing on the movement of PII inside the enterprise
Data Quality	Data catalogs do not actually inspect data – so they do not assist with data quality issues	Complete failure to identify issues and communicate who those issues impact downstream from a data quality issue

Data Discovery

Data catalog solutions provide powerful frameworks for collecting metadata and facilitating documentation of datasets. The collection and aggregation of metadata is automated, and data engineers/analysts are, hopefully (but rarely), compliant in documenting their work so others can leverage it.

Unfortunately, data catalogs fail to identify how datasets and processing rely on each other. There is no way to reliably identify which users access and rely on particular datasets, both directly and indirectly.

Tasked with this objective, a data engineer can manually answer these questions by searching and analyzing the relevant data processing code. This approach uses expensive data engineering resources, takes time and results in reduced analyst and data engineering productivity.

Data Governance

When new data sets are created, the authors must tag the data to control data governance. Not only does this rely on compliance from the author, but the rules must be propagated to determine who can use the newly processed data. How should the rules be propagated? This is a complex issue. For example, the city in an address may be restricted, but aggregates across that city may not be. Defining these rules, in and of themselves, is complex — applying them is even more complex.

Reliance on data engineers and analysts to comply with data governance documentation and systems for enforcement is error-prone and expensive. Sensitive datasets are at high risk of being exposed.

Infosec Compliance

Current solutions to information security compliance rely on:

Users manually documenting if their data processing activities use (or produce) infosec-sensitive data.
Automated identification of infosec-sensitive data by inspecting the actual data and using algorithms to identify how the data needs to be secured.

Data catalogs are used to facilitate user documentation of data processing activities to comply with infosec compliance requirements. This process relies on humans manually maintaining documentation and systems (and performing actual data inspection.) These processes present significant security risks in and of themselves, because both data inspection – which itself presents a risk – and skilled manual work is required.

Data Quality

Data engineering already has processes to identify when there is an issue landing data in a table. Automated systems are available that inspect data values and make intelligent decisions (some even based on machine learning) about whether or not the data is acceptable. These systems exist outside of traditional data catalogs.

The question is, who will that problem impact? You must identify who are the immediate users of the data. You must also identify the indirect users of the poor-quality data.

The current approach to this problem is to have skilled data engineers review all data processing code and manually identify users to warn – this is both expensive and time consuming. Beyond this, the risk is that a user is not identified; what if the error impacts a report that lands on the CFO’s desk and they are not warned?

Data catalog solutions are good at solving the problem for which they were initially designed – finding datasets that are of interest to data engineers and data analysts.

The risk in DevOps is that data catalogs are relied upon for data discovery, data governance, infosec compliance and data quality issues that they fail to address in today’s complex data processing environments.

May 12, 2021

From ‘Cloud Zombies’ to Multi-Cloud Experts
A significant risk associated with cloud migration is overspending. For years, zombie-like obedience to AWS has resulted in price lock and escalating fees at many organizations – this is a tale for fighting the cloud zombies.

Cloud costs around data egress or application bandwidth are often not transparent and thus highly unpredictable. Complicating this problem is the need to rapidly develop and scale large application ecosystems, leading to snap judgments by project engineers to buy tools. Or, cloud purchasing decisions fall to the CFO, who may not be qualified to understand technical nuances. Without cloud spending forethought, it’s easy to enter unsavory pricing commitments.

However, all of this is changing. As cloud service providers (CSPs) mature their offerings, competitors have risen to take on AWS, unlocking more consumer choice throughout the cloud landscape. In a multi-cloud world, cloud economists are leveraging lift and shift capabilities to optimize performances. Ironically, companies are also seeing cost benefits in retaining some on-premises servers.

I recently spoke with Scott Leatherman, chief transformation officer at Virtana, to get his take on preventing cloud cost escalation and how IT should streamline multi-cloud environments to stem the cloud budget.

Zombies in the Cloud

When the cloud arrived, it promised high performance at a low cost. Companies were happy to offload on-premises servers for the ease of cloud arrangements. For years, there was a “zombie-like loyalty” to cloud platforms, said Leatherman.

Not much was thought about costs rising in parallel with the astronomical growth of the web, mobile and IoT. Executives penned long-term pricing contracts without a second thought. Then eventually, reality hit. And it came with a hefty price tag.

Spooky Cloud Expenses

As cloud-enabled businesses took off, they encountered many hidden fees. To name a few:
- Egress: Fees for moving data in and out of the cloud.
- Application: Fees for moving applications to new instances.
- Workloads: Mounting costs for increasing bandwidth and workloads.
These and other add-ons could quickly raise a cloud bill from $200 to $200,0000 a month. Complicating this was the fact that many large organizations purchased multi-year cloud subscriptions locked at a specific price.

Counterintuitively, such cost-savings guarantees are “completely backward,” Leatherman said. As computing power increases exponentially per Moore’s Law, servers become cheaper to maintain.

“You’ve literally committed yourself to something that is depreciating in cost,” he said. Yet, those savings don’t trickle down to consumers and SLAs have remained pretty flat. “What’s the most frustrating thing for me is that the best way to run this business is to lock people in.”

The Zombies Become Aware

In the wake of COVID-spurred digital transformation, cloud zombies are waking up to slash the budget, finding cost-effectiveness in multi-cloud strategies. Some companies are even reaffirming ties with on-premises servers.

If performance isn’t worth the cost, it defeats the benefits of moving to the cloud entirely. So, how can we smartly optimize our environments for quality performance with leaner financial margins? Leatherman shared some steps to become cloud-smart:

1. Baseline Your Footprint

The first step toward reducing cloud costs is maintaining a baseline of all cloud dependencies and private data centers. This should cover a deep cost and performance analysis.

“You need a baseline in your data center—what can I move? what should I move?” said Leatherman. Unless you have a baseline of what your data center is doing (both private and public), you can never understand what resources should move to the cloud. Powerful apps for monitoring the public cloud could help audit architecture.

2. Optimize, Optimize, Optimize

Only after mapping topologies can you can begin to optimize performance. Since usage differs depending on the season, cloud optimization should respond to yearlong application usage trends. For example, ERP software may pull significantly more requests at the end of the quarter and e-commerce spikes during the holidays.

Organizations should ensure the elastic cloud truly scales to application needs, but only when needed. Ensure systems of orchestration and routing function properly, and diligently search for hidden fees. “The cost that sneaks up on you in the elastic cloud always comes up,” said Leatherman.

Heavy computation processes, such as machine learning over big data, will require enormous server consumption. For resource-intensive scenarios, Leatherman recommended keeping code lean. Whereas CPU relies on the laws of physics, “application developers are only limited by creativity,” he said.

Application developers should whenever possible optimize code to avoid redundancies. Doing so yields a positive environmental impact, too—eliminating operational inefficiencies can decrease cloud server usage, which reduces carbon footprint.

3. Lift and Shift

GCP, Azure, Oracle and other CSPs mature, the zombie-like loyalty to AWS is fading. “Now that there is parity across cloud vendors, things are changing,” he noted. “The flexibility in the marketplace is really dynamic.”

In a multi-cloud world, organizations can leverage a hybrid cloud and “lift and shift” toward the best cost optimizations for their specific needs. This is where maintaining a strong baseline will help continually test cost efficiencies among cloud competitors.

4. Refine Your Tool Procurement Process

Another solution involves scrutinizing your cloud procurement process and the personnel involved. As mentioned above, new cloud tooling purchasing may occur at the DevOps level or large contracts may be escalated to the CFO office.

Unfortunately, neither side is in an ideal situation to make smart financial decisions—one faces an urgency to deliver and the other can’t see the trees from the forest (and especially not the termites eating the valuable wood). Many believe a new role is required—The Other CFO—to help optimize the cloud stack.

5. Retain and Optimize Private Data Centers

By now, most enterprises are accustomed to having both a data center and a cloud center team. Leatherman argued that ultimately these two teams must work together to avoid outages and reduce the exponential costs of moving to the cloud.

From a security perspective, there are still significant advantages to maintaining on-premises private clouds and data centers. By leveraging private assets, you aid data protection efforts and have increased control. “Datacenters have value,” he said. “Unless you started there, no one will be 100% cloud.”

Although cloud migration seems inevitable, through software automation and Moore’s law there is value in on-premises. As a result, Leatherman predicts a rise in private cloud data center optimization and management services by third parties.

6. AI for Cloud Cost Awareness: DevFinOps

Finally, AI soon may be available to automatically scan for cloud cost impacts before applications are even deployed.

Most DevOps teams use container security scanning or automated compliance validation as part of their DevSecOps process. Leatherman suggests future DevOps tooling will similarly evolve to display automated notices that estimate cloud costs for your actions in real-time.

For example, as engineers are pushing code, a popup could display server utilizations and a cost impact breakdown. Or, inline code validation suggestions could find redundancy to decrease waste. Perhaps we’ll need a new abbreviation for this Developer + Finance + Operation focus—DevFinOps, anyone?

Such an AI would likely not negate the need for a cloud economist to help manage multi-cloud complexity. “We’ll still need them,” said Leatherman.

Surprises in a Multi-Cloud World

COVID–19 is accelerating deployments tagged for much shorter timeframes, forcing many more instances to public clouds. Businesses need to get performance up and meet their SLAs. To reduce expenditures, they must quickly lift and shift to the most performant and cost-efficient environments.

The gravity is apparent, yet hasty moves now could be disastrous for security and data privacy in the future. Besides rising cloud costs, there are other risks associated with the cloud that increase exponentially with multi-cloud environments. “It’s not just the financial spend, but the security and risk that goes along with it,” Leatherman said. “It becomes really scary how many people have access.”

Not only must cloud spending decisions shift left, but security must arise earlier in the development cycle as well. Compliance with regulations such as GDPR or HIPPA should also influence how teams manage data between clouds.

The Cloud is Relatively New

An IBM-endorsed study found enterprises are only 20% of the way through their cloud journey. Many enterprises are still just at the beginning of their cloud transformations, and 90% plan to expand their cloud footprint in the next year.

The cloud journey is quite long, and many have quite a way to go. Knowing this, cloud economics will only rise in importance. As I recently covered, cloud expertise is now superior to a university degree.

Current forces, like the COVID–19 crisis, will no doubt continue to “drive people to the cloud,” said Leatherman. While responding to the imperative to deliver more and higher speeds, it’s essential to be smart, leverage existing assets, and optimize cloud usage at every level possible. “It shouldn’t just be cloud-first,” he said. “It should be cloud-smart.”
October 27, 2020
Strengthen API Security With These Tips and Patterns
If you haven’t noticed, digital organizations are building more and more APIs. ProgrammableWeb tracks more than 23,000 public web APIs at the time of writing, and the API market is estimated to be worth $5.1 billion by 2023. Building with APIs increases internal interoperability, reduces development time and can extend product functionality tremendously. In short, the value of APIs is rising. However, opening up with APIs brings security caveats that, if not addressed, could result in serious breaches that negate these benefits.

I recently spoke with Roey Eliyahu of Salt Security to get his take on the top security concerns for API-enabled companies, and how organizations can safeguard their services. In short, he recommends thoroughly cataloging and understanding your API inventory, designing for legal challenges and mitigating OWASP’s top 10 API security vulnerabilities.

Challenges for API Security

APIs have been called “the next frontier in cybercrime.” Rightly so, as API breaches continue to pop up nearly every day. Take the recent API vulnerabilities at Cisco Systems, Shopify, Facebook, U.S. presidential campaign apps, and GCP as evidence. The most infamous was likely the Equifax breach—not enforcing formats on incoming API calls resulted in a massive data breach, which cost the company a $700 million lawsuit.

So, why have API breaches increased recently? Well, a few years ago, there were simply fewer APIs. And, these older services exposed less sensitive data. Nowadays, companies typically have many more APIs. Banking applications, for example, might expose hundreds of highly valuable endpoints that are very appealing to hackers.

Companies also build APIs for different audiences: A 2020 study by SmartBear found 72% of companies develop both internal and external-facing APIs. These APIs could easily hold hundreds of parameters, and with every new method added, a company’s potential attack surface grows considerably.

3 Steps To Secure APIs

To mitigate the above API security issues, Eliyahu suggested three main steps: Understand your API inventory, audit against privacy policies and data privacy regulations, and track the most pressing API vulnerabilities like a hawk.

Know Your API Inventory

“You need a great inventory of APIs to know your attack surface,” said Eliyahu. Many API providers document their APIs with OpenAPI definitions. But, these files don’t always match the production realities—they may be manually generated or leave out certain methods. “In every organization, there is up to a 40% gap between APIs that are actually deployed versus what has been inventoried,” he said. By maintaining a clearer API inventory, organizations can avoid unprotected shadow IT.

Include Legal Auditing

A change to an API could result in a change in how it collects user information. Updates involving sensitive data could break privacy policies or regulations such as GDPR. “After you have an inventory, you need to map all parameters manually,” said Eliyahu. He recommended performing a legal audit against your API inventory and establishing a repeatable testing process that plugs into a continuous delivery pipeline.

As API teams release new updates, they must consider security and legal implications earlier on in the development flow. “Now, with CI/CD, these changes have to happen very frequently,” he noted.

Watch the OWASP API Security Top 10

There are many known API security gaps. For example, there may not be proper authorization in place, keys could be exposed, unexpired tokens could be hijacked and so on. Thankfully, OWASP has compiled a resource of top API security vulnerabilities, which Eliyahu and many others in the API economy have rallied behind. “The world should move to adopt the OWASP issues,” he said. These vulnerabilities are as follows:
1. Broken Object Level Authorization
2. Broken User Authentication
3. Excessive Data Exposure
4. Lack of Resources & Rate Limiting
5. Broken Function Level Authorization
6. Mass Assignment
7. Security Misconfiguration
8. Injection
9. Improper Assets Management
10. Insufficient Logging & Monitoring
Using AI To Go Beyond ‘Whac-A-Mole’

Plugging these above holes is a good start. However, these are only defensive measures against known patterns; they aren’t helping you keep guard for nuanced attack types. Ideally, security systems should stop cyberattacks before they even get started.

Yet, Eliyahu recognizes a big gap in how APIs are monitored. Organizations “don’t have proper tools to know who is looking for vulnerabilities,” he said. “Tools are typically not that advanced. They mainly look for injections and known patterns.”

Imagine a hacker probing an API. They will likely do so by trial and error, testing undocumented endpoints, sending malformed requests and so on. “They need to probe for hours or days before they find something,” said Eliyahu. If an AI could detect these odd attempts in minutes or seconds, IT could significantly reduce risk.

Security solutions should thus leverage big data and AI to create baselines of typical behavior, then deter malicious activity the millisecond any nefarious probing begins. Eliyahu said such a security AI must consider dozens of behaviors such as, How is the API being accessed? What parameters are being used? What are the relationships between parameters? What is the flow of API calls? What type of data can be exposed?

Instead of waiting for a breach to occur, an AI-enabled API security monitoring could instantly pause behavior deemed “out of the norm” by referencing a knowledge base that’s constantly refined using machine learning. Because, if you are only blocking malicious actors without knowing how you need to improve API security posture, “you are just playing Whac-A-Mole,” said Eliyahu.

DevSecOps Requires API Security Forethought

Mark O’Neil, analysis at Gartner, predicted that by 2021, 25% of organizations with public APIs will have discontinued or rebooted their public API strategy, citing security concerns as a common breaking point. APIs are now at the core of many application developments. Yet, without the proper DevSec proposition, teams risk wasted efforts and costly breaches.

To summarize, here are our tips on API security:
- Ensure everything is well-documented: API inventories cannot be audited if undocumented. Shadow APIs increase the likelihood of an attack.
- Watch the top 10 OWASP vulnerabilities: These points are foundational to API security. Though not a cure-all, they present a starting point.
- Build a security and legal review process: Ensure APIs are never deployed without the proper security review. Create validation systems that plug into continuous development processes, so new versions are never breaking privacy policies.
- Each API is unique: Another thing to keep in mind is each API is unique. Therefore, API security monitoring also requires more context into unique business logic.
- Continuously monitor your APIs: Continuous search for anyone looking for vulnerabilities.
- Remove deprecated APIs: Remove old API versions. “You need a mechanism to identify and remove old API versions.” Otherwise, “zombie APIs will bite you!”
- Learn from the attacks you block: Don’t just play Whac-A-Mole. “You need a mechanism to learn from attacks you block.”
October 6, 2020
How Logs Can Empower Your CI/CD Delivery Pipeline

Change is the currency of software success. This need to change and grow is the basis for continuous integration and continuous delivery (CI/CD). Each year, new industries embrace this philosophy of constant change, but without the necessary data underpinning their decisions, every deployment is a risk. When you enrich your software delivery with log data coupled with best practice log management, you mitigate this risk and enable rapid, safe change in your organization.

Logs are often the most common source of information about an application’s behavior. They provide context as well as result, opening up far more information than a simple count metric will. Despite the immense insight that they can offer, organizations regularly overlook them. Many leave their logs in files on servers, far away from the CI/CD pipeline that needs them. It doesn’t have to be this way; with some best practices, every part of your business can collect, analyze and utilize the hidden knowledge within your logs.

Add Life to Log Data With JSON

Logging in an unstructured format dramatically increases the complexity of detecting patterns in your logs. By logging your application output in JSON, you open up the possibility of analysis of all of your logs. This gives you a top-down view of your entire system while still maintaining readability. This empowers your CI/CD pipeline by allowing you to query and filter your logs, which then enables you to focus in on the problem and precisely diagnose any unwanted side effects of your latest change.

Create Actionable Alerts

If alerts are defined and channeled correctly, have context and can be interpreted easily, they will be actionable, add context and offer greater value. It is difficult to provide context when a single metric exceeds a threshold, but log lines are more sophisticated. They can include additional information to provide context and allow us to pinpoint the source of the issue fast. This is an essential capability for a company embracing CI/CD.

Prioritize Your Logs

Structured logs often come attached with a severity, which indicates how seriously we should investigate an event. An INFO level log signals business as usual, but an ERROR demands immediate attention. Once your logs have these tags, you can make intelligent, automated decisions that will automatically respond to unwanted changes in your system.

Benchmark Each Version to Understand ‘Normal’

“Normal” is a complicated idea when working in a microservices architecture. It may be safe to ignore a minor slowdown of a single service, but an unfortunate combination of events can lead to disaster. Benchmarking enables us to see these disasters well before our users. Benchmarking is the act of recording what the normal behavior of an application is. After each deployment from your CI/CD pipeline, compare your new behavior with your previous benchmark. If something is out of the ordinary, logs provide the information you need to act decisively. They provide an outstanding baseline signal for your benchmark.

Analyzed Logs Can Level up Your CI/CD Pipeline

Building a CI/CD pipeline is not the most difficult part. Neither is the deployment of new features. In fact, the greatest challenge facing any organization wanting to deliver their changes via a CI/CD pipeline is observability. With a best-practice approach to the preparation, curation and analysis of application and system logs, we can overcome this challenge and confidently change at a pace that propels us to the forefront of our market.

September 22, 2020
InfluxData Video Series: Dealing with Large Amounts of Data

The data deluge is becoming a daily issue, as companies collect ever more data from ever more devices and endpoints. But collecting data is one thing; deriving value from it is another.

In this video, InfluxData VP of Products Tim Hall discusses some of the considerations in dealing with large amounts of data and how a time series database can help.

“The vast majority of data systems that exist today are not really set up to ingest and analyze that data at speed,” he said, which can pose a real issue for organizations looking for actionable insights from their data in real-time. Time series databases enable ingestion and analysis instantly, with an ingest pipeline that allows users to gather and shape data to be useful upstream.

To learn more, watch the video and download the “Benefits of a Purpose-Built Platform” infographic here. Also, check out the first video in the series here, the second video here and the third video here.

— Charlene O’Hanlon

December 19, 2018
How Data Drives the Business of a Clean Energy Company
Imagine using kerosene lanterns to light your way into the evening, at home so you can cook, read, shower and the kids can do homework, or outside to keep a business or medical center open or play night games in a park or field. This is life in undeveloped nations. Not only is the light source insufficient, but burning kerosene is harmful to individual health and pollutes the air.

These issues are important to the team at BBOXX, a new generation utility company that develops clean renewable solar energy products for developing countries. Its team has developed its own “Battery Box,” a solar-powered battery box that powers lights and appliances for rural Africa—providing clean energy for numerous products and appliances including TVs, lamps and mobile phones and any other item that lights up or needs to be powered.

BBOXX describes itself as a service company, not a technology company, and as such focuses on being a data-driven company, capturing customer data, analyzing and manipulating it to provide better services and to continue to improve the battery box. Customer data is critical to its overall success: Knowing how and why customers buy and use the battery, when they use it, length of time … there are a number of data points that help BBOXX understand the customer. In addition, BBOXX needs to monitor the batteries to ensure they’re working properly, help maintain connectivity and disable units, if need be.

BBOXX’s development team captures data by remotely monitoring units. They are able to access data that is being sent from sensors via a 2G network and through the Amazon Web Services infrastructure into the cloud. Once the raw sensor data is in AWS, it is then stored in a database designed for metrics and events. From here, they are able to analyze it, provide business dashboards, generate predictive maintenance alerts and notifications and predict and find interesting business trends such as energy consumption spikes during soccer events, for example.

In addition, the team at BBOXX can also correlate and aggregate usage information, which helps them upsell additional power capacity based on upcoming events. For example, a football match between Kenya and Zambia increased demand for more energy to support extended viewing times.

Becoming a data-driven business meant that BBOXX needed access to real-time data at its core, which had another set of challenges:
- Securely monitor a distributed set of devices across low bandwidth 2G networks
- Provide real-time 24-7 monitoring to a distributed team who expects the data to always be right and available
- Provide the ability to monitor and react to high-speed time-series data
- Ability to remotely monitor and manage the distributed devices
- Ability to track usage statistics and bill based on these statistics
- Proactively monitor units so they could be repaired before the customer lost power
- Provide insight into customer usage patterns to develop compelling pricing plans
- Gather, harmonize and use three different sets of data from the various components:
  - Raw telemetry data (voltage, current, temperature)
  - Logs from the box firmware
  - Derived data (power, daily energy usages, state of charge)
- Lack of staff to manage another solution
- Keep latency systems at a minimum – their systems were already hosted in AWS and the performance was meeting the needs of the company
BBOXX is now able to gain insight into its data and apply lessons learned from analyzing past data to develop new and exciting products. The company now uses data stored in a whole new way:
- Developers analyze the data for ways to create new data streams
- Analysts look for irregularities in usage and write algorithms to detect alerts
- Business managers look for customer usage patterns
- Technicians analyze time series voltage and current patterns for remote diagnostics
- Centralized call center operators reconcile more technical issues for customers by seeing the reason for the outage (hardware failure, over-usage, non-payment)
In addition, their investors are highly interested in their data collection and analysis as a marker for competitive advantages, reduced delinquencies, growth potential and decreased risk.

BBOXX uses Amazon Web Services (AWS) to receive store and analyze its steadily increasing portfolio of products. By designing its software architecture with scalability in mind, BBOXX takes advantage of a wide range of AWS’s integrated features including load balancing, auto scaling groups and code deploy to maintain fast, reliable and consistent access both internally and externally.

Without access to user data, BBOXX would not be able to offer the unique services it does today, which is a fully vertically integrated service, controlling every part of its customer experience.

About the Author / Mark Herring

Mark is a zealous marketer who believes that the road to marketing success always leads with the developer. Before InfluxData, Mark was VP of corporate marketing and developer marketing at Hortonworks, SVP of Products at Software AG, VP of Middleware, Java and MySQL Marketing at Sun Microsystems, and VP of Marketing at Forte Software. Earlier in his career, Mark was a developer and technical support engineer for Oracle. Mark holds a B.S. Degree from the University of Witwatersrand, South Africa. Connect with him on LinkedIn and follow him on Twitter.
November 29, 2017