staging-devopsy.kinsta.cloud

Category: DevOps Toolbox

  • How to Optimize Data in Software Development

    How to Optimize Data in Software Development

    Data can drive significant business value for every company. Globally, we are collecting more of it than ever before, and it keeps growing—from 45 zettabytes in 2019 to a projected 175 zettabytes by 2025, according to IDC. However, companies are struggling to monetize all the personal, behavioral, engagement and transactional data from customers and operations and turn it into actionable intelligence to drive their digital strategies. Leveraging insights from your data is critical for success and the development of new digital solutions to better serve customers, grow sales, improve operations and more.

    Data helps organizations develop the right software for the right customer at the right time. Data-driven organizations are 23 times more likely to acquire and retain customers than their peers, and are growing at an average of more than 30% annually, according to Forrester.

    Organizations with a deep and dedicated focus on data and analytics tend to follow three common paths to successfully drive their software development initiatives.

    Discovery Matters

    It is nearly impossible to draw value, understanding, and insights from the massive amount of data businesses collect without a proper discovery system in place. Having a comprehensive process of collecting data from various databases and silos and consolidating it into a single source that can be easily and instantly evaluated is critical for any software development project.

    During client engagements, data scientists perform deep-dive research into clients’ data pools and workflows to support client roadmaps. We recently partnered with a global e-commerce company to create a recommendation system for their customers. The discovery process allowed us to verify the feasibility of recommended products using their existing data before moving ahead with the project. Using machine learning, the team was able to clean and analyze information from multiple angles. With that knowledge and confidence, the rest of the development project proceeded quickly knowing the data was in place and the project was viable.

    Align KPIs

    Software development metrics help organizations and their partners track and measure how an application is performing and how effective the development team is in its work. IT organizations rely on a range of KPIs to fully understand software engineers’ progress, as well as gauge software quality including performance, engagement and user satisfaction. While each engagement is different and requires a different set of KPIs, ensuring that the organization and development team are on the same page is crucial.

    During the e-commerce project, success was measured through conversion and user retention rates. Other KPIs that are regularly used include developer productivity and performance metrics like cycle time and team velocity, and software developer metrics like response time, usability and UX data, among others.

    Deploy Agile

    By using agile methodologies at all stages of development, developers are able to focus on business results, not just architecture and new technologies. A similar agile approach can also be applied to data management. When it comes to data science, it’s all about extracting useful information from raw, structured and unstructured data and implementing machine learning models to analyze vast amounts of information. This process requires more creativity and is a key driver for understanding potential risks and failures over the course of any given project.

    With agile, data models and analyses are broken down into manageable bite-sized chunks that can be built and tested quickly—the same way software is developed. If the analytics do not yield the expected results, data scientists see the failures immediately and can correct their course. They could ask the team to analyze different data sets, adjust the models, tweak algorithms or make an educated decision about the project’s viability.

    A multinational credit company took this approach to the next level when it used agile to develop a new, big data hybrid cloud platform. After anonymizing, aggregating and storing information in a data lake, the team quickly realized the database it was planning to use was limited to a certain number of partitions and entities. By running agile sprints and developing and testing the minimum viable product (MVP) continuously, the team was able to seamlessly switch to a new database that had more scale and functionality.

    The value of data is only going to increase with time. Data-driven organizations embrace the use of data and analytics to gain unique business insights to support their software development projects and overall business. By focusing on the data in each and every development project, companies have a tremendous opportunity to gain a competitive advantage and grow their businesses.

  • F5 Networks Planning Open Source Projects Beyond NGINX

    F5 Networks Planning Open Source Projects Beyond NGINX

    F5 Networks today revealed plans to launch multiple projects in the months ahead that will extend its commitment to open source beyond the NGINX proxy software the company acquired in 2019.

    The announcement was made during an online NGINX Sprint event; two of the projects are extensions of NGINX that will address networking requirements between microservices-based applications in addition to the management of fleets of NGINX instances. The third project is a platform-agnostic open source developer tool for building modern applications that does not necessarily require NGINX proxy software.

    At the same time, F5 Networks also unveiled an open source NGINX Modern Apps Reference Architecture that gives IT teams access to a complete, fully operational microservices-based application that can be deployed in minutes on a Kubernetes cluster. Based on a modular approach made up of open source components that come pre-wired and pre-integrated, F5 Networks is committing to extending this framework in collaboration with multiple third-party vendors and organizations.

    F5

    Rob Whiteley, general manager for the NGINX product group at F5 Networks, said the company is also committing to modernizing its NGINX proxy software to leapfrog rival open source Envoy proxy software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). NGINX is still more widely employed than Envoy largely because it is lighter-weight and easier to deploy, but Whiteley conceded that Envoy has more advanced capabilities.

    Envoy, of course, is at the core of the Istio service mesh. F5 Networks supports both Istio as well as a service mesh it has developed on top of NGINX. Depending on the number of application programming interfaces (APIs) that an organization needs to support, an IT team might make use of proxy software, ingress controller or an API gateway for a service mesh to manage them.

    Finally, F5 Networks said it will participate in the Kubernetes ingress project and that it is joining the gateway API community as its implements its own NGINX-based gateway controller to ensure its proxy software and Kubernetes remain tightly aligned in the future.

    In general, Whiteley said F5 Networks also is making a commitment to be clearer about where the line between the open source and commercial editions of its offerings is drawn. As a rule, the open source editions are designed for developers and small teams while the commercial version is designed to scale within enterprise IT environments. The overall goal is to promote innovation without enabling cloud service providers to reap all the revenue benefits of F5’s research and development by creating a service based on that open source software, said Whiteley.

    As part of that effort, F5 Networks is expanding the number of open source projects it supports to make it easier for developers to contribute code beyond the core NGNIX proxy software, said Whiteley.

    It’s not clear to what degree F5 Networks will be able to rally the open source community to advance additional projects beyond NGINX. The company is promising to provide additional details at an unspecified future date. However, Whiteley said there is still plenty of room for innovations not only in emerging technologies such as service meshes but also proxy software that is already widely employed within many enterprise IT organizations. Especially since most of them are embracing microservices-based applications that will ultimately change every aspect of IT.

  • Why You Should Use GitOps to Experiment With AI

    Why You Should Use GitOps to Experiment With AI

    As the pace at which artificial intelligence (AI) models are being constructed and inevitably updated starts to increase, it’s becoming more apparent that the pace at which data science teams are currently experimenting also needs to dramatically quicken.

    Many data science teams today are fortunate if they manage to successfully deploy two AI models in a production environment over the course of 12 months. It’s already apparent, however, thanks in part to the rise of digital business transformation, that there are potentially hundreds of use cases for AI models within each enterprise IT environment. Truth be told, it’s hard to envision an application being developed today that would not incorporate one or more AI models. The challenge is, the demand for AI capabilities is now far outpacing the ability of data science teams to deliver.

    The best way to solve this issue is to recognize that AI models and the components employed to construct them are, in reality, just another kind of software artifact. In fact, the need to accelerate the development of applications using artifacts is largely a problem already solved. Rather than reinventing the wheel, the time has come to apply what we already know to the development of AI models.

    The Rise of GitOps-Based Processes

    Modern software development teams today routinely make use of Git-based repositories to share code and collaborate. These repositories are jam-packed with vetted software artifacts that development teams continuously reuse via an opinionated instance of a DevOps workflow based on a set of GitOps best practices.

    As a subset of DevOps, a GitOps workflow enables data science teams to employ a Git repository that already exists within their software development environment to manage artifacts. Git enables versions of files to be created using what is known as a Git branch. They also employ concepts known as tags (Git Tag) and remotes (where the server is located). Those three different concepts, or references, exist under the hood in Git. In fact, the entire flow is based on those concepts. When a new feature is implemented, a new branch is created and then merged into the production branch with a tag that says something like “This is a new version that needs to be deployed.”

    An AI model, like any other software artifact, needs to be updated. As new data sources become available, an AI model is likely to drift beyond the parameters originally intended. A rapid change in business conditions may also require changes to an AI model. The same processes that DevOps teams apply to update any software artifact can also be applied to rapidly update an AI model. Employing a Git repository to also manage AI artifacts extends the benefits that software development teams enjoy to data science teams.

    Often, organizations will have an MLOps solution to streamline AI model development that is completely separate from their DevOps tools. These solutions will have dedicated features like version tracking for data and experiments specifically for AI models. But many MLOps solutions exist in a closed ecosystem, meaning they don’t work off of existing tools like GitHub used on the DevOps side. 

    Also, data science teams have to separately buy, deploy, and manage these closed MLOps solutions. Going with a GitOps-oriented solution streamlines management of added MLOps tools and makes it easier for both application developers and data science teams to gain visibility into each other’s respective efforts. The end result is not just faster AI experimentation at lower cost; the likelihood that more AI models will successfully be deployed in a production environment increases dramatically. 

    Software and AI Teams Working Together

    The convergence of DevOps and MLOps is already inevitable. The longer organizations continue to employ isolated MLOps platforms to build and deploy AI models, the more expensive it will become to meld MLOps and DevOps processes. 

    Data science teams are conservative by nature. The difference between success and failure often comes down to how thoroughly an AI model is validated. However, time to market doesn’t have to be sacrificed to ensure an AI model works as promised. There is no such thing as a great AI model that arrives late when rivals are racing to develop similar AI capabilities—there are only missed opportunities. Speed enabled by the reusability of artifacts that drives faster experimentation using GitOps workflows is of the essence now more than ever.

  • Low-Code is Table Stakes for Enterprises

    Low-Code is Table Stakes for Enterprises

    Enterprises across all industries are facing mounting pressure to deliver business value with speed and agility. In the digital age, companies that can’t keep pace often fall to the wayside, especially in light of increasing customer expectations and dynamic regulatory requirements.

    Speed has become table stakes. And business leaders have risen to the challenge across the board—focusing on efficiency and streamlined operations. However, despite enabling speed, many organizations still struggle to deliver value and continuously improve performance using technology.

    Traditional software development methods can no longer keep up with the demands of modern users. And so, to achieve speed and generate significant value, business leaders must leverage customized software with low-code capabilities, thereby empowering professional IT developers to develop and deploy business applications rapidly and seamlessly.

    Speed is Table Stakes, But So is Low-Code

    Low-code application development is certainly not a new phenomenon—in fact, the term low-code was originally coined in 2014. However, since its advent, an amalgamation of digital disruptions, hyperautomation and new technologies has led to an exponentially rising demand for low-code tools.

    According to Forrester, “Low-code development platforms are emerging as a key strategy to accelerate app delivery to support digital business transformation. And they have the potential to make software development as much as 10 times faster than traditional methods.”

    So, not only is low-code an absolute imperative in today’s business environment, it is the key to unlocking speed and agility in your operations.

    Fabrizio Biscotti, research vice president at Gartner, further states that, “Globally, most large organizations will have adopted multiple low-code tools in some form by year-end 2021. In the longer term, as companies embrace the tenets of a composable enterprise, they will turn to low-code technologies that support application innovation and integration.”

    What Does Low-Code Mean?

    When you Google “low-code” you’ll find a number of different definitions. Low-code means different things to different people, depending on their industry, their business needs and their experience with IT.

    Perhaps the most common interpretation of “low-code” refers to “no-code” development, wherein citizen developers are enabled with declarative, drag-and-drop tools to create applications without having to depend on IT.

    This type of low-code works well in cases where users are attempting to streamline simple workflows and automate standardized use cases. However, when business needs become more complex, the “no-code” approach can fall short.

    A low-code platform, in the truest sense, offers a broader and deeper set of capabilities, enabling professional IT developers to use a modeling-driven environment to rapidly develop and deploy mission-critical applications and address complex, content-driven business needs.

    Generating ROI From a Low-Code Platform

    The right low code platform enables far more than just speedy app development. By investing in one comprehensive low code platform, you can:

    • Simplify your complex, content-driven business needs
    • Enable mobile app development
    • Develop customer-facing and customer support applications
    • Streamline internal workflows
    • Create transaction-oriented standalone applications
    • Develop content-driven workflow applications
    • Modernize legacy systems
    • Reinforce backend systems to handle larger volumes of data and transactions
    • Build portals for internal and external users
    • Deliver an enhanced customer journey across multiple touchpoints
    • Deploy your applications in a cloud-based or hybrid environment
    • Scale your applications with agility

    Low Code is Here to Stay

    Gartner predicts that “Low-code application building would gather more than 65% of all app development functions by the year 2024 and with about 66% of big companies using a minimum of four low-code platforms.”

    So, not only is low code a business imperative today, it is only increasing in utility and ubiquity as we move into the future.
    For enterprises striving to stay on the cutting edge, the right low-code platform is an investment that will keep on generating significant returns. From enabling speedy app development to providing the agility and scalability to keep up with dynamic business needs and customer expectations, low-code is the key to continuous innovation and ongoing success.

  • Micronaut Taps Moderne to Automate Java Framework Updates

    Micronaut Taps Moderne to Automate Java Framework Updates

    Micronaut, a provider of a framework for building applications based on microservices and serverless computing platforms, today revealed it is allying with Moderne, a provider of a tool for refactoring code, to create a “recipe” that will automatically update the Micronaut framework.

    Moderne CEO Jonathan Schneider said the open source OpenRewrite refactoring tool for code, originally developed at Netflix, makes it possible to automate updates to codebases, remediate vulnerabilities and provide a foundation for high-level building blocks that will make authoring custom transformations simpler. That approach will eventually be embraced by other providers of application development frameworks.

    Fresh from raising $4.7 million in initial funding, Moderne makes OpenRewrite available via a software-as-a-service (SaaS) platform to enable IT teams to search and transform millions of lines of code in minutes. OpenRewrite currently supports Java, but the company is working toward adding support for additional languages, including infrastructure-as-code (IaC) tools such as Kubernetes, YAML and XML.

    Moderne has also developed “recipes” for common search and refactoring operations that can be assembled into more complex recipes for framework migrations involving, for example, an upgrade from Java 8 to Java 11. Micronaut is the first provider of a development framework to create a recipe for its own framework using OpenRewrite.

    At a time when applications are more distributed than ever, updating codebases is a major challenge. In fact, the manual effort involved in making those migrations is one of the primary reasons many organizations continue to employ older versions of frameworks such as Java 8. OpenRewrite is designed to automate that process at scale in a way that leaves it up to the individual developer to determine how involved they want or be or if they simply prefer to have a DevOps team manage the upgrade process on their behalf, said Schneider.

    In the longer term, Schneider said he expects recipes will be created to address everything from mass migrations to remediating specific types of vulnerabilities across an entire codebase.

    Too much reliance on legacy development frameworks generally stymies innovation. The latest versions of Java, for example, are designed to make it simpler for developers to build microservices-based applications.

    It remains to be seen to what degree DevOps teams will embrace automation to upgrade code. Many developers are hesitant to rely on automated tools of any kind. However, given the need for most developers to spend more time writing new code, it’s only a matter of time before most of them come to terms with automated approaches to updating existing code.

    In the meantime, the cost of supporting legacy frameworks should decline as organizations begin to migrate their codebases faster. One of the reasons there is so much technical debt in the enterprise is because different development teams tend to adopt different framework versions at different times. DevOps teams can find themselves being asked to simultaneously support multiple versions of the same framework. There may always be multiple versions of those frameworks, but, over time, DevOps teams should eventually see the bulk of deployed applications all using the same framework as the upgrade cycle becomes more automated.

  • The Future of IT: Smart Tools, Intelligent Technology

    The Future of IT: Smart Tools, Intelligent Technology

    In the last few years, the lines between information technology and development have been blurred. Developers need to know about infrastructure, and administrators need to know how to code. Advances like containerization, orchestration, continuous integration and infrastructure-as-code (IaC) happened in tandem with DevOps’ rise. The advantage of DevOps is that you can quickly build and deploy products. That comes at a cost, though, and that cost is infrastructure complexity. The answer to managing that complexity lies in technologies such as artificial intelligence (AI) and machine learning (ML).

    How can AI Help IT?

    Intelligent technology has come a long way in recent years. Machine learning and artificial intelligence are finding their way into more and more technology. New tools are beginning to emerge that can monitor and manage complex systems with relative ease. There are several clear upsides to using these systems in IT:

    • Instant notifications when there is a problem.
    • The automation of tedious IT tasks.
    • Dynamic infrastructure adjustments to help with traffic spikes and other changes.

    Training an AI to keep systems running optimally, apply updates and alert IT staff to problems in real-time is entirely possible today. By automating away a lot of tedious, manual work like digging through logs, IT is free to focus on mission-critical objectives instead of babysitting servers and monitoring dashboards.

    What Does That Mean for IT Jobs?

    Some people worry that advances in AI will lead to a world where robots take human jobs. While there will undoubtedly be some domains in which AI and ML make more sense than a person, those same technologies will also create new jobs. As IT teams are unburdened from tedious tasks, they can expand into working with ML and AI technology. IT teams are ideal for guiding business decisions about what tasks benefit the most from intelligence technology. So while IT jobs may be transformed by AI, there is no reason to believe that they will disappear. It is more helpful to think of AI as a new tool for IT instead.

    Where can IT Implement AI Tools?

    AI is best used for tasks where there is a lot of data. Having a human comb through log files, databases and dashboards is a slow process. An AI could continuously monitor everything, all the time, freeing up the IT staff to focus on the solution instead of tracking down which piece of the infrastructure is experiencing a problem.

    One area in particular that could benefit from AI/ML is fraud detection. Machine learning can analyze data to find fraudulent transactions much faster than a person ever could. And over time, it will get faster and more accurate at detection.

    Another area with potential for AI is the customer-facing helpdesk. Intelligent systems that have access to ticket resolution histories and device data may be able to handle a majority of requests without needing to use the time of an actual technician. If the AI fails to solve the problem, it can escalate the issue to a technician. As these tools mature, they’re impacting customer service processes, too; let the AI go as far as it can and then escalate less common and/or more complicated issues to a technician. This is an excellent example of how AI and IT can work together to provide the best possible outcome.

    AI and ML tools are the future for IT teams. These tools will be able to handle a lot of the tedious work that is currently done by people such as parsing logs, monitoring traffic and providing some measure of technical support. That leaves IT to focus on mission-critical issues and problems that AI and ML  cannot easily solve.

  • Foundation Proposes Advancing eBPF Adoption Across Multiple OSes

    Foundation Proposes Advancing eBPF Adoption Across Multiple OSes

    The eBPF Foundation, which proposes to advance adoption of an approach that enables sandbox programs to run faster at the kernel level, is being launched today as an arm of the Linux Foundation.

    The technology known as eBPF was originally developed for Linux. The eBPF Foundation is now committed to expanding use of eBPF across all operating systems. Members of the eBPF Foundation include Facebook, Google, Isovalent, Microsoft and Netflix.

    Thomas Graf, CTO and co-founder of Isovalent, a provider of networking and security tools that leverage eBPF, said IT organizations should expect to see the technology become part of Windows, along with other operating systems, as the core technology is tested and validated across a much wider range of platforms.

    In effect, eBPF changes the way operating systems are designed. It bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that, historically, were completely independent of one another. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput to improve overall scale at a time when the volume of cybersecurity attacks being simultaneously launched continues to increase.

    Currently, eBPF is widely employed by web-scale companies such as cloud service providers. Facebook is using it as the primary software-defined load balancer in its data centers while Google is using open source Cilium networking software within its managed Kubernetes offerings.

    However, adoption within on-premises IT environments that have deployed Linux has been more limited simply because the number of vendors that have optimized networking, security and storage offerings to take advantage of eBPF is still fairly limited.

    Sysdig, for example, recently launched made available an instance of the open source Falco container security platform available that employs eBPF. Tigera has also made available an instance of its container networking platform that leverages eBPF at the kernel level of Linux.

    Ultimately, Graf notes the biggest benefit of eBPF is efficiency. The total cost of processing for security, networking and storage platforms should decline as more vendors take advantage of a capability that will one day be widely employed across multiple operating systems.

    In the meantime, IT organizations would be well-advised to ask their vendors when they plan to support eBPF. Cloud service providers generally require it to deliver their own managed services more efficiently, so any vendor that hopes to sell a platform to those providers will need to support eBPF to drive performance at scale. The issue then becomes determining which platforms that have embraced the technology are also available in on-premises IT environments looking for similar benefits.

    Regardless of approach, it’s clear there will soon be a reason to upgrade not just networking, storage and security platforms but also instances of operating systems that don’t yet support eBPF. DevOps teams would be well-advised to plan accordingly because the number of platforms that will ultimately benefit from eBPF spans the entire enterprise.

  • Choosing an Incident Management Platform

    Choosing an Incident Management Platform

    When you’re feeling the stress and pain of manually managing incidents and incident response, making the decision to find an incident management tool is a no-brainer. But how do you choose the one that will work best for you, your team and your business?

    You might be asking yourself, “Where do I start? What do I need to know? What questions do I ask? What are the options? How can I be sure we’re choosing the right tool?”

    To help identify the most appropriate solution for your organization, here are the top four things to consider.

    1. Do you have the culture and processes you need to support an incident management solution?

    Different business models require different approaches to incident management. You should be looking for a platform to fit the unique needs of your organization. More importantly, even if you find a solution that seems like a fit, do you have the people, culture and processes necessary to support a new tool? And if not, do you have a plan to put them in place?

    Implementing solutions successfully takes planning, so you’ll want to make sure that you and your team can handle the workload.

    2. What are your pain points and goals?

    You’re looking for an incident management solution for a reason, but do you know exactly which pain points you hope to solve? Your top goal may be to “make it easier,” but every solution is likely to tout that they make things easier. Determine what you’re hoping to accomplish. Do you want to automate processes? Bring your team together more quickly? Track every action from declaration to retro? All of the above? If you prioritize the problems you want to solve, it’ll be easier to check off boxes when you investigate your top choices.

    3. Who should be involved in this search?

    You may be the one taking the lead on searching for a better way to manage incidents, but knowing who else should be part of the decision-making process will make the search more efficient. You’ll need to know if you have support from your team and from leadership; who’ll authorize the budget for your new tool and who has to sign off on your decision.

    And don’t forget other stakeholders outside of your team! Incidents impact all parts of an organization, so consider who needs to be a part of the selection process—from your client success team to your sales team.

    4. How will you make a decision?

    Finding and implementing an incident management solution requires time and resources. It’s especially important to account for the time it will take to onboard your team and configure your solution, including assigning individual roles, setting up integrations with your existing tools and test-driving your incident management process.

    The worst-case scenario is that you purchase a platform but don’t have time to implement and use it. You won’t be happy, nor will your team or executive stakeholders.

    So have a clear idea of what you need to make a final decision—whether it’s considering who’s involved, how much budget you need to secure, what your timeline is or how you’ll address any legal or security issues.

  • Google Unveils Tool to Better Secure GitHub Repos

    Google Unveils Tool to Better Secure GitHub Repos

    Google today launched a GitHub app that provides automated continuous enforcement of security best practices for GitHub projects.

    Kim Lewandowski, a product manager for open source software security at Google, said the Allstar application enables IT teams to assess any project on GitHub to check for security policy adherence. In addition, Allstar sets desired enforcement actions and automatically applies those rules when triggered by a setting or file change in a repository.

    The goal is to provide the open source community with a tool that makes it possible for organizations to have more confidence in the open source software that is being employed within a software supply chain, said Lewandowski.

    Allstar is intended as a companion application for Security Scorecards, a tool that Google made available last year to assess whether, for example, an open source project employs branch protection to ensure that malware isn’t inadvertently committed to a project. Allstar continuously checks expected GitHub API states and repository file contents against security policies and the enforcement actions defined by an IT organization. After it detects a policy violation, the tool can be configured to simply send an alert or to automatically enforce a specific policy to remediate the issue, said Lewandoski. Specifically, she said the options available are to log the security policy violation without taking any additional action, open a GitHub issue or modify the GitHub setting to match the original Allstar configuration.

    At present, Allstar provides a limited number of security policy checks, but more will be made available over the coming months, said Lewandowski. Current security policy checks include branch protection, vulnerability disclosures, access controls and detection of binary artifacts that can’t be scanned. Additional checks that will be added include automatic dependency updates and frozen dependencies.

    Lewandowksi said Allstar and Security Scorecards are part of an ongoing Google effort to give back to the open source community that is being targeted by cybercriminals attempting to compromise software supply chains. Just about every application is now dependent on open source components, to some degree. The maintainers of those projects, however, don’t always have the tools or expertise required to check for malware that has been injected into a codebase, she noted.

    In general, Lewandoski noted that achieving DevSecOps best practices will require a lot more automation of application security. It’s not possible for each developer to become a world-class cybersecurity expert. The challenge is finding a way to introduce that automation into the application development process as early as possible, which, she noted, in many cases starts with open source components that are incorporated in applications.

    It may take a while for every maintainer of an open source project to revamp the processes through which code is added to their project. However, the easier it becomes to continuously review that software, the more confidence organizations that use it will have in its security. In fact, the maintainers of those projects should expect the downstream users of that software to be asking some pointed questions about how security is being managed in the wake of a series of recent high-profile breaches of software supply chains.

  • DataOps Vs. DevOps: What’s the Difference?

    DataOps Vs. DevOps: What’s the Difference?

    There is a mindboggling amount of data today; to even measure it requires using a byte measurement called a zettabyte, which is one sextillion bytes (that’s 21 zeros). Currently, because such a ridiculous amount of data exists, there is a growing urgency to end wasteful data processes. From this environment, DataOps was born. Similar to the way that enterprises adopted DevOps to formalize and streamline wasteful development practices in the past, today, many large organizations turn to DataOps to formalize modern data management practices.

    What, Exactly, is DataOps?

    Primarily, many companies are adopting these principles because they are either trying to avoid or rectify data debt, which is the amount of money required to fix data problems due to the mismanagement of data processes. Data debt is a strong motivator for revamping outdated processes and policies, particularly when decision-makers and stakeholders require metrics before implementing change. Unpaid data debt can be detrimental to a business; the longer it remains unpaid, the more it costs to maintain a data landscape.

    By implementing DataOps principles and data governance, an organization can effectively reduce its data debt and prevent it from growing any larger. Moreover, DataOps practices and software engineering can be used to detect inefficiencies, minimize knowledge loss and capitalize on missed opportunities related to data usage.

    Similarities Between DataOps and DevOps

    Much of the processes that enable DataOps were borrowed initially from the same foundations that built DevOps. Likewise, just as companies need DevOps to provide a high-quality, consistent framework for software and feature development, data enterprises also rely on these same features to realize rapid data engineering and analytics development. For organizations that already have a DevOps framework in place, leveraging DataOps is relativity straightforward. Several important DevOps concepts adopted by DataOps include:

    • Agile development
    • Focus on delivering business value
    • Continuous integration and continuous delivery (CI/CD)
    • Automated testing and code promotion
    • Reuse and automation

    The Differences

    Despite the similarities between the underpinnings of DevOps and DataOps, there are several major differences.

    The human factor: The people using DataOps and DevOps have divergent personalities and skillsets. DataOps participants may be tech-savvy, but often their knowledge is theoretical. DataOps professionals can include data engineers, data scientists and analysts who focus on creating models and visual aids. DevOps, however, was made for software developers and engineers–coding is in their DNA.

    The process: The life cycles of DataOps and DevOps do share similar interactive properties. But, the former deviates in that it consists of a data pipeline and an analytics development process, both active and intersecting. While conceptually, the pipelines of DataOps resemble the development processes of DevOps, typically, experts note that the DataOps process is more challenging.

    Orchestration: In the DevOps process, application code does not require complex orchestration. However, for DataOps, both the data pipeline and analytics development orchestration is an essential component. Although orchestration in the DataOps pipelines occurs frequently and drives data flows, there is usually no such coordination of pipelines in application development and DevOps processes.

    Testing: Again, the two pipelines of DataOps create a significant difference from DevOps; testing in DataOps occurs during both the data pipeline and the analytic development process. These tests attempt to catch anomalies, flag abnormal data values and–unlike DevOps–validate new analytics before deployment. Likewise, these tests get embedded into a data quality framework for continual monitoring.

    Test data management: In most DevOps environments, test data management hardly takes priority; with DataOps, it’s vital to accelerate analytics development so that innovation keeps pace with agile iterations.

    Tools: DevOps is the ‘father’ of DataOps, and as such, the tools needed to support the latter are still in their infancy. While testing in DevOps is primarily automated, DataOps doesn’t have the same luxury – most users modify testing automation tools or build their own from scratch.

    Exploratory environment management: Generally, data teams use more tools than software development teams. Moreover, exploratory environments in data analytics are more challenging from a tools and data perspective; data teams also naturally depart from data islands across the enterprise.

    While foundationally, the concepts of DevOps serve as a starting point for DataOps, the latter involves additional considerations to maximize efficiency when operating data and analytical products. Nevertheless, both serve their intended audiences, reducing data debt and evolving data products or shortening systems development life cycles or providing continuous delivery. For businesses looking to make internal data-related processes more efficient, they should start by examining best practices associated with DevOps.