staging-devopsy.kinsta.cloud

Tag: multi-cloud environment

  • Building Hybrid and Multi-Cloud Architectures for Analytics and AI

    Building Hybrid and Multi-Cloud Architectures for Analytics and AI

    We are in the early stage of the data revolution. Today’s conventional wisdom states that hybrid latency prevents you from running analytic and machine learning workloads in the cloud with the data on-prem. As a result, most companies either do not burst their workloads into the cloud or have to copy their data into a cloud environment and maintain that duplicate data—and even this can take days or weeks.

    Counter to conventional wisdom, you can create a high performance hybrid/multi-cloud data platform for analytics and machine learning by using data orchestration technology. Container orchestration technology (e.g. Kubernetes) enables containers to run in any environment agnostic to the hardware that is running the application, similarly data orchestration technology enables active working sets and data to be orchestrated on demand. This enables enterprises to create a compute agnostic, storage agnostic and cloud agnostic stack to serve all their data driven workloads, including analytics (e.g. Apache Spark, Presto, Apache Hive) and AI (e.g. Tensorflow, PyTorch).

    An enterprise hybrid cloud analytics strategy should burst compute that spans on-prem and cloud data stores. By bringing the hot data to the analytics and machine learning applications, the performance is the same as having the data co-located in the cloud. Also, the on-prem data stores will have offloaded the computation and minimized the additional I/O overhead.

    Specifically, a data orchestration system should:

    • Abstract data access across storage systems. Today, data is siloed in various storage systems and deployments. For example, lots of on-premise data is stored in systems such as Apache HDFS, Ceph, EMC Isilon and IBM Cleversafe, and data in the cloud is stored in systems such as AWS S3, GCP Store, Alibaba OSS, etc. Historically, there have been many attempts to unify all data into a single storage deployment with limited success. As a result, we’ve seen more of a trend around embracing these data silos by abstracting the data in various storage systems/deployments. To do this, the data orchestration system needs to abstract and virtualize across various storage deployments/deployments by server side API translation, which enables users to run existing applications without modifying any code. The system also needs to provide a single global namespace to enable apps and users to access all the data in their environments without the need to understand which storage the data is stored in.
    • Bring data locality in a separated compute and storage environment. While cloud vendors and leading companies in data-heavy industries have adopted a decoupled compute and storage architecture, it can still be very challenging to implement this type of architecture, especially when it comes to bursting on-premises data into the public cloud. A data orchestration system resolves this challenge by providing caching functionality that enables the system to automatically orchestrate the hot data to be closer to compute while at the same time keeping cold data in a cheap storage. Because each workload may have a different usage pattern, the system must have a policy that enables customized policies best suited for those specific workloads. Moving forward, it is also very critical to have policies across various clouds to enable a multi-cloud environment.
    • Provide security for data in motion and at rest with secure access methods. A data orchestration system plays a critical role in the success of an organization’s data strategy. While open source software is important and can help in avoiding potential vendor lock-in, it is also critical to provide necessary security features in this environment given the system touches a lot of data in the organization.

    Besides empowering data driven application developers, a data orchestration system also brings value to data and infrastructure engineers. It de-risks vendor lock-in by providing organizations with flexibility at the infrastructure level. Transitioning to different storage systems (including cloud storage), adopting another application framework, or even having a hybrid or multi-cloud environment are all possible without the need for many different technologies or incurring large development costs.

    — Haoyuan Li

  • How Not to Sabotage Your Multi-Cloud Strategy

    How Not to Sabotage Your Multi-Cloud Strategy

    The immense pressure for enterprises to deliver new applications and insights that exploit new business opportunities has been a driving force for digital transformation. This, in turn, has led IT leaders to embrace a multi-cloud strategy—whether that’s deploying a combination of cloud platforms such as AWS, Azure and Google, or extending private data centers to the public cloud—to achieve greater agility, optimized performance and cost savings resulting from deploying workloads across multiple cloud platforms.

    However, maintaining multiple cloud platforms and services adds complexity and even confusion to enterprise IT environments already crowded with a range of technologies, applications and processes. While a recent Forrester study shows that enterprises are increasing investments in using multi-cloud strategies for business-critical applications and workloads, the majority of these enterprises also report that they experienced issues with deploying and using multi-cloud environments.

    The difficulty in managing IT environments made more complex by multiple cloud services involves three major issues: ignoring the importance of cultural transformation, not addressing the widening skill set gap and implementing a DIY approach to cloud management.

    Here are three ways to deal with these challenges, so you don’t end up sabotaging your multi-cloud strategy.

    Make Culture Change a Priority

    The cultural transformation required to support digital transformation and to fully take advantage of multi-cloud’s power is more than shifting the organizational mindset from capex to opex. It’s also more than picking one or a combination of the big three cloud providers and simply saying, “Let’s do this.” It requires building trust and breaking down the silos of technology knowledge—whether vendor, legacy or emerging technologies—across disparate IT teams to achieve the business goal of speedier application development and faster time to market.

    Specifically, this culture change requires better collaboration between teams and arming central IT with up-to-date processes and management platforms that are focused less on managing infrastructures and more on achieving a balance between visibility and resource control, the linchpin for IT, and creativity, the true north for developers. It will have parameters in place for provisioning resources while helping developers get the resources they need—compute, storage, network services and more—as quickly as possible.

    Culture change also enables the maturing of DevOps processes toward streamlining and bringing order to the clutter and smooth partnerships with other IT teams, especially security and operations. These include processes to eliminate bottlenecks in workflow, to advance automation and self-service, to consolidate resources and to make security integral to—rather than an afterthought in—the development process.

    Invest in Building Expertise in Multi-Cloud and Emerging Technologies

    The more cloud environments an organization has, the more skills and tools are needed to be able to deploy and optimize these environments efficiently. Despite increasing commoditization in the industry, there are still differentiators between cloud vendors and their services, and ways to leverage these to align with an organization’s specific needs. To do this, it is crucial to train the internal DevOps team or hire experts in multi-cloud environments and in key technologies for advancing multi-cloud such as containerization, microservices and serverless computing.

    Additionally, a cloud management platform that provides self-service IT and enables any user to provision, manage and orchestrate IT resources, will allow both end users and admins more time to focus on getting high-quality work done.

    Select a Cloud Management Platform that’s Right for You

    Cloud management platforms address the complexity of hybrid and multi-cloud environments by providing a central platform that allows a selection of the clouds and services that best fit an organization’s needs. They also enable IT to connect to any third-party resource, whether that’s VMware on-prem, AWS in the public cloud or container orchestrators such as Kubernetes, to gather the inventory of cloud computing resources and to provision new resources to be used by anyone in the organization from one place. Self-service IT can also be enabled when appropriate.

    In addition, by ensuring cloud management platforms are built on a single pane of glass, IT can gain a holistic view of hybrid and multi-cloud environments. It also allows effective monitoring of resource utilization across those environments, thus ensuring that cloud spend is not only reined in but also optimized.

    Finally, a cloud management platform should connect both legacy and new technologies within cloud environments. Although the complexity will not go away, it gets easier to manage. Instead of having multiple teams from different departments managing and accessing resources in separate environments, a central team handles complexity with enterprisewide visibility and control.

    The way to succeed is to embrace the complexity by configuring it and managing it behind the scenes, so developers and end users consume IT resources from one consistent way, regardless of where it comes from. Leave the complexity to the cloud and IT architects and let your end users and developers get to work.

    — Brian J. Kelly

  • HPE Rises to Multi-Cloud Management Challenge

    HPE Rises to Multi-Cloud Management Challenge

    Hewlett-Packard Enterprise (HPE) is making available a suite of tools to unify the management of public and private clouds via a software-as-service (SaaS) applications.

    Bryan Jacquot, HPE worldwide user experience architect, says HPE OneSphere provides IT organizations with a means to manage a multi-cloud IT environment using a SaaS application they can consume in the same on-demand manner they employ to deploy cloud applications on-premises or on a public cloud service.

    HPE OneSphere makes uses of application programming interfaces (APIs) to unify the management of diverse IT environments under a common software-defined framework.

    Within that management application, HPE has also provided Project workspaces for different types of individuals and teams, including IT operations, business users, developers and data scientists.

    Jacquot says HPE makes use of Docker and Kubernetes container technologies to provide the foundation for HPE OneSphere. The goal is to make it easier for IT organizations to manage any combination of hybrid cloud computing environment they choose to craft.

    To one degree or another, most IT environments today are multi-cloud. They typically employ at least one public cloud alongside an on-premises IT environment that is becoming more cloud-like. At the same time, many IT organizations either already have or are planning to add an additional public cloud service, if for no other reason than to play Amazon Web Services (AWS) against Microsoft Azure. However, each stack of public and private cloud computing requires them to master a separate management plane, which drives up the total cost of ownership (TCO) of an IT environment—which creates a challenge at a time when many IT organizations are looking for both lower costs and increased agility. Through OneSphere, however, HPE is resolving that issue by providing access to one common management portal for both on-premises IT environment and public cloud services, says Jacquot.

    There are other means to develop a common management plane spanning multiple clouds. But building, deploying and maintaining a management framework spanning multiple clouds can be a complicated endeavor. HPE OneSphere presents IT organizations with a portal through which all the underlying APIs are abstracted away from the IT operations team, making the process simpler, says Jacquot.

    It’s unclear at what rate IT organizations will move to unify the management of multiple clouds in 2018. Different parts of an organization often fund adoption of different classes of public cloud services. At the some point internal IT teams are tasked with managing those services. But the constitutional crises that drive those events might not occur until 2019 or beyond within many organizations.

    Regardless of the approach taken to managing multiple clouds, there’s no doubt that the distribution of application workloads across multiple internal and external platforms represents arguably the biggest DevOps challenge of 2018. Organizations that get in front of this issue today are likely to find managing IT to be much easier in the years ahead. Those that don’t are likely to experience more pain than they might have ever bargained.

    — Mike Vizard

  • Moving from legacy to DevOps is a journey that takes time…and the right tools

    Moving from legacy to DevOps is a journey that takes time…and the right tools

    If you wanted to start a new company tomorrow it would be easy to look at the tech landscape and the prevalent trends and decide to just build everything around a DevOps culture from the beginning. Most of the companies in the world already exist, though, and they already have processes, tools in place. Making the move from legacy to DevOps isn’t just a moment-in-time decision—it’s an arduous journey over time.

    That’s where CloudBolt comes in. I had a chance last week to speak with CloudBolt CEO Jon Mittelhauser and CloudBolt CTO Bernard Sanders about the company, its vision, and the latest version of the CloudBolt platform. Mittelhauser explained that what customers appreciate about CloudBolt is the ability to transition from the past to the future on their own schedule.

    One of the problems with change is that it takes time. While the new process or tool you’re moving to might be better in every way, getting there can be a challenge. In the meantime you’re forced to manage and maintain disparate systems or platforms—both the old and the new. CloudBolt wants to help organizations reduce the chaos by providing a single platform that enables customers to administer all of the above.

    Large enterprises in particular don’t just adopt a new technology en masse. One site or department might start to use DevOps tools and processes out of necessity. Another department might also embrace DevOps—but choose different platforms and tools. Mittenhauser explained that even if the entire organization is on the same page about which DevOps tools and processes to use the transition occurs slowly over time—often by attrition as licenses expire or platforms reach the end of their support lifespan. Those customers still need a way to view and manage their entire ecosystem throughout the slow evolution.

    CloudBolt also shared that one of its newest customers—the London Borough of Camden—has adopted CloudBolt to get better insight and control of its vast virtual environment. “At a time of austerity in government, running a low-cost, flexible hybrid cloud service for our customers is essential. CloudBolt provides a service that enables our customers to get the compute power they need, on demand, increasing their productivity and reducing our administrative overheads,” said John Jackson, Assistant Director (ICT), London Borough of Camden.

    Sanders claims that CloudBolt v5—released Wednesday, May 27—makes it easier for IT admins to manage activities, applications, and services across a multi-cloud, multi-platform environment. He described a variety of new features in v5. According to the press release the list of new features includes:

    • powerful new service catalog functionality to facilitate complex, multi-tier app deployments
    • provisioning of service catalog blueprints available through both the CloudBolt web interface and a powerful REST API
    • deeper out-of-the box integration with technologies such as VMware, OpenStack, AWS, Azure, Infoblox, Puppet and Chef
    • ability to define arbitrary server actions so DevOps experts can provide simple buttons for users to perform complex operations on servers
    • ability to snapshot VMs from the CloudBolt user interface
    • increased control for administrators, including the ability to set rate-based limits and set limits on environments in addition to groups and handlers
    • significant UI improvements, including adding social aspects to the interface
    • even deeper extensibility – additional trigger points for hooks/actions
    • revamped docs, now made public: http://docs.cloudbolt.io
    • new trigger points for power on/off, rebooting servers and post-order completion process have been introduced and currency units are no longer limited to USD

    In many ways it would be easier to just switch gears and embrace the DevOps culture throughout an organization, but with large enterprises—and even some medium or smaller organizations—that is simply unrealistic. For organizations that are maintaining multiple technologies and environments during that transition a tool like CloudBolt can help IT monitor and manage everything efficiently.

    Existing CloudBolt customer can download the latest version from the support portal.