Category: Enterprise DevOps

  • SREs: The Happiest – and Highest Paid – in the Industry

    SREs: The Happiest – and Highest Paid – in the Industry

    According to the 2019 StackOverflow survey completed by 90,000 developers around the world, whether you are already a Site Reliability Engineer (SRE) or thinking about becoming one, looks like you are onto something.

    SREs, also called DevOps specialists, are a pretty unique type of talent; one that has knowledge in both software development and system infrastructure. And because the world of infrastructure is so vast, these professionals also need to juggle networking, security, system administration, hardware and anything else that could make an infrastructure unstable or down (yes, I am looking at you, A/C). That makes any day at work unique.

    SREs, for the most part, are in charge of making sure that your favorite website apps are loading, and fast. Their work is not customer-facing but highly critical. According to Gartner, the average cost of IT downtime is going anywhere from $5,600 per minute to—when you work for Amazon—$2 million for every minute down.

    That’s one of the reasons why SREs are among the highest paid in the industry, well, right behind their fellow managers! They also rank among the ones with the most coding experience—it takes time and hard work to get there.

    And all of this leads to happiness! With an interesting job and high pay, SREs satisfaction is among the highest in the industry, and they rank in the top three of the ones not actively looking for a job. SRE for a day, SRE forever!

    But here’s the problem: There aren’t a lot of SREs out there and hardly any computer science programs cover the topic. A recent tweet from Target software engineer Jessica Schalz illustrates the point very well:

    As TechRepublic reported, these professionals are among the most wanted in the tech industry and 33% of IT leaders report having a hard time hiring profiles with these skills. A potential reason for this lack of talent might come from the fact that the profession is failing at attracting about half of our population: women. SREs are 30 times more likely to be men than women. Training non-bearded SRE talent is possible. The Holberton School, a coding school I co-founded where 40% of the students are women, is leading a lot of students to be hired as SREs.

    And the good news is that we are not alone: CloudFoundry and Infrastructure Masons are also taking action to move the needle. And industry leaders such as Tammy Bütow and Abby Kearns are showing the way.

    As former software architect Eduards Sizovs pegged it: Great developers are raised, not hired. The industry is spending tons of money and energy and trying to attract the best talent when it would often be more efficient to take juniors that can grow to their need.

    “There is a new generation of developers of all races, genders, backgrounds coming up who can do these jobs,” said Max Johnson, DevOps engineer at Pypestream. “Businesses should widen their hiring scope and consider non-traditional sources that train as good, if not better, talent and coach them further for these highly skilled jobs.”

    LinkedIn is among the first to understand the value of investing in people—so it’s probably not a coincidence that it’s among the best places to work in the U.S. From an internal, employee-led perspective, the company has developed a number of in-house programs for both career and technical development. For example, its Women in Tech (WIT) organization leads a number of programs to connect, inspire and grow women technologists. From an external standpoint, with its REACH engineering apprenticeship program, LinkedIn is redefining the way it identifies, recruits and trains talent. REACH is an apprenticeship program that creates opportunity for every individual with the passion and potential to either develop or restart their career in engineering. The program guides participants through various levels of technical roles at the company, with the end goal being that each individual builds their skill set and gains the experience needed to become an engineer.

    “Reinventing our approach to recruiting and developing junior talent has been a great way to expand our funnel and has brought forward more diverse perspectives, forming a more robust team,” said Casey Brown, senior manager, SRE at LinkedIn and core team member of the REACH program.

    As you have probably understood, SREs are a niche talent (43% of them even picked Vim as their favorite editor). But as I recently tweeted, they are ruling the world, and more of them are needed. Let’s get creative to train and hire more of these amazing professionals!

    — Sylvain Kalache

  • Different Organizations, Different DevOps Outcomes

    Different Organizations, Different DevOps Outcomes

    The main motivation behind inventing DevOps platform was to help close collaboration between developers and IT operations. Basically, DevOps emerged as a great platform to bring closer together developers and beneficiaries of the app development process. DevOps initiates a cross-disciplinary practice spanning across development and operations.

    The ultimate benefit of the DevOps approach is removing the barrier between development and deployment of software applications in specific business processes. Regardless, not all businesses reap the benefits of DevOps platform.

    Does DevOps really suit all businesses of different sizes and niches? What are the key considerations for utilizing DevOps platform? These are the questions we will try to answer through this article.

    Do All Businesses Require DevOps?

    The common and widespread consensus is that any software development company geared for innovation must embrace a DevOps environment. Any business that considers innovation and smooth deployment in the software development process—a key factor in their long-term growth strategy—should take advantage of the benefits of a DevOps platform. Whenever any service provider needs to adopt new strategies to keep pace with the evolution in the software space, DevOps will come as an irreplaceable platform choice.

    Despite this consensus and widely accepted perception, there are different opinions in the market. Many industry experts believe that to reap the benefit of the platform you need to have a dedicated DevOps team in place. If you have a DevOps culture and in-house DevOps team in your company, you can gain a competitive advantage.

    The Key Benefits of DevOps

    The main benefit of DevOps is faster delivery of software with much-improved quality. There can be several other benefits depending on the industry.

    While some companies refer to the huge reduction in downtime since they adopted DevOps culture, there are still many others who simply give credit to the platform because of its integrated approach that makes the process more agile and fast-paced.

    For most companies, the domino effect of getting better software at a faster speed ends up delivering sustainable growth and revenue opportunities.

    Here’s a look at the role of DevOps platform across three types of businesses: small companies and startups, midsize companies and large business houses.

    Small Companies and Startups

    In smaller companies and startups where resources and human capital remain under-supplied, the responsibilities ascribed to developers, engineers and deployment professionals often tend to get blurred or overlap. Because of the scarcity of human resource, developers and engineers are often assigned multiple responsibilities. Naturally, in such companies, it is common to see cross-functional teams ready to engage into various development projects with ease. This makes them ideal business organizations to embrace and grow with DevOps culture.

    Midsize Companies

    As the companies tend to grow bigger in size and resources, they have a bigger (and often unbreachable) divide between verticals such as operations and development. In such organizations, staff are less likely to be open to cross-functional responsibilities, as many have been in the same job doing the same tasks and feel settled. This brings in a stagnant organizational process, where strong departmental divide rules the business process outcome.

    This stagnant work culture makes such companies less appropriate candidates to implement DevOps environment. For these companies to reap the benefits of a DevOps platform, mass organizational changes should be brought in the team structure. Building a separate cross-functional DevOps team to handle some focus projects can also be good practice for them.

    Large Companies

    Large companies enjoy the luxury and abundance of resources, and this makes them much better-placed compared to other organizations. Companies such as Amazon, Facebook, Google and Microsoft have the diversity, huge experience and robust abilities to bring together cross-functional specialists from development and operations to work on ambitious projects. These organizations are known for having product-oriented units, comprising specialists from both development and operations.

    Apart from their cross-functional expertise, product-focused approach and abundance of resources to roll on new projects, large companies also can make a big difference to the teamwork and motivation through a highly unified vision and strategic long-term objectives shared by every member of each team. This is why large companies always stand as suitable candidates to embrace DevOps approach and environment for their projects.

    Fixing the Right DevOps Strategy: 3 Things to Consider

    Beyond the organizational differences and core strengths and weakness for embracing a cross-functional DevOps approach, we must look into the key aspects that make a right strategy for any organization to incorporate DevOps into their scheme of things. Question is, despite the shortcomings and lacking, how can you embrace DevOps environment and reap the benefits? Here we provide some clues:

    • Making Change a Habit. Some strategists believe that any organizational change will face a lot of friction, causing substantial harm to the immediate business outcome. But there should be no reluctance to change for this. Rather, by bringing constant changes, people get used to the friction and pain and it becomes normal. This is one way to replace traditional IT mindset with DevOps.
    • Starting Small with a Neat Team. Obviously, finding the appropriate and credible skilled employee for your DevOps projects is not always easy. This is why it is advisable to create small startups or units, in which sharing the common vision and embracing a cross-functional approach becomes the order of the day. On top of that, do your best to bring out the best from your team and hire some experienced ones with knowledge of DevOps.
    • Continuous Development Mindset. We all know how the DevOps platform always focuses on continuous development approach. It is easier for the software developers to embrace such approach, but for operations staff, you have to slowly make them think of infrastructure as evolving and slowly building components. That’s something challenging but not impossible.

    Conclusion

    DevOps has emerged as the most promising development approach for modern software developers. Though for many development companies the friction emanating from strict role divides is becoming an obstacle, the positive push by DevOps for cross-functional approach looks indomitable.

    — Atman Rathod

  • 5 DevOps Trends Worth Knowing in 2019

    5 DevOps Trends Worth Knowing in 2019

    DevOps today is more than the buzzword it was 10 years ago. Back then, the idea of combining development with production in your IT infrastructure was not common; the logistics were nonexistent and most of the time businesses decided it was far too much effort.

    However, if the last nine years has taught us anything, it’s that DevOps is worth every penny businesses invest. It helps companies streamline production, keep customers happy and provide agile services to attract new audiences.

    But as with anything in the IT field, there is a paradigm of change. This article discusses some of the top DevOps trends happening now and how to implement them in your own system.

    Automation Will Be a Bigger Focus

    Automation has always been the way for businesses. Whether it’s speeding up the pipeline with machines, providing better response times for angry customers with chatbots or better inventory management with tracking and logging system, automation is behind it all.

    DevOps is no exception.

    But as with everything, there can be too much automation. The best thing to do is break down processes and find simple, redundant tasks that can save time if automated and apply the changes.

    The most important aspect of automating DevOps is to understand the six pillars in the DevOps cycle, which are:

    • Continuous business planning
    • Collaborative development
    • Continuous testing
    • Continuous release and deployment
    • Continuous monitoring
    • Collaborative customer feedback and optimization

    DevOps Assembly Lines: The Next Big Thing

     

    To track the development of your application, we have pipelines. These visually represent the product from its initial stage right through until final development and release.

    However, continuous integration (CI) is no longer the priority focus when it comes to effective monitoring and implementation. Continuous delivery and automation are a major part that businesses have already started delving into in 2019.

    This is where DevOps assembly lines enter the picture.

    Assembly lines are focused on bridging the gap between manual and automated tasks, ensuring there is no redundancy while providing several tools for teams to implement, to ensure apps are hitting the production lines as reasonably quickly as possible.

    Microservices Architecture Adoption Taking Off

     

    Microservices architecture may as well be that new kid in school trying to fit in. These smaller services compound into agile, implementable tools that DevOps teams can use to scale and test software over the course of weeks—not years.

    As you can imagine, this can help some businesses streamline their development and production lines, setting them way ahead of their original production estimates—which is never a bad thing.

    Implementing microservices into your DevOps team can also help companies focus on the runtime and efficient delivery of new development sources, without the hassle of creating dependency errors when something decides to bug out for no apparent reason.

    Security Needs to Step Up

    Cybercrime is on the rise—we all know that’s old news. But as a business, cybercrime can either make or break your sales and weigh down your DevOps team.

    That’s why more companies are hopping onto enterprise security options. Although there are third-party programmers for almost every situation you need, none of them have the same security levels as the tools from the enterprise version provides. Safeguarding both your company and your clientele.

    Riding the Artificial Intelligence Wave

    Finally, we have artificial intelligence (AI) and machine learning (ML). When you think about it, these two elements are literally the sum of everything we want in the DevOps community—the ability to crunch out hours upon hours of menial labor and saving your teams hours to focus on higher-priority tasks.

    The combination of AI, used to predict algorithmic influencers, and ML’s ability to adapt to new situations will be key for DevOps teams to get to the next level in 2019.

    — Gil Artmoore

  • Digital Transformation: Coming from Behind

    Digital Transformation: Coming from Behind

    For many years, enterprises have been in a race to digitally transform how they work through the digitization of all areas of their business, to reduce costs and business frictions through creating efficiencies and delivering increased value to their customers. While many organizations have attempted a transformation, few have been successful.

    Research from the consultancy McKinsey & Company finds that less than one-third of transformations are successful.

    How do organizations that find themselves today, perhaps after several failed transformations starts or having procrastinated getting started, get themselves back on track before they find themselves too far behind the competition to ever hope to catch up? We reached out to several experts to find out.

    First, many of those advised organizations that currently find themselves digital laggards should not rush into a digital transformation out of fear of missing out. “Don’t pursue digital transformation with a FOMO mindset,” said Andy Lin, senior vice president of strategy and consulting at Nexient, a software services provider. “Approach it with clear, measurable business results in mind, such as increased market share or increased revenue. Focus on how the end state of your transformation—business capabilities, technical capabilities, digital experience capabilities and similar—will make the biggest impact, and prioritize accordingly,” he said.

    Research from business consultancy West Monroe Partners, which ranked respondents by maturity, found 21% of respondents are digitally mature and 17.9% as very digitally mature. There remains a sizable percentage, 35.9%, who are either not digitally mature at all or slightly digitally mature.

    Trailing in digital maturity likely means those organizations find themselves feeling the pressure from some constituents: shareholders, customers, competitors and others, said Justin Rodenbostel, vice president of delivery services at digital transformation agency SPR. “If pressure is coming from lagging customer engagement, then one should become more familiar with customer desires through research or analyze how other industries are innovating and transforming,” he said. “Pressure from competition might require competitive analysis, market research or research into applications of potentially disruptive technologies,” Rodenbostel advises those who are trying to identify where they should focus on their catch-up efforts.

    Scrum.org CEO and product owner Dave West agreed: “Enterprises embarking on a digital transformation should look at their business strategy, their customer base and start with that; building teams that align to the customer, empowering them to drive technology and business decisions to make that customer successful.”

    Also, when they set a course, enterprises need to be sure to set goals to be set to such business outcomes as well, said Nick Polachek, senior principal strategy for Maven Wave. Instead of prioritizing technical results, such as building an enterprise DevOps pipeline, it’s better to focus on results associated with business-driven outcomes, such as improved customer experience. “To prioritize effectively, you need to understand business drivers and divide technology initiatives into attainable slices that create real business value and quick wins,” he said.

    West Monroe found many attributes—from the use of digital in sales to leveraging data—that organizations should prioritize based on their digital maturity. “An organization can fast-track its maturity by prioritizing certain characteristics over others, regardless of industry. For instance, those with the lowest maturity should prioritize innovation that addresses customer pain points and those with the highest maturity should prioritize best-in-class cybersecurity. More accelerators can be found in our research,” said Kyle Hutchins, senior director at West Monroe Partners.

    Finally, as the organization attempts to increase its digital transformation maturity, others advise organizations prioritize their digital transformation efforts that will get widespread support throughout the enterprise—which usually translates to initiatives that will increase sales. “Since most organizations are designed to support sales efforts, prioritizing transformation within the sales organization is an effective way to motivate the organization to transform,” said Craig Besnoy, program director and digital strategy consultant at Mindtree, IT services provider.

    Still, and this could be why so many digital transformations fail, a sizable percentage of organizations—41%, according to research and strategic consulting firm Altimeter—make investments in digital transformation without ever seeking the guidance from thorough customer research.

    — George V. Hulme

  • The 5 Core Principles of Lean Management

    The 5 Core Principles of Lean Management

    Lean project management is rooted in cost-saving manufacturing methodologies. The ultimate goal of lean methodology is to reduce waste. Companies in all industries have adopted lean principles into their own processes, but lean is most suited toward physical manufacturing, engineering and related industries.

    Regardless of your company’s size or the activities of your organization, lean project management principles can be extraordinarily helpful in optimizing your everyday operations.

    Below are the five core principles that guide lean thinking for project managers.

    Identify Value

    Defining the value of a given product is crucial. Before beginning any manufacturing project, or any project for that matter, the amount of value a finished product will generate must be assessed accurately. How much customers expect from a given product must also be ironed out.

    Value questions that lean teams will ask:

    • What do customers want?
    • How will they benefit from our product?
    • How much are they willing to pay for the benefits of our product?

    Lean teams can, for example, estimate how much customers are willing to pay based on related data points, then set costs low enough to reach an agreeable price. The goal is to eliminate all waste in every process. Any area where value is not added—or worse, cost is exceeding value—must be addressed early on using lean management principles.

    Map the Value Stream

    The value is the end goal of the project. Once you’ve established value as a definite endpoint, you can begin mapping the way to get there (or value stream). Every single process and every single step involved in getting to the end product is part of the value stream. From gathering the materials needed for production to coordinating efforts with PR, every related process is represented in the value stream.

    Kanban is one of the most popular ways to map value streams. Tasks are broken down and are posted in a “just-in-time” fashion. That way, each task is taken one at a time and only done when it is absolutely necessary to complete.

    Using Kanban flowcharts and other diagrams, managers can set up visual aids for value streams, mapping every single step onto a single page. By mapping the flow of activities, lean project managers can easily identify waste and inefficiencies with the process, allowing them to ameliorate these issues swiftly.

    Create Flow

    By mapping the value stream, project managers can spot waste. Any tasks that are done too early, too late or too often creates waste. By analyzing the map and removing waste, lean project managers can create the optimal flow of activities.

    Lean project managers check for possible bottlenecks and delays. If there are any inefficiencies present, tasks may have to be reordered. Sometimes, project managers may even decide to append value-adding tasks to improve processes.

    Ideally, every step should flow into the other, beginning at the factory (or organization) and arriving smoothly as an end product to the customer.

    Employ a Pull Approach

    Many organizations use a push approach, stockpiling supplies in anticipation of demand. Lean organizations, however, employ a pull approach, wherein production increases as demand increases. Customers can “pull” directly from the manufacturer, getting what they need in an on-demand fashion.

    The pull approach requires extremely accurate market research and exceptionally efficient dissemination of information within the organization. Also, all processes must be streamlined; otherwise, it will take too long to ramp up production to meet demand.

    Seek Perfection

    At its core, lean management is about continuous improvement. Lean management requires a thoughtful approach to every action. Optimization must be part of the corporate culture itself. Every member of the organization must be working toward removing waste, adding value and delivering quality. There is always room for improvement in lean organizations.

    Conclusion

    Lean organizations are removing waste. Lean project managers ensure that activities within the organization flow properly, without interruption, delays or bottlenecks.

    Identifying value means finding out what your product will do and how much customers will pay for it. Value streams are then mapped, following every step in production. Project managers ensure that tasks flow and processes run smoothly. Market data is collected during development and these data points drive production. The most important principle that guides lean thinking is to seek perfection. The lean management philosophy demands a dedication to continual improvement.

    — Ellie Martin

  • Survey Finds Wide Gap Between DevOps Adoption and Success

    Survey Finds Wide Gap Between DevOps Adoption and Success

    A recent HBR survey shows there is work to be done to advance DevOps in organizations

    A survey of 654 subscribers to the Harvard Business Review found that while 86 percent of respondents said it is important for their organizations to be able to build and deploy software quickly, only 10 percent said they are successful at it.

    Conducted by Harvard Business Review Analytic Services on behalf of Google, the survey found nearly half (48 percent) of respondents said their organization always relies on DevOps methodologies when deploying software, while another 21 percent rely on DevOps selectively. An additional 8 percent said their organization plans to adopt DevOps practices.

    Among those respondents employing DevOps practices, the top benefits cited were increased speed to market (70 percent), productivity (67 percent), customer relevance (67 percent), innovation (66 percent) and product/service quality (64 percent). Despite those benefits, however, most organizations still have a significant amount of work to do in terms of implementing best DevOps practices—the three biggest hurdles organizations face are organizational silos (50 percent), legacy technologies (49 percent) and resistance to change (46 percent).

    Melody Meckfessel, vice president of engineering for Google Cloud Platform (GCP), said these issues are precisely why Google is sharing how it operates in the form of site reliability engineering (SRE) principles, which are a prescriptive set of guidelines for implementing best DevOps practices. Not every organization needs to implement those principles to the same level, but just about every organization can benefit from adopting a set of well-defined engineering principles, said Meckfessel.

    The challenge many organizations face is the fact that most of the IT staff are not engineers by trade. Most IT staffs are made up of administrators who tend to have limited programming skills. But Meckfessel said even IT organizations that don’t necessarily have a lot of engineering expertise still can benefit from automating operations based on the SRE principles.

    Many of those organizations will need to slow down the rate at which they are building applications today to be able to speed up tomorrow. It takes time for organizations to master the cultural change required to implement DevOps practices. But once they begin to master DevOps, the benefits to the organization compound quickly. Meckfessel said two of the most important attributes of DevOps is adopting an immersive approach to training and establishing a no-blame culture within the IT organization when it comes time to conduct post-mortems. That latter goal is especially difficult to achieve sometimes, because most interactions between developers and IT operations teams historically have been less than positive.

    Meckfessel also noted it’s critical now for organizations to incorporate cybersecurity into its DevOps processes as well, which requires not only additional training for developers but also closer alignment with cybersecurity teams.

    It’s now only a matter of time before the complexity associated with emerging technologies such as microservices based on containers force many organizations to dive deeper in DevOps. The real challenge now is charting a path of least resistance that also generates the maximum level of benefit for all concerned.

    — Mike Vizard

  • Blame Replaces Strategy: Enterprise IT War Rooms Causing Incalculable Losses

    Blame Replaces Strategy: Enterprise IT War Rooms Causing Incalculable Losses

    The concept of a “war room” is time-honored. One of the early examples is the Churchill War Rooms, an underground bunker that allowed Britain’s leaders to plot the allied route to victory during World War II. British Prime Minister Winston Churchill, as plain-speaking and self-assured as ever, said, “This is the room from which I will direct the war.” 

    Today, the war room concept also serves the world of business—specifically, organizations’ IT departments when troubleshooting a major issue impacting the business applications and their supporting IT infrastructure. The problem? So many parts of the infrastructure, whether hardware or software, can be implicated as the root cause of any problem. As a result, IT war rooms can encompass many stakeholders who are experts in those individual parts. And, while the goal is to collaborate to resolve the problem and minimize the damages from it, IT war rooms often result in a case of too many cooks in the kitchen, all speaking different “languages.”

    War rooms are a symptom of a much bigger problem within the IT organization. At the highest level, it results when IT operations teams have insufficient visibility and have limited proactive control of the infrastructure. For one thing, in a siloed organization (which most enterprises are), discussion across silos is rare. Plus, siloed monitoring tools for resolving infrastructure issues lack a common context and have no inherent understanding of the business impact of the applications. By contrast, the hardware, software and data that span silos do “talk.” In a sense, they collaborate. That’s the nature of connected infrastructure. Data, for example, flows throughout the enterprise, altogether oblivious of silos.

    The lofty view of war rooms, in the words of Dennis Drogseth, vice president at analyst firm Enterprise Management Associates, is this: “Our war rooms can be either physical or virtual. Highly automated or not. Made up of consistent, well-defined teams, or not. But what makes them war rooms is the need for collaborative decision-making across silos.”

    How Low Can IT War Rooms Go?

    The reality is that, despite their noble charters, IT war rooms have devolved into blamestorming—rancorous debate and finger-pointing to indict application development, database analysts, IT operations or DevOps for a failure. Drogseth described a bleary view of war rooms as “disastrous assemblages of finger-pointing adults caught up with siloed versions of the truth.” In an even more disillusioning example, one vendor stated that “mean time to innocence” becomes the IT war room’s chief metric”—rather than isolating and resolving the root causes of issues.

    With so many factors in play, dollar costs of IT war rooms are difficult to calculate. Taking aim at the problem, Enterprise Management Associates said, “The average War Room comprises 15 people and takes 5-6 hours to respond to and resolve the incident that caused it to convene. 29% of War Rooms take more than 11 hours to respond and resolve.” And those figures are over and above the substantial losses that occur if a server or switch, for example, goes down and war room staffers bicker over who’s to blame. Revenue losses can easily reach into the millions of dollars per hour of outage.

    Furthermore, the problems and costs are growing. Infrastructure complexity has ballooned with the growth of hybrid data centers. The hybrid model uses cost-effective, flexible compute and storage via the cloud, while retaining the control and security provided by on-premises infrastructure. However, the reality is that highly virtualized, multi-cloud environments are amazingly complex and problematical when it comes to ensuring application service delivery. Out-of-control infrastructure leads to outages and slowdowns—and finger-pointing.

    Remedies

    There are, however, several ways to mitigate the downsides of IT war rooms:

    • Discover technology solutions that provide cross-domain insights. Stated more simply, that means finding tools that can trace infrastructure issues to their source—exactly where the issue originates—even if the symptoms appear elsewhere in the infrastructure. Achieving this level of sophistication effectively eliminates finger-pointing.
    • Learn from history. Issues repeat themselves; if an issue occurs but is not traced to its source and remedied the first time, it will almost certainly recur. Often, it requires analytics-based tools to make the necessary correlations and remedy an issue.
    • Create what EMA calls an “informed context.” An informed context can help resolve an issue, moving from triage and diagnostics, to remediation, to validation. Knowing where applications live on the infrastructure at any given time and the business priority or tier of each application is one example of “informed context.”

    IT will likely always have its war rooms, and many organizations will claim that they’re necessary. At its best, a war room is a highly motivated, solutions-driven SWAT team. Of course, SWAT teams, like anything to be valued, will forever come at a cost to the enterprise.  One way to view the IT war room is that it’s just one more cost center. Maybe the better way to view it is by quantifying its value: If a war room saves minutes in getting a server up and running after an outage—or prevents an outage altogether—the savings could run in the millions of dollars. But too often, the opposite is the case: IT war rooms further contribute to the lost revenue from a failure.

    Let’s hope that enterprises remember the original intent of IT war rooms and reorient them around strategy rather than blame. If not, enterprises might want to consider closing the door of the IT war room for good.

    — John Gentry

  • DevOps and Retail: Transforming Brick-and-Mortar to Brick-and-Click

    DevOps and Retail: Transforming Brick-and-Mortar to Brick-and-Click

    Brick-and mortar-retail is rapidly transforming with the digital age. As physical storefronts turn digital, it’s safe to say that traditional retail is quickly becoming a mesh of “brick-and-click.”

    So, how are retail giants surviving in this new landscape? Interestingly enough, the remaining brick-and-mortar retail are reinventing themselves by applying helpful DevOps principles.

    In this post, we’ll peek into cases of innovation and horror stories from historically brick-and-mortar retail chains and brands. From Target, to Balian Group, Sears and other companies, we’ll discover the importance of DevOps in this new brick-and-click multi-channel retail environment.

    Target

    In an effort to increase agility, Target engineering teams embrace something they call “the Dojo”—immersive, six-week training sessions with hands-on DevOps and Agile coaches. As Target CIO Mike McNamara recently told the Enterpriser’s project: “The Dojo has been fantastic in getting teams engaged with agile and DevOps, removing the natural resistance and fear of change, and then supporting the team through the changes while maintaining productivity. It’s been a huge success for Target.”

    The result of these sessions is increased comfort with methods including containerization and tools that enable continuous integration and continuous deployment. McNamara also noted their grassroots approach; they encourage open source activity and listen to their engineers when they petition the use of new tech.

    “Our engineers have fully embraced DevOps,” he said.

    Bailian Group

    For another case study, take Bailian Group, a large Shanghai retailer that operates 6,000 grocery and department stores. By the time Bailian entered the online marketplace, it suffered from a severe lack of agility, non-native web apps and underutilized server assets.

    “Our transition from traditional brick and mortar to omnichannel business presented a great opportunity but an equally large challenge,” noted Lu Qichuan, director of IaaS and Cloud Integration Architecture at the Bailian Group.

    Qichuan and Bailian realized that to stay relevant, they had to cut long development cycles and embrace “new cloud application development tools and processes that foster a CI/CD and DevOps culture and increase innovation and time-to-market,” he said.

    They ended up implementing a private cloud using OpenStack. Now, midway through their cloud transition, they estimate to have 10 million virtual machines by the end of 2020.

    Sears

    While the previous two examples had positive outcomes, digital transformation has not gone over well for other retailers. Sears declared bankruptcy in October 2018.

    In its final years, Sears was in a sticky situation. To the chagrin of former CEO Eddie Lambert, surveys found their customers preferred to shop in stores as opposed to online. Disregarding customer desires, however, Sears attempted a big data play with its “Shop Your Way” program, under which Sears gathered user data to then sell. The program floundered, resulting in consumer irritation and loss of sales as reported by Business Insider.

    Writing for InformationWeek, Emily Johnson reminded us that evaluating DevOps using the wrong metrics will ultimately lead to failure, citing customer satisfaction as the No. 1 goal.

    In its final years, an abrasive Sears leadership halted any sense of DevOps to flourish. A disengaged Sears CEO Lambert created “a climate of fear,” according to Busines Insider reports. It appears the retailers that stay afloat encourage agility and intrapreneurship—this means encouraging innovation, not fear.

    DevOps Aids the New Omnichannel Retail Benchmark

    Brick-and-mortar are evolving into omnichannel retailers that bridge physical and virtual spaces. Take IKEA’s augmented reality app, which visualizes items in a house. From Disney, to Startbucks, Nordstroms and elsewhere, physical storefronts embrace digital offshoots of their core model to augment the traditional retail experience.

    Surprisingly, physical storefronts aren’t completely dead. They still offer conveniences for urban environments with high walking scores. See the emergence of Amazon physical locations for ironic evidence of their benefit. Other storefronts and pop-up shops have taken on a museum-like quality. The flagship Funko store, for example, creates impressive displays and interactive exhibits, pulling in flocks of weekend shoppers that want more than a dull e-commerce shopping cart.

    DevOpsGroup similarly describes a “reverse showrooming” movement; consumers research products online to then purchase in person. This enables brands such as ASOS, which is emphasizing social media and free worldwide deliveries, to stay competitive.

    “Retailers have been forced to re-evaluate their business strategies and understand how new technologies will affect their offering,” according to the DevOpsGroup website.

    DevOps Turns Bricks to Clicks

    Online shopping has sufficiently disrupted retail, and brick-and-mortar must reinvent themselves to stay relevant. An open-minded leadership and a commitment to fast development, dynamic scaling, uncompromised availability and open source technology underpin a successful evolution.

    Of course, this increased reliance on technology necessitates a hardened look at how development and operations communicate, and brick-and-mortar retailers must adopt the behaviors of tech companies to retain agility.

    This may come in many forms: new user-facing digital experiences, microservices for composability in niche omnichannel environments, containerization to package code that scales well, distributed data stores, heightened user data security or smart incident management techniques, to name a few.

    Traditional retailers, then, should dissuade their doubts against agile. DevOps can aid retail by reducing redundancy, cutting costs and streamlining production and distribution.

    — Bill Doerrfeld

  • Best of 2018: The Beauty of the COBOL Programming Language

    Best of 2018: The Beauty of the COBOL Programming Language

    As we close out 2018, we at staging-devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the first in our weeklong series of the Best of 2018.

    Well-written code is a work of art. Always has been, always will be. A programmer pulls a thought pretty much out of nowhere and transforms it into a working idea that can be used by others. It’s abstract expression made real. Computer programming requires a depth of creativity and discipline of logic that is hard to find elsewhere. Maybe architecture and the theoretical sciences come close, but computer programming stands apart. Computer programming is special and I love it!

     

    Thus, I am always interested to learn a new programming language. Every programming language offers a new way to expand my thinking. It’s always time well-spent.

    Recently I decided to learn COBOL, for no other reason than the fact that there are a lot of mainframe installations out there and it’s the mainstay language for many of them. Mainframes are critical to the operations of many banks, insurance companies, transportation systems and governmental agencies. Learning COBOL been on my Bucket List for while.  So I took the plunge.

     

    What does the term, COBOL stand for? Common Business-Oriented Language

    Working with COBOL in a Modern IDE

    The first thing I needed in my journey to learn COBOL was an IDE. I am a big supporter of coding in an integrated development environment (IDE). I like being able to write, test and run code all in one place. Also, I find the support features that an IDE provides, such as visual code structure analysis, code completion and inline syntax checking, allow me to program and debug efficiently.

    The IDE I found is an open source product, OpenCobolIDE, as shown below in Figure 1.

    Figure 1: OpenCobolIDE provides many tools to make COBOL programming easier

    OpenCobolIDE allows me to write, compile and run code all in one place, without having to go out to the command line, which is very good because I am the world’s worst typist!

    Programming COBOL in the Real World

    When it comes time to do real world, mainframe based COBOL programming, you will do well to look at the IDEs that IBM provides. The tools are designed to go about against its Z Systems environment seamlessly as a set of Eclipse plugins. These tools allow COBOL developers working in the mainframe environment to code, debug, unit test and do problem determination.

    Having set up my development environment, the next thing to do was design my first project.

    Designing My First COBOL Program

    Typically when learning a new language I like to start small and work my way up into more complex tasks. I did the typical Hello World program. You write the code to display string, Hello World, to standard output. Every developer does one. It’s a trivial program, as shown below in Listing 1

    beauty-of-cobol-listing-01
    Listing 1: A Hello World program written COBOL

     

    While the Hello World required that I learn about the concept of a DIVISION, which I will discuss in detail below, in the scheme of things, I needed more. I wanted to write a program that would force me to learn how the following:

    • How to create and use variables.
    • How to structure data into a hierarchy.
    • How to structure code into encapsulated procedures.
    • How to do some basic arithmetic.
    • How to accept user input and then do some manipulation around that input.

    Thus, I create the program, RESEL-WORLD, as shown in Listing 2, below.

    beauty-of-COBOL-Listing-02

    Listing 2: The COBOL program, RESEL-WORLD accepts and manipulates users’ input

    The user input and program output is shown below in Figure 2.

     

    Figure 2: The RESEL-WORLD COBOL program asks a user for first name, last name and age, then manipulates the input

    The RESEL-WORLD program taught me a lot about COBOL and how to program in the language. In the spirit of giving, I am going to share what I learned from writing the program. However, before I delve into the details, please be advised that the information I’m presenting is but a high-level overview. As with any programming language, it takes about a year of consistent coding to become an entry-level professional. My sincere hope is that the information I provide gives the reader a good sense of the language and enough motivation to want to learn more. As I mentioned early, learning COBOL is worth the effort.

    COBOL is Code in a Structured Document

    The most important thing to understand when learning COBOL is that is very strict in terms of code layout. The layout rules relate to the use of columns and characters. Also, the format uses a hierarchical outline structure. The following sections describe the details of layout specification.

    The COBOL Column Specification

    In COBOL a line of code can be no longer than 80 columns in length. You can also think of a column as a character. Columns are segmented into groups, with each group serving a particular purpose. The columns groups are as follows:

    Columns 1-6 is the group in which a programmer defines a sequence number. A sequence number is similar to a line number.

    Column 7 is reserved for special characters. Asterisk (*) starts a line of comment. Hyphen (-) indicates line continuation and slash (/) is form feed.

    Columns 8-11 is called Area A. Area A is the group of columns in which you start a DIVISION, PARAGRAPH and SECTION. We’ll talk more about these below in the next section, The COBOL Structural Hierarchy.

    Columns 12-72, also known as Area B, is where you write code statements.

    Columns 73-80 is reserved for developer use. You can write poetry in there, if you so desire. Just make sure you don’t go past column 80.

     

    Figure 3, below illustrates the columns grouping described above.

    Figure 3: Each column in a COBOL file is specified to serve a specific purpose

     

    COBOL is a compiled language. When it’s time to run your code, the compiler will check to make sure that the code layout adheres to the column grouping specification. If there is a violation, the compiler will error.

    Understanding Performance in Terms of Compilation and Compilers

    COBOL is a compiled language, as are others such as Java, C# and C++. Compilation is the process of taking textual source code and converting into a binary format that the computer can understand.

    There’s a lot of variety in the COBOL world when it comes to compilers, particularly when it comes to cost and performance efficiency. For example, how a compiler orchestrates the way numbers are loaded and then computed in memory matters a lot! A good compiler will make code run really fast.

    IBM has been in the COBOL business for a long time and has a keen understanding about making compilers that are fast and cost-effective. For example, the z14 System compilers improve efficiency by taking advantage of almost 24 new low-level instructions. As a result, computation speeds improve dramatically.

    As you work more with COBOL, you’ll come to appreciate the power that an industry-leading compiler such the IBM Z Systems brings to the coding experience.

    The COBOL Structural Hierarchy

    The concept behind the file format of a COBOL program is based on the structure of a document outline, with a single top-level heading followed by subordinate levels. The organizational units that make up the hierarchy are PROGRAM, DIVISION, SECTION, PARAGRAPH, SENTENCE, STATEMENT and CHARACTER. Figure 4 illustrates the hierarchy.

     

    Figure 4: The structural hierarchy of COBOL program entities

    PROGRAM is the root level of the COBOL code hierarchy. PROGRAM represents the unit of code that the mainframe job scheduler, the JCL, loads into memory to run. The program is identified by the PROGRAM ID statement in the IDENTIFICATION DIVISION. The IDENTIFICATION DIVISION is part of the next level of hierarchy that descends from PROGRAM. PROGRAM must contain the IDENTIFICATION DIVISION.

    There are other DIVISIONs that can be included. These other DIVISIONs are: ENVIRONMENT DIVISION, DATA DIVISION and PROCEDURE DIVISION,  which occur after IDENTIFICATION. Also, these subsequent DIVISIONs must appear in the order defined: ENVIRONMENT, DATA and then PROCEDURE. You can read a detailed description of each DIVISION here. DIVISION names are terminated with a period, for example,

    DATA DIVISION.

    The next organizational level down from DIVISION is SECTION. (Please see Figure 5.) Each DIVISION will contain SECTIONs that are special to it. For example, the DATA DIVISION can contain FILE SECTION, a, WORKING-STORAGE SECTION and/or the LINKAGE SECTION. You can read the details about DIVISIONs and SECTIONs here.


    Figure 5: A COBOL program is structured in a hierarchical manner

    A SECTION name is terminated with a period, for example:

    WORKING-STORAGE SECTION.

    A SECTION contains zero or many PARAGRAPHs. (Typically a SECTION will have at least one PARAGRAPH.) A PARAGRAPH name is terminated with a period, similar to a DIVISION and a SECTION.

    A PARAGRAPH contains SENTENCEs or STATEMENTs. A SENTENCE is a group of STATEMENTs. Usually a PARAGRAPH contains one or more STATEMENTs. A STATEMENT is a line of execution. A STATEMENT is made up of CHARACTERs. A CHARACTER can be an alphanumeric symbol or a special character. CHARACTER is at the bottom of the COBOL code format hierarchy.

    Variable and Data Types

    COBOL allows you to declare variables in a variety of data types. Special to COBOL is the concept of a variable level as represented by a level number. A level number defines a variable in terms of being or having a parent variable. (You’ll see more about this in the section, COBOL Supports Hierarchical Data, later on.)

    The following statement, which will be declared in the WORKING-STORAGE SECTION of the DATA DIVISION, is the declaration of a variable, WS-QUANTITY. WS-QUANTITY will hold a numeric value.

    01  WS-QUANTITY PIC 9(2) VALUE 12

    The expression above above declares a variable , WS-QUANTITY to be a two-digit value with an initial value of 12. Also, the variable is declared to be at level 01, which is the highest level possible. The variable has no parent. The way we know that WS-QUANTITY is a variable for a two-digit value is due to the PIC (Picture) clause. You can think of a PIC clause as a type declaration. Table 1 provides a high-level description of the various PIC clause symbols.

    Symbol Description Example Declaration Sample Value
    9 A numeric value where each occurance of 9 represents a digit 99 or 9(2) 35
    a Alphabetic aaa or a(3) “Bob”
    x Alphanumeric xxxx or x(4) “R2D2?”
    v Implicit decimal* v(3) .175
    s Sign s9(2) -76
    p Assumed decimal* p9 .6

    Table 1: Symbols for a PIC Clause

    *COBOL has a special way of storing numbers such that the numeric conversion take place when the characters are loaded into memory. For example, a variable with a PIC of p9(2) will be stored as 56 and loaded into memory as .56. For a more detailed discussion of usage and the PIC clause, go here.

    COBOL Supports Hierarchical Data

    Storing data in a named group of variables makes programing easier. For example, in the C programming language, we can use a struct to name a group of variables, as shown below in Listing 3.

    struct User { 
      char  first_name[10]; 
      char  last_name[10] 
      int   user_id; 
    };

    Listing 3: A simple structure in C

     

    One of the real nice things about COBOL is that it allows you to organize data according to a named group of values similar to that of a C struct. What’s interesting in a historical sense is that COBOL has had this capability well before Kernighan and Ritchie created the C programming language.

    The way COBOL accomplishes named data grouping is with construct called a record. Figure 6 below shows a graphical representation of a record named, WS-USER.

    Figure 6: The record, WS-USER contains subordinate variable, WS-FIRST-NAME, WS-LAST-NAME, WS-AGE

    The code snippet in Listing 4, which is from the RESEL-WORLD program at the start of this article, shows the COBOL code that declares the record, WS-USER.

     

    beauty-of-COBOL-Listing-04

    Listing 4: The record, WS-USER contains 3 subordinate elements, WS-FIRST-NAME, WS-LAST-NAME, WS-AGE

    Notice in Listing 4 above the there are two levels of data variables in play: 01 WS-USER and 05  WS-FIRST-NAME, 05 WS-LAST-NAME, 05 WS-AGE. The variable, WS-USER is a parent to the subordinate, level 05 variables, WS-FIRST-NAME, WS-LAST-NAME, and WS-AGE.

    Assigning variable to a numeric level is special to COBOL. The declaration logic is that a record variable (the root of the record) has a level number of 01.  Levels 02 to 49 are used to declare subordinates elements. (You can think of a element as a member of the record.)

    You use the reserved word OF to access an element of a record. Listing 5 below shows two statements. The first asks a user to input their first name. The second statement takes the value entered and assigns it to the element, WS-FIRST-NAME of the record, WS-USER.

    DISPLAY "What is your first name?".
    ACCEPT WS-FIRST-NAME OF WS-USER.

    Listing 5: Accessing an element of a record in COBOL

     

    It make sense that records were part of COBOL from the start. COBOL was intended to be a language used for business application. Businesses have been organizing values according to named groups since before the customer form was created. COBOL was designed to reflect business needs. It’s interesting that the fundamental needs of businesses in terms of logical data structures have been surprisingly consistent over time.

    COBOL Has a Natural Language Syntax

    Expressing a statement in COBOL is very similar to the way a person speaks naturally. Listing 6 below shows the statement that adds together the values of the two variables, WS-AGE-DELTA and WS-AGE (from the record WS-USER), and stores the result in the variable, WS-NEW-AGE.

    
    ADD  WS-AGE-DELTA WS-AGE OF WS-USER TO WS-NEW-AGE.

    Listing 6: Adding two integers, WS-AGE-and DELTA WS-AGE, then storing the sum in the variable, WS-NEW-AGE

    Notice that the statement in Listing 6 is very close to saying, “Add WS-AGE-DELTA and WS-AGE from WS-USER to WS-NEW-AGE.”

     

    What I find pretty astounding is that many modern programming paradigms try to capture the ease of natural language expression that COBOL has had for years. What comes to mind immediately is the use of natural language expressions in the NodeJS Chai package. Chai is used in NodeJS unit testing to express BDD assertions. Here’s a Chai expression that checks a variable, myVar to assert that it’s a string.

    expect(myVar).to.be.a('string');
    
    FAST FACT: Is COBOL a case-sensitive language?

    No. COBOL will consider a variable named WS-FIRST-NAME to be the same as one named ws-first-name.

    COBOL supports DRY

    For programming language to be really useful, it needs to support DRY. DRY is an acronym for Don’t Repeat Yourself. Most programmers don’t want to write the same code over and over again, nor should they. What you want to do is to encapsulate code in to a single area of execution that can be called repeatedly. Being able to program by DRY is common for most languages—BASIC, Java, C#, Javascript, Python, to name a few and … you guessed it: COBOL.

    COBOL supports code segmentation and reuse in a variety of ways. One way is through linkage, calling routines between programs. Another way is to segment code inline under a PARAGRAPH in the PROCEDURE DIVISION and then call that PARAGRAPH. (See Figure 7.)

    Figure 7: You can make code reusable in COBOL PARAGRAPHs

    The little RESEL-WORLD program I wrote uses the inline PARAGRAPH technique. Before I go in to the details of calling code encapsulated into a PARAGRAPH, it’s important that you know that in COBOL the entry point of execution into a program is the first line of code after the declaration of the PROCEDURE DIVISION, as shown below in Listing 7.

    beauty-of-COBOL-Listing-07
    Listing 7: COBOL supports reusable code by using PARAGRAPHS to segment execution

    Notice the PERFORM statements right after PROCEDURE DIVISION. These four statements are calling PARAGRAPHs. (For those of you familiar with JavaScript, you can think of each PARAGRAPH as a function definition.) Essentially what the code is saying is, “Execute the code at the paragraph named, GET-DATA, then perform the code at CALC-DATA, SHOW-DATA and then FINISH-UP.”

    Encapsulating code into STATEMENTs within a PARAGRAPH prevents code from become spaghetti that is hard to maintain. Also, the encapsulation enforces the sensibility of DRY. Any language worth its salt needs to support some sort of encapsulation. COBOL meets the need with room to spare.

    An Expressive Language for Now and the Future

    The more I learn about COBOL, the more I like it. The language continues to evolve to meet the needs of our fast-changing times, with revisions as recent as 2014. Since its inception there have been a dozen enhancements to COBOL including a continuing stream of formal standards.

    Today’s COBOL supports modern programming paradigms such as object orientation. The IDEs have grown to keep pace with the demands of modern users. And, given the immense installation base out there, there is a lot of money to be made doing COBOL programming. To quote Wikipedia:

    In 2006 and 2012, Computerworld surveys found that over 60% of organizations used COBOL (more than C++ and Visual Basic .NET) and that for half of those, COBOL was used for the majority of their internal software.

     

    What’s even more amazing to me is the cleverness of engineering the language has promoted. COBOL developers addressed problems in the past that still vex many today. We modern developers can fall prey to thinking that before we came along, there was no cool stuff. It’s like the young, aspiring guitarist who thinks thinks he has his chops down and can shred to the top of the heap. Then one day his music teacher gives him a 1936 recording of Django Reinhardt playing guitar, and playing with only two functional fingers on his left hand! At that point the young artist realizes that virtuosity transcends era and that creativity has no bounds. This is what it’s like for me as I learn COBOL. It’s a beautiful, expressive language that was cool then and is very cool now. Learning it is making me appreciate how much amazing thinking went on back then and continues to emerge. There’s a lot of rich opportunity at hand to make great code for mainframes using COBOL. It’s only a matter of mastery, creativity and discovery.

    Special Thanks

    If I have seen further it is by standing on the shoulders of giants.

    —Isaac Newton, Letter to Robert Hooke, February 5, 1675

    Learning any programming language is hard work. Years ago, when I wanted to learn a new language, I would get a few books on the topic and then hunker down to spend the hours necessary to absorb the required knowledge. If I had the time, I might take a course. Maybe I had an expert friend I could call up when I hit a wall or needed real-world guidance.

    We’ve come a long way since that time. The internet is a game-changer. Online tutorials, videos and interactive, digital books make things a whole lot easier than in earlier times. My experience learning COBOL testifies to the value of these modern benefits. But still, as I’ve been learning, I’ve hit more than one wall that required professional guidance and review. Allan Kielstra and Roland Koo at IBM (who also wrote this great article) provided the expertise that made my learning COBOL not only less difficult, but also fun. I am in their debt. Their enthusiasm for COBOL was infectious. Their commitment to technical excellence is inspiring.

    — Bob Reselman

  • DataOps: DevOps Plus Big Data

    DataOps: DevOps Plus Big Data

    In traditional DevOps, there are the complimentary forms of development operations (that I call DEVops), and development operations (that I like to call devOPS). Between them they automate the toolchain and bring the people working on getting an application out to users onto the same team—not necessarily the same organizational team, but using their strengths in tandem to meet the needs of the business.

    DataOps is an interesting extension of DevOps. You still need all of the coding side to get data into the system and queries consistently maintained. You still need all of the operations side to get your database/noSQL/whatever up and running. In fact, while the development side in a DataOps environment might be (but isn’t necessarily) lighter, the operations side is almost always more complex. No matter the big data engine in use, it is a complex system in addition to other supported systems in a normal environment. My first installation of a big data environment (Cloudera, as it happens) was a weeks-long learning voyage. Only after I’d completed it did I use an automation tool (which is no longer available) to make it easy. My second round was hours … But it assumed the knowledge I had gained in the first attempt.

    The other bit of DevOps—ongoing monitoring and management—is also more complex in a big data environment, but we’ll come back to that in a moment.

    Once the normal DevOps systems are in place, ETL/data import tools will need to be supported also. The volume that these tools crunch through in a day makes their inclusion in DevOps in a data-heavy environment critical. If the data uptake is slow or the data itself inaccurate, there is an impact on the organization. This step also requires inclusion of data scientists, people who traditionally have not been pulled into the DevOps model. But they are the ones that can gauge the accuracy of data, and normally are responsible for data acquisition anyway.

    Which brings us to the monitoring and management stage of DataOps. Normally when we talk monitoring and management in DevOps, we are talking about tools that ultimately help with things like availability, responsiveness, auto-scaling and recovery. DataOps needs all of these things, like any other application does. And we’ve already mentioned that it has a complex environment embedded in your complex environment. But it also (some would argue more importantly) needs data monitoring. A knowledge of the state of the data is imperative. Whether from a piece of code gone wrong or data poisoning, monitoring the health of big data archives is essential to keep results from getting out of whack. This monitoring will again involve the people who know the data—data scientists by whatever title. This iterative process will need to run relatively constantly, checking on whatever makes the data valuable, normally using cross-field checks to verify that data hasn’t somehow been corrupted, or comparing trending averages to current average to monitor for sudden swings in results.

    Of course, for a system (and in research there are many) that spins up, finds results and spins down, the monitoring part is less essential, but for anything that keeps accessing and adding to the data pool, it is the key to a healthy application or application portfolio based upon the dataset.

    If all that sounds a bit intimidating, you’re not alone. The phrase “DataOps” has been around for a few years, and while data scientists use it, it hasn’t yet gained serious traction in the enterprise. Here’s how I would approach it as CIO: Given that DevOps is running on other projects, that is the low-hanging fruit. Given the rate of change for servers (as queries—particularly long-running ones—increase and decrease, systems needs change is one example), traditional DevOps is a greater benefit to big data than many other corporate systems. So roll out DevOps first without serious concern for DataOps beyond how DevOps impacts ETL—not ETL data quality, but ETL data availability/performance. Only when the system and the team are in place and the software/hardware/code part of the overall system is fully in a DevOps environment, start looking at ways to move data into it. Data monitoring will be the big one, but ETL under the initial round of DevOps implementation will have shown places it could be improved and ETL data integrity can be added, so improving those areas also. A measured, step-by-step approach that builds off of organizational experience is key here. Data monitoring and management will require different tools than normal DevOps monitoring and management, so having the data scientists look into available toolsets and how they mesh with existing tools early on will help with eventual implementation.

    Make no mistake, it is a big job, moving large data driven environments to DevOps. But the benefits are increased awareness of data quality and data quality issues, along with more responsive systems and teams that DevOps offers every business application. The amount of data available to any give organization is currently on a massive increase as IoT, open datasets, partnerships and ML drive greater pools. It is worth getting the automation and cooperation of DataOps into the mix as early as possible.

    — Don Macvittie