Category: DevOps Toolbox

  • A New DevOps Test Environment Platform is Needed

    A New DevOps Test Environment Platform is Needed

    Test environment management is bottlenecking DevOps transformations and is driving the need for a new test environment management platform solution. On-demand test environments in which developers and testers do not have to wait for a shared environment are sorely needed.

    The evidence is clear. In my book Engineering DevOps I indicated that finding and setting up test environments is a primary source of bottlenecks in DevOps value streams.

    In my February 2021 webinar with Plutora, I presented Best Practices for Developing Test Management Environments. I indicated that orchestrating and managing test environments is often a primary source of delays and waste within value streams.

    The State of Testing Report 2020 by PractiTest indicates that DevOps practices increased 57%; from 26% in 2017 to 41% in 2020. The report revealed that 46% of testing has shifted to using CI/CD processes, 55% are actively involved in shifting testing left and 36% are testing in production. This is clear evidence that DevOps testing is on the rise.

    Wolfgang Platz’s December 2019 article Why Software Testing Remains a Bottleneck indicates that testing is responsible for more delays than any other part of the development process – accounting for 52% of the bottlenecks in the development process. And  53% of respondents to the World Quality Report 2020-21 indicated that test environments are not available when needed, 67% lack the capability to book and manage test environments and 71% lack visibility of available test environments.

    An article by Velocity echoes the problems with pre-production environments. “With the rise of microservices and cloud-native applications, environments are becoming more complex. Often you need to wait or compete to get access to develop on a specific microservice, or can spend time waiting to deploy to a shared staging environment only to realize something already changed, and you’re back to square one.”

    The Continuous Testing Report – 2020 by Capgemini defines test environment management (TEM) as managing infrastructures, test data and applications to support testing on demand. These environments are one of the greatest impediments to continuous testing. In this report, 36% of respondents said they are spending more than half their time building and managing test environments; 44% of respondents indicate they spend weeks of their time searching, managing and generating test data.

    Test environments should be regarded as functions to be spun up or destroyed at will, with virtualization, infrastructure-as-code (IaC)  practices, service virtualization, containerization and improved test data management.

    Given all the above evidence, it is clear that a new approach to DevOps test environment management is needed.

    The following are examples of benefits that DevOps test environment management platforms need to offer:

    • Compatible with current tools:
      • Automatically map and configure application infrastructure to develop and test multiple services in an environment that runs just like it would in production. This needs to be fully agnostic to cloud and tech stacks, such as Kubernetes and other environments.
      • The platform needs to work on applications in parallel without disrupting others, with full access to data and third-party dependencies.
    • Automatic configuration and maintenance:
      • The platform needs to automatically configure and update any changes within the production infrastructure’s mapping in real-time so ephemeral environments will always be synced with what is in production. Instead of having to spend time making sure any changes are configured, the platform will ensure the right environment that replicates production, without having to wait.
    • Data and third-party dependencies:
      • The platform needs to load the relevant data and third-party resources that are vital parts of the environment to test, work and debug against, so test environments will, as closely as possible, mimic working in production.
    • Team collaboration:
      • The platform needs the capability to share ephemeral test environments with other stakeholders so they get visibility and can collaborate for faster feedback cycles. This is also important for collaborating for debugging.

    What This Means

    DevOps testing is on the rise, and test environments are a major bottleneck for DevOps testing. A new DevOps test environment management platform is needed to reduce the bottlenecks. On-demand test environments in which developers and testers do not have to wait for a shared test environment are needed. Such a solution would improve agility and reduce wasteful orchestration bottlenecks in test, improve quality and risk management through comprehensive and automated testing and enable continuous improvement by providing more intelligent insights and feedback.

  • Cycloid Tool Can Reverse Engineer Provisioned IaC

    Cycloid Tool Can Reverse Engineer Provisioned IaC

    Cycloid this week unveiled a tool that makes it possible to reverse engineer code used to manually provision cloud infrastructure. The Infra Import tool is part of an effort to create a more consistent and reliable version of that code using open source Terraform software.

    Benjamin Brial, Cycloid CEO, said Infra Import is based on Cycloid’s open source TerraCognita software, which enables IT teams to reverse engineer infrastructure-as-code (IaC) created using open source Terraform tools.

    Infra Import extends that capability to cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform or an IT environment employing the open source OpenStack cloud platform that may have been manually provisioned. IT teams can then create a configuration based on that Terraform code that could be employed across multiple clouds.

    Infra Import is designed to connect to a cloud provider and then automatically reverse engineer Terraform files and IaC stacks based on the existing deployment. It also ensures that the latest edition of Terraform is being used to create that configuration code, noted Brial. That capability eliminates the need for DevOps teams to keep track of what version of Terraform might need to be updated as new versions of the IaC tool become available.

    Just as significantly, Infra Import also makes it a lot easier to onboard new members to a DevOps teams because the tools uses valid secure configurations they know can be relied on. Misconfiguration of cloud services is one of the biggest security issues that IT teams today are wrestling with, simply because most developers lack the security expertise required to configure cloud services properly.

    Cycloid is in the process of rolling out a comprehensive suite of management tools that complement a lightweight framework it created to make DevOps best practices more accessible. The challenge is many IT organizations have already manually provisioned cloud infrastructure in a way that is, at the very least, often suboptimal if not downright insecure.

    Misconfigurations are rife across cloud computing environments, but most organizations are not prepared to manually address the issue. Automating the reverse engineering of those configurations is the first step toward enabling IT teams to address the problem at scale.

    It’s not clear to what degree configuration concerns have impacted the number of applications that would otherwise have been deployed in the cloud. Security is always listed as the top concern IT leaders have when employing cloud platforms. The issue, however, is not the security of the platforms themselves but rather the processes employed to deploy applications on the platforms. Cloud service providers have promoted a shared responsibility model for security under which developers are accountable for both securely configuring services and ensuring the overall security of the software environment. The trouble is, not every developer understands the implications of that shared responsibility model much less how to actually secure an application workload.

    It’s not likely security concerns will slow down the rate at which applications are deployed in the cloud. However, as more cloud security issues are encountered, more scrutiny will be applied.

  • EP 12: DevOps Unicorns

    EP 12: DevOps Unicorns

    What should you look for when choosing a DevOps vendor? What criteria are others in your industry using to inform their decisions? How do you choose a vendor that is a good fit for your organization? And how do you know if you made the right choice? In this episode of DevOps Unbound, host Alan Shimel welcomes Larry Maccherone of Comcast, John Willis of Red Hat, Ravyn Manuel of the Smithsonian, Garima Bajpai of Capital Carbon Consulting and Mitch Ashley of Accelerated Strategies Group for a panel on DevOps unicorns. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hello, everyone. Welcome to another edition of DevOps Unbound, where we talk about DevOps, hence the title. Thanks for joining us. This episode is titled DevOps Unicorns: The Quest for End-to-End DevOps.

    You know, I’ve been in IT 30-plus years and there’s always been this tension or this pendulum that swings between the one throat to choke versus the best of breed solutions in any area. Today it’s DevOps and software delivery and deployment. But in anything, in the security world, in everything, in hosting, in infrastructure it’s always been a common theme. But it’s really, really becoming acute within the DevOps sector now. And we’ve assembled what we think is a great panel to discuss this issue. Let me introduce you to them and we will go from there.

    First of all I want to welcome, well, to a panel for the first time in a long time, my friend Larry Maccherone. Larry, welcome to DevOps Unbound. Give people a little background, if you don’t mind.

    Larry Maccherone: Sure. Thanks for having me here, Alan. First of all I want to say so I launched the Dev PsychOps Transformation Program at Comcast and grew that for five years. And I’m a developer at heart, though, so I was sort of the ideal person to take what was actually becoming more common outside of big enterprises, Dev PsychOps, and try to actually do a diverse approach inside of a large enterprise like Comcast. And so all of my experience is around that recently, the last five years or so, around that.

    Shimel: Excellent. Next let me introduce you, she’s a repeat visitor to DevOps Unbound. Garima, if I mess up your last name I apologize in advance, but Garima Bajpai. Close?

    Garima Bajpai: Almost. Yeah. So I’m Garima Bajpai. I’m based in _____, Canada and I’m the founder for the DevOps Community of Practice here in Canada, which has three chapters: Toronto, Ottawa, and Edmonton. I’m also the organizer of the DevOps Summit in Canada, which happens in November 8th and 9th. And I’m also associated with organizations like DevOps Institute as an ambassador, as well as CD Foundation as an ambassador.

    Shimel: Fantastic. And, Garima, I apologize on your last name. My bad. I’m trying to read it and the monitor’s too far away from me without my glasses. But thank you for correcting me, and welcome.

    Next up, another good friend of mine, my actually business partner on Digital Anarchist. And though that’s not his full-time gig these days; he’s very busy with his red hat career. The one and only Botchagalupe, John Willis. John, welcome.

    John Willis: Thank you, sir. Thank you, Alan. Hey, yeah, John Willis, Botchagalupe on Twitter. My current day job is in a team called Global Transformation Office at Red Hat. I’m sort of blessed to be on a team with Andrew Clay Shafer; Kevin Behr, the co-author of The Phoenix Project; and Jabe Bloom. Just quickly, you know, I’ve been in the industry for 40-plus years, 10 or 11 startups, 12 books, if somebody’s counting. Probably the most notable is the co-author of DevOps Handbook.

    Also was involved in basically the starting and building of the original DevOps Days. Was the only American at first DevOps Days and Damon Edwards and I created the first DevOps Days event in the USA.

    Shimel: Yep. And that’s only a partial list, John. Thanks for joining us today, man.

    And then I want to introduce – she’s off-camera right now, but hopefully when you’re watching this at least you’ll get to see her picture, but she was having some video difficulties, but happy to have her on audio, my friend, Ravyn Manuel. Ravyn, welcome. And if you wouldn’t mind, give people a little background.

    Ravyn Manuel: Thank you, Alan. I do appreciate being on your show. It is awesome. It makes me laugh. I am the senior application developer for the National Museum of African-American History and Culture. And I’m also the DevOps engineer for that particular unit. As a developer I implement some of the hands-on interactive that you play with in the museum, and as a DevOps engineer I am in charge of just about everything that’s DevOps, from the cloud solutions and how we – from the application development side, how we’re using cloud and the tools and writing strategies and keeping my leadership informed of all the good things that are out there.

    Shimel: Fantastic. Ravyn, welcome and thanks for joining us today.

    Last but not least, my co-host here on DevOps Unbound, Mitch Ashley. Hey, Mitchell, welcome. You want to just introduce yourself?

    Mitchell Ashley: Great. Great to be here. Man, what a panel. I am so excited to be with these folks on today. Yeah, I head up Accelerated Strategies Group, analysts from focusing on, of course, DevOps and digital transformation, full cybersecurity and cloud. And I also serve as CTO with MediaOps, so heading up the platform strategy and technology, so we used to deliver programs like this.

    Like John, I think we’ve been probably in the industry about the same amount of time and have zero books to my credits, so he’s made a lot more progress in those areas, but also a startup person and run a lot of software projects, product, as well as IT groups, as well as on the vendor side. So this will be a fun discussion.

    Shimel: Absolutely. I should mention DevOps Unbound is sponsored by our friends at Tricentis. To give you an idea of what a great sponsor they are, they don’t even have anyone on our panel today. But they do enable this show to go forward every other week, so many, many thanks to Tricentis.

    So, John, I laid it out, right, it’s the one throat to choke versus best of breed, how many vendors can we support, how do I package solutions that are more than just point solutions, or maybe I just want point solutions. What’s the answer here, John?

    Willis: The answer is, Alan, I would say if I had the right answer I’d be on a Grady-White right now in the Gulf of Mexico.

    Shimel: Exactly. And I’d be right with you.

    Willis: But no, I think this is a great panel. I agree with Mitch; I’m excited about just the discussions we can all have. I think the question – you know, I was thinking just that sort of Chinese curse translation, “May you live in interesting times,” right? And that’s clearly where we are right now. And I think your question starts with a shift, right? Like there was this – talking about you, Mitchell, some of us have been here forever, and like back in the day there was this question of one throat to choke versus should I do an ELA with computer associates versus piecemeal things, right? And probably less than one-percent of the enterprises were do-it-yourselfs.

    Now today you will find probably – I’m just going to swag the number, but it’s probably north of 10, maybe 15-percent, maybe even higher of companies that are now being challenged to do it yourself. You know, Kubernetes is a great example. That’s a can opener for companies to make a decision of are my vendors keeping up with what comes from basically sort of upstream technology that I should build myself.

    So I think that’s the sort of question we move into, is more about do you build internally to solve your problems, to keep up with the changes in the market and the technology, or do you try to find a pair of vendors or a single vendor – and there really is very few single vendors that can give you the whole scope of what’s going on in modern technology today. But I think that’s the question that I think we’ll have fun just talking about today. _____.

    Shimel: I don’t disagree. Panel?

    Manuel: So, Alan, this is Ravyn. I agree with you, John. I think one of the things – because that’s a struggle for me and my organization, is we need help with implementing DevOps, but I think the vendors are very tool-focused and the tools are like the third priority in the stack of there. And what I’ve been doing when I’m building the strategy for how we’re going to implement DevOps, I looked, and we’re talking about tools. I look at the standards, because if they’re making standards and the tool has to follow that standard then it’s plug-and-play, you don’t get – like if Docker decides that they’re going to go out of business then you can have another container-type technology to do that.

    So when we are looking at benders we actually kind of do a piecemeal, because we’re really unicorns. We’re not a software company, we don’t have software products; we have digital products, and they’re all one-offs. And so it really just depends on what fits for us. So we don’t really go for the one vendor solution; we actually go for who can help us get to where we need based on our business problem.

    Bajpai: Excellent. And I wanted to add a few things, because when we drill deeper into this conversation we will find out that there are two sides of this coin. One part is where an organization from a maturity perspective is. And the second thing is what are the vendors and key capabilities you’re looking at. But if you look at both sides of the coin there are three things which are needed to calibrate these DevOps tools in the overall context. The first thing, and this is – I want to reiterate the fact, because I come from a very large complex organization background, that it is very essential to have an enterprise architecture view. So I would suggest and recommend that before even, you know, taking this approach of single vendor or multivendor discussion, we need to kind of break down what are the key capabilities which are needed. So whether it’s a single vendor which can provide you that enterprise IT architecture capability, or it will be your organization who will be doing it. So that’s essential.

    The second thing I think which is also essential is orchestration layer. You know, you will have to have that governance of total cost of ownership, compliance, security, identity access management, and that orchestration layer has to be there to stitch all you calibrated DevOps tools. So it’s a question of whether you want to kind of build it in house in that organization or you kind of have a vendor-centric approach where this capability should be or could be brought in.

    So these are like two things, and then you start calibrating these DevOps tools. So you don’t have to look at the foundation aspects of compliance, security, governance, total cost of ownership, and the job becomes much more easier. The question is that, you know, how to bring these DevOps tools and enable business value.

    Maccherone: So I want to riff off of that orchestration layer idea, but I want to backtrack a little bit on sort of the – we have a binary sort of difference here between unicorn and best of breed. But I think within the unicorn bucket there’s really two major flavors of unicorn that are wildly different, and so I want to sort of bring that out. You have these Frankenstein vendors that are piecing together acquisitions into this monster and those are not pleasant experiences. The only benefit you get out of one vendor in that case is really the contracting benefit of having a single master service level agreement. You’re not getting an integrated product that really allows you to actually smoothly do the workflow, insist in the workflow that developers are using to produce value for your company.

    And then you have this other class of unicorns in the DevOps world, but less so in the security world, and I’ll talk about that in a second, which have taken a marketplace approach. And this is not a DevOps vendor, but I think the first vendor to really pull off this marketplace approach was Atlassian. You know, they had one product that was really great, but they said, “Okay, plug into this product” and they created APIs and they created toolkits and they created a whole community. Most of those people did it for free and they just sort of did it for themselves and then they shared it with the community. But some of those people became vendors and sold their product in the Atlassian marketplace, and then Atlassian acquired those. They were already pre-integrated with Atlassian’s other offerings when they got acquired, and so their acquisition unicorn approach doesn’t look like a Frankenstein’s Monster in the end, because they were built with the APIs of the vendor in place.

    And I think Github is definitely taking that marketplace approach and pulling it off really well, even into the security space. And so I think that kind of unicorn – and Github to a lesser degree, that kind of unicorn doesn’t suffer from that Frankenstein Monster’s sort of problem. No one in the security space has really pulled off the marketplace approach yet, though. We have some Frankensteins and Frankenstein Monsters over there, but we don’t have a full sort of marketplace approach. And I’m waiting for somebody to do that. I think that’s going to be the big win, somebody _____.

    Willis: I’d like to do a whole podcast on that, because that’s all I’ve been thinking about right now, is what is the end-to-end on security. But like if I got into it now it would take up way too much time. So I did want to comment on some of the things. You know, the first, Ravyn, I think the interestingly about standards, I agree, right? The problem we have right now is figuring out which standards are actually going to last, are malleable. I mean that’s one of the things I’ve really been working hard on and looking at like OSCAL versus OVAL versus S-CAT versus, you know, just NIST _____ – in order for – I mean there’s just so many things and they all sort of have a little bit of different approach.

    So I agree, the question we have to deal with now in these sort of, you know, may we live in interesting times is it’s like picking sort of an early vendor, like I like this vendor technology, but are they going to be around? Are they going to be acquired by a company that I don’t really want to do business with? I think the standards is just it’s such complicated.

    Just the other thing I wanted to say is I think that for vendors, you know, and I’m not _____ ’cause I work for Red Hat. Because I will say right now the thing that like in the grand scheme of technology, if you’re going to go all in a vendor, look around. You know, I’m going to say this. I’d love to be corrected and I’m not saying, you know, Red Hat because I work at Red Hat. Anybody that knows me, I’m not a shill for any vendor I’ve ever worked for. But it’s VMware and Red Hat, right? They can keep up with – because you’ve got to – in this world you have to have people on staff, their full-time job are contributors to an open source project.

    And if you think about what’s going on with like Kubernetes and clusters and server list, you know, you think of the two companies that are positioned really well at that sort of scale version to be able to keep up with the complexity and change of technology because they’ve put the resources in. I mean a longer list that you can probably make after that. And I’m not including the cloud providers either, ’cause that’s a different story.

    But anyway, those are the two comments I wanted to make on some of the things people talked about.

    Ashley: I just wanted to jump in. What’s fascinating about this conversation is how diverse and complex it is. There are a lot of aspects to consider, whether it’s an enterprise architecture like Garima is talking about, John’s talking about, who are the vendors that can pull it off. Larry is too, if you’re talking about this sort of roll-up strategy, but retro-ally integrated strategy.

    I think another dimension to consider is every organization has a DNA about how they think and operate when it comes to vendors and software and tools. And while I’m not always an advocate of go with the flow, sometimes when you’re thinking about how do we have something adopted, think about how the organization operates. If you are kind of a large tool vendor, let’s work with, you know, Red Hat or VMware or Broadcom or whoever might be – think about that. That’s oftentimes, you know, save yourself some brain cells and live a little longer.

    There’s also the innovation curve, and some of the best ways to ride the innovation curve is the open source route and the vendors that are built around that kind of a strategy. There’s combinations of all those.

    So I guess the thing I would add to the consideration of the conversation here is think about how you match up your strategy with the DNA of the organization so you can accomplish as much without killing as few brain cells as possible.

    Bajpai: And thank you, Mitch, for bringing up this point of DNA of an organization. I think that this is a pathway of dialogue which we are having. I do believe that the mindset is shifting, and even I see in large legacy enterprises that they have started to realize that, you know, this vendor-based approach is something that needs to shift, and they need to bring people as partners in this context. Because when you think about DevOps it’s basically exploring your opportunities, breaking the barrier for, you know, your technology onboarding, the gaps – the alignment and outcome gaps you have created in the past.

    I think this is the opportunity where the organizations can shift and bring more access to innovation through these partners. So I truly believe that there are softer aspects of this conversation as well which we need to kind of consider that traditionally how contracts are _____, you know, how we look at these DevOps tools from a vendor perspective, how can we bring them as partners, what we need to change if we start doing that.

    It’s not kind of done overnight, right? So we have to kind of set what kind of modeling practices we need to bring in. And this opens up another door of possibilities. And probably during this conversation we will have some opportunity to discuss these SLA constraints which we have built in the organizations which steer us to a specific vendor or steer us to like a specific window through which we can outsource a lot of capabilities and just, you know, a choke the throat approach, right?

    So let’s maybe in the upcoming discussions we also can bring these points and highlight.

    Shimel: Ravyn, I know Smithsonian is sort of quasi-government, or it’s not true government. But to Garima’s point about SLA and being able to only deal with chosen vendors like that, it is; it is a real governor on your ability to choose best of breed, on your ability to have choice in how you want to do things. Interested in your experiences around that.

    Manuel: It’s been difficult. I’ve been listening to this conversation and it just hit – I don’t know why it just hit me, but the vendors – and I love Red Hat, so I’m not trying to talk Red Hat or anything, and open-ship. But when you get a vendor that you have worked with and you trust, and Smithsonian, we use Red Hat a lot in our OCIO. Red Hat is doing open shift and Kubernetes and they are driving their tools to these specific tools. And for me, as a unit within the Smithsonian, I may not want to use open shift and I may not want to use Kubernetes, but I don’t actually have a choice, right? Because that vendor is a great vendor and they’ve got great tools, but they have chosen for me what I am going to be using, and that makes it really hard for me to say “Does that really fit what I really need? Is that too much?”

    Because I can tell you that a lot of the vendors that I talk to, they’re expecting us to use this like suite of products, and it’s too much for what we want to do, and so I’m left with I either go with this vendor and overspend and use tools that I don’t really need, or I have to do it myself because I can’t really find somebody that can customize that for me. So it’s a challenge.

    Shimel: Yeah, it sounds like it.

    Willis: You know, one of the things I struggle with, you know, I sort of love committees and hate committees, right? Because like I so – HashiCorp has this cluster manager called Nomad, and I’ve actually seen people use it effectively with like incredibly less toil, 80-percent of the features of it, but it’s never going to get any escape velocity. It hasn’t and it won’t.

    And I fear for our industry that we’re like so focused on there’s just only one way to do cluster management today where like – so to Ravyn’s point, like whether it’s Red Hat or whatever, it seems like you’re not even able to explore possible emerging containerized cluster management solutions because all of the oxygen in our industry right now is consumed by Kubernetes.

    Manuel: Amen.

    Shimel: Thoughts on that? I’d like to jump in with some thoughts on that. I mean this is sort of a dream/nightmare scenario. Right? I’ll take it – let’s take it off Kubernetes, let’s go back to PC operating systems. It was great that we had only Windows to develop on, and this way if you were an inept developer you only had to worry about developing on Windows. You had one operating system, you could make your app on it and it worked and you got 90-percent market share with it. What a fantastic thing for app developers. Of course, it gave Microsoft the inside track on Office and everything they did. And from a security point of view what a big, fat juicy payload that was, right, to go attack.

    So then we started seeing, for as long as I’m in IT, “This is going to be the year of the Linux desktop.” Right? How long have we heard that? And so, you know, and then of course Mac resurfaced.

    But this is we strive for some uniformity. We strive for Kubernetes to be the standard, ’cause it’s open source and it’s run by a foundation, and we want a standard. But then we chafe that, “Hey, we all can’t fit into that same standard. We want to be our own unicorn.” And that’s really what we’re talking about here today. If you go back to early, John, you sold the company to Docker before Kubernetes dominance, right? And there were several cluster management solutions vying. Right? There was Mesosphere and Rancher out there _____.

    Willis: Mesosphere and _____, yeah. [Laughs]

    Shimel: You remember. Yeah, _____ of these, exactly. So, you know, this is – but this is the world we – you know, to quote Hyman Roth in the Godfather II, “This is the world we chose to live in,” right? We strive for these single kind of standards that one-size-fits-all. And maybe that’s a huge mistake; I don’t know. Maybe it is a mistake. Larry, what do you think?

    Maccherone: Yeah, I do see that problem, John, you’re sort of focused on here, that sometimes the de facto standard is not the best technical choice. You can go all the way back to Betamax versus VHS, for those of you who are old enough to remember that, right? VHS won out, but Betamax was considered by everyone to be a superior technology. Why?

    You know, so I think if you’re going to take a best-of-breed approach you’re going to have to – when you have a true de facto standard like Kubernetes you don’t have to isolate yourself from it, but when you’re making a bet on which one is going to win out – and John is really good at this; John talks about this all the time. John sort of like doesn’t want to put his chips down on something that’s not going to survive long-term, so he wants to think about what’s going to survive.

    So in order to hedge your bet there I think you need this orchestration layer that was talked about earlier. You need to sort of isolate the tool vendor from the way it’s consumed to a certain degree, so you can plug-and-play a little bit as time goes on. And this is what we did at Comcast, we built a layer between all of the tool vendors for security tools, because they don’t have a good unicorn solution there. There is no good unicorn solution there anyway, even if I wanted to. And so we had more than one vendor in every tool category competing internally, and sometimes we had three tool vendors. But from the developer’s perspective it was really easy for them to switch out from one to the other, and they got the same feedback channels, they got the same visualization, it aggregated and scored the same way across all these different vendors.

    I think that orchestration layer is really key when you have a best-of-breed approach, and I think you have to build it yourself. I don’t think you can buy it today. I really don’t think you can do that.

    Willis: I’ll try to shut up quickly. I just want to make a comment. I think there’s two books that help us understand our world. One is Guns, Germs, and Steel by Jared Diamond, which actually shows you why we get into these predicaments. If you haven’t read it, it’s a fabulous book. And then Start With Why by Simon Sinek is the sort of anti-pattern to that, which I would say like there’s clusters – atomic computed things like sort of serverless or functions or containers will be around for 10, 15 years. Clusters of those things will be around for 10 years. So if you think in those terms instead of thinking about Kubernetes and Docker – and Docker is already slowly dying, then you get that sort of start with why isolation against these sort of Guns, Germs, and Steel trends.

    Bajpai: Right. Then I also wanted to kind of bring another aspect of this, which is connected, but I think companies might start looking at it from a scaling perspective. We talk about a lot of tools and how to calibrate these tools in our DevOps pipeline. Of course when you have a large organization you have multiple product lines, and these product lines will choose to be flexible, so they need to have that flexibility to deploy their DevOps pipeline in the way they choose. And that’s the whole power of DevOps, right?

    But I also feel that the conversation in this discussion is leading us to a point where we start looking at our technical oversight committee from a governance perspective, where we can stitch the dots, we can actually lay that common foundation which is required in order for people not to kind of rewind the _____ over and over again. So I think that’s the key here.

    And of course when we talk about a specific random _____ with huge capabilities it has pros and cons, right? So one pro would be like they can mutually enforce new capabilities. So when they bring one feature in one product it’s mutually enforcing another product, which is a beautiful advantage they bring in. But that also limits our possibility to look beyond what the vendor is bringing in. So we are kind of essentially not exploiting our own innovation cycles to introduce more capabilities around how we want to experiment, how we want to kind of build innovative new features into that life cycle.

    So these are kind of good considerations to look at and I truly believe that the technical oversight committee is something that you will see in large organizations a wave kind of starts in a bigger way when it comes to DevOps.

    Ashley: And there’s interesting too, when you think about different sizes of companies that are in the market for technologies, you know, Larry mentioned what he did at Comcast or what he was part of that was created as sort of an architecture where tools can be plugged in, unplugged, replaced, etcetera. It sounds simple, but I know that they have an architecture to do that. And some organizations have the wherewithal to create those things. What we see in the data, looking at the market, is in the small business, the kind of sub-200 people or so companies, they have really good high-performance teams that can work to create software or set up their own environments, kind of create a very productive setup for themselves, tool chains, etcetera.

    Then on the high end or the large end, the enterprises, you know, they have a lot of resources available to them that they can potentially marshal to figure out how to do some of those things. The mid-tier market, the medium-sized company is often where they’re most challenged to – they want to do what the enterprise can do, but they just don’t have the resources. They don’t have the money or the vendor relationships across the board to do all those things and they’re expected to deliver at almost an enterprise-class level, but they don’t have all of the resources to be able to do that. And that’s often where somebody – the non-Frankenstein vendors that do a good job of kind of delivering a suite of solutions can work in those situations. Not that they can’t work in the others; they can, too.

    But depending where you are in that kind of a company, that size of a company, makes a big difference on what strategy you can take.

    Maccherone: I think that’s a great point and my recent experience last five years has really been enterprise. But prior to that I was talking to lots of companies, and you’re right, the mid-sized companies are the ones that are sort of in that no-man’s land sometimes, where they need some standardization, they need some consistency, but they don’t have the wherewithal to put it all together. And a vendor unicorn approach might actually be better in those circumstances.

    Shimel: Hey, guys, I want to – Ravyn, I’m going to let you in a second. John, the name of the first book that you mentioned – I want to put these books in the notes somewhere.

    Willis: Oh yeah. I did a presentation with Josh Corman way back with Guns, Germs, and Steel. Anyway, it’s an amazing book about how civilizations move from sort of – you know, how they go from sort of agriculture and they move into these clusters. If you can see the forest beyond the trees you can see how we get into these like Kubernetes is the only word we can use. Anyway, Jared Diamond, Guns, Germs, and Steel, it’s incredible.

    Shimel: All right, we’ll try to get it in there. Ravyn, did you want to say something and I cut you off?

    Manuel: No, I don’t think that was me.

    Shimel: Garima?

    Bajpai: Yeah. I wanted to actually enhance what Larry was saying about midsized companies. This is totally based on my experience and exposure with the Canada DevOps Community of Practice. We are seeing more and more of these midsized companies, not only the consulting companies, as well as the companies who want to build a DevOps capability coming to the community of practice and starting to steer this discussion of evaluation and assessment of tools. And I’m also seeing that the winter side. You know, specifically we are talking about tier two companies here. The winters and coming and exposing the tool capabilities in the community of practice, and they want some support and help for us to kind of steer this conversation “Is this a good tool? Is this a good capability where they are in their context?”

    And it’s a very interesting discussion because the technical side is one side of the story. I would also like to kind of emphasize the fact that the questions we ask these vendors and these companies are like, “Okay, do you have a product owner for this product?” Because essentially, you know, what it means is that how accountability is built in a specific tool perspective. I also kind of ensure that we ask them what kind of themes they have, like what kind of cross-functional themes they have, so that we do not miss out on the point of interoperability, integration, automation, all these like are important points.

    So I just wanted to mention that how we take that approach from the community of practice perspective when we’re asked to kind of assess.

    Willis: You know, I totally agree with all of that. One of the things I was thinking about, I’ve had this luxury of being involved in these different communities. Like I’m in Alan’s community, you know, so I’ve gotten hooked into usually what’s interesting and going on in Alan’s world, which is a pretty big world. I’ve been hooked into Gene’s world pretty heavily, you know, DevOps Enterprise Forum/Committee. And then, you know, I’m one of the co-founders and advisors for the DevOps Days.

    And one of the things that I get to see is true leaders. And I don’t want to put all this sort of weight and gravity on like you have to have a great leader to be successful, but like as I think about the people – Larry is a great example. Like, you know, you can talk all day long about what Comcast does. I’ve seen Larry in action. Like that’s the kind of guy you want in your enterprise.

    Courtney Kissler now, you know, was at Nike and now she’s moved on to a startup. Ross Clanton now at American Airlines, Heather Mickman. I mean I can go on on the list, but these are people that impact change in larger organizations that where it’s incredibly hard. And there’s just something about that.

    Maccherone: There is an immune system at large enterprises, though, that makes it hard for someone like me to actually be hired and be successful. You know, the idea of disruptive change is not something that enterprises are sort of seeking out ever. And so if you’re sort of like grassroots and you’re sort of like ground up and you get a few pilots going, that’s easy; there’s not a lot of resistance. It’s when you get enough momentum that somebody feels like their domain is threatened that you get the political pushback in an enterprise. And, you know, that’s really hard to overcome. Companies have to be very conscious about that. And Comcast has done a great job of that for me in the past, but I see it in the nature of the enterprise immune system essentially.

    Ashley: We probably could take any one of those last three comments and turn that into a whole other dialogue. Maybe we can get some help from the folks in the production department for pulling us back together.

    Alan, do you want to close this out?

    Shimel: One thing’s for sure, is we’ve got a lot more to talk about on this subject. I’d love to – I invite you all back to continue the conversation. But for now, Ravyn, I appreciate you soldiering through this on audio only. If we do it again we’ll make sure to get that video working. Garima, always a pleasure to have you on here; you bring so much. Larry and John, what can I say, it’s a pleasure to see you guys. I can’t wait till we can see each other in person. I know a lot of us have gotten our shots; it won’t be long. But thank you so much for adding so much. Mitchell, fantastic job.

    Most of all, hey, thank you people that are watching this, because that’s why we do this, is for you guys to get something out of it, join in the conversation, let us hear what you’re thinking. Many thanks to Tricentis for sponsoring DevOps Unbound.

    Until next time, though, this is Alan Shimel, Media Ops for DevOps Unbound. Have a great day, everyone. Bye-bye.

    [End of Audio]

  • Logz.io Allies with Microsoft on Open Source Observability

    Logz.io Allies with Microsoft on Open Source Observability

    Logz.io this week announced an alliance with Microsoft that enables DevOps teams to deploy its observability platform—based on open source platforms such as ElasticSearch, Logstash, Kibana (ELK), Prometheus and Jaeger—on the Azure public cloud.

    Jonah Kowall, Logz.io’s CTO, said the multi-year effort enables those teams to deploy the Logz.io platform via the Azure Console with the click of a button. The Azure platform has a data collection capability that aggregates operating system events in a way that makes it simpler for the Prometheus monitoring solution to consume, he added.

    DevOps teams, however, will still have to collect data from the applications they instrument on the Azure platform, he added.

    Logz.io integrates multiple open source monitoring tools based on metrics, logs and traces within a single observability platform that otherwise would require considerable time and effort from internal IT teams on their own.

    Interest in observability is rising as organizations attempt to provide internal IT teams with more context when troubleshooting issues. Whenever there is an IT issue, most teams convene a “war room” where they attempt to discover the root cause of an issue by comparing analytics generated by a wide range of monitoring tools. Observability platforms promise to enable a DevOps team to replace many of those tools in a way that provides more context by integrating log data, metrics and distributed traces within a single platform.

    In many instances, existing application performance management (APM) platforms are being transformed into observability platforms by adding tools to monitor IT infrastructure. Conversely, platforms for managing IT infrastructure are adding support for tools to monitor applications. Elsewhere, a range of startups have emerged to offer observability platforms they have developed from the ground up.

    Regardless of the platform employed, the rise of microservices-based applications within many organizations is forcing the observability issue. The dependencies that exist within microservices-based applications are too complex to manage without a platform that provides some level of increased observability. Of course, most existing monolithic applications are not being retired any time soon, so the need for an observability framework for both legacy and modern cloud-native applications has only become more pressing.

    The Logz.io platform can, of course, be deployed on other cloud platforms as well as within on-premises IT environments. As hybrid cloud computing continues to evolve, it is apparent that observability platforms will need to extend their reach from the cloud to the network edge. Over time, artificial intelligence (AI) capabilities will be added to make it easier to, for example, streamline the number of alerts that are generated across complex IT environments.

    In the meantime, the most important thing with regard to observability may be simply getting started. Most IT teams start to analyze logs before moving on to metrics and then traces using the open source Jaeger agent software. After all, the only thing more difficult to instrument than an application before it is deployed is, arguably, one that is already running in a production environment.

  • WhiteSource Tool Automatically Fixes Code Vulnerabilities

    WhiteSource Tool Automatically Fixes Code Vulnerabilities

    WhiteSource today announced that it has developed the first-ever tool that automatically remediates vulnerabilities discovered in custom code.

    Rami Sass, WhiteSource CEO, said WhiteSource Cure surfaces recommendations for fixing security vulnerabilities in code that developers can then apply with a click of a button. WhiteSource has a long history of providing tools that discover vulnerabilities in open source software that it tracks via a database it manages, but Sass said customers are making it clear they need a way to automatically remediate those issues in a way that doesn’t adversely impact developer productivity.

    The average developer spends, on average, half a day fixing a single vulnerability, so the impact security issues are having on the rate applications are being developed is significant, added Sass. WhiteSource Cure gives each developer the equivalent of their own personal security expert, noted Sass.

    Developers can either apply the remediation suggested by WhiteSource Cure—that manifests itself as a pull request in a DevOps pipeline—or they can fine-tune it as they see fit, said Sass.

    WhiteSource

    WhiteSource Cure arrived as friction and tensions between developers and cybersecurity teams rise in the wake of a series of high-profile attacks against software supply chains. As a result, organizations are instituting more thorough security reviews of application development projects. The concern is those reviews will dramatically reduce the rate at which applications are being delivered.

    Of course, many organizations are looking to shift responsibility for application security further left toward developers to minimize the number of vulnerabilities that might need to be remediated either just before an application is deployed in an on-premises IT environment or, worse yet, after it is deployed. Trying to achieve that goal within the context of DevSecOps best practices, however, is challenging. Organizations first have to acquire vulnerability scanning tools that developers will actually use, and then provide developers with the training required to identify various classes of vulnerabilities.

    The other big challenge is the time it takes to train developers to recognize vulnerabilities. WhiteSource Cure eliminates the need for developers to become security experts, said Sass. In addition to a commercial offering, WhiteSource is also making available a Community Edition of WhiteSource Cure for open source projects that will forever be available for free to developers working on those projects.

    Sass said that rather than relying on machine learning algorithms and other forms of artificial intelligence (AI) to build WhiteSource Cure, the team that developed the tool found that more traditional approaches to software composition analysis (SCA) provided more reliable results.

    It’s unclear whether the ability to automatically remediate known vulnerabilities in code will render much of the DevSecOps debate moot. However, it’s clear that the simplest path forward toward achieving DevSecOps is, as always, to automate as much of the remediation process as possible. The challenge will be waiting to see how much confidence developers and cybersecurity teams will have in the recommendations generated.

  • Shoreline Platform Automates Repetitive IT Tasks

    Shoreline Platform Automates Repetitive IT Tasks

    Shoreline emerged from stealth today to make available a namesake incident automation platform that eliminates the need to manually perform the same repetitive tasks.

    Anurag Gupta, Shoreline CEO, said the platform is based on domain-specific language, dubbed Op, that provides a simple pipe delimited syntax to integrate real-time resources and metrics in a way that allows DevOps teams to invoke an orchestration engine to automate the execution of a task from the Linux command prompt. Op employs a syntax that is familiar to any IT professionals that currently use shell commands and scripting tools to resolve IT issues, said Gupta.

    That same Shoreline tool, however, can also be employed to create remediation loops that check for issues, collect diagnostics and apply repairs automatically in the background, he added.

    Shoreline

    Shoreline achieves that goal by first collecting metrics on a per-second basis via data collected from open source Prometheus monitoring software. Armed with that data, it then becomes possible to distribute queries to both collect data in real-time and run local loops in parallel to automate repairs, Gupta said.

    The platform employs the JAX and XLA frameworks for machine learning algorithms to reduce the amount of CPU resources required to compute metrics and create alarms. Wavelets are also employed to compress the input metric stream. Finally, queries are portioned across historical data stored in the S3 cloud storage service from Amazon Web Services (AWS) and real-time data collected from each node.

    All Shoreline components are designed to be fault-tolerant, added Gupta. Automation can still run locally to Shoreline backends, even when connectivity is lost. All processes have supervisors to restart when failing and reestablish connectivity.

    Finally, Shoreline provides controls to limit the scope of a manual operator action and automate executions in a given time period to help prevent any errors from cascading through an entire IT environment. That approach makes it possible to safely automate tasks in a way that makes it less likely outages will occur because, for example, a misconfiguration winds up being propagated across every system, noted Gupta.

    While IT environments are arguably more complex than ever, Gupta said it’s the repetitive tasks that take up so much of developers’ time. In the absence of any way to automate those tasks, an IT team will, over time, become increasingly burned out, which Gupta noted inevitably leads to mistakes being made.

    At the same time, Gupta noted the Shoreline platform makes it simpler to onboard new members of an IT team because much of the tribal knowledge that once had to be learned on the job has been captured within an automation platform.

    The arguments for automating IT have, of course, been growing louder and more frequent for nearly two decades. Most platforms that promise to automate IT have required a fair amount of expertise to implement and maintain. The Shoreline approach, in contrast, starts with a Linux prompt that many IT teams already routinely use to execute a range of tasks. The difference now is simply the level of scale at which those tasks can be performed with a minimum of programming skills.

  • Shopify Gives Devs More Creative Freedom

    Shopify Gives Devs More Creative Freedom

    Shopify’s Unite 2021 conference was held on June 29, 2021, and the e-commerce platform had some big, developer-centric news to share. Alan Shimel spoke with Shopify’s Glen Coates about the developer-first business model that is giving developers the most opportunity to build their businesses with a new 0% revenue share model, and how developers can leverage as much creative freedom as they want, from low-code capabilities in Online Store 2.0 to “max-code” custom storefronts. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hi, everyone. Welcome to another TechStrong TV segment. My guest this segment is Glen Coates. Glen is with Shopify. Glen, welcome to TechStrong TV.

    Glen Coates: Thanks, Alan. It’s great to be here, man.

    Shimel: It’s great to have you on. Glen, no pressure, but I think you’re the first person from Shopify who has been on TechStrong TV.

    Coates: There you go. _____ [inaudible due to audio distortion] it’s going to be all up from here, like you’re starting at the right place.

    Shimel: We’re starting at a low bar.

    [Crosstalk]

    ‒ it only gets better. We’ll see. Anyway, Glen, I think our audience doesn’t need an introduction to Shopify. We’ve all bought things online through Shopify storefronts and so forth, but why don’t we talk a little bit about you, your background, your position with Shopify?

    Coates: Right. I’ve been at Shopify a couple of years now. Before Shopify, I founded a startup called Handshake, and Handshake was a wholesale-focused e-commerce platform, so very much like Shopify but not focused on direct-to-consumer selling, just on wholesale. Handshake was acquired by Shopify about two years ago, so I’ve been at Shopify since the acquisition. In my time in the last couple of years at Shopify, I initially started out focused on wholesale, obviously, given my background. I spent most of last year focused on Shopify’s checkout, which is, obviously, a pretty important part of Shopify, and then about six months ago, _____ _____ product for what is called Shopify Core.

    If you’re a merchant, Shopify Core is like what’s in the box when you buy Shopify. It’s the online store, it’s the checkout, and it’s the back office admin that lets you set up and run your business. If you’re a developer, Shopify Core is the core developer platform, so the APIs, the dev tools, the docs, all the things that let you build apps and themes and work on the Shopify platform. Yeah, and, obviously, it’s a big responsibility and a really exciting role to be able to help the world of entrepreneurs start and try their business ideas.

    Shimel: Absolutely. Glen, just to give a little color, how many different Shopify merchant stores are there, whether it’s sheer number, percentage of commerce, or what have you?

    Coates: Yeah. Today, Shopify powers almost two million merchants. I mean, obviously, the last year has been kind of an accelerant to businesses going online, and that’s both been new businesses, people trying businesses for the first time, and it has also been traditional businesses going digital more quickly than they might have otherwise. But yeah, today, Shopify powers almost two million merchants worldwide, and it wasn’t that long ago that that was only a million. Even at massive scale, even being the world’s largest commerce platform in terms of number of merchants, it’s still growing very quickly, which I think just goes to the pace at which the world of business is going online, and how many people want some help with that.

    Shimel: Got it. Excellent. Thanks for all that info. If you don’t mind, let’s turn to the topic of today’s conversation, and I’m going to let you explain it, if it’s okay.

    Coates: Sure. Next week, it is Shopify Unite, which is our annual conference. Over the years, Shopify Unite has been focused on a combination of merchant and developer topics, but this year we’re really doubling down on developers, and really making it a developer-centric conference. The reason for that is sort of going back to the question you asked me a second ago, is Shopify was founded in Canada, and, if you go back five, six, seven years, Shopify’s mission has always been to make commerce better for everyone. But, if you go back half a decade, it was mainly North American merchants mainly selling apparel, mainly direct-to-consumer, which that’s a big market but it’s not the world of commerce.

    Over the last few years, Shopify has expanded considerably into new industries, new countries, new ways of welling, so wholesale as well as retail, and so we’re now in a much more diverse world of commerce, different businesses doing different things. As we sort of march down the road and try to fulfill that mission of making commerce better for everyone, that everyone is becoming a lot more diverse as we expand. Shopify can’t build all the solutions for everyone, all those different niches. While we have a great core product that includes many of the out-of-the-box things that people need to start and run a digital business, it’s not everything. We need the help of the world of developers to help build those solutions for the markets for the spaces that they are experts in to complement what is in Shopify Core, and really deliver everything that merchants need to successfully start, run, and scale a business.

    This year, we’re really focused on developers and helping them get onboard with our platform. There are really three main legs to that, I would say. The first one is mainly technical. I think for your audience, some of those details are important, like Shopify really is expanding to be PlanetScale commerce infrastructure, really becoming the power lines, the plumbing, the electricity grid that runs the world of commerce. So, we’ve done some things to really scale up our infrastructure that if you’re a developer building on Shopify, you can be confident your stuff is going to run at global scale without you really having to do anything. Our checkout can handle seven times the load that it could before. We’re now at greater than 30,000 checkouts per minute on a single shop.

    It’s interesting what’s happening to the world of commerce, like the world of social selling has made, and the way algorithms reinforce themselves in terms of distribution to drive traffic towards things that are trending. It’s become more important than ever that a merchant who blows up, whether it’s because of them being re-shared by an influencer or whatever it is, that they can capture that demand in the moment. Traffic is basically getting more spiky on the Internet. When things trend, they trend fast, and so Shopify already was the highest throughput checkout on the Internet but it’s now 7x higher, so merchants should never worry about not being able to make the most of those moments when they’re trending becoming more and more important.

    Shimel: It’s hyperscale, is what it is, Glen. We’re seeing it not only in merchants and commerce, but across everything. The Internet is big business, it is worldwide business, and especially with COVID, right? If you haven’t built your infrastructure to hyperscale, you’ve got trouble.

    Coates: Right.

    Shimel: That’s what digital transformation is about. That’s what the world is demanding right now, hyperscale.

    Coates: I imagine the readers, viewers of your site are not strangers to the world of what it takes to scale websites, and I think a lot of what Shopify tries to do is say, “Well, listen. We’ve figured that out for you. You’re a merchant with an idea. You’re going to come up with a new shoe or a new candle,” or whatever it is. Entrepreneurship is a risky endeavor, not everyone succeeds, but, when you succeed, like we really want you to be able to just capture the whole thing. The checkout works at one checkout a minute when you’re just starting out, and then, on your biggest days, it goes to 30,000 and you just don’t have to do anything. It just works, which is great. Kind of piece number one of Unite this year is Shopify really is PlanetScale infrastructure for commerce, and we already were the leaders and we’ve kind of gone up another multiple from there. Point two is really like, “As a developer, what things can I build on your platform? What are the opportunities for me to innovate, like where can I do that, and why is that exciting?”

    We’ve made some major updates to our online store, to our themes platform, which is obviously every merchant basically runs an online store with a theme on it. There is a new themes platform that allows developers to properly separate basically data and presentation into different layers in a way that makes themes more modular and extensible and kind of upgradable in the future. We have this powerful new data infrastructure that allows merchants and developers to essentially build custom data models inside a shop, and not just be confined to the out-of-the-box data model that ships with Shopify. There is now a much more powerful customization framework for both theme developers and app developers to integrate custom content into themes without having to hack the theme by hand and try to take it off the upgrade path, which is really great for merchants. We have also recognized that the Shopify online store is based on a web templating language called Liquid, which is similar to the templating languages that all developers are basically familiar with.

    We are also announcing, if you want to, you choose your own stack. If you want to build your website using React or something else that’s not our out-of-the-box stack, we’re also making available a new platform codenamed Hydrogen that will allow developers to basically choose their own stack, build online stores using whatever technologies they want, but ‒ and this is the crucial part ‒ still leverage a global infrastructure and hosting in scale and, “I don’t have to think about it. It’s just going to work,” on the Shopify platform, which is really great and really unique. Last but not least, I mentioned earlier that I spent last year on checkout. We are opening up for the first time the ability for developers to build checkout apps. Checkout is the piece of Shopify that it really matters. It’s where the money changes hands. It has to scale, it has to be super reliable ‒

    Shimel: Super secure.

    Coates: Super secure, super trusted, right, and trusted not just in the technical security sense of the word, but even just in the emotional sense of the word, like when I’m a buyer and I’m buying from an online store, when I see this checkout, does it feel familiar? Does it feel like something I can rely on? Am I comfortable handing over my money? Checkout is really where, in the _____ journey, the emotions shift from at the top of the funnel it’s kind of, “Okay, your brand, it’s really different. It’s exciting. I want to see differentiation. I want to be challenged and see new things.” But, when you’re handing over your money, the emotional tone is different, right? You’re not looking for excitement, you’re looking for predictability, right?

    Shimel: Yeah.

    Coates: Historically, Shopify’s checkout has been less customizable than the actual storefront pages higher in the funnel, and so now we’re opening up a new extensions framework for checkout that allows developers to build in apps that securely extend the function of Shopify’s checkout without compromising on the trust, the scalability, the speed, and all those kind of baseline, crucial things.

    Shimel: Very cool.

    Coates: That’s really cool. The last thing I’ll mention is we’ve talked about, “Okay, as a developer, if I build on your platform, is it just going to scale to the world?” The answer is yes. “Okay, so then what am I going to build?” Okay, so there are a bunch of new programming surface areas that we’re opening up for people to build apps and themes into, and then the last piece is, “What’s in it for me? Where are the dollars? If I start building your platform, am I actually going to make any money as a developer?” One of the things we’ve announced is _____ economic opportunities for developers. One is we’re cutting the rev share on apps in the Shopify app store to zero for the first million dollars in revenue per year, and, after the first million dollars, we’ve actually cut our rev share from 20 percent, which was already one of the lower rev shares in the industry, down to 15 percent. We’re really trying to help developers who have an idea for an app or an idea for a theme get that off the ground, no rev share on that first million bucks, and we really just want to help developer entrepreneurs get going as easy as we try to help merchant entrepreneurs get going ‒

    [Crosstalk]

    Then, the second one is that the Shopify theme store has historically not been open for submissions from any developer who wants to build a theme. There has been a sort of a curated set of themes there until now, but we’re actually opening up the theme store, so developers who want to build a theme and sell it to merchants will now have that opportunity that they didn’t before, which is also a huge market that is new.

    Shimel: Look, I think emphasizing the developer and the developer interaction in Shopify is huge. Glen, this is all great. I didn’t want to stop you when you were talking, but of course we’ve pre-recorded this show and it’s now playing after Unite has already taken place, I believe, on the 29th.

    Coates: Right.

    Shimel: But, for people who maybe want to see a replay of it or maybe get the highlights, is there a place we could send them to check it out?

    Coates: Yeah, if they just go unite.shopify.com, they’ll be able to watch the keynote and then dive into the other sessions that were part of the event and dig into whatever they would like.

    Shimel: Fantastic. Glen, thanks for a great recap on what happened at Unite and what’s going on at Shopify. Please come back on and keep us posted. You were wrong. You set a high bar. It’s going to be tough for people to beat, so come in and please keep us up to date on what’s going on at Shopify, especially with this renewed focus on developers and development options on the platform.

    Coates: I appreciate your time, man. Thanks for having me.

    Shimel: All right. Glen Coates on Shopify, about Unite, the Shopify conference here on TechStrong TV. We’re going to take a break and we’ll be right back with another guest.

  • Cycloid Brings Open DevOps Framework to the US and UK

    Cycloid Brings Open DevOps Framework to the US and UK

    Cycloid, a Paris, France-based provider of an open framework for implementing DevOps best practices, today announced it is expanding its operations into the U.S. and United Kingdom.

    Benjamin Brial, founder of Cycloid, said the Cycloid framework provides a lightweight approach to striking a balance between enabling developers to employ the tools they prefer and the need for centralized governance. The framework allows roles and policies to be pre-defined to ensure governance while service catalogs are used to deploy software using a set of drag-and-drop visual interfaces. There are also a set of management tools and features that provide additional visibility, automation, monitoring and financial controls to DevOps workflows.

    In effect, the portal that Cycloid provides to manage those workflows also serves to make DevOps more accessible to a wider range of IT professionals, added Brial.

    Cycloid

    Many organizations that embrace DevOps wind up building their own custom portal to achieve that same functionality. Rather than devoting resources to building and maintaining such a portal, Brial said Cycloid maintains and updates a portal platform that can be employed across any combination of tools that developers prefer. Other DevOps frameworks are often rejected because they require developers to adopt a specific set of tools, noted Brial. Without support from developers, it becomes nearly impossible to adopt DevOps best practices at scale, he added.

    After more than a decade of advocacy, adoption of DevOps within most organizations remains spotty. A significant factor in the lack of adoption is cultural issues involving skillsets. Many IT administrators lack the programming skills required to programmatically manage DevOps workflows. Cycloid is trying to establish a middle ground that enables IT operations teams to manage and govern DevOps workflows without having to acquire programming skills. The goal is to make it simpler for developers and IT operations teams to collaborate, said Brial.

    It’s not clear to what degree such an approach might increase adoption of DevOps best practices within enterprise IT organizations. However, it’s already apparent that the cost of hiring and retaining, for example, a small army of site reliability engineers (SREs) to manage IT environments at scale is often cost-prohibitive for many organizations.

    There’s no shortage of DevOps frameworks, but most organizations generally wind up crafting their own as part of an effort to tailor a DevOps workflow to meet their specific requirements. However, if there is a readily available framework that is highly customizable, Cycloid is betting more organizations would rather buy than build and maintain the entire framework themselves. Otherwise, DevOps teams would spend half their time performing the same repetitive manual tasks over and over, said Brial.

    One way or another, DevOps as a methodology for building and deploying applications needs to become more accessible. Proponents of DevOps have made a case for the methodology for the better part of a decade, with mixed success. The challenge continues to be finding a way to scale DevOps processes across the entire enterprise versus allowing each individual development team to define application development and deployment processes that, today, vary widely in terms of efficiency.

  • Using ML/AI to Support Infrastructure Monitoring

    Using ML/AI to Support Infrastructure Monitoring

    Successful infrastructure monitoring enables IT teams to ensure constant uptime and performance of their company’s systems. Technologies like machine learning (ML) and artificial intelligence (AI) benefit infrastructure monitoring by more quickly collecting and analyzing data from all of the hardware and software components that comprise the IT stack. Infrastructure changes are occurring faster than ever before, but complex systems, the unique nature of applications and lack of IT skillsets can cause challenges when integrating with these newer technologies. However, it’s more important than ever that sysadmins and DevOps teams understand how ML and AI can mitigate these roadblocks, support them in staying on top of infrastructure performance and rapidly address issues that arise.

    Intelligent Monitoring Support for Complex Systems

    The most tangible result of intelligent infrastructure monitoring tools and processes is near-immediate alerting of performance and uptime issues, which can then be addressed in an efficient and effective manner so no business interruptions occur. However, complex systems can stunt these benefits if ML and AI are not being used and manual monitoring protocols are still in place.

    Tools that use ML or AI lessen the work of IT staff immensely, freeing up critical business resources and aiding in overall productivity. Both technologies can automatically identify and update all IT stacks that comprise an enterprise’s infrastructure to keep systems up-to-date and aligned with established key performance indicators (KPIs). In addition, intelligent offerings can detect and factor those metrics against set standards so that early alerts to an “unhealthy” section of infrastructure can be identified, even as the IT stack is constantly changing. This drastically speeds up troubleshooting efforts.

    Differentiation in Applications

    The different applications supported by the various IT stacks will most often have unique service-level agreements (SLAs) for their performance and uptime, as well as remedies or penalties should those service levels not be achieved. Plus, system loads that stress the underlying infrastructure are frequently changed. For these reasons, it is important to identify what constitutes a “healthy” IT stack so that these minute parts of the infrastructure are not overlooked due to the variation involved.

    ML and AI can be programmed to track system baselines that support a “healthy” IT stack. These technologies are particularly great at finding novel and unusual patterns in data. As the monitoring and observability landscape becomes more complex over time, driven by real changes in how developers build applications and systems, the ability to spot and detect such patterns in data can be crucial in helping make sense of it, further cutting down efforts on manual searching, detective work and “death by dashboards,” which we’ve all experienced at one time or another.

    Supporting IT Team Skills with Intelligence Technology

    The role of sysadmins—and to a greater extent, developers—has shifted over the past few years to become nearly as complex as the infrastructure they oversee. Nowadays, it seems as though developers are required to have expertise in all aspects of infrastructure, from monitoring to Kubernetes to machine learning. This can take quite the toll on developers who possess such skills, but in a more realistic sense, developers that can do all these things are very hard to come by. The lack of these skillsets is pervasive in the industry, which is why ML and AI can be seen as supporting technologies—they can fill in these gaps, to an extent.

    With built-in intelligence and automation, ML/AI can enable even the most inexperienced sysadmin or DevOps professional to monitor complicated infrastructure like a pro, taking on most of the time-intensive work around collecting and analyzing the data and identifying where to troubleshoot. The main goal is to put humans in the driver’s seat, utilizing ML and AI for granular discovery of system issues, providing the metrics or charts that might be most relevant to IT staff as they troubleshoot their system and reducing the cognitive load of developers.

    With the vast benefits that intelligent technologies possess, integrating them into your IT stack can help mitigate challenges experienced with complex systems, application differentiation and the skills deficit experienced in the IT team. The important ingredient in making ML and AI effective in infrastructure monitoring is using tools that incorporate the right formulas, algorithms and automation that can best help determine success when it comes to your desired outcome.

  • Rocky Linux Emerges as a CentOS Replacement

    Rocky Linux Emerges as a CentOS Replacement

    Rocky Linux was created to offer the community an alternative following Red Hat’s recent decision to shift its focus away from CentOS—the open source version of Red Hat Enterprise Linux (RHEL). However, the Linux distro’s popularly can also be attributed to how its creator sought to offer a superior alternative based on lessons learned during CentOS’ development—and less as a perceived backlash to Red Hat’s controversial move, said Gregory Kurtzer, Rocky Linux’s creator, who is also founder and CEO of Ctrl IQ.

    “As one can probably imagine, there were a lot of lessons learned from the founding of CentOS—we did a lot of things correctly, but we made some mistakes, as well, that set into motion some of the difficulties the project went through,” said Kurtzer. “I was not—nor am I now—upset at Red Hat or anyone else. It’s important to understand that companies have a legal responsibility to do what they need to do. But I will say, this is why open source communities should not be controlled by a single company, especially if that company has a competing product to the open source project.”

    In addition to applying experience gained from the CentOS project to help “recreate the initial goals of CentOS in a new project,” Kurtzer said he was prompted to create Rocky Linux following Red Hat’s decision to end-of-life (EOL) CentOS. The move “affected my company, our customers, as well as the entire industry,” said Kurtzer.

    “To be blunt, my knowledge—in terms of doing it myself is severely dated. Luckily, after I made the announcement that I was considering recreating CentOS, thousands of people reached out to me wanting to be part of the project and contribute,” said Kurtzer. “From that, our development team and open source community was born.”

    Additionally, CentOS Stream offers value to the community by allowing “us and others” to take part in developing the upstream Enterprise Linux standard, said Kurtzer, who added that his team will continue to debug Rocky Linux so that solutions are properly integrated into the Enterprise Linux Standard.

    “The necessity was absolutely clear, so I founded Rocky Linux and the Rocky Enterprise Software Foundation (RESF),” Kurtzer explained. “Together with our contributors, sponsors and partners, we all have united for a common goal: to create an open source community that will provide us all with an enterprise operating system to stand the test of time, attrition and commercial agendas.”

    Interest in the development of a CentOS alternative was strong when Kurtzer first floated the idea of the project to the community. Since its release in June, the total number of ISO downloads is 56,000, according to Rocky Linux Tier0 mirror data.

    “We have zero visibility as to any other ISO downloads from any of our other mirrors, but if I were to guess, I’d say it would be about two to three times more, in total,” said Kurtzer. “When we all get a moment to breathe, it will be fun to see how many people are using Rocky Linux, and take in all the feedback the community has provided,” he said.

    With this team of developers,  Kurtzer said it took about seven months to build the general availability  (GA) release. “We didn’t just jump right into building the operating system because the goal was to create an infrastructure where many contributors and developers can take part without putting the security of the product at risk. Building that infrastructure took about four months,” Kurtzer said. “Once the infrastructure was completed, we spent about two months building the first-release beta candidate and then spent the following month ensuring that we did everything correctly.”