staging-devopsy.kinsta.cloud

Tag: devops

  • Resolving CI/CD Permissions Issues to Address Delivery Needs

    Resolving CI/CD Permissions Issues to Address Delivery Needs

    Security concerns and DevOps process issues can stall CI/CD operations. DevOps teams are familiar with the frustrations plaguing their pipelines, whether regulatory or process-based. Operational hurdles are all too common in DevOps pipelines, leading to miscommunication between team members and the broader organization. One of the leading operational issues DevOps teams encounter is permission issues. Permission issues are a seemingly small yet significant roadblock to smooth CI/CD pipelines. Fail to address them and the result is a lack of cohesion between development and organizational objectives.

    Here’s how to streamline these processes, boost security integration within the broader CI/CD framework, and maintain robust security postures.

    Review Pipeline Tools for Permissions and Access

    The DevOps life cycle often contains multiple tools, all with different access needs and permissions. As such, this situation is a security nightmare. Jeremy Hess, head of DevRel at secrets management platform Akeyless, called this a “secrets sprawl.”

    “The combination of proliferation and decentralization of secrets creates an operational burden, if not a nightmare,” he said. “For organizations that operate in both a cloud-native environment and classic IT infrastructure, a duplication issue is created due to having their own secrets managed with different tools and cloud-native solutions.”

    There is also the risk of these tools exposing user credentials and permissions to malicious actors. For instance, configuration tools like Jenkins use plugins to determine access and artifact deployment. Because these tools communicate with other pipeline tools, credential details can often be present in configuration details.

    Developer passwords are not visible on the frontend but could accessible from the system backend. Any user with ‘configure’ permissions could request any credential and then inject them into agents. The result is that AWS keys, Git credentials and/or passwords are at risk.

    To mitigate this issue, the first step involves deleting hard-coded secrets from CI/CD tool files. Distributing secrets between multiple tool configuration files also reduces the possibility of attack while easing developer and engineer access. Password managers are also a good choice, but validate them for security before implementing a solution.

    Practice Least-Privilege Access 

    Access and permission issues often create a lot of frustration among DevOps teams as they are forced to assign blanket access to the majority of users, regardless of members’ roles or job functions. While this situation encourages rapid development, it also creates massive security issues.

    Balancing security with CI/CD needs is tough to get right. That’s where the principle of least privilege comes in. Team members receive access to secrets on a need-to-know basis. Note that this principle applies to everything from apps to systems and connected devices.

    While most teams put this principle into practice, they make the mistake of leaving their process intact. The lack of access audits creates DevOps frustration, not the level of access. CISOs should regularly involve DevOps teams when reviewing access to mitigate issues quickly.

    For instance, embedding a security role within every delivery team will mitigate access-related risks quickly. The security team member will have insights into risk-based access needs and can quickly approve or reject requests.

    Creating an access management repository will also remove any confusion related to role-based access. In addition, record time-based and task-based access permissions in the repository. The result is every DevOps team member understands their access paths before projects get started. It allows them time to offer feedback and request one-off access to sensitive secrets.

    Review segmentation rules within your systems when assigning role-based access. Often, these rules will have to change depending on delivery timelines. Involving all stakeholders in these discussions is good practice and prevents frustration down the road.

    Implementing OTPs and other authentication factors is also a good idea when validating user access to secrets.

    Review OSS Projects

    Open source software (OSS) projects are essential to industry growth but might pose security risks if access is mismanaged. Zan Markan, developer advocate at CI platform CircleCI, summarizes the problem aptly.

    “Often, the company that initiated and that owns a popular OSS project continues to employ the core contributors,” he said. “They will probably be joined by other regular contributors and maintainers that are not part of that company. And then there is everyone else—anyone who occasionally might contribute a fix or a feature.”

    As user access grows, security concerns grow exponentially. Enforcing rigid user-based access is unrealistic and detrimental to an OSS project. However, CISOs or other security-focused managers must review whether sensitive secrets are being passed during builds for pull requests. Monitoring permissions—and who can place requests and roles that review them—will ensure a good level of security.

    Establishing machine identity is also critical, given the degree of non-human access pipelines face. Authentication can be based on verifying whether client runtime container attributes match the characteristics of the valid container. Once authenticated, role-based access can take over, limiting access to secrets.

    It’s also a good policy to destroy containers and VMs after they’ve been used.

    Streamlining DevOps Operations is a Top Security Priority

    DevOps is critical to every organization’s success. Access and permission-related issues are common occurrences, but are easily avoided. Reviewing access and establishing a balance between delivery and operational needs is essential to maintaining a competitive edge.

  • DevOps Unbound EP 18 – Standardizing DevOps – TechStrong TV

    DevOps Unbound EP 18 – Standardizing DevOps – TechStrong TV

    Alan Shimel: Hey, everyone, welcome to another DevOps Unbound. If this is your first time catching a DevOps Unbound, welcome. DevOps Unbound is a biweekly video series where we talk about relevant topics in DevOps. In addition to the biweekly show, we do, it’s every four to six weeks, a live roundtable show where we invite the audience to participate and drive the discussion with online questions and comments and so forth, so we open it up to a live audience. Unfortunately, this show is not one of our live roundtables, it is prerecorded, but we have a great panel and I think you’re gonna enjoy it.

    I’m gonna get into today’s topic in a second, but before we do, let me introduce you to today’s panel. First of all, I wanna introduce a business partner of mine and my friend, Jayne Groll. Jayne, why don’t you introduce yourself to the audience, if you don’t mind?

    Jayne Groll: Thanks, Alan. Hi, everyone. I’m Jayne Groll, I’m CEO of The DevOps Institute, we’re a global member association and a certification body around topics related to DevOps. I’m really, really excited to be here and I’m hope everyone listening joins us at DevOps Institute and becomes a member.

    Alan Shimel: Fantastic. Welcome, Jayne. Joining Jayne, I’m happy to have Paul Bruce. Hey, Paul, welcome, and if you wouldn’t mind, you know, introducing yourself?

    Paul Bruce: Yeah, hey, Alan. Good to be back. My name is Paul. In my full time job, I work with Tricentis in a number of different areas. I’m historically a performance and reliability nerd. In my off time, I organize a number of events, help to co-organize volunteer things like DevOps Days Boston, just did o11yfest this year about observability, we had that conversation, I think, with Christine Yen a couple months ago.

    Alan Shimel: Good stuff. Mm-hmm.

    Paul Bruce: And generally, I’m a people advocate. I had to pull DevOps out of my LinkedIn title because there were too many recruiters that I don’t need, because I’ve got a pretty good job. But also, because people advocate really describes what I care the most about, right? People are some of the hardest part of the DevOps people, process, and technology, really, software. So, I really focus my time on that.

    Alan Shimel: That’s great to have you back on, too, Paul—good stuff. Next, we have a first time guest on DevOps Unbound today, a new member of the band, Sarah Baker. Sarah, welcome to DevOps Unbound, if you wouldn’t mind introducing yourself?

    Sarah Baker: Thank you. Wonderful. Welcome. My name is Sarah Baker, my pronouns are she and her. I’ve been working in Operations my entire career and I’ve moved into DevOps as something I always—I needed a name for it. I always wanted the people part of Ops and technology and DevOps was that. And I’ve been doing that as soon as I learned the term. And I’ve been moved and started working with Paul Bruce in the IEEE work group 2675, who is trying to create a standard for DevOps.

    Alan Shimel: Fantastic.

    Sarah Baker: I work for—and today, I work as the Manager of Biztech in Airbnb.

    Alan Shimel: Thank you, and thanks for joining us. That sounds like a great place for you to do this. And then the last member of our panel today is my co-host and another business partner, a long-time friend as well, Mitch Ashley. Mitch, if you wouldn’t mind introducing yourself?

    Mitch Ashley: You bet. Always good to have a good panel on with you, Alan. Mitch Ashley, I’m CEO of Accelerated Strategies, an analyst firm that’s focused on DevOps, and also on cloud native, cyber security, and digital transformation. I come to DevOps both as a practitioner and a vendor leveraging it, so very interested in hearing the perspective of folks on our topic today. So, thanks for joining us, everybody, too.

    And by the way, our DevOps Unbound is sponsored by Tricentis. I think you may have mentioned that, right, Alan?

    Alan Shimel: No, I think I didn’t, Mitch, but thanks for picking it up for me. [Laughter]

    Jayne Groll: [Laughter]

    Mitch Ashley: I was trying to give you credit where credit may not be due.

    Alan Shimel: Yeah, no, I don’t wanna take credit when I messed up. Yeah, DevOps Unbound is sponsored by Tricentis, and many thanks to them for their sponsorship. It makes all of this happen.

    So, let’s turn to the topic at hand today, guys. You know, when I first started staging-devopsy.kinsta.cloud eight years ago, I met Jayne maybe a year, within a year of that or a year and a half of that. One of the interesting things about DevOps is, there was no manifesto. There was no hard definition of what is DevOps and what is not DevOps, you know, contrary.

    And ever since then, it’s been a double-edged sword. In some ways, it’s helped DevOps grow, it’s helped DevOps stay current over the last 8 to 10 years as the world has changed, DevOps has changed along with it. On the other hand, though, we’re still dealing with the, “So, what is DevOps?” Right? “What exactly is DevOps and what exactly is not DevOps?” which is sometimes harder than what is DevOps.

    And as we sit here now, August, 2021, there really still is not a quote-unquote standard for DevOps. There is no standard DevOps. It is what you want it to be, right? As I think Andrew Clay Shafer said, “The DevOps you get is the DevOps you deserve,” right? But have we outgrown that stage? You know, that seems like a terrible twos stage of DevOps to some; to others, it’s what kept it cool. Has DevOps matured? Do we need standardization? Will it be something that kills the goose that lays the golden eggs?

    We’re gonna discuss that today, and I’m interested in all of your views. Jayne, as the CEO of DevOps Institute, how many people have taken certified DevOps classes and certifications with DevOps Institute? It’s gotta be tens of thousands.

    Jayne Groll: Yeah, about 25,000 so far. So, it’s a pretty large community.

    Alan Shimel: Mm-hmm.

    Jayne Groll: And if you don’t mind, I wanna kind of just lay the stage, because I’m really interested in the standardization initiative. So, you know, I had a privilege of seeing, going to DevOps Days in 2012 in California. So, if you can imagine, it was in an old, dusty data warehouse, right, and it was at the very, very early stages. And so, one of the reasons DevOps Institute emerged was because there was a lot of really cool things here culturally, from an automation perspective, right, doing things faster, collaboration.

    And so, we started up DevOps Institute because we wanted to make sure the humans were represented. And I love that Paul and Sarah in particular both represented people. The problem with that is, at that point, it was very, very open, it was very kind of organic. You know, it was really a group of passionate people that were trying to just change the way IT operates, right, from a people, process, and technology perspective.

    What happened after that, though, is that there’s so much noise about DevOps that if I were an enterprise CIO right now, my head would explode, right? Now, is there one right way to do DevOps? No. Is it only gonna get you there with automation? No. You know, is there a very, very strong human element—absolutely.

    So, it’s a double-edged sword, Alan, as you said, is that we wanna be able to kind of scale down, right, some of the common language, some of the common definitions. I always say I’m married to a CPA, they have generally accepted accounting principles; maybe someday, we have generally accepted IT principles. But how do we distill the noise, right, and everyone having a different interpretation of how to do DevOps and what it is compared to, let’s get past the, “what is it?” We’ve spent, what, 10 years trying to figure out what it is, to, “how do we do this?” Right?

    Paul Bruce: Mm-hmm.

    Jayne Groll: And it isn’t just one roadway that takes you there. So, I think it’s a challenge. I mean, we introduced foundational certifications around DevSecOps and SRE and DevOps Foundation and DevOps Engineering, really, to create kind of a common vocabulary more than the, “here’s the single way to introduce CI/CD.”

    So, I don’t know if that helps kinda set the stage, but I think it’s a really big challenge right now that, if we standardize it, we run into the risk of what happened with ITIL where it couldn’t keep pace, right, and it became religious. I come from the ITIL space, right, so it’s like, open chapter 1, verse 3 and sing a hymn before you introduce change management to how do we then, on the other end of the spectrum, it’s so open and so undefined that it’s really complicated for legacy, I call them heritage organizations, to be able to move from this step to this step.

    Paul Bruce: Mm-hmm.

    Alan Shimel: The church of ITIL. Paul, what’s your feeling on that?

    Paul Bruce: Well, everybody’s on a journey, right? Your personal journey, your professional career, whatever, maybe you go do a walkabout on a particular topic to learn more about that. And so, we started this venture, there was an early round of, “Who wants to be involved in something like this?” in 2016. I answered a call from Bob Aiello and Lynn Carter, who were kind of forming this idea of, there’s a bunch of people who really care deeply who have practitioner practical experience in the field. And the early part of this process was just learning, kinda level setting on what we have done, right?

    Jayne mentioned that it’s hard, maybe, to have accepted IT standards, even though there are standards, right, that not every organization applies the same way. I don’t think that invalidates some of the general principles that we know to be true, which is, you know, you ship broken software, you hurt people, you hurt yourself, right? But those are high level. How do we break this apart and start to make sense of the world, as you said, the heritage, Jayne, you said the heritage organizations—what is so wrong, what’s going on that we have these kind of security problems, these performance problems, this inability to be competitive in terms of acceleration and stuff.

    And so, I think the purpose of a standard is not necessarily to force people to do something a certain way. I think it’s simply just kind of like, if you’ve ever had the Christmas ornaments or just ornaments box that has the insert that allows for things to be put in careful places and also work together as a unit, right, to provide a normative, not a prescriptive standard, but a normative standard to say, “Look, these processes that you already know? Well, you already followed these from, let’s say, 12207, right?” These processes around supply acquisition, knowledge management, human resources? Right, human resourcing in DevOps is a big problem, at least in the Boston area. Try to hire an SRE—there’s only about 15,000 to 20,000 in the United States, legitimately. There’s a reason for that demand.

    So, how do we think about the processes we already know and start to say, how are these things different because there are some accepted principles and practices around DevOps? Now, the practices are not things that you would ever wanna dictate in a standard. This is not the same granularity level as some other standards that dictate how to encode time formats. That’s very important at that level. But the level that we were working on was how to level set across the things that, especially organizations in high compliance and high risk industries, right, risk-averse industries—how do they take the practices and processes that they know from a process standard and start to go, “I understand how to plug, now, what the DevOps principles are into the standard to encourage both of those practices in a more predictable manner?”

    So, when we did all this work—it’s taken years, right, of a lot of learning collaboration between a whole host of people I don’t have even time to list out, but a lot of people from a lot of various different aspects of software have a lot of experience in those aspects. You know, it took synthesis and refactoring, and I was up nights and weekends just rewriting some of the statements, you know? And each of these, it’s a line to 12207, so there’s multiple processes, process declarations like quality management, quality assurance, verification and validation.

    Like I said, these project enabling processes and then these technical processes, and each of those process sections have outcomes, activities, and tasks. Things that are likely to occur in this area, right, as well as, regardless of how you do this, right, not dictating what the DevOps looks like, it’s outcomes of these things that are auditable, that not compliance but conformance. That you can confirm at various levels to these outcomes to say, if it’s important to get this right, when you hit the piñata, what comes out should be candy, not garbage, right?

    So, I’ll pause there, because I wanna give Sarah a chance to kinda provide perspective, because she’s also been working a lot with us, for all those years. [Laughter]

    Sarah Baker: I think I counted 27 people on the final author list for the IEEE.

    Paul Bruce: Yeah.

    Sarah Baker: And it was many, many weeks of arguing, reading together the exact language. And I agree very much with some parts of what Paul says, and I wanna step back and talk a little bit about how I see, you know, what, to me, was the big shifts. And so, the way we wrote the doc was to kinda look at the 12207 and say, “How does DevOps change it? How has it changed the game? What are the perspectives that you need to have in mind in order to get the outcomes that Paul’s talking about, there?”

    And so, when we were writing it, we were writing it to that. How is this different, and to what extent do you wanna consider these areas? It’s a framework of thinking. At its largest level, it’s a framework of, for me, the value chain of delivery of software. And it’s not—and it uses many of the concepts that come out of learnings of industrial engineering, way back in the ‘40s, ‘50s, ‘60s. But it had never really come into a lot of the thinking of software, because software is different. And the more I was working and kind of digging into what are the concepts underneath DevOps, I came quite aligned with a lot of industrial engineering concepts about how things flow and how things move back and what the peaceful aspects are in that.

    And so, that—actually, if you read through the standard, you’ll see a lot of those flow pieces and all of the pieces that you then say, “Oh, it’s the feedback loops, yeah, lots of feedback. Collaboration, it’s all there.” And it’s just applying a lot of those learnings into software engineering, to me. And it was stuff I was like, “Oh, it’s so wonderful to have this.”

    And the other thing, one of the big learnings that I think I brought into the conversation is how—and Paul will back me up on this—how security for a lot of us changed slightly as we started walking our way through the 2675 work. The terms of risk became a very interesting concept that flowed through the document more than I would have expected.

    Paul Bruce: Mm-hmm.

    Sarah Baker: And so, there was some learning within the totality of our team as we wrote this. It was quite fascinating.

    Mitch Ashley: I’m really fascinated to hear from both of you that have been working on this, because it’s not something I’ve been involved in. I sorta have a mixed history with standards, because I’ve worked in areas like networking and things like that where standards are very beneficial or really fundamental, but I’ve also learned standards are kind of what you’re gonna do, not how you’re gonna do it. Because we ran a test lab that, even though it says we’re gonna do least cost routing in a network device this way, people implement it differently and they don’t always work together. So, it’s not a nirvana, but it’s also not a pariah, either.

    Paul Bruce: Mm-hmm.

    Mitch Ashley: So, what I wrestle with here is that, I think some of the thinking—and Jayne can speak to this, but innovation happens so quickly now. So many things are created without standards; maybe they evolve into standards, maybe they don’t. And it doesn’t bother me that there’s lots of noise around DevOps, because we’ve all seen the cycles, the hype cycles, right? If it’s worthwhile, everyone and their brother, sister, cousin, uncle will all jump on the bandwidth and call what they’re doing fill in the blank—it’s AI, it’s DevOps, it’s whatever. That’s just part of our industry. And it’ll either fizzle—you know, value stream management, is it a real thing? We’ll see. It’ll fizzle or it’ll happen, right, if it’s a real thing. And DevOps has proven itself not only is it a real thing, but other parts of the organization now see an opportunity to play, to participate, to become part of the software creation process.

    Paul Bruce: Mm-hmm.

    Mitch Ashley: And I think that’s what’s so powerful about DevOps—it’s not about a technology, it’s about how we create software. And if you were gonna start out, you know, Jayne, when you went to that meeting in 2012 when you said, “We’re gonna go create a standard for completely doing software differently because we have cloud and we have the ability to automate and we were gonna change the software architecture,” you couldn’t have known how to do that. But being the industry would’ve, you know, the antibodies would’ve crawled all over all of us and stopped it. So, I think there’s a lot of power in the innovative, organic nature of it.

    And I don’t want DevOps to turn into ITIL, [Laughter] because that’s sort of taken on a life of its own, and now it’s trying to catch up and it’s done some good things, too. So, I wrestle with trying to constrain it when I think it’s real power is the innovation behind what we’re doing.

    Paul Bruce: Yeah, so, before I—

    Sarah Baker: If I could take a moment—

    Paul Bruce: – yeah, go ahead.

    Sarah Baker: – I’d like to take a moment and call out something. Part of the problem with ITIL is like, it’s ITIL. And, you know, a gazillion managers said, “ITIL, let us get a hammer and hit you over the head with it.” And DevOps, fortunately doesn’t have that problem. It has, you know, a little fuzzier in its definition, but the great thing about DevOps is that it has that sort of fuzzy edges. You can’t, it’s really hard to beat someone over the head with it. You can try. I’ve seen it happen, you know? Everybody’s a DevOps engineer now, but it’s not necessary.

    And what people talk about is not that, but the work, you know? It’s not, “We’re conforming with ITIL,” it’s like, “We’re doing DevOps.” The doing part is the important part, not the compliance part. So, when you say, when you hear people talk about it, they talk about it differently, and that’s the good thing.

    I use ITIL, I just don’t tell anybody the word.

    Jayne Groll: Which is a shame. It’s such a shame because, you know, I come from the ITIL space, and in the early days, this was not what the intent was to be. It was very much like DevOps in terms of, “Let’s help IT grow up,” right, in terms of some things.

    And, you know, to what you’re all saying, you’re absolutely right. A framework, whether it’s a standard, is a guideline, right? It helps you with the journey, as Paul said. And so, if I wanna kinda check, check, check—you know, I’ve worked with some of the ISO standards, right? So, in the ITSM space, there was ISO 20,000, 27,000 for security, right? And it basically, like, in the one or service management says, “You should record all your changes.” It doesn’t say how to record it, it doesn’t—it’s a really great practice, you should do this.

    The problem, sometimes, with standards—and I’m sure you know this and it’s a risk—the pure intention is pure. The ability to weaponize it, right, from the auditors and from others is a real risk, right? Because we’ve seen that with other standards as well, where it becomes more about the compliance than it is about how having it being a guiding light. So, I think there’s always a risk there.

    And then the other risk is, you know, think about observability, right? So, observability has been trending, what, for a year, maybe two, right? Four—okay, but really mainstreaming, like, talking about observability as kind of a mainstream practice, right? And I will share with you, Mitch, to your point, when we launched DevOps Institute in 2015, we were not welcomed with open arms, right? We were vilified for trying to put some boundaries around this.

    Now, you know, move forward seven years and I think we’ve overcome a lot of those and we have some that were detractors or really great supporters. But I think they were concerned that it would become weaponized or commoditized or whatever.

    So, I’m just gonna say one last thing because I think it’s important. Commoditization of knowledge, which, this is kind of in support of the standard, right? Commoditization of knowledge where we make sure it’s available to everyone, some people see that as a negative—it’s not. What would happen if the whole world, right—every IT organization, every IT professional decided to take this standard and use it as a guiding light, according to the needs of their organization? They weren’t worried about the audit, they got trained, they shared knowledge—wouldn’t that be amazing? But that’s commoditization.

    So, I think the risk of good versus evil on initiatives like this is real, but I also think that distilling it to everybody, kind of like accountants have done, right, is important because IT needs to grow up.

    Paul Bruce: Yeah, well—

    Mitch Ashley: Well, and if you look at the accounting standards, they don’t tell you how to do it, they tell you what to do.

    Jayne Groll: Right.

    Mitch Ashley: It isn’t prescriptive in the how very much, so it’s very much like an IEEE standard or whatever.

    Paul Bruce: Yep. Well, first off, I forgot to mention when we first started—I speak as an individual contributor. If it isn’t abundantly clear, [Laughter] I don’t speak for the IEEE, and I don’t speak for the other people who have spent years contributing to this meaningfully.

    So, when I say certification processes inherit all the exact same things you just mentioned as problems—weaponizing, actually low ROI maybe in some cases—we see these deficiencies and these outcomes that aren’t so great. And if you’re somebody who’s just gonna sit back and say, “I’m not willing to hold myself accountable” to stand up and say, “What does this thing actually mean like a real engineer should, you know, to start to break apart the problem and see what are the components, how does this thing work and how do I want it to work? How does it work best for the people around me as well?” If you’re not willing to do that, then I would call you the nay-sayer, you know what I mean? Like, not any one in particular, but that would be a nay-saying, like, let’s avoid accountability, let’s avoid doing what we know needs to be improved?

    Continuous improvement is a huge element that is all throughout the entire standard. It’s one of the things that’s identified as some of the core principles in an ISO study that was done before we started in on this standard to see what are the common themes of people, when they’re really feeling like there is this DevOps mindset, this culture, these practices—what do they mean by it?

    And a number of things came out. One was a high degree of automation, right? Okay, well, that’s very tactical. No matter how you do it, right—high degree of automation. Duh, right? How about effective communication and collaboration? How about shifting things and continuous everything? The notion of, that these things are not this one time and then three years later, you get to come back. No, no, no. If it’s a good idea now, maybe don’t bite off a three-year apple. Maybe take a bite, but know that you started in, there’s a continuous improvement process.

    I can say, we mentioned earlier about a holy war that might exist when you start getting into what is DevOps and what is not DevOps. Like, I started—the best way I can describe to people, because like I say, half of my day is spent on and off of Slack with people who are actually doing, working this out in a highly statistical, large distribution across the country, you know, and sometimes outside the U.S., too, just time zone supply.

    But I can say, for sure, after seeing the complete garbage Dumpster fire where nobody is willing to stand up and say, “Hey, aren’t these things common and standard?” And oh, by the way, not vendors that have something to push, right? I mean, that’s, I think, Jayne, where you were talking about maybe observability is now a thing. You know, the only reason why it’s a thing for the most part is because it’s in our face now because all the vendors are hash tagging the crap out of it.

    Same thing happened with DevOps five, six years ago—everybody had to get on the bandwagon. I remember CEOs way back then going, “What is this DevOps thing?” Oh, man. You know, just talk to some of your engineers.

    And what I can say, right, I have developed at least an internal personal perspective on what is not DevOps. So, if I can borrow the last 30 seconds of the mic for a second, here are some things I wrote as notes as I’m going back over the standard, re-reading it, going what I violently agree with from both others and myself, we did not have a chance to see this before it went live many times, right?

    So, what I got from one of the higher order sections in Section 5 was this feeling of, like, what is not DevOps. And those things are burnout. DevOps is not burnout. You know, you’ve probably been burnt out if you’ve worked in DevOps before, so where is this coming from? Why is this the case? It’s not burnout, it’s not no help. You don’t get any help, go do this thing, you know? It’s not heroics. It’s not superheroes in capes, right? Single points of failure, as I call them. It’s not cliques—which, by cliques, I mean power silos. Huh-uh, there is no such thing as The DevOps Department, people. There’s no rogue decisions. There’s traceability in decisions. There’s accountability in decisions.

    Leadership thrash—not easy to, if you’re really adopting this DevOps mindset and way of thinking about how to do things better, leadership thrash doesn’t really fit in. Toil and debt, right? One of my biggest things is, you know, I seek to eradicate toil, which is not easy. [Laughter] Just reducing it a little is nice, but it’s, and it’s probably pragmatic to say so, but really eradicating toil from other people’s plates and from your own, that’s a challenge, right? That’s kind of a life goal. And if we do this stuff together, right, if we say these are not the things we want, the question is how do we move towards something that isn’t those things, right, and that is closer to where we all align and agree, like, this is a better way of delivering software and doing it faster, right?

    So, anyway, I’ll pause there.

    Mitch Ashley: You know, one of the things I’d love to hear from both Sarah and Paul is, there’s always sort of that, “So what?” answer—question, right? “If we did this, what is the result? What’s the impact of it?” And I mean that in a positive way, I’m not being critical.

    Paul Bruce: Yep.

    Mitch Ashley: If this standard was widely adopted, how would it improve? How would it—what things are we wanting to improve and change from where we are today? Can you describe that in any way to kinda give myself and others maybe an idea about that?

    Paul Bruce: I just did. People already implemented it.

    Sarah Baker: Well, at the base, there’s the vocabulary, yeah.

    Paul Bruce: Sorry, people are—

    Sarah Baker: At the base, that is part of it. Sorry. Paul and I always get [Cross talk].

    Alan Shimel: Sarah, go ahead. Paul spoke—you go.

    Paul Bruce: Yeah, yeah.

    Sarah Baker: Vocabulary is that first piece, making sure we’re all talking about the same framework. And so that, when we’re exchanging ideas, how did you do it, how did you do it—you know, not any one way to do it is gonna it, but knowing what your options are and talking about it with others and trying something out and figuring out if it works or not in your environment is one way to get everybody on the same playing field. But if you’re not talking about the same thing, you can’t do that.

    So, it’s having the ability for a large corpus of options to apply to your situation. To me, that’s one of the biggest things that it provides, and it also gives you ways of interacting with the rest of the business that is not IT, and kind of, “Oh, this isn’t—it’s not just IT. You’re a stakeholder, you’re a customer.” And it includes all of those pieces for the collaboration and really helps support it.

    So, to me, that’s what the standard brings. It widens the organization’s perspective a lot, and allows that cooperation to seep into the organization across all of these different pieces. Because to me, that’s how it started as DevOps, DevSecOps, DevOps Observation Ops. I mean, it grows because that’s part of its DNA is to include.

    Paul Bruce: Yeah.

    Sarah Baker: And so, it’s very natural that it has done that and it includes other parts of the organization into that basic thought processes. I totally could see DevOps marketing security—I mean, you see all that. That’s what’s, to me, the real advantage of thinking this way is that it includes the entire org.

    Jayne Groll: Well, and first of all, utmost respect to Paul, Sarah, and the other 25 people that are working on this, because this is a gargantuan task, right? Trying to encapsulate something that creates a common framework, that has the purest intention and also, you know, if you think about the heart of why DevOps emerged in the first place, it emerged because we were not collaborating together, right?

    So, you know, in the early days of IT—I always say I’ve been in IT longer than I really want to admit to—you know, there were 25 people that sat on the same floor, in the same office building, and we all knew each other and we went to happy hour. And then, you know, the tree grew and there were lots of branches and other things. So, you know, giving a central source of truth that everybody can kind of internalize, process, speak the same language—you know, IT, we have so many different languages, we’re like countries that are butt up right next to each other.

    And I said, I have so much respect, because this is a gargantuan task. And Sarah, I agree with what you said, the purpose is really to take this need for collaboration and optimizing automation, right, so making automation essentially members of your team, right, and then taking that out into the business because every business is a technology organization, it’s—I’m sure you’re aware of the risks. You know, we’re not telling you anything that you’re not aware of, but as I said, I just have so much respect. Because it’s not gonna be an easy road, right? There are so many different opinions from vendors and from everyone who’s involved in this space that are going to have an opinion or whatever. And I think, you know, that’s something that everybody’s gonna have to deal with.

    Again, the only thing I would suggest is build it in a way that it’s nimble—dare I use the word agile, so that—

    Paul Bruce: So, this is already done, right?

    Jayne Groll: Yeah.

    Paul Bruce: Like, this is not a draft, this is not an initiative. This has already been done for the past four years and is now an official standard. So, the good news is, to your point, I think, about suggestions about how to do this, we are working on the second version of this draft already.

    As soon as it was out, even before it was out, we knew there were things that we just couldn’t—we couldn’t spend the time properly addressing. Like I said before, there was a lot of refactoring, but there was a lot of careful avoidance of not already repeating what was already in a good security standard over here, or in a good risk management thing, or one of the folks that worked with us, Tafline Ramos, works at the ISO, right? And what is it, 29119, right, verification and validation, the processes—that was already stuff that’s already been, and great thinking and not slow, you know? Can fully be put into a modern, unicorn sneeze rainbows and fart sprinkles type approach as well as the large Fortune 100.

    So, just to say, this is already—we’ve already put a ton of, it’s already out there, and the easiest way is, I mean, some people might be wondering, “Oh, yeah, but it’s a standard, so now I gotta pay for it?” Some standards, you do; some standards, you don’t. This one, IEEE, they cover their costs with the standards. The easiest way, if you wanted to get a free copy, become part of the new, the next version of this, right? See what we’ve done, poke holes in it, have collaboration, let’s work on it together, because that’s the point of this is to bring a framework of how we are working together in a more effective manner.

    Jayne, I’m really sorry I cut you off.

    Jayne Groll: No, no, you didn’t cut me off. Just a cautionary tale. One of the reasons ITIL kind of lost a lot of its momentum—and I’ll be very frank, I owned an ITIL training company for 15 years and I was a chief evangelist, but it took 8 years between version 3 and version 4. And so, it wasn’t built, it was a monolith.

    Paul Bruce: Mm-hmm.

    Jayne Groll: And I think that was—you know, in the meantime, lots of other things happened, and I think there’s a cautionary tale. I’m not suggesting you’re looking at eight years, please don’t misunderstand, but this market is so—

    Paul Bruce: No, it’s probably about three, right?

    Jayne Groll: Yeah, this market’s pretty fast. Sorry.

    Paul Bruce: Our target is about three, between where we officially started and now it’s out and the next revision, at most, would be another three, right, that’s our target. It could be earlier or later than that.

    Think about, we’re all citizens of the United States here, right? Or anybody in Canada?

    Alan Shimel: Is that a HIPAA violation to answer?

    Paul Bruce: Yeah, right.

    Alan Shimel: Just kidding, just kidding, just kidding.

    Jayne Groll: [Laughter]

    Mitch Ashley: I’m not showing you my vaccine card.

    Paul Bruce: I ask because, I ask because—

    Alan Shimel: I didn’t wanna go there, but go ahead.

    Paul Bruce: – we all know what the U.S. Constitution is and where it came from.

    Alan Shimel: Mm-hmm.

    Paul Bruce: Do we think that it was perfect and never has to be revised? Absolutely not. That’s what the Amendments are for. And arguably, I people wanna go back to the original version, there are some serious problems with that original version. Same thing with ITIL, same thing—remember with, I think you might have heard about The Agile Manifesto. You know, that was a drunken party with a bunch of white dudes, okay?

    Jayne Groll: I know. [Laughter]

    Paul Bruce: So, yeah, there were some good ideas that were encoded in there, but do they not have to be revised and worked out? Even Dave Thomas says agile is dead, long live agile. Like, what people have taken it to is not the point.

    So, the goal of this is to really get a full perspective. To what Sarah was saying, it’s not just—I’ve talked with a lot of the DevOps Days, various different people from DevOps Days, from a number of different, like, The DevOps Enterprise Summit and stuff, and there are some people out there that are like, “No, DevOps is just about developers and operations and it’s about just these things, and if you start getting biz in there—no, no.” And it’s like, what a narrow perspective, because you know what, you had a good idea and then you screwed it up by not including other people, by not realizing there are meta learnings that come out of what you locally did—maybe, arguably, you did well—that apply broadly. And when they do apply broadly, they bring us back to the same premise, which is, we’re all in this together, right? We all win when we all win. As Tim O’Reilly might say, provide more value than you consume, right?

    So, that’s where we intend on taking this. It’s not to turn into a bunch of agile consultants again or ITIL consultants, right?

    Mitch Ashley: [Laughter]

    Jayne Groll: Framework growers, that’s what we call it.

    Alan Shimel: Yep. Guys, I gotta pull the plug on ya, I’m sorry, guys, and ladies. We are out of time. I think a great way to end this, though, is—Sarah, Paul, where can people get more information on the standard from the IEEE?

    Paul Bruce: Sure. Google it, Google IEEE 2675, you’ll get to that page if you wanna buy a copy.

    Alan Shimel: 2675.

    Paul Bruce: I think without being a member, it’s like 99 bucks—so, look, you know, if you wanna buy it for yourself—

    Alan Shimel: It’s not breaking the bank.

    Paul Bruce: You probably also have, in your organization already somebody’s a member, your Risk and Compliance are probably already members to IEEE.

    Sarah Baker: Yeah.

    Paul Bruce: And again, like I said, the best way to get a free copy is to come work with us on what does the next standard mean. It’s not a huge effort and commitment like Sarah and I have put in. [Laughter]

    Sarah Baker: [Laughter]

    Paul Bruce: You can dip your toes in, see what makes sense, what works for you, and we love to—we include people in that effort.

    Alan Shimel: Great. Well, Paul, Sarah, Jayne—thanks for joining us on our panel today. Mitchell, as usual, great job. This is another edition of DevOps Unbound in the books, check it out. Go check out the IEEE standard. We’ll be back in two weeks with a fresh show and a fresh topic. Until then, this is Alan Shimel, be strong, be well—Tech Strong. Take care, everyone. Bye bye.

    [End of Audio]

  • What the New OWASP Top 10 Changes Mean to Devs

    What the New OWASP Top 10 Changes Mean to Devs

    The open web application security project (OWASP) recently updated its top 10 list of the most critical security risks to web applications after four years. It represents the most radical shake-up since the list was introduced in 2003. The changes will undoubtedly have a big impact on how businesses address application security going forward and how developers and DevOps teams approach their jobs. This article will look at three of the most significant changes that have emerged in the new top 10 list.

    OWASP

    New Methodology for Current and Future Threats

    This time, OWASP took a more data-driven approach to their research to get better insight into current and future threats. Members provided more than 1.5 million data points on the security threats they see. OWASP categorized the data and assigned an impact score before deriving their overall ranking.

    OWASP also included survey data from security professionals about emerging threats. For example, the current incident rate of server-side request forgery (SSRF) vulnerabilities is low, but security professionals consider this attack very serious and expect it to increase significantly in the future. SSRF enables attackers to use vulnerable servers to request and receive data from protected internal sources, which is a very serious risk. So, SSRF became a new category (A10:2021) this year.

    With this new methodology, OWASP is now able to offer comprehensive insight into the most serious current and future threats.

    OWASP 10 Expands Security to the Left

    One key change in the new top 10 list is the inclusion of many categories (e.g, Insecure Design—A04:2021, Software and Data Integrity Failures—A08:2021) that recognize the industry has to start with better application design practices to improve security.

    Many application vulnerabilities creep into software because secure design principles are not followed from the outset. In the race for faster app development, corners are being cut. The CI/CD approach to application development is a major contributor to the use of plugins, libraries or software modules of dubious integrity. This problem is getting worse. Businesses must ensure that all their software components are from reputable sources and should use software supply chain tools to check for known vulnerabilities.

    Major Ranking Changes due to the Evolving Threat Landscape

    Injection attacks, which have been ranked as the number-one risk since 2003, are now ranked number three. While this is welcome news, we cannot claim victory just yet. Your valuable data is still very much at risk from vulnerable apps that allow bad actors to run unauthorized commands and access the sensitive corporate information your business depends on.

    Injection attacks have been replaced by broken access control (A01:2021). OWASP reported that in their data set 94% of applications were tested for these vulnerability types and 3.8% showed one or more weaknesses. A staggering amount!

    Due to the increased adoption of standardized authentication frameworks which are more readily available and easier to implement, identity and authentication failures (A07:2021) have plummeted in the ranking from number two to number seven as a risk.

    This shift demonstrates that as businesses have improved their determination of who can access applications, they have neglected to enforce controls over what an individual user, process or device can do in that application. It is crucial to consider authentication and authorization together for a better security posture.

    Lots of Work Ahead to Bring Left and Right Security Together

    The 2021 OWASP top 10 list is a big step forward. OWASP’s expansion of security to the left with the inclusion of new categories and significant changes to their rankings will require businesses to revaluate their application security posture. Addressing security earlier in the application development life cycle will likely prevent many of the more common attacks, but businesses must complement this with robust, proven and scalable security protections on the ‘right’ like web application firewalls. It is not just about shift left, it is about expand left. You need both left and right security for a better multilayered security posture.

  • Four Steps to Avoiding a Cloud Cost Incident

    Four Steps to Avoiding a Cloud Cost Incident

    The recent Flexera 2022 State of the Cloud Report found that organizations waste 32% of their cloud spend, up from 3o% last year. This can be due to cloud cost incidents triggered by unused resources, malicious activity or overambitious projects and which have a massive financial impact if not found and corrected promptly.

    As an organization’s cloud infrastructure footprint matures, mapping cloud resource use and costs accurately to product lines, teams and business units becomes progressively more complicated and problematic. However, accurate cloud cost attribution is vital to understanding team resource use and determining accurate cost of goods sold (COGS) margins and is imperative to budgeting and forecasting. Organizations need reliable, granular views of the cloud environment to attribute costs back to departments and optimize a meaningful budget.

    Cloud cost spikes can blow up an organization’s budget fast and can be hard to track down, but enterprises can help protect themselves from a cloud cost incident with just a few simple steps.

    Have a Strong Tagging Strategy That Attributes Cost Granularly

    The cloud operations team needs to identify what is important to business reporting, which will help define and create a strategy for tagging. For instance, when an organization has two teams managing a particular service, the teams should safeguard the services tagged with a team identifier. Also, the teams should add another tag for the environment in which they’re running so they can allocate cloud costs across teams and environments.

    When an application uses numerous resources, like a database and storage, teams should consider adding a specific application tag and break down costs by each application. It should be noted that there is no one-size-fits-all tagging solution, so enterprises will need to determine what works and try out different strategies by adding and removing tags later.

    Understand Cloud Cost Ownership to Avoid Chaos

    Ask yourself how well your organization has attributed cloud costs to teams, applications or COGS. If the answer is “Not well,” then you are missing vital information about how your cloud costs relate to your business.

    The lack of this context or visibility into resource allocation makes it unmanageable and impossible for cloud operations owners and senior leaders to agree on who is responsible for varied expenses and resources. Manual cost attribution doesn’t scale accurately or account for shared cluster resource use across teams and applications. This creates confusion that can escalate when users arbitrarily slice data and claim ownership without accountability or clear visibility.

    What happens is that costs can sometimes be double-counted with two owners—or even missed completely with no owner whatsoever. The resulting data emerges as an inferior quality that is untrustworthy and not actionable. The inability to visualize resource use in context can dilute, hinder and even defeat cost improvement initiatives and optimization.

    Cloud costs should have an owner who can ensure that cloud spending is tied to business value. Understanding who owns which pieces will help pinpoint the cause.

    Establish and Monitor Cloud Budgets

    Historical data on cloud spending is gold—it can help forecast and budget for what should be happening with the cloud’s cost footprint. If that’s not happening, then it should be flagged and investigated.

    Once you get cost data under control, organizations can get an accurate picture of where cloud spending is taking place. However, does this match the expectations of spending? By establishing and monitoring budgets at every stage of a project, organizations can have guardrails to cloud spending and prevent costs from soaring out of control when engineers overallocate on performance “just in case.” Once budgets are in place, organizations can monitor spending and right-size resources to balance actual usage, rather than estimated usage at the start of execution.

    Over time, right-sizing resources will allow for optimal spend, as well as identify unused resources that can be shut down. Organizations should identify development and test instances left running when no longer needed and shut them off.

    Implement Cost Anomaly Detection

    Organizations can shield themselves from cost anomalies by continually monitoring and optimizing cloud costs. Cost anomaly detection can help act as a barrier to early recognition before they become problematic. This is especially effective when prompt, actionable alerts can be delivered directly to resource owners via Slack or email when their forecast exceeds the set budget. The best option would be adopting a cloud cost monitoring and optimization (CCMO) tool that gives alerts in real-time when a cost anomaly occurs. These tools must be able to detect anomalies and analyze their root cause, while also allowing visibility into what went wrong to avert similar future errors.

    Executing continuous cost monitoring and anomaly detection tools for dynamic cloud environments is essential in identifying any activity that doesn’t align with expected expenditure or diverges from an established pattern. With proper monitoring, organizations can take corrective measures before the fact.

    Unwanted and unexpected cost incidents can clean out an enterprise’s budget quickly, especially as its cloud infrastructure grows. Organizations need to practice these steps to gain a full picture view of their cloud environment.

  • How FinOps Can Optimize Cloud Costs and Drive Innovation

    How FinOps Can Optimize Cloud Costs and Drive Innovation

    Two mainstays drive most application migration and modernization initiatives. First, organizations are moving workloads to the cloud as an innovation catalyst. Achieving their strategic goals requires the speed, agility, reliability and advanced technical capabilities available to them in a hyperscale cloud.

    Second, they’re intent on ensuring the return on investment (ROI) through cost optimization, which is different from the abstract “cost savings” that was part of many early cloud pitches. It’s a recognition that attaining ambitious strategic goals depends on ruthless attention to the bottom line. Cloud enables the capabilities described above not just by making them available at the click of a button but via a finance and business model far more palatable to CFOs and boards than if the company were to build everything in their own data centers – including the data centers themselves.

    Cloud Usage Costs

    Therein lies a fundamental problem—and one of the major application modernization trends we’re seeing in 2022: Cost optimization in the cloud is far more challenging than many organizations first expect. And as cloud use bills balloon, the costs begin to undermine the benefits of migrating there in the first place.

    Identifying cost savings in the cloud isn’t really the problem. Doing so isn’t easy, per se, but there are various ways to go about it—such as changing a storage type, instance type, or retention policy, among many possibilities.

    Implementing those cost savings is the real issue. Companies know where and how to optimize utilization and spending but struggle for various reasons to make the changes needed to realize these gains. This challenge is why we’re seeing the growth of FinOps in 2022. FinOps is the culture and practice of creating visibility and accountability to manage cloud spend throughout an organization.

    FinOps responds to two significant problems that hinder cloud cost optimization measures.

    The first is conflicting incentives. The IT team or a consultant or managed services provider may identify opportunities for saving money in a company’s cloud bill, but that same IT team—the ones typically responsible for the “keyboard work” needed to implement the changes—may not truly own the application in question. Moreover, the business unit or team that does own the application may not, in fact, have any financial accountability for the infrastructure that powers that app. They don’t have any skin in the game and are often reluctant to approve changes especially if the app is meeting their performance goals.

    Second, many organizations migrate to the cloud only to realize they lack dependable visibility into how they are allocating costs in their new environment. A CFO or even CIO with a multi-million-dollar cloud spend may have very little visibility into who in their organization is spending that money—much less why they’re spending that money.

    These are significant issues that can spiral into runaway cloud bills—and ultimately suffocate the business value that the cloud was supposed to unlock in the first place.

    FinOps is to Dollars What DevOps is to Code

    Enter FinOps: FinOps is to dollars what DevOps is to code. It is essentially about driving awareness of cloud spending throughout a business, from C suite to a junior developer. In doing so, FinOps makes possible the kinds of cost optimizations required to attain the cloud’s transformative potential.

    Just as DevOps fosters values like shared responsibility and visibility (without playing the blame game) to sustainably increase development velocity, systems reliability, and other priorities, the FinOps framework brings a similar approach to cloud resource utilization. DevOps is ultimately concerned with that first mainstay cloud goal I described above: Achieving speed, agility, reliability, etc. FinOps focuses on the second mainstay: Cost optimization.

    In this way, FinOps is becoming a vital trend in application modernization because you cannot have innovation without cost optimization. (If you think otherwise, invite your CFO to lunch and see how they feel about it.)

    FinOps engenders cost optimization in various ways, but there are two key facets I’ll highlight here. The first is that FinOps culture and practices typically entail implementing “showbacks,” a system that clarifies who is using (read: spending) what in the cloud.

    Showbacks take some time to set up—they involve things like resource labeling and tagging and creating a reporting structure, for example—but are well worth it. They form the foundation of shared visibility into resource utilization and spending. With showbacks in place, you can see who is spending what and optimize accordingly. For example, you might discover a business unit that uses a high percentage of overall spend for a relatively low percentage of revenue. There might be a good reason for that, but you may have uncovered an opportunity to reduce spending if not.

    Chargebacks are the second phase, in which application owners, teams and business units – virtually any org structure can work here – become directly responsible (via their budget) for their cloud spending. With showbacks and chargebacks, you have visibility. You can also reduce or eliminate misaligned incentives since there is more granular control and responsibility for spending down to the engineering level if desired.

    It can even create healthy competition among cross-functional teams or the broader organization—who can decrease their spending the most, who has the best spend-to-income ratio, etc. At 2nd Watch, we use an “innovation score” model to assess cloud spending in a client’s organization, for example, that factors costs into the overall score.

    FinOps enables everyone to speak the same language, not just about cloud but about cloud spending.

    It’s worth noting that the showbacks and chargebacks aren’t new concepts at all. Yet we’re seeing growing interest in them in the industry—and a corresponding increase of tangible goals attached to implementing them, such as “implementation of showbacks and chargebacks by Q1 2023.”

    This is a reflection of the FinOps movement—and the fundamental necessity of optimizing cloud costs.

    You don’t have to call it FinOps, but you do have to address how you allocate and use your resources in the cloud. All of the cloud’s good stuff—the outsized benefits and business value it creates—depend on it.

  • TechStrong Con 2022: Exploring What’s Next in Tech

    TechStrong Con 2022: Exploring What’s Next in Tech

    We have become increasingly reliant on technology and it’s hard to imagine life without all the great tech inventions we use in our daily lives. As technology keeps progressing, more is expected of IT teams, and security is a top priority.

    While we are still trying to get used to the new normal, the IT community continues to shine as a beacon, demonstrating how humans are leveraging technology to overcome the many challenges we face. On June 21, business and tech leaders who are working on transforming the world and moving it forward will come together at TechStrong Con Virtual Summit to celebrate the power of technology, share their stories, provide practical insights and lend valuable advice to help other professionals succeed in their careers. More than a virtual event, it is a community gathering that offers learning, networking and upskilling opportunities. Now in its third year, TechStrong Con continues to serve as a testament to the ways in which technology can improve human existence.

    Things You’ll Learn

    The agenda features a mix of topics including live coding, cloud-native, cybersecurity, digital transformation, observability, DevOps, open source and more. Attendees will have the opportunity to learn from some of the brightest minds in the industry about:

    • Managing the security of APIs
    • Key observability trends for 2022
    • How to accelerate digital transformation
    • Habits that will help improve your security posture
    • The state of the software supply chain post-Log4Shell
    • The distributed cloud
    • The future of data engineering
    • Building a diverse cybersecurity workforce
    • Crypto and NFTs
    • Low code and citizen development
    • The value of feature flags
    • How DevOps must be part of your future to attract the right talent
    • Managing and motivating teams in a hybrid environment

    Meet the Speakers

    The speaker lineup features globally recognized business leaders, DevOps, cybersecurity and IT experts including:

    • Sholmi Ben Haim, co-founder and CEO at JFrog
    • Chenxi Wang, founder and general partner at Rain Capital
    • John Willis, senior director of the global transformation office at Red Hat
    • Caroline Wong, CSO at Cobalt
    • Thomas Krane, principal at Insight Partners
    • Lisa Plaggemier, executive director at National Cyber Security Alliance
    • Tami Mazel Shackar, CEO at Incredibuild
    • Tom Sweet, VP in IT solutions at GM Financials

    Visit the Exhibit Hall

    In addition to the sessions, attendees can visit the virtual exhibit hall to interact with partner sponsors including Chronosphere, ConnectALL, Tidelift and TechStrong Research. The expo hall will be open throughout the event, giving attendees the opportunity to chat with the sponsors and download information at any time.

    Registered attendees will also receive a link to access the virtual environment on-demand. To see the full agenda and to register for free, please visit the TechStrong Con website.

  • How to Securely Manage Secrets Within Jenkins

    How to Securely Manage Secrets Within Jenkins

    The continuous integration and continuous delivery (CI/CD) pipeline is a fundamental component of the software delivery process for DevOps teams. The pipeline leverages automation and continuous monitoring to enable seamless delivery of software. With continuous automation, it’s important to ensure security for every step of the CI/CD pipeline. Sensitive information like access credentials is often necessary at various stages of the pipeline. Securing this information is vital to maintaining a strong security posture. This post discusses how to secure secrets within Jenkins CI/CD pipelines.

    Using Secrets in CI/CD Pipelines

    As previously mentioned, CI/CD pipelines often need access to sensitive information like passwords and API keys. Hardcoding them in plaintext represents a critical security risk and violates auditing guidelines in most compliance frameworks. A mechanism to make use of these credentials without exposing them unnecessarily is important for automating software delivery.

    The collection of tools and techniques used to store, manage, and access these credentials securely is known as secrets management. Secrets are broadly used in IT, DevOps, and cloud environments.

    Secrets include:
    ● App-generated passwords
    ● System-to-system passwords
    ● API keys
    ● Private encryption keys
    ● One-time passwords
    ● Certificates
    ● Privileged account credentials

    CI/CD pipelines integrate multiple configuration management platforms, requiring secrets to enable service-to-service communication. These platforms also need secrets to authenticate entities requesting access to SSH servers, databases, HTTP services and other resources.

    The Purpose of Secrets in CI/CD

    Secrets are used to manage access permissions for entities within the CI/CD pipeline. Some use cases include source control management, infrastructure connection, collaboration providers and verification. Let’s discuss each of these use cases in more detail.

    1. Source Control Management
    Version control systems, such as Git, foster collaboration between multi-personnel teams. These systems help maintain cohesive application development by managing and storing code contributions from multiple developers. Since source code can comprise some of an organization’s most valuable assets, it is important to properly authenticate and manage access permissions for Git repositories.

    Secrets management systems are important in securing credentials used to connect contributors’ machines to the repositories. The secrets themselves are not stored within version control, as malicious users can access them and orchestrate security attacks. Instead, they are stored in special, sensitive files that are designed to be excluded from the repository’s storage and logs.

    2. Infrastructure Connection
    CI/CD pipelines are often composed of different services residing on multiple hosts distributed across a disparate infrastructure. These hosts constantly communicate with each other, as well as with services through API calls, to achieve the required application functionality. Secrets such as API keys and system-to-system passwords enable a secure connection between the infrastructure running these services, allowing for multi-tenant and multi-cloud CI/CD pipelines.

    3. Collaboration Providers
    CI/CD pipelines encompass collaborative activity across multiple development, operations and quality assurance environments. These teams rely on multiple tools and frameworks to collaborate and share information. Using secrets for collaboration helps teams ensure secure communications and provides a layer of trust between both parties.

    4. Verification
    Identity verification is a crucial element of CI/CD and application security. Secrets are used to ensure security and privacy for CI/CD pipelines by connecting applications to identity and access management (IAM) solutions. These solutions provide an external database of user roles and permissions, letting users log in to the applications without exposing their credentials within the CI/CD pipeline.

    Examples of Secrets Usage in CI/CD

    To set the stage for improving secrets management, it’s helpful to understand some simple use cases. In this section, we’ll provide some basic examples of CI/CD configuration that requires access to sensitive credentials or other data for functionality.

    1. AWS Credentials
    As one of the leading cloud SaaS platforms, AWS is a common choice for most cloud-based infrastructures. Interacting with the AWS API to provision and query resources typically requires the use of a secret key/access key credential pair. Protecting these credentials is critical to maintaining the security of an AWS account.

    2. Infrastructure-as-Code (IaC)
    Modern application architecture often depends on infrastructure as code (IaC) to deploy the underlying systems that the workloads run on. Codifying this configuration makes it much easier to maintain a repeatable, scalable and secure infrastructure. In the course of provisioning this infrastructure, engineers will often need to deploy sensitive information to the nodes, such as SSH keys and API credentials. Automating infrastructure provisioning via IaC and CI/CD is a common pattern and managing secrets is an important component of that workflow.

    3. Environment Variables
    Environment variables, a common convention for configuring processes on a computer, see consistent usage in most CI/CD pipelines. The ability to dynamically set values and retain them for consumption by other processes and scripts is a powerful abstraction when dealing with CI/CD workflows. In many cases, sensitive values will need to be passed via environment variables, again highlighting the need for effective secrets management.

    Securing Secrets Within Jenkins

    Jenkins requires access to a large number of credentials to interface with all the platforms and applications that make up a complete CI/CD pipeline. Jenkins ships with a default credentials plugin that offers a built-in secrets store. It also comes with the credential binding plugin which enables the binding of credentials to environment variables.

    Storing secrets in the Jenkins controller enables applications in the pipeline to reference secrets multiple times, even when they’ve only been created once. While the Jenkins credentials plugin offers a simple way to store and manage secrets, any Jenkins administrator and pipeline author can access them. Therefore, DevOps teams should ensure proper configuration of access rights to keep the CI/CD pipeline secure.

    This section discusses some of the available options for securing secrets in Jenkins.

    Types of Secrets Jenkins Accepts

    Credentials supported by the Jenkins secrets plugins include: Secret text, username/password pairs, secrets file, SSH username and certificates. For a limited number of secrets, these can be stored individually, whereas a large number is typically best managed with a credentials file.

    Managing Secrets in Jenkins

    If you’re ready to inject secrets safely into your Jenkins pipelines, follow these best practices.

    1. One-Time Passwords (OTPs)
    One-time passwords help prevent user account takeovers by ensuring that a username-password combination can not be used more than once. While the account retains the username throughout, the user is sent an OTP every time they log in. This keeps Jenkins administrators’ and users’ accounts safe, preventing secrets from being exposed to unwanted individuals.

    2. Avoid Hard-Coding Secrets
    Remove any secrets that have been hardcoded into CI/CD configuration and Jenkinsfiles. This keeps sensitive credential information away from the pipeline, making it inaccessible to malicious actors.

    3. Reduce Attack Surface by Storing Secrets in Multiple Jenkins Files
    Storing secrets in multiple files helps enforce the principle of least exposure for credentials. This makes credentials available to the minimum number of applications and users possible.

    4. Use Authorization Controls to Manage Secrets Permissions
    Developer teams should configure security checks to ensure the application requesting resource access has the properties of a valid application. These applications should be able to access various resources based on permissions assigned in RBAC policies.

    5. Follow the Principle of Least Privilege
    Only grant permissions to applications that absolutely need access to secrets within the pipeline. Teams should consistently audit permissions to maintain the principle of least privilege throughout the pipeline’s lifecycle.

    Managing and Storing Secrets

    This section walks through three basic examples of actually using Jenkins with secrets and the Akeyless secrets management platform.

    ● The first example will highlight a completely insecure example: Using plaintext environment variables to store sensitive credentials.
    ● The second example will focus on using native Jenkins features and plugins for secrets management.
    ● The third example will show some of the benefits of using a third-party tool for secrets management.

    Plaintext example
    To get started, we’ll create a basic “freestyle” project in Jenkins, and connect it to a GitHub repository:

     

    Figure 1: Project creation

    Regardless of how we choose to manage our secrets in code, it makes sense to use the Jenkins built-in mechanism for credentials to access the GitHub repository:

    Figure 2: Configuring GitHub credentials

    Our actual code will be a very basic Python application that makes an API call. The API requires a simple API key in the form of a text string for authentication:

    #!/usr/bin/env python
    import os
    import requests
    api_key = os.environ[‘API_KEY’]
    response = requests.get(“https://thedogapi.com/v1/breeds?api_key={}”.format(api_key))
    print(response)

    Jenkins provides the ability to configure environment variables; however, without additional plugins, they can only be set globally to the Jenkins environment or instance:

    Figure 3: Configuring a global environment variable

    The code will read the API key from the environment variable that was just configured. Here’s the configuration for the build steps:

     

    Figure 4: Build configuration

    The build steps run some simple linting and then actually execute the code. Here’s a look at part of the log from a successful build:

     

     

    Figure 5: Build log with exposed secret

    The API key has obviously been exposed in a plaintext log with a simple directive in one of the build steps. A minor misconfiguration could potentially cause a breach or compromise.

    Jenkins Secret Text Example

    Now we can try to improve the security of our build environment by using the built-in credentials handler:

     

    Figure 6: Configure build environment to use secrets

    In this example, we’ll use a simple “Secret text” credential binding to store the API key:

     

    Figure 7: Adding a secret credential

    Next, we configure our build job to use the new credential variable:

     

    Figure 8: Updated build configuration

    Now, after running another build job, we check the build logs again:

     

    Figure 9: Build job with obfuscated secret

    Even though we configured the build step to print the variable contents to the log, the Jenkins credential handler knows that the value is sensitive and obfuscates the actual value in any log or output.

    Although this measurably improved the security of the build, there are some caveats. While this works well enough for a small Jenkins deployment with a limited number of users, managing access and correct permissions scoping for multiple users and teams across a large, distributed Jenkins architecture quickly becomes complicated and difficult to manage effectively. In addition, the secret value stored in the credential variable is still accessible to every Jenkins instance in the cluster, as well as any user with the appropriate permissions.

    If any node is compromised, secrets across multiple deployments and contexts could be at risk. Users still have to manually secure and configure Jenkins infrastructure that now doubles as secrets storage. What if there were a way to use sensitive credentials without ever having to store them in Jenkins itself?

    Secrets Management Tool Example

    Let’s try using Akeyless to manage our secrets. Since we’re integrating with a Jenkins deployment, we can follow the documentation for working with Jenkins. Akeyless integrates with the publicly available Hashicorp Vault plugin, so it’s simple to get Jenkins set up.

    Next, we’ll create a free account with Akeyless, and configure a secret:

    Figure 10: Creating a secret in Akeyless

    Figure 11: Creating a secret (cont).

    The next step is to create an Access Role that has permission to access the secret that was just created:

    Figure 12: Create a role

    Now we need to give our Access Role the correct permissions to view the secret:

    Figure 13: Giving the role permissions

    The last thing we’ll need to create on the Akeyless side is an authentication method. This is how we can give external entities, like our Jenkins deployment, the ability to make API calls to Akeyless and retrieve secrets:

    Figure 14: Create an auth method

    Once the authentication method has been created, be sure to save the Access ID and key by downloading the CSV file; the values will not be displayed again otherwise. Finally, we need to associate our newly created role with this access method:

    Figure 15: Associating the auth method with the access role

    The next step is to configure the Jenkins build environment according to the Akeyless documentation referenced in a previous step:

    Figure 16: Jenkins build configuration with Akeyless

    Now we’ll run another build, this time configured to utilize the Akeyless vault we’ve configured:

    Figure 17: Build log with Akeyless configuration

    In the logs, there is now a step present where Jenkins retrieves the secret from the Akeyless Vault. The credential value is also still obfuscated in the build log. We’ve now got a working, third-party secrets management platform integrated with our Jenkins instance.

    It may seem like extra steps just to achieve the same outcome, but an external tool focused entirely on secrets management is a powerful abstraction. Jenkins is a CI/CD tool first and foremost, with secrets management provided as an add-on. In that context, it will never be able to provide the first-class features, management capabilities, and security guarantees that a dedicated secrets management platform can:

    ● Managed infrastructure with SLA guarantees
    ● Zero-knowledge encryption
    ● Intuitive, accessible secrets management and organization
    ● Integration with industry-standard DevOps tools and platforms

    A tool like Akeyless also does away with having to manage complex, self-hosted infrastructure. We were able to have a working secrets endpoint in just a few clicks.

    Conclusion

    Since security is a top consideration for CI/CD pipelines, it is important to follow best practices when using secrets in Jenkins. Jenkins pipelines rely on secrets for identity verification, collaboration, and infrastructure management. CI/CD secrets management presents a challenge for most teams because of how complex it is to implement access control at a granular level.
    With a secrets management tool, Jenkins users get a centralized and secure resource to manage the many credentials that are required to operate CI/CD pipelines. They benefit from using a simplified, standardized approach to access all secrets they need, without worrying about security. Simply put, a secrets manager accelerates DevOps workflows.

    Meanwhile, security teams benefit by gaining full visibility into where secrets are, and how they are used. With automated processes and centralized access policies, a secrets management platform helps security teams stay compliant with security regulations as well.

  • 5 Tips If You’re the First SRE Hire

    5 Tips If You’re the First SRE Hire

    Site reliability engineers (SREs) have a considerable set of tasks to juggle no matter where they work or how long their company has had an SRE practice. But if you’re the very first one to join an organization—as many SREs are these days, given that the trend is trickling down into smaller and smaller companies—you face a special group of challenges.

    You may find it difficult to get buy-in for SRE from other technical teams. You may struggle to know how to convince your boss that the company’s decision to hire an SRE is paying off. You probably worry that you’ll end up being overextended because you are the only SRE. And so on.

    These five tips are for site reliability engineers who find themselves in the position of being the pioneer at their companies. Whether you’re an experienced SRE who has just been hired at a company that had none previously or you’re brand-new to SRE and your first job happens to be at a company that is also brand-new to the SRE role, the following best practices will help you make the transition as smooth as possible for all involved.

    Collect SRE Metrics Early and Often

    For starters, you’ll want to be prepared to justify the business’s investment in your new role as soon as a boss or fellow engineer asks.

    Do this by collecting metrics about the technical and business impact of SREs starting as early as you can. Your goal should be to quantify how you are benefitting the organization so that you can encourage more support for yourself and any others who may be hired after you.

    On top of this, collecting metrics is also important for establishing early feedback about which SRE strategies are succeeding and which could be improved. Since you’re building an SRE practice from scratch, getting this insight early and often is critical.

    Treat Devs and IT Engineers Equally

    As you start the role, you may find some uncertainty among existing engineers about whose ‘side’ you are on. Are you there to support developers or IT engineers?

    The answer, of course, is both. Although SREs tend to borrow from software engineers’ playbooks more than they do from those of IT Ops, an SRE’s goal should be to empower and support both of these teams. So, unless you’re given explicit instructions to interface with only one group or the other, make yourself everybody’s buddy within the IT organization.

    Introduce Yourself to Non-Technical Teams

    SREs spend most of their time doing technical things with technical tools. But ultimately, the only reason they are doing what they do, in most cases, is to support non-technical users by improving the reliability of the systems they use.

    As the first SRE at a company, it may be well worth your time to get to know these non-technical stakeholders and ask them what they see as the greatest reliability issues in the organization. Building these relationships early on will help you communicate the value of your role across the organization. It’s also likely to come in handy during incident management routines, when you may need to interface with non-technical stakeholders as well as other engineers.

    Build Your SRE Tool Set

    Deciding which tools to use is an obvious early step for any SRE who is building a practice from scratch. But we didn’t put it first on this list because it’s important to get to know the rest of the organization before committing to a particular set of tools.

    As you meet the rest of the organization, ask detailed questions about which tools they use to address needs like observability, security and incident management. Then, by factoring in feedback about which tools work well and which don’t, choose which tools you’ll deploy to support SRE. Be prepared for those tools not to match your own personal preferences perfectly. You may need to prioritize the preferences of the rest of the organizations.

    Plan to Grow the Team

    Unless you have been hired at a very small company, it’s likely that the business plans to hire more SREs over time (assuming that you prove the value of the position, of course).

    Toward that end, plan for how the team will expand. Think about things like whether to use embedded SREs or create a dedicated SRE team. Think, too, about how you’ll document your company’s SRE practice and ensure it keeps operating smoothly as individual engineers come and go.

    Conclusion

    The job of a newly hired SRE is not an easy one, especially if you’re the first person at your company to fill those shoes. But by being deliberate and strategic about how you work, which data you collect and how you introduce the company to SRE, you set yourself – and your future colleagues – up for success.

  • Why is Security Still in the Way? A Look at DevSecOps Right Now

    Why is Security Still in the Way? A Look at DevSecOps Right Now

    According to industry trend reports for 2022, DevSecOps is now considered to be one of the most effective approaches to building software quickly and securely. This effort, of course, means development, security and operations teams commit to addressing security as early as possible in the software development life cycle (SDLC). The goal of the shift into DevSecOps revolves around the need for more automation, streamlined CI/CD pipelines, less friction among teams and plenty of time for productive innovation. Incorporating DevSecOps practices into an organization unites security, compliance and development issues into a collaborative concern, one that allows DevOps to move fast without jeopardizing code quality. 

    As we approach the middle of 2022, it’s worth looking back and considering why security is still standing in the way of smooth software development. Security vulnerabilities have always been present in the process, and yet these pesky bugs continue to be the biggest threat to software performance. The question is, why?

    The Truth About Security

    Software has always been deployed and shipped with some existing bugs. But as software and services continue to dominate our everyday lives, the reality of this statement continues to plague both businesses and the end user. On some level, software development has yet to evolve into the next phase of security. Code is still rolled out regularly, only to discover security vulnerabilities in production. Some say the way software is developed today actually lends itself to defects, mostly because the pace of work required supersedes best practices around AppSec. 

    While the larger cybersecurity industry—and every DevSecOps team in the world—is striving to make software more secure, many organizations still prioritize speed and productivity over digital risk. This means the state of security (and, to some extent, the DevSecOps world) has become somewhat lopsided, with some enterprises making great efforts while others embrace a more lackadaisical approach. To make matters worse, many professionals today believe there has been a general under-investment in cybersecurity over the past several decades. 

    The Next Phase of DevSecOps

    The shift to remote and hybrid work models has led to mass adoption of cloud technologies and digital transformation, across all sectors and industries. As a result of this evolution, organizations today need to bear in mind that the more services they use, the more digital threats and overall risk they face. As this exposure to vulnerabilities increases, so does the need for new approaches to security. 

    The majority of modern enterprises do, in fact, have a security process in place, as well as other risk management solutions. But with the cybersecurity reckoning of 2021 and the stakes of cloud computing being higher than ever, businesses right now are feeling the pressure to think long and hard about why security still isn’t a guarantee. The trend of slowing down software development to address security more thoroughly, also known as the secret sauce of DevSecOps, is not free from roadblocks—but it is also far from optional. For many organizations, the implementation of DevSecOps practices feels less like security and more like speed bumps that just slow down the shipping of software. 

    Even so, the effort to improve security is happening now. Organizations do not need several solution platforms or tools to get started. They can begin right away; evolving their security tools and practices as they grow the business. With this growth comes one of security’s biggest challenges—scalability. Because toolchains across the cloud have become more and more complicated, setting policies and/or workflows in one tool is challenging. While these issues can be largely addressed by the adoption of an end-to-end security solution, organizations must continue to focus on strengthening supply chains, improving security oversight, and bolstering their approach to DevSecOps. Some ways to make this happen include:

    • Making sure CI/CD pipelines are integrated with DAST and SAST testing in development
    • Ensuring developers and DevOps teams complete vulnerability scanning before commit or merge, as they write code
    • Minimize manual security as much as possible with automated AI/ML tools to support more effective scanning and monitoring
    • Invest in management solutions that improve security in a multi-cloud environment

    Addressing the Supply Chain

    According to an industry survey, 60% of companies have decided securing the software supply chain should be a top priority for 2022. When we think about software supply chain attacks, we need to think about better controls to validate its integrity through all stages of the SDLC. Obviously, DevOps and automation have provided considerable help in this area; however, they cannot pull all the weight of security without proper governance and guardrails. 

    With a secure supply chain comes better security, as well as compliance, privacy and transparency. When a supply chain is secure, it can automatically enforce policy requirements for things like instrumentation, testing and architecture. As a result, software development and deployment become a repeatable process that is also reliable. Teams across the organization can rest assured that the products and services they are building comply with security expectations, such as validating code before production with static analysis and scanning tools. 

  • When DevOps-as-a-Service (DaaS) Meets Security

    When DevOps-as-a-Service (DaaS) Meets Security

    One of the most recent IT methodologies to be offered as a service is DevOps, a cultural and practical approach that brings development and operations teams together under one umbrella of work. DevOps-as-a-Service (DaaS) ensures that everything related to the selection, management and maintenance of DevOps tools and infrastructure, including all policies and procedures, are addressed through a centralized team who are able to provide this service to development teams across the organization. 

    From a 30,000-foot view, DaaS represents DevOps capabilities made accessible to practitioners through portals and APIs. Because DevOps capabilities vary among organizations, there is no standardization of these services. For the business, DaaS enhances competition. Application users and developers find DaaS improves overall software development, while specialized DaaS teams say it improves the management of DevOps capabilities.

    What Are the Other Benefits of DaaS?

    DaaS offers many benefits, especially to development teams who no longer want to spend all their time configuring and deploying the necessary security tools to run DevOps pipelines. This means it’s no longer necessary to understand all the differences between DevOps tools and the underlying infrastructure, as the appropriate tools and technologies are automatically deployed for development teams. As a result, developers are able to focus their efforts on creating innovative software at the speed of business. 

    Further, Daas delivers clear business and operational benefits in the form of a more streamlined process for facilitating DevOps across the enterprise. Daas supports governance while also enabling centralized oversight and visibility into development workloads and productivity. As we know, this shift leads to increased accountability among teams and an overall optimization of business processes. Add on resource efficiency and cost savings through tool selection, and it’s easy to see why DaaS has found a home in the world of software development. 

    But despite the benefits, Daas also presents new security challenges that can’t be fully addressed by traditional management solutions. These approaches are often sluggish, expensive, and/or incredibly complex to manage.

    What About Security and DevSecOps?

    Because the principles of DevSecOps proactively address the risks and threats associated with DevOps processes, it has seen rapid adoption in recent years. By integrating AppSec tools and strategies within a DevOps pipeline, developers are able to find and fix security vulnerabilities earlier within the software life cycle—not after the fact. This ability is what allows developers to generate secure and high quality software from the beginning, while also preventing any deployment delays stemming from last-minute security checks in production. With DevSecOps processes in place, AppSec testing becomes seamless and transparent for developers, who need a frictionless experience to facilitate steady and reliable software development and deployment. 

    DevSecOps is a  natural fit for the as-a-service delivery model. Similar to DevOps, AppSec involves various tools, policies and processes that demand a certain degree of professional expertise—not to mention the time and resources it takes to to select, maintain, and deploy them to drive DevSecOps practices. Security and development teams looking to  implement and manage DevSecOps can reap major benefits from a centrally defined and managed process, to be administered by a specialist team and delivered “as a service” to those in charge of DaaS.

    Standardizing AppSec through a DevSecOps-as-a-Service offering would guarantee security is applied evenly across all DevOps activity, thereby allowing businesses to produce secure and high-quality software consistently while also supporting AppSec compliance demands. This shift also establishes a foundation where a holistic and homogenous view of AppSec can be found, one that is critical to the assessment and management of application and business risk. 

    Developing DaaS Metrics

    First, don’t try to boil the ocean by measuring all applications at the same time. Instead, choose an application that’s supported by DaaS processes and appoint a specialized team to develop metrics for the application. This team can look many ways—but would likely include stakeholders, developers and other DaaS-supporting roles. Service-level indicators (SLIs) and service-level objectives (SLOs) are also baked into this model, which are chosen to best suit the requirements of an individual application. 

    Once the most valued SLOs of DaaS are determined, the service can be tailored to identify the most critical SLIs. With this knowledge, things like instrumentation, telemetry and analysis requirements can be identified and used to design alerts, dashboards, analyzers and more. Next, setting up policies and procedures around accountability establishes a complete solution. And once the solution for the chosen application is tested, confidence kicks in and creates an easier process for adapting and expanding the process around other applications. 

    This means DaaS can provide many key capabilities to both the organization and the teams who rely on them. A specialized DaaS team can then figure out the best metrics for larger implementation on any application. As a result, SLOs can be evolved for other future applications, to be used as a baseline for future improvements.