staging-devopsy.kinsta.cloud

Tag: devsecops

  • How DevSecOps Can Help Avoid Catastrophic Breaches

    How DevSecOps Can Help Avoid Catastrophic Breaches

    Earlier this year, Tesla’s Cloud was hijacked and used to mine cryptocurrency, exploiting a vulnerability in the company’s Kubernetes cluster. A mountain of FedEx data was recently exposed, affecting 119,000 individuals. The Equifax breach garnered international attention after an estimated 145.5 million Americans were jeopardized. In other news, we’ve reported on how the Vine Docker registry fiasco was hacked, leaving an embarrassing PR trail in its wake.

    What is common across all these scenarios? They arguably could have been avoided with basic safeguards underpinned by a healthy dose of DecSecOps. In hindsight, let’s see what specific strategies developers can adopt to avoid such horrendous leaks.

    Basic DevSecOps Can Prevent Vulnerabilities

    DevSecOps is described as placing security at the forefront of every action. Within the world of cloud tooling, this means instilling protection at every point of the build life cycle. You would think this equates to advanced container and orchestration security, powerful access management and establishing hardened oversight for internal applications, right? Actually, many modern exploits simply involve a lack of basic password protection.

    Case Study No. 1: Tesla Cloud Cryptojacking

    In mid–2018, Tesla’s Amazon servers were hijacked by malware and used to mine cryptocurrency for rogue agents. Similar cryptojacking has transpired at Gemalto, Aviva and others. According to RedLock’s report, the hackers infiltrated the Kubernetes console, which was not password-protected. In one pod, they found access credentials to Tesla’s AWS environment. The breach exposed sensitive telemetry data, though was limited to “internally used engineering test cars only,” according to Tesla.

    How to Prevent

    Password protection for Kubernetes administrative consoles is a readily apparent lesson here. All cloud accounts, even if used internally, must be better-equipped and access credentials sequestered. Surely, instilling a “security by design” mentality can help limit the number of such consoles left accessible.

    Case Study No. 2: Vine Registry Breach

    In 2016 a white hacker by the alias avicoder was able to infiltrate Vine, the video-sharing social network. The hacker used tools to discover subdomains and behind one—https://docker.vineapp.com—found an open Docker private registry that housed the Vine source code. Such a vulnerability could have been used to collect user details or inject malware for malicious purposes. Thankfully, no users were compromised, and the Twitter Bug bounty program awarded avicoder a handsome sum for the discovery.

    How to Prevent

    In this particular incident, there was nothing at fault with Docker or Docker containers. The service (which was meant to be private) was simply left public and devoid of access controls. The lesson is that URLs on the world wide web are just that—publicly exposed to the world. It does not matter if no documentation links to them; such subdomains are easy to discover with crawlers and will be exploited if found.

    Case Study No. 3: Equifax Exploits

    In the Equifax breach, millions of customer records were stolen. The Equifax report notes that hackers exploited a “website application vulnerability.” Further reports detailed that the 2017 breach utilized flaws in Apache Struts, likely through a zero-day exploit, according to the Apache Struts Project Management Committee.

    How to Prevent

    Some note that the deserialization of untrusted data inherent in Apache Struts applications leaves some major vulnerabilities and potentially malicious code execution.

    Containers, due to their short lifespan, are not persistent, therefore making consistent hacking harder. They can also isolate functionalities so as to decrease attack vector. Regarding the Equifax breach, Aquasec speculates that container usage could have lessened the impact:

    “… it’s likely that such a breach would have been more difficult with containers, and that if successful, it would have been less persistent, not as widespread, and mitigated sooner.”

    Some view containers as a formidable force if armed with behavioral analysis and firewalls for network connections. Still, others note flaws in assuming containers are safer.

    Regardless of architecture, In DevSecOps, security for customer data isn’t drafted into the background; rather, it is imbued into the entire tooling process and holds equal footing within each build.

    Instilling a DevSecOps Culture

    Instead of a build-now, secure-later attitude, DevSecOps seeks to elevate security into every decision. With each build, security is melded with greater forethought regarding the repercussions for how data is treated.

    Admin consoles for Kubernetes clusters must be armed and password-protected, and the same goes for private GitHub repositories or Docker registries. Any traversal of data through cloud SaaS services must have authorization in place. A most obvious lesson is to password protect those “hidden” registries with discoverable URLs.

    Bounty programs for white hackers continue to lead to successful bug detection. At least for these institutions, there is less evidence of compromised user data and remediation is quickened.

    Much of this can be distilled by due diligence and through breeding a culture of security. While that’s a little fluffy for some, it apparently still requires repeating for institutions that continually de-privilege these basic safeguards.

    — Bill Doerrfeld

  • Alcide Report Finds 75% Will Increase the Number of Cloud Security Tools They Rely On Over the Next 12 Months

    As Hybrid Cloud and Serverless Continue to Gain Ground, Organizations Rush to Keep Up; Fewer than Half Have Dedicated Cloud Security Teams

    Tel Aviv – November 6, 2018 – Alcide, provider of the most comprehensive full-stack cloud-native security platform, today released the findings of a new industry report: 2018 Report: The State of Securing Cloud Workload based on responses from close to 350 security, DevOps and IT leaders. The report reveals that as more organizations embrace hybrid cloud – with more than 50 percent claiming a hybrid cloud setup – and serverless, now used by close to third of organizations, they lack the tools and specialization to keep up. 75 percent of respondents expect to see an increase in the number of security tools they rely on in the next year, while over half share they still manually configure security policies. The resulting complexity has the potential to slow critical business functions in the absence of an integrated security approach to distributed cloud environments.

    According to a recent report from 451 Research:

    The pace of innovation on cloud-native environments places significant burden on traditional security practices. Not only is there a need to support new technology options – quickly moving from traditional virtual machines to containers, serverless, and newer constructs such as service mesh – but there is also a difference in how security and DevOps teams consider their needs and workflows.

    Alcide’s report, conducted in August 2018 in conjunction with Informa Engage, reinforces the idea that new practices and technologies are disrupting traditional security practices, with findings including:

    Cloud complexity increasing, with hybrid cloud as the new infrastructure normal:

    • While virtual machines (VMs) remain the most common cloud computing environment (83%), containers (37%), serverless (28%), and service mesh (21%) are gaining traction.
    • Hybrid and multi-cloud approaches now make up more than three-quarters of all configurations (77%).

    Serverless running in production; most bullish about its security

    • Despite some security concerns, the majority (57%) of serverless users are currently running it in both production and development.
    • The majority currently using serverless have a high degree of confidence in its security, while one third (32%) express a lack of confidence in the security of their environments.

    As cloud infrastructure complexity grows, security becomes a shared responsibility with DevOps

    • Fewer than half of organizations (45%) now have a dedicated security team responsible for the cloud, with 35% of all organizations now using either a DevOps team or dedicated DevSecOps team for security.

    Hybrid cloud complexity pushes Dev, Sec and Ops teams to look for more tools to secure their distributed environments

    • Over two-thirds (75%) expect to increase the number of tools in use over the next twelve months – with no one expecting to retire any tools in use
    • One-third of organizations reporting using more than five tools for cloud security

    Proliferation of cloud security tools leave the enterprise vulnerable, point to need for intelligent policy automation

    • More than half (60%) of organizations rely on manual configurations of security policies for their apps, while almost all organizations (90%) rely on multiple individuals to configure and set policy rules.

    “Our report validates what we’ve seen with our own customers – modern organizations are striving for a consolidated security approach that will support business velocity and tackle the challenges associated with the overhead of multiple tools in use,” said Karine Regev, VP Marketing of Alcide. “Modern teams can’t assume that emerging technologies like serverless are secure, and need a practical and uniform enforcement and management of security policies to control disparate and cloud-native services, infrastructure and environments.”

    Access the full report here for additional insight.

    About Alcide:

    Alcide’s Cloud Native Security Platform protects any combination of container, serverless, VM and bare metal. Offering real-time, aerial visibility, threat protection and security policies enforcement, Alcide secures the cloud infrastructure, workloads and service mesh against cyber-attacks, including malicious internal activity, lateral movement and data exfiltration. For more information, please visit www.alcide.io

  • Building Great Cloud Security Guardrails

    Building Great Cloud Security Guardrails

    Security guardrails are an incredible way to keep our cloud deployments safer without slowing things down. Taking this structured approach will minimize friction while increasing protection.

    When I first started working hands-on in cloud (AWS) around nine years ago, I quickly realized both the power and risks of just how darn fast and agile cloud could be. Being, on occasion, an optimist, I decided this capability was a significant upgrade to our security capabilities. Sure, I could accidentally spin up internet-exposed servers, but in exchange I gained levels of central control and visibility unheard of in traditional infrastructure.

    One of the most powerful new capabilities is the ability to leverage cloud APIs and automation to build automated security guardrails without having to spend a gazillion dollars for a big box that actually delivers as promised, but also adds tremendous friction and slows things down. Guardrails are automations that constantly watch your deployments, find deviations from desired baselines and can even automatically remediate issues. Yet, even cloud-native guardrails can become problematic if they aren’t designed properly. Having both built a fair few myself and worked with many organizations as they built out their cloud security programs, here are some of the lessons I’ve learned along the way and some code to show you how it works.

    Building Awesome Guardrails

    My first guardrails were a bit of a hodgepodge in terms of structure and capabilities. And practically speaking, there are a lot of ways to attack the problems. Even today I’ll take different approaches depending on the particular challenge in front of me, but most of my guardrails (including operational ones) tend to follow a consistent pattern:

    • Define the problem: I like to start by defining the business outcome we want, and then move onto the technical problem. I used to start with a narrow technical problem to solve but found that didn’t always lead me to the right outcome. For example, is the problem that we want to stop all internet-facing port 22 access, or is the real outcome we don’t want to expose internet-facing admin servers to the open internet and we want to use a guardrail to restrict to our known corporate IPs instead? This helps prevent a guardrail from becoming a blocker.
    • Set the scope: Multiple cloud accounts (or subscriptions/projects for you Azure/GCP people) are now the standard. The first step is to determine how broad a guardrail you want, which will also define some of the technical specifications. For example, if you want it to apply to multiple accounts in Amazon Web Services, that affects where you run it from and how you manage the required IAM privileges.
    • Build or pick the deployment model: There are a lot of technical specifics regarding where and how to run your guardrail. Today’s example will run in AWS within a single account, but if you manage a larger, enterprise-class deployment you will need to look at how and when to balance centralized and localized guardrails. More often than not, I prefer a serverless approach (FaaS with other supporting PaaS), but I’ve also found longer-running guardrails often need containers as well. Thanks to API gateways, it isn’t bad to build deployments that can support both.
    • Define filters: It’s awesome that you can automatically close all internet-facing security groups with 20 to 40 lines of code—right until you break a production application. Filters allow you to tune your guardrail for a particular environment or project’s needs. Basic filters might work on a simple resource tag, while more complex filters might evaluate a combination of include and exclude rules for greater flexibility. This can, for example, treat development and production environments differently from the same guardrail.
    • Determine triggers: We have a rich range of trigger options on the major cloud platforms today. The main categories are time-based assessments and event-based triggers. I tend to use both, with event-based for more deterministic events (an S3 bucket policy was changed) and time-based sweeps to catch conditions I might miss purely on an event trigger.
    • Perform the analysis: This is the meat of our guardrail. Based on the trigger, collect the information we need and, in combination with our scope and filters, make a decision. For example, if we find port 22 open to the internet, if it is a Dev environment we might decide to update the security group to restrict to known corporate IP addresses. If it is in Prod we might delete the rule completely unless it was implemented through a CloudFormation template that went through an approval process.
    • Send notifications: Even when you have a completely automated guardrail, you likely want a parallel notification. These can go straight to email/text/Slack or drive into a ticketing system. For security guardrails, I also recommend sending them to your SIEM or other security alert/management system. If you’re fancy, you can turn these into the first step of a chatOps process.
    • Take actions: Actions are what separates guardrails from all the annoying alerts of the past. Based on the analysis and filters, actions can be fully automated remediations or request human intervention. Actions also can follow different paths based on the conditions to provide the best options for that situation. The key is that even if a human is called to make a decision, the decision should be a simple click to run the action, saving the person from having to fix things manually. Unfortunately, we still sometimes need human beings to make a decision. Then again, the alternative is bowing down to our robotic overlords. It’s a trade-off, I suppose.
    • Validate and log: Don’t assume the action actually did anything. Rechecking is vital, as is logging and sending the results as a second notification to the ticketing system/SIEM/etc. I fully recommend sending both incident open and close notifications to the destination system, even if the guardrail fixed everything within seconds. This keeps a consistent record and also allows you to do nifty things such as measure your mean resolution time and compare automated versus manual.

    Running a Demonstration Guardrail

    One of my favorite guardrails to demonstrate is automatically reversing security group changes in AWS. Here’s how to set it up for yourself and run my demonstration code to see what it looks like:

    • The first step is to make sure you don’t start running this in production or anyplace else where you might break things before you actually tune the filters. If you do, it isn’t my fault. The desired outcome in this case is ensuring we don’t have network exposures due to unapproved security group changes on designated resources.
    • The scope of this guardrail is a single local account, with a corresponding deployment model. We will use CloudTrail, CloudWatch Rules, Lambda and SNS.
    • There are three prep steps to configure your account before building the guardrail:
      • Log into the AWS console and set up a destination for notifications. If you haven’t done this before, you want to use SNS (Simple Notification Service). Create a topic for security alerts and a subscription to wherever you want to send the alerts. I recommend starting with a simple email subscription.
      • Make sure you have CloudTrail turned on and streaming to CloudWatch. For you newbies, CloudTrail records (almost) all the API calls on your account and we need this for our trigger. Sending the logs to CloudWatch allows us to build near real-time triggers in CloudWatch based on events such as the API call we will look for.
      • Create an IAM Role for Lambda to use. It will need the rights to AuthorizeSecurityGroupIngress, DescribeSecurityGroups and RevokeSecurityGroupIngress. To save logs, it also needs PutLogEvents, CreateLogStream and CreateLogGroup. These are also in a JSON template in the project repository for cut and paste.
    • With that in place, create a new Lambda function (Python 2.7) and load in our code located at https://github.com/disruptops/RevertSecurityGroups.
    • Now go to CloudWatch Rules and create a new rule. For the trigger, use the following:
      • Service Name: CloudTrail
      • Event Type: AWS API Call via CloudTrail
      • Specific Operations: AuthorizeSecurityGroupIngress
    • Then set two targets: your Lambda Function and your SNS topic for security alerts. This will revet the changes and simultaneously send you a notification.
    Screenshot of configuring the trigger in CloudWatch

    This should now revert any new ingress rules added to a security group. If you play with the code you can also use this for egress rules. This is just demo code and there are edge cases it will miss.

    The code also includes a number of filter options, including:

    • Restricting to a region or VPC.
    • Restricting based on a tag or a specific security group ID.
    • Allowing the change if it comes from a designated IAM user, and reversing if it was originated by anyone else.

    It takes the action automatically; if you want to manually review first you would modify the code to send a notification instead. We have some alternative examples posted at https://github.com/disruptops/demos-tools. These includes some samples that protect S3 and also show how to embed notifications in the Lambda itself for more-flexibility.

    Hopefully this gives you some ideas to refine your own guardrails to dramatically improve security while still moving blazingly fast.

    — Rich Mogull

  • DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    Shifting security left during the software development process can be tough for companies because of the sheer number of operational things that go along with implementing security that must occur somehow in the realm of DevOps. It can be easy to lose control over their security processes without some sort of guardrails in place to help.

    A brand new company is helping do just that. I’m excited to introduce disrupt:Ops, which bills itself as a guardrails platform to help companies include and enforce best practices across security and operations, bringing security into the DevOps process at the beginning. My friend Mike Rothman, disrupt:Ops co-founder and president, joins me in this DevOps Chat to discuss the vision of his company and how it hopes to help make security truly a part of DevOps.

    You can read more about disrupt:Ops and their guardrail approach to cloud security and devsecops from another co-founder of the company, my friend Rich Mogull at https://staging-devopsy.kinsta.cloud/building-great-cloud-security-guardrails

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel, staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Breaking some big news today on DevOps Chat, the launch of a labor of love of some friends of mine. I’m happy to be joined by Mike Rothman, cofounder and president of disrupt:Ops. Mike, welcome.

    Mike Rothman: Alan, it’s great to be here, thanks. It’s always fun to catch up with old friends.

    Shimel: Yep, and old friends it is, Mike. We should let our audience know – maybe we shouldn’t let them know. But we are old friends. We go back a long time now, back to the beginning of blogging and security –

    Rothman: That’s right.

    Shimel: It’s got to be 2004, 2003 or something like that.

    Rothman: Yeah, that’s right, that’s right. When I had dark hair. Actually I didn’t have dark hair back then, but you know, it’s –

    Shimel: I had hair.

    Rothman: You had hair. That’s right.

    Shimel: I had hair. But you know what, Mike? I said it was a labor of love and everything else. Just in the way of background, why don’t you share with our audience your personal sort of journey in tech and, you know, work history.

    Rothman: Well sure. I started out, God, in what feels like 100 years ago, as a mainframe programmer. Ended up doing some local area networking stuff. Kind of found my way to network security. Went into research for quite a while. Did a couple of companies and then went back to research and for the last 10 or 11 years at this point I’ve been partnered up with Rich Mogull and Adrian Lane at Securosis, doing a bunch of kind of information, security-centric research. And actually about a year and a half ago we spun out some technology that we had prototyped within Securosis in cloud management and cloud automation into our new company disrupt:Ops, and have been just making a lot of progress over the last year with that. And that’s what we’re announcing this week.

    Shimel: Yep. So Mike, you’re being humble. When you say you did research, you were an analyst at Giga, were you not?

    Rothman: META Group, yes. META Group, exactly.

    Shimel: META Group. And then you were also – you had executive positions in several cyber – we didn’t call it cyber – infosec companies back then. 

    Rothman: It was. Yeah, you bet. So companies like TrueSecure and CypherTrust, you know, back early in the e-mail security days. I did a network monitoring company called EIQ. So yeah, I’ve been there, done that. Have the road rash and kind of the screw-ups under my belt to prove it. But you know, it’s interesting in terms of just how long and how small an industry security is for being actually a really big thing nowadays. 

    Shimel: Being so big, yeah. It really is. And Mike, again, for many people in our audience who come from DevOps or even from security – but they’re worldwide and they not be familiar with Securosis. You know, Rich Mogull comes over from Gartner. Adrian’s been CTO at major companies. You know, you guys are probably the leading, let’s call it independent security analyst research firm – you said 10 years-plus. Certainly in my time you guys laid the fundamental kind of – a lot of the fundamental backbone of what Cloud Security Alliance teaches and so forth. 

    Rothman: Yeah, that’s exactly right. So it was interesting. So I guess it was probably eight and a half years ago we partnered up with the Cloud Security Alliance and helped them build their curriculum for their CCSK certification. And that got us to be much deeper than your typical analyst in terms of having to play around with cloud, having to understand automation, having to build a number of the kind of scripts and – I’ll call it the inherent messiness of building out a stack in the cloud and really making it operational, because we were eating our own dog food. Our website, a lot of the stuff that we had prototyped was really demonstrative of the challenges that organizations face as they move a bunch of their stuff into the cloud, and really as they embrace a DevOps mindset of trying to be agile, trying to be innovative, trying to move towards more of a continuous type of deployment model. We really came across a number of the challenges these organizations have in terms of just really maintaining control, right? Once you have multiple DevOps teams doing things in multiple kind of accounts, you start to run into challenges relative to how you enforce best practices.

    And that’s really one of the challenges that security people have with this whole DevOps revolution, which is, “Hey, this sounds great, but we’ve got a playbook, a set of practices and policies that we really require all of our environments to adhere to in order to protect that data.” And obviously the faster you move, the more agile and continuous you try to operate, the harder it is to enforce those best practices.

    Shimel: More about that in a second. You touched on two things there that we’re going to dig into. Let me just finish laying my groundwork. Mike, besides you, Rich and Adrian, the Securosis gang, there are at least two other key people on the founding team of disrupt:Ops who also happen to be friends of ours for years –

    Rothman: Yeah, that’s right.

    Shimel: Give them their due. Why don’t you give them a shout out?

    Rothman: You bet. So back when we spun the technology out of Securosis, we brought on two guys, very experienced guys in the security space. Our CEO is Jody Brazil. So Jody was a cofounder and long-time CEO of a company called FireMon, which I know you’ve done work with for a long time –

    Shimel: Sure.

    Rothman: So Jody kind of is really the driving force behind a lot of what we’re doing at disrupt:Ops. And our CTO is a guy named Brandy Peterson, another longtime security professional. He was actually the CTO of FishNet Security for 15 or 16 years.

    Shimel: As was Jody before Brandy, right?

    Rothman: Yeah. I met both of them around the same time.

    Shimel: Yes, when I first met Jody, he was CTO and FishNet, and he said, “Who is this Jewish guy with the New York accent?” I went out to visit him in Kansas City to tell him why you shouldn’t use Qualys, and he should use StillSecure, and he made fun of my accent. [Laughs]. Good times. Anyway. But Mike, let’s turn back to a couple of things you were saying about kind of the genesis driving disrupt:Ops. You touched on one thing; you said, “multiple DevOps teams.” So this is something we see, and I call it almost second-generation DevOps. You know, when a company or an organization is first embarking on their DevOps journey and they do sort of a trail balloon, and they pick a project and they have a DevOps team – you know, and they may not call it DevOps. They  have a team that’s practicing DevOps sort of principles on this one particular project. And oftentimes, wow, great success, because you are – you know, it’s an isolated environment, right? In some ways you’re not in the quote/unquote real world.

    And then you say, “This is great. Let’s take it out. Let’s expand that beachhead. Let’s expand the bubble.” And all of a sudden – you know, larger organizations; small teams or smaller organizations not so much, but larger organizations – now you start having two, three, four, six different projects that are being run with a DevOps mindset and you have multiple instances of Jenkins and you have three different – this team’s using Puppet; that one’s using Chef. And serverless. And all of the complexities that we see in today’s modern software factory. And it’s chaos, right? And then at some point the CIO or VP or director of IT says, “Oh my God, this DevOps thing isn’t working so good. It’s all over the place.” Let alone the security guy’s screaming, you know, right out of the Phoenix Project, “No, no, no!”

    Rothman: That’s right.

    Shimel: And that’s a problem that we see with a lot of second-gen, if you will, DevOps installs, where it goes beyond the initial team to several teams.

    Rothman: That’s right.

    Shimel: That’s what you guys see.

    Rothman: We do. And it’s interesting, because what you have are really kind of two paths at that point, in terms of how these teams start to behave when starting to be put under scrutiny in terms of, kind of again, their ability to enforce operational and security best practices. You know, one group kind of buries their head in the sand and continues to move forward and really don’t do much, right? They’re like, “Oh, it’s okay,” and you know, they’ll maybe run a vulnerability scan to kind of prove that they’re sort of within policy, but they really don’t have an operational environment that is built to work in concert with the rest of their factory. I love that term, right? That really does encompass a lot of how software is being built today. And then the other part ends up being folks starting to build their own scripts, right? They take maybe an open source-looking thing, they start to build a bunch of their own scripts and code around it, and then what they find is they’ve got a suitcase full of these scripts that they end up having to maintain, upgrade. They don’t have any means of reporting or any scheduling for how they enforce and run a lot of these scripts.

    It’s really – you know, we call it automation, but it’s manual automation, which I get is a total oxymoron. But that’s really how it is. You know, you have a bunch of folks, they’re building these scripts out, they’re executing them manually or through very simple triggers, without any means of maintaining, upgrading or scheduling or reporting on what these really key aspects of their operational environment are doing. And it becomes, again, a big problem, where you have – it’s unwieldy because of just the sheer number of things that are happening in a modern DevOps environment with multiple teams. Then you compound it with a number of very tactical scripts that are built to surround the environment, if you’re doing anything. And then at some point somebody says, “Hold on a sec, folks. This is just not where we want to be. We really need to think about it more from an operational excellence perspective and make sure we’ve got an environment that we can enforce our best practices for security and operations.”

    And that’s kind of when they get the sense that, “Well, I’m either going to have to make a significant investment in this and become like” – Rich calls them the unicorns of cloud land, right? You know, the Netflix and the Capital One and, you know, the folks who have whole teams of environments that are building tools to do this. Or they have to start looking for some kind of commercial option, because there really is no option, but you’ve got to do something. You can build it yourself or you can look to buy it. And candidly, until the last handful of months, there really wasn’t a realistic commercial offering to start implementing some of these capabilities.

    Shimel: No, no. I think the bigger – you know, a lot of what you’re saying certainly obviously rings true. The other fundamental piece of this is, even if you are using one of these homegrown, cockamamie, you know Rube Goldberg kind of things –

    Rothman: Yeah.

    Shimel: – is it the security guys who are? You know what? Shame on me. It’s not the guys. Is it the security team who’s doing this, people? Is it the DevOps cross-functional team? Is it a little bit of both?

    Rothman: Yeah, yeah.

    Shimel: You know, how many of – look, here’s my basic, fundamental belief. You are not going to find a developer who raises his hand who says, “I want to write shitty code. I can’t wait to put something insecure out there.” No one says that. No one says that, right? They’ll tell you they’ve got time pressure, there’s pressure to release, pressure to release, they wish they could do more on security. And some of them, the enlightened ones, as you said, they’ll even try. They’ll run a vulnerability scan, because there are certainly open source kind of tools out there for them to mess with.

    The problem is, they go to a lot of the security guys, and as you alluded to before, security guys say, “Well we have our rules and procedures here. This is a no.” Right? And they don’t want to hear no. They’ve got to go.

    Rothman: You know, what’s interesting, Alan – I want to hit on a couple things that you alluded to there, because they’re important, right? The first is kind of where you start to see the agitation for better operational and security practices. And it’s not that it’s not the security folks. Obviously they’re always going to push for that. But it really is – you mentioned it, right? The DevOps teams that we’ve worked with, they’re very cognizant, they’re very serious stewards of kind of the corporate data, and they want to do the right thing. But in a lot of cases they don’t know what to do, right? So you have a whole bunch of folks who don’t know what they don’t know and they don’t want to go to the internal security team because of a lot of those things you were just alluding to. They’d say, “No, we can’t do that. We have all of these rules.” And in a lot of cases the security team has not updated their set of conditions, their set of policies – 

    Shimel: Exactly.

    Rothman: – to work in a DevOps type of environment, so these folks really struggle. They talk to – again, on the consulting side of our house at disrupt:Ops, we do a lot of work with those teams in terms of helping them understand what cloud native architecture looks like and how to integrate cloud security into their environment, and really just to help them build a program in order to do that. So then you do that with one DevOps team, then you end up talking to another DevOps team and maybe the third DevOps team, and finally IT, at some point, they’re like, “Um, what are you telling our folks? They seem to like what you’re doing and they seem to be doing things in a more secure way. So can you help us?” And then IT starts to get religion. The security team starts to get a little bit of religion about what they have to evolve to. And it really is kind of that interesting transition where it has started kind of out in the DevOps community with, again, those enlightened ones going, “I don’t know what I don’t know. Let me get some help,” and then it kind of back-ending into security. That’s not always the case, but probably three quarters of the time, that tends to be the transition that we see.

    And the other thing I want to hit is just about velocity, right? JA very forward-thinking DevOps leader that we’re working with, he said it the best. And he’s just like, “I’ve got 20 DevOps teams. I cannot slow these people down. They are doing stuff; they are making progress. The last thing I can do is put a bunch of onerous security requirements on what it is that they’re doing. They’ll ignore me, they’ll go around me and that’s counterproductive. I can’t do that. What I really need is to be able to build and enforce a set of guardrails around their environment.” Right? That term, “guardrails?” It just really kind of opened up and was a light bulb in our head when we were having this conversation. It’s not about stopping them from doing things. If anything, it’s about allowing them to go faster, yet making sure they don’t drive the car off the road, right?

    And that’s really the metaphor that resonates relative to what we’re doing at D:Ops, and really just kind of where the environment is going, in that we have to be able to move faster. Slowing things down is not an option, yet you want to make sure that you are not putting corporate data at risk. You are not opening up significant new attack services just – because people are moving so quickly. Sometimes they make mistakes. Sometimes automation isn’t done in the perfect way. Sometimes humans get involved and certainly are fallible from that standpoint. But with a lot of the inherent, underlying automation orchestration that’s possible within the cloud and DevOps and continuous, you have options to enforce these best practices in a way that we couldn’t do in our traditional data center with our traditional waterfall types of development and operational motions.

    Shimel: Absolutely. And you know, Mike – I mean, I’ve spoken about this a lot at various conferences and stuff. A lot of that has to do – you know, we use this term “shift left” in security and all that. A lot of it does have to do with getting to security earlier, faster. Yeah, we’re going to have to automate it. Yes, we can’t be the roadblock. But we can be the appendix after – when I say, “we,” I mean security. We can be the appendix after the delivery or after the deployment.

    Rothman: I’ll raise you. I see your point and I’ll raise you one, right? I’m of the opinion that security isn’t a thing. Right? Security should be within everything.

    Shimel: Yeah.

    Rothman: And that’s really the mentality that we’ve been trying to push on the industry for a long time.

    Shimel: Amen.

    Rothman: And it’s one of these things where we really can’t afford to wait and be in a position where we’re trying to retrofit poorly architected systems that are moving faster than we’ve ever seen before and then say, “Oh hey, we’ve got to change it because of security,” or because of any other operational gap that we see there. Right? What we have to do is be building in a way that allows us to use architecture as security control, right? Isolation, segregation, multi-factor authentical, identity management – all of these things that are fundamental building-blocks of what we do in the cloud and DevOps can be utilized to build in a much more secure fashion. And it’s really about those kinds of practices that are going to make the biggest difference relative to what it is that we see and what it is that we deploy over the next couple of years.

    Again, like I said, I don’t think security is a thing five to seven years from now. Because if it’s not embedded within everything that we deploy, we screwed up.

    Shimel: Yep. You know, I heard – because I never come up with original ideas. I just listen. But I heard something awhile back – “Security needs to become synonymous with quality.” And we’ve had QA as a concept in software technology forever or for a long time. Security needs to be just built in, like as part of the quality control. Right? The same way shit or stuff gets tested, whether it be hardware or software or whatever. Security needs to be part of that quality. 

    Rothman: You bet. To use a software term? Right? Why is a functionality defect different than a security defect?

    Shimel: Exactly.

    Rothman: Right? And they should all be within the same thing. And again, that’s a lot of when you start talking about immutable, when you start talking about kind of building security testing into your pipeline, which again, is stuff I know that you guys at staging-devopsy.kinsta.cloud write about quite a bit. Again, it’s really not any different than you managing and making sure that you hit your functionality kind of specs from the standpoint of just building code, right? Part of that factory. So yeah, the code has to work, but it also can and should be secure, and there really aren’t different operational motions to make that happen. And that’s what’s so exciting. For a guy who spent 25 years trying to retrofit and get people to think about security as part of their core structure and to treat it as really a business function, it’s really enlightening to be in a situation where you actually can implement a lot of those functions within the core processes that you use to develop the technology.

    And again, are we there yet? Of course not. Is there a lot of education to be done? Without a doubt. But we at least have the option now, right? That’s the aspirational goal to what it is that we all do every day, which is to make sure that we’re pushing for the right types of decisions to be made as early in the process as we can, because we all know how much it costs to fix it later.

    Shimel: Absolutely. Mike, I’ve got to tell you, listening to this conversation, I’m thrown back to, “NAC: is it a feature or stand-alone?”

    Rothman: You’re dating both of us, man. Stop that out. Cut that out.

    Shimel: It’s such a pleasure to talk a little theory here with you. But let’s turn for a second to real stuff. Not that what we’re talking about isn’t real. But I wanted to give our audience a little bit of a taste – so disrupt:Ops – great people, great big problem to tackle out there. Give us some specifics. What are you guys doing?

    Rothman: Yeah, so what we’re doing is we’re basically building a cloud management platform to implement a lot of these ideas that we’ve been talking about for the last couple of minutes, Alan. Initially it’s a guardrails platform. So what we have is the ability to, in an AWS environment, enforce a set of best practices across security, operational and also cost management. So one of those things we don’t talk about too much is that developers don’t really like to shut their stuff down because, “No, it’s going to mess with something. I don’t have to. The CFO bitches but we don’t really care.” What we can do is also implement best practices in order for folks to clean up their room, right? We try to do that with our kids all of the time, too. You know, so we try to do that with our developers as well.

    So really implementing a platform for security, operational and cost management guardrails, to again, immediately not just find the issues – there are a lot of tools, as you said. A lot of vulnerability scanners can kind of look at your cloud environment and tell you you have problems. The issue with that is that most operations people are overwhelmed with the number of other things they have to fix every day. Adding more stuff to their list doesn’t really help them. So disrupt:Ops is really focused not just on finding the issues, but also fixing them and enforcing those best practices. Let me give you an example of that. So let’s just say some person decides they want to open up a server to the internet because it’s just easier for them to do that. Well obviously that would be a pretty significant security problem. So as opposed to just taking that server offline, what you can do is you can basically change the security group to make sure that it’s only accessible via a known, good IP address range within your corporation. That’s a good example of a guardrail. Or making sure that you’ve got MFA turned on for every account that gets spun up by your magical Terraform environment that may not build that out as a core security requirement.

    So again, there are 50 to 60 of these guardrails, in effect, that you can implement in your environment, to again, make sure that you are not just alerting when you get out of policy, but actually fixing the issues, in a lot of cases before you even know they’re issues, so that you’re not sitting there trying to pick up the pieces after half of your data is in Chechnya.

    Shimel: Exactly. Or Romania.

    Rothman: Or Romania. It’s somewhere.

    Shimel: Mike, how is the offering packaged? Is it a SaaS kind of offering? 

    Rothman: Yeah, it is. What we found is since – you know, remember the days where people were like, “Oh, I don’t know about this cloud thing or SaaS. I’m not sure I’m comfortable with that”? Now it’s like, “Wait, you want me to install software in my environment? What? What’s wrong with you?”

    Shimel: That would be kind of oxymoronic for a DevOps tool, but –

    Rothman: That’s right. That’s right. So we deliver as a SaaS service. It’s actually a very cool kind of one-click provisioning process where, when you’re logged into your AWS account, you run basically a cloud formation script that really kind of gives us cross-account privileges to your environment. And again, we’re security guys, right? So at the end of the day, we are basically not even just assuming but enforcing least privilege. So we only have read-only access. When we need to make changes, we grant the privilege as we need it, then we revoke it. Because again, what we’re trying to do is make sure we enforce operational best practices; not add more attack service. So again, we’ve built the environment to be secure by design. We use all of our own dog food, to make sure that we are adhering to kind of common best practices. And again, we’re all 25-plus years security folks, so we do take that pretty seriously.

    Shimel: Excellent. Mike, for people who maybe want to download – not download, excuse me – sign up for it or check it out more, where can they get more information?

    Rothman: Yeah, disruptops.com. So by the time you hear this little interview, the site will be live. There’s a place you can sign up. You know, first we start with a demo and show you what it’s about and then you can sign up for early access. And again, we’re kind of expanding towards the end of this month, in October. We’re going to add a bunch more folks to the system with the goal of kind of being ready to actually sell the product by the end of the year.

    Shimel: Fantastic, man. Well, I don’t have to tell you, Mike. I wish you, Rich, Adrian, Jody, Brandy and the rest of the team nothing but success. We’ll be checking in, I have a feeling, pretty often here, whether it be through Security Boulevard, staging-devopsy.kinsta.cloud, Container Journal or what have you. And you know, fantastic. It’s good to have you back in the game.

    Rothman: Yeah, no, I’m excited. Again, really it’s not just a big promise and not just a huge opportunity. You know, and what Rich, Adrian and I, and now Jody and Brandy have always done is – you know, we’re trying to do the right thing to evolve the practice. Right? And that’s what it’s about. It’s a big world and there’s a lot of opportunity. There’ll be a lot of folks talking about guardrails over the next year. And that’s great, right? Because what we want to do is get to a position where we can move faster but move more securely, and I a lot of cases, in the world that we grew up in, those were counter indicators. Right? You couldn’t move fast and be secure. You couldn’t do operational excellence and make sure that you’ve got a lot of these things enforced. And now you can, and that’s what’s really exciting to us. Again, we’re just excited to partner with you, as always, and again, just go out and solve some bring problems for a lot of organizations.

    Shimel: Fantastic. All right. Mike Rothman, president and co-founder of disrupt:Ops, a hot new entrant in the DevSecOps space, with a great pedigree and a real reason for being our guest today on DevOps Chat. I hope you’ve enjoyed this. This is Alan Shimel for staging-devopsy.kinsta.cloud. Until next time, everyone, have a great day. Bye-bye.

    — Alan Shimel

  • DevSecOps and Development: Making the World Safer One Application at a Time

    DevSecOps and Development: Making the World Safer One Application at a Time

    If you have been around software development much at all in the past five or 10 years, then you’ve certainly heard of DevOps and know it’s here to stay. And it is very likely that you have also heard the term “DevSecOps” even more recently. With today’s extensive proliferation of application capabilities being delivered through services—and all the inherent complexity that comes with the delivery systems for them—it’s no wonder that cybersecurity has moved to the forefront for many companies. In fact, according to research from Stratistics MRC, the current global application security market is growing at a CAGR of 23.4 percent and is expected to reach more than $10 billon by 2023.

    But it is one thing for there to be an interest in cybersecurity; it’s another thing entirely to implement it well—and that is where the conversation gets more interesting. Today, the name of the game in application security is all about bringing the role of the developer into the fold and equipping developers to code, fix and maintain security as part of the software development process. In an article on DevSecOpsDays.com, DevOps thought leader and author Jon Willis offers the following comments:

    Fast forward to February 2017. I was speaking at the RSA Conference in San Francisco and I started seeing for the first time people talk about security as an interesting new way of integrating it into DevOps. They were unapologetically calling it DevSecOps. They were talking about this in a way that included security as a systems (end to end) approach to the traditional DevOps pipeline.

    This was not the first time I had heard people talk about security and DevOps.

    The thing that most caught my eye was that this approach was clearly putting the ownership of security on the developer. I immediately thought back to Werner Vogels’, the CTO of Amazon, statement of “You Build It, You Run It”, back in 2006. However, what I was hearing this time was “You Build It, You Secure It”. This thing they were calling DevSecOps was describing a holistic and systems approach to security as part of the software delivery pipeline.

    This notion of developer empowerment being a critical aspect to success was something I heard again and again at the recent 2018 Black Hat USA Conference. You can read more about my experience at Black Hat in this DZone article. But how do you effectively bring developers into the security fold? That is what we will explore in this multi-part series.

    In this first article, we will focus on what I (and many others) believe is the single most important factor for long-term success: the organizational culture. And by culture I am talking about the day-to-day environment of the organization the developer is a part of, and how they are (or are not) allowed to do their work. A successful DevOps culture is one that promotes collaboration to foster innovation and speed delivery, and the following two sections of this article depict perhaps the two most critical components for a strong DevOps culture: leadership and trust.

    It All Rises and Falls with Leadership

    Many would easily agree with the notion that long-term success for DevSecOps in an organization needs executive support and buy-in—and they would be correct—but real success needs much more than just budget approval and a shifting of resources. It needs ownership. It needs visibility. It needs truth-telling and that is where leadership comes in. According to the PuppetLabs “2018 State of DevOps Report,” 64 percent of C-Suite respondents firmly believed that security teams are involved in technology design and deployment, but only 39 percent of actual team members agreed. Bridging that kind of a gap requires breaking down traditional silos and evolving for better communication and transparency. The same survey also found that high-performing organizations were 24 percent more likely to use automated security policy configurations—a clear indication of tighter cooperation between security teams and the operations teams maintaining production environments.

    So, what are some of the key ingredients of successful leadership in this space? Glad you asked. Here are some things to look for:

    • Balance: Is the organization able to balance lowering technical debt with new work?
    • Shared ownership: Does everyone see how they contribute to business success and how their role impacts it? Is security included in that?
    • Shared learning: As projects progress and complete, is information moving across teams and projects? Are there best practices, lessons learned, etc., that are communicated both internally and externally?
      • Another way to think of this one: Is the level of “tribal knowledge” in your organization getting reduced?
    • Shared Vision Goals: Does everyone see and understand what matters most? Do they know the core metrics and KPIs, and how what they do impact them? Can they see their own success and bottlenecks?

    To have great success, you need leadership that defines the key standards, actively facilitates learning and collaboration and then provides the freedom necessary for innovation to occur. And there is another reason why you need it: Great leadership builds trust. Let’s explore that next.

    Your Speed of Delivery is Directly Proportional to Your Level of Trust

    The best depiction I have seen that correlates the speed of delivery with trust is this simple equation shared with me by DevOps Technical Evangelist Al Wagner from our mutual friend and Senior IT Specialist Lee Reid. It can also be found here. It simply states:

    Where “T” simply stands for time. In this simple equation, you can see that the HIGHER the percentage of trust is in the organization, the LOWER the overall value of the Time to Delivery will be, which is what we are all after. What that tells us is that today, businesses cannot tolerate a trade-off between speed and quality—you need both simultaneously.

    But how to you calculate trust?  In this model we are not talking so much about the work habits of the individuals as much as we are considering the outcome of each part of the process.  We  have to consider how often each of these areas “gets it right” versus how much we have to go back and rework.  Lee provides this example based on a client scenario:

    On a given project team the plan has about a 50/50 chance of being right, and the design is about 85 percent right, and the developers will get about 90 percent of the implementation right, and the testers have about 90 percent of the test cases right, and the release team has about 95 percent reliability of having the right stuff together to release, then your trust factor is something like

    .5 x .85 x .9 x .9 x .95 = 32 percent Trust

    Putting that back into the simple equation above, this level of trust means that your delivery time is three times longer than it could be.

    When my children were in elementary school and doing homework, I used to tell them that sometimes the best way to go fast is to go slow. The point was that if you wanted to solve problems you first had to understand them. Rushing to a formula or just trying combinations of numbers doesn’t work well and it ends up taking you twice as long to get done. This notion of trust and DevSecOps is similar and when you make the initial investment to build trust it can make a real impact.

    While it is true that adding application security to the Tplan and Tdeisgn parts of the process will initially increase those values, having application security as part of the software delivery process from the start—instead of at the end of a release cycle—will naturally cause the values for TDevelop, Tfix, and TEvaluate to shrink. This is largely because you are NOT having to go back to rework security into those areas midstream. It could also actually shrink Tplan and Tdesign over the life of the project for the same reason. In addition, the overall level of trust goes up because the risk that is associated with the delivered items decreases. We are more confident that artifacts are secure if vulnerabilities were identified early and fixed then.

    If you would like to learn more about how you can build trust, lower risk and make real business impact with application security, or to learn more about the potential impact of cognitive capabilities and artificial intelligence on your development efforts, download this complimentary Ponemon Institute study.

    — Rob Cuddy

  • DevOps Chat: The New Security Breed with Veracode’s Mark Curphey

    DevOps Chat: The New Security Breed with Veracode’s Mark Curphey

    DevSecOps has changed the way developers approach security in their application development cycle. But it’s also been the source of confusion, as some developers grapple with understanding how DevSecOps is different from application security. Mark Curphey, VP of strategy at CA Veracode, is clued in to the differences, which he discusses in this DevOps Chat.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hey, everyone. It’s Alan Shimel, staging-devopsy.kinsta.cloud, Security Boulevard, and you’re here for another DevOps Chat. Today’s DevOps Chat is all about DevSecOps, AppSec, what’s happening on the front lines. I’m joined by an old friend of mine, Mark Curphey. Mark is currently VP of strategy at CA Veracode. Hey, Mark, welcome to DevOps Chat.

    Mark Curphey: Thanks, Alan. Great to talk to you again.

    Shimel: All right. So Mark, let’s start off with this. I know you’ve fairly recently joined Veracode. Give our audience a little bit of your background and how you came to be where you are.

    Curphey: Yeah, sure. So misspent youth, went back to university late, did a master’s degree in information security, and wound up working for a bunch of banks in the city of London. Moved here to the states in 2000, then ran software security for Charles Schwab. So Microsoft were creating the SDLs. You know, sort of Gates had sent out the famous memo and I was doing the same for an online service basically at Charles Schwab. So that’s where I started. Then at OWASP, which some people sort of associate my name with, which is the Open Web Application Security Project.

    And since then I’ve worked at Microsoft around the security tools team and then I think with MSDN, the Microsoft Developer Network. I more recently created a startup called SourceClear which was focused on helping developers use open source code safely in the modern DevOps world. We were acquired by Veracode about five months ago. So I’m now the VP of strategy at Veracode helping the Veracode platform and how we really sort of work with DevOps and the new way of doing things.

    Shimel: Fair. Okay, so Mark, let’s start off with this though, and I think there’s some confusion out there with both security people as well as non-security people in terms of what’s the difference between AppSec and DevSecOps? Is one really just synonymous with the other? Is DevSecOps another name for AppSec? I don’t think it is. What do you think and how do you define the differences between them?

    Curphey: Well, certainly every vendor’s got a DevSecOps product these days, right? So it’s easy why people are getting confused. I mean I think the reality is the way you have to do AppSec in a DevOps world is fundamentally different to the way we’ve done it before. It used to be about educating developers, writing policies that people had to read, human processes to do final check off, things like threat modeling, you know, all of these sort of things. They kind of worked okay in a waterfall world were kind of modified a bit to work in an agile world. But, you know, DevOps is changing the game completely. So we’re in a world of automation, and, you know, the modern ways that we have to secure it are fundamentally different. So for me DevSecOps is about how do you apply security in this modern, you know, DevOps world and that’s predominantly focused around, you know, automation. What can you automate and, you know, how can we put automation in those pipelines that are happening so that security happens inline as code is created and it’s moved through the pipelines?

    Shimel: Fair enough. We talked pipelines. We talk left to right, Mark. You know, and Veracode is probably one of the leaders in this space when we talk about DevSecOps and shifting left and working. But I think what people want to know is really how does the security person or the security team work with the development team? It’s easy to say, but what does it really –

    Curphey: Sure.

    Shimel: What does it really mean?

    Curphey: Yeah. I mean, look, I think there’s a new breed of security people which is exciting to see. Like the old world that used to be the development team and then there were the security team. There were kind of the two separate teams. The security team needed to sort of bless things before they went into production, but we get appointed a data center and whatever. These days what you see is it’s all about velocity.

    So, you know, there are security people that are embedded into the security teams and they’re kind of acting like consultants and helping the dev team do security as and when it’s needed and make the right decisions without slowing things down. So instead of it sort of being this procedural thing where they were required to go through whatever a control grate, they’re just tired of the flow. That’s exciting to see because I think we’re seeing a paradigm shift of how that happens.

    Shimel: Yep, I think we are seeing a paradigm shift in this. So here’s going to be a question you’re going to say, “Alan, you’re crazy,” but Mark, does it make a difference? Is it definitively improving our security here?

    Curphey: Yeah. First, Alan, I’m never gonna call you crazy, but that’s good to know, I think.

    Shimel: I figured you would.

    Curphey: No, but look, I mean like, for sure, like think about like we all know what happened with Equifax, right? Like biggest data breach in history, result of open source that was insecure and hadn’t been updated. That problem is solvable and it’s solvable at scale in a pipeline. So what we were doing at SourceClear – I don’t want to turn into a product pitch in any sense, but we were basically trying to close the loop of this library is out of date, issue the pull request, figure out if it’s going to break things, and if so, update it. And the developers loved that, not because they’re fixing security but they’re fixing the dependency hell problem.

    So we’ve not been able to do that before because, you know, we hadn’t been injected directly in the pipelines. So these are opportunities that really make a difference and companies that implement these tools like us and there are others around, totally can avoid the whole Equifax thing. It’s not a case of like no one got notified or no one knew. Like it happens automatically in the pipeline. So there are great opportunities to do things like that.

    And I think moving forward we can also look at things like are people using dangerous APIs? Are they calling, you know, external services that we don’t want them to call? Like to a large extent the pipeline has become the new firewall and it’s a place where we can put controls, inspect the traffic, and decide what we want to allow and what we don’t want to allow which is a great place to be.

    Shimel: I agree. I agree there. So Mark, I guess where I wanted to go next with this is a lot – again, I’m a man who lives with a foot in each world, right? I spend a lot of time with my security friends and colleagues that I’ve known for years and years and years, but as part of staging-devopsy.kinsta.cloud I’m out speaking to developers an awful lot, right?

    Curphey: Mm-hmm.

    Shimel: And some of the questions that I hear from both sides is, okay, you can’t serve two masters, there can’t be two people in charge. How does the day-to-day of this flow? You know what I’m saying?

    Curphey: Yeah, no, for sure. I mean look, the reality is in all organizations the digital transformation and everything else happens that it’s all about value creation and the value is created by developers. So they’ve been empowered to choose their tools. They’ve been empowered to figure out which process allows them to innovate faster and all those things. With that we’ve seen the rise of the cloud and everything else. So it can’t be an us or them scenario. It’s how do you support those developers on their mission to change things and create this digital transformation? So the old ways, you still see it but you still see the old school of security people saying no and stopping people; but those battles are not going to be won. It’s about how do we help people move forward at the right rate.

    I mean I kind of equate it like someone was saying the other day, like similar to autonomous cars, like you’re not going to stop them. Like figure out how to make them work properly, safely, and securely. But if you don’t like it, you’re not going to stop that, have that technology happening and being created so you’ve got to embrace it. That I think is the philosophy that is going to win out the day.

    Shimel: Fair enough. I don’t disagree with you there, Mark. Let’s talk a little bit about what you see near term and maybe a little bit longer term in developments here that, you know, our listeners, you know, could fundamentally change their day to day. Where do you think we’re heading?

    Curphey: Yeah, sure. So, you know, I think machine learning and artificial intelligence and big data were sort of touted and, you know, the hype cycle kick off and, yeah, people didn’t always deliver on that. But there are exciting things happening in that which allow us at scale and at speed to do things that we couldn’t do before. So a good example of that is in the security world most of the vulnerabilities no longer exist in the CV system open source code. They just can’t keep up with the scale. We’re releasing thousands of libraries. In the DevOps world, developers find things, fix them, push them out. They don’t go back to monitor and publish these vulnerabilities, etc.

    But the great news is, you know, there are big data systems, machine learning that can process that data and we can identify those things at scale. Those same kind of techniques have been applied by Google and the Android App Stores and things like that. So that application of like that modern tech I think has great opportunities for us to do things which we’ve not been able to do before, and you’ve certainly seen some of the big vulnerabilities that have been released have been a result of that stuff coming out.

    Shimel: Got it. Got it. I just had my phone ringing there. I apologize. Mark, let’s, if I can and if you’re not too shy about it, I doubt you will be, let’s talk specifically a little bit about Veracode.

    Curphey: Yes.

    Shimel: You know, you’ve been there five months. What do you see? What gets you excited there in the morning?

    Curphey: Yeah. We had loads of places to sort of land. We came to the conclusion that people wanted to have less tools than more. So they wanted to buy a suite. When you look around, what the Veracode guys had done is built sort of best-in-class SAST and it started working on software composition analysis. We had best-in-class software composition analysis. So it made sense from that perspective.

    Also as you know, the heritage of the company, you know, it isn’t a fly-by-night company. It’s Chris Wysopal and Christien Rioux, right? These guys were in the loft, right? So that legendary around the place. So the company has this real DNA about it around security. So for me it’s kind of this exciting security DNA in the company embracing the way we build modern software. So that’s what attracted us to join forces. We’ve been extremely excited since. A lot of exciting things happening that for sure will start to come out of it at the other side no doubt.

    Shimel: I agree with you. I actually have Chris on a panel we’re doing on DevSecOps next week. I’m really excited to have Chris as well as Dave Duncan from CA, I think is on the panel with us.

    Curphey: Right. Right.

    Shimel: So Mark, you know, running out of time here, but I wanted to talk a little bit more about, you know, you came into Veracode, as you said, about five months ago via the acquisition. There’s a lot of people out there seeing a lot of buying and selling, M&A kind of stuff going on in especially around DevSecOps and DevOps related companies. I mean obviously a rising tide lifts all boats and there’s consolidation going on, but is there anything else at play that you’re seeing that’s kind of driving this stuff?

    Curphey: Well, I mean, you know, in the place that you look on the horizon is actually where the dev tools are going. So you know, if we look at companies like AWS, it’s not really just a hosting platform. It’s a full tool suite, right, of everything from machine learning libraries in the cloud to real-time databases and things. Google and Azure are building the same. As we start to think about like what is a next generation of developer look like, they’re likely to use an IDE and a browser. It’s not going to run on a desktop anymore. And you know, co-changes happen and everything happens up in the cloud. So I think there are a lot of exciting things happening there and how do you integrate security into that world for sure. So that’s a lot of the stuff that I’m focused on. Then you can get scaled I think.

    Shimel: Got it. Got it. Mark, you know, I appreciate you talking to us today. I think we’re about out of time. But for people who want to get more information on what’s going on, veracode.com obviously, correct?

    Curphey: Yeah, veracode.com is an excellent, well-trafficked blog with Wysopal and various other people and myself contributing to it which is great and we’re out speaking at conferences and talking about some of the both current work and sort of some of the futuristic stuff as well. So love to engage with the community and the industry for sure.

    Shimel: Perfect. All right. Hey, Mark Curphey, VP of Strategy, CA Veracode. Thanks for being our guest in this episode of DevOps Chat and we look forward to having you on again soon.

    Curphey: Yeah, great, Alan, thanks a lot. Great to reconnect and look forward to doing so the next time.

    Shimel: Okay.

    Curphey: Take care.

    Shimel: This is Alan Shimel for staging-devopsy.kinsta.cloud. You’ve just listened to another DevOps Chat.

    — Alan Shimel

  • DevSecOps: Helping Contain the Security Problems Containers Introduce

    DevSecOps: Helping Contain the Security Problems Containers Introduce

    The idea of building security into applications as they are developed versus bolting it on at the end always made sense but proved difficult to pull off in legacy environments. Monolithic software was too complex and the teams were disparate. But the arrival of containers 1) demands the need to attack the problem earlier because the attack surface changes so rapidly, and 2), thankfully, vastly simplifies the security process when DevSecOps principles are employed.

    The DevSecOps movement, which builds on DevOps by stirring in security, has been steadily building momentum as containers shoulder increasingly critical roles in larger organizations. Heck, DevSecOps even has its own manifesto.

    Although not all organizations adhere to the approach spelled out in the manifesto by DevSecOps.org (which says it was founded by “security practitioners dedicated to the science of how to incorporate security within agile and DevOps practices”), the core elements are the same: The goal is to shift left, as some people say, meaning start the process of securing applications earlier in the development process, improve communications, and repeatedly test along the way.

    Adding the Sec into DevOps

    Obviously, DevSecOps builds on DevOps, which is all about development and operations teams working more closely together to focus on the full life cycle of applications rather than one team focusing on the beginning of the cycle and the other on the end. DevOps is about breaking down boundaries of responsibility and modifying processes so the output is more resilient and code can iterate faster.

    DevSecOps is all that plus security. “DevSecOps requires changing mindsets, processes and technology,” Gartner says. “Security and risk management leaders must adhere to the collaborative, agile nature of DevOps to be seamless and transparent in the development process, making the Sec in DevSecOps silent.”

    The need to embrace the new approach is necessary because containers upend so many conventions.

    For example, in the DevOps world security tends to be viewed as a tech thing. “Are we encrypted?” “Yes.” “Great. We have secure connections. That’s all we need to do.”

    As containers become used for more mission-critical things, that approach is too limited because there are so many more moving parts.

    Before you might have had to worry about securing a few physical servers and a few dozen VMs supporting an application, and now that application requires you to secure 200 to 400 containers, spread across a couple dozen virtual machines in both AWS and Google Cloud. The amount of work has multiplied by 10. The older security methodologies just don’t scale.

    What’s more, there are more layers to worry about—your container runtime (Docker, RKT), your orchestrator (Kubernetes, OpenShift) and your build environment (Jenkins).

    And with containers we release things faster. It isn’t once per quarter or even every month. With containers and microservices architectures and continuous integration/continuous development, we are pushing new versions of software into production on a daily or weekly basis. Existing security practices can’t keep up.

    How Do You Do It?

    There are a number of ways to realize DevSecOps, but typically organizations aren’t going to fully integrate security personnel into DevOps groups, or require developers to become security experts or security experts to become developers. That’s what Gartner was getting at with keeping the Sec in DevSecOps silent.

    The movement is all about building processes and inter-team communications and workflows, which is also how DevOps started before it become more of a delineated role/persona. Yes, developers will have to learn more about security. Yes, security team members will have to learn some particulars about containers. And, yes, workflows will need to morph.

    But the good news is that once experts from the various disciplines agree on what is important, many of the DevSecOps processes can be automated.

    When to Test

    Job one is figuring out where in the container development process code should be tested for security. The goal is to set up a series of gates developers have to go through to make the resultant code more secure. Typically—and most critically—it starts with scanning source code repositories.

    So, before an app is even built, the code the developers are writing is being checked. Are they referencing a package with known vulnerabilities, or pulling in a library that has a vulnerability or is open source and against company policy? This a workflow change that starts the security process much sooner than traditional approaches.

    The next check in the cycle typically comes when the container image is being built, say in Jenkins, the popular open source build environment. When the image is being pulled together, packaging the code up into a container and defining dependencies, a scan is done to ensure it is free of vulnerabilities and is configured to meet the best practices and policies defined by the organization.

    And finally, when the image is pushed to the registry, the image is scanned yet again before it can be put into production.

    All these checks are tied to your CI/CD pipeline and just become part of the developer’s build process. The scans themselves are automated.

    “The goal of CI is to provide rapid feedback on disparate code changes, allowing developers to correct errors as soon as possible by identifying functional defects introduced into the larger code base,” according to eSecurity Planet. “In this environment, integrated security testing is needed to provide developers a real-time threat assessment of all changes they’ve made, regardless of their operational success in the larger code base.  Without integrated security testing, there’s a risk of re-engineering solutions multiple times to address security threats detected long after functional solutions are accepted.  That wastes valuable time, money, energy and effort.”

    Gates = Slow?

    Developers traditionally have viewed security as a necessary evil. They would build something, get it in staging, test it and then right before it hits production, some security review would identify 10 things that need to be changed.

    Testing more often, earlier and using automated scans vastly reduces the chance there will be significant problems with the resulting application. So, while the idea of introducing testing gates into the development process might seem like something that would put off developers, the end result will more than make up for any inconvenience along the way. Instead of huge changes at the end, there are incremental fixes at different phases of development.

    Companies are adopting many of the DevSecOps practices today even if they aren’t calling it that. DevSecOps is, after all, more of a process and a way of working rather than a specific role. Those practices enable organizations to move faster and improve the quality of the services they deliver from a risk, security and compliance perspective.

    Of course, there is no such thing as 100 percent security, and vulnerabilities will still crop up and be exploited. But even here a DevSecOps culture can be a huge step forward for container environments.

    In legacy enterprise environments, when a vulnerability is found in something that is running live, the organization would identify it as critical and have a service level agreement that called for it to be patched in two weeks. The patch needs to be developed and the deployment planned, and probably includes some amount of infrastructure management and updating.

    With DevSecOps and container environments, you can make the change in development and push it out within a day.

    So, even as containers change the security calculus by introducing more and new things to worry about, with the adoption of DevSecOps best practices organizations should come out being more secure and fleet of foot.

    — Loris Degioanni

  • DevOps Chat: Container Security and More with StackRox’s Ali Golshan

    DevOps Chat: Container Security and More with StackRox’s Ali Golshan

    Security has many faces, which is why it’s so difficult to implement effectively. Each environment is different; each security solution is different; the needs of each organization is different. That’s why, as Ali Golshan, StackRox founder and CEO Ali Golshan put it, security has gotten a bad rap.

    Golshan and his company are approaching security from a more DevOps-focused view, providing the necessary tools to enable DevOps teams to include security throughout the entire process. In this DevOps Chat, he and I discuss the many facets of DevSecOps, container security and more.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hey, everyone, it’s Alan Shimel, and you’re listening to another DevOps Chat. Today’s chat is a little DevOps, a little security, a little containers, little bit of everything. Happy to be joined by StackRox founder and CEO, Ali Golshan. Ali, welcome.

    Ali Golshan: Thanks for having me.

    Shimel: And, Ali, I think I mispronounced your name. It’s “Golshan.” Correct?

    Golshan: Correct. Yes, thank you.

    Shimel: Cool. I try to do it right. So, Ali, let’s start with the obvious, though, maybe some people out here have not yet heard of StackRox, so, before we dive in, just a quick sorta elevator pitch so they have some context.

    Golshan: Sure. We started StackRox three-and-a-half years ago. The premise for the company was we saw a couple of major shifts in the industry. One happened to be the introduction of containerization and microservices, which is a sub-effect of this larger trend that we’ve been seeing, which is application security and development becoming full product life cycle, so ensuring that you can cover security and compliance and requirements from build to deployment to runtime. And containers provided, really, a great packaging and a construct for building applications and microservices.

    And then the second part of it is is that, when you look at this model, we realize that there’s really three fundamental things that are disrupting a $30-billion security industry. This is the whole notion of a full product life cycle needing to be automated. It’s immutability and ephemerality. And, because you can’t forklift traditional security solutions and deal with these three constructs, we decided to go after solving this much bigger problem for what we like to see is for the future of cloud.

    And where we started was containerization just as the fundamental building blocks. As time and market has progressed, we’ve progressed with it, which is focus more on things like APIs and orchestrators and managed cloud environments, but very much with kind of a host-application-centric view of the world. We’re not a network-based tool. We’re not a log-ingest based tool. We tend to focus a little bit more on security automation, detection, and creating a multifaceted risk view of your environment, specific to containers and microservices.

    Shimel: Got it. So, you know, there’s a lot in there and we’ll come back to StackRox maybe a little later in our chat today, but I wanted to – let me hit on a couple of things you mentioned. So the first thing is this whole idea of cloud-native, container-native. You know, when we first saw the shift to the cloud, the security industry’s response was what some analysts called “cloud-washing” existing security solutions. So they took their existing security solutions and they said, “Okay, now it could run in the cloud too. It’s a cloud security solution,” but it really wasn’t. It was a on-prem security solution moved to the cloud. It wasn’t cloud-native and it really was a little bit of a round peg in a square hole.

    I think we’re seeing the same thing when we talk about security solutions built or trying to provide security for containerized applications. Right? What’s involved? Can you – you know, I don’t wanna coin the phrase, but can you “container-wash” an existing security solution and make it a container security solution, Ali, or do you really need sort of container-native-designed security?

    Golshan: To your point regarding traditional cloud environments, I think that was a smaller chasm, where security companies had to cross, because that original movement was a little bit more about virtualizing your environment. So there was still some kind of applicability of running things like WAS or endpoint solutions, to be able to broaden your security posture to the cloud. Obviously, the fundamental problems we saw were created were around how do you automate and provision workloads and then the nature of those workloads remaining. Those were kind of – the new capabilities in the cloud were actually the pieces that created gaps for security.

    So, if you then think about that and expand that further, the capabilities and designs for containers are even kind of another order of magnitude different than traditional. It’s not just virtualization and then outsize your boundaries. It’s virtualization; it’s microservices, every function broken down into a particular service; controlling it end to end. So, if you think about that there’s a large portion of the security industry that doesn’t even look at some of the most critical components of this process, which is the build and deployment. Building constant context there. Building declarative policies from that environment and then using it further. So I think that’s one of the major reasons we don’t see the applicability of traditional security solutions, even more so in containerized environments.

    Shimel: Yep. I mean, Ali, I agree with you and I think that’s why, frankly, a lot of what is today passing for container security seems to be people doing vulnerability scans of containers or, as I think I had spoken to you about the last time we chatted, you know, network IDS for the container. It just doesn’t – it seems very much square peg, round hole. And I’m not saying you don’t need vulnerability management, whether you’re in a container or not, and I’m not saying you don’t need intrusion detection, whether you’re in a containerized environment or not. But to say that that is, quote unquote, a “container security solution” seems just pushing it and just trying really hard to sell.

    But wanted to come back to another thing you mentioned in your opening remarks. And that was around this whole idea of not only are we living in a containerized age, but we’ve seen the rise – and I don’t wanna say the dominance of – but, you know, the whole CI/CD software factory pipeline model of which DevOps is an integral part of and we talk about shifting security left into the software development life cycle. And this is yet another way that traditional security has to give way to how things are done today, so how does that influence or how does that play itself out, in terms of StackRox?

    Golshan: The way we think about that is, if you think about who were the first companies who moved towards containerization and microservices, they were all companies that needed to deal with massive scale, building velocity in their development and to the offerings they have for their customers. And, as a result, they realized that kind of traditional model of building and deploying and managing was just not sustainable because it wouldn’t scale with the human element. So this notion of DevOps, and even containerization as a result, were really tools and methods for companies to be able to increase their level of automation and sophistication as to how they managed and run their infrastructure.

    So the way we look at this is is that, also, it’s the same reason why a lot of those first movers build their own security models around this environment ’cause nothing like it existed. So we see security kind of mapping to the same model, which is, if you think about it, things used to be a little bit more around a model of batch process, respond, do analytics, take those sorts of options. And, again, the way we look at this is is that what containers have done is break that fundamental workflow. That fundamental workflow isn’t kind of “Build something. Check. Build. Check. Build. Check. And have all these gating factors applied by security.” It’s a lot more fluid now. So, as a result, security has to kind of integrate into that whole workflow.

    And this is one of the big things we talk about, which is is that the reason you can’t take traditional security solutions and apply ’em here, it’s not just the form factor of what they’re applying to, but it’s also the entirely new workflow that’s been created by DevOps and these types of environments that impact how a security tool should run. Because it’s not just tying it to a form factor or choke point. It becomes more of a service mesh – not to borrow the word “service mesh” from kind of the service mesh project, but that’s why it lends itself very well. It’s a number of services integrated across the process and then, as you automate these processes, they become force multipliers for each other. That’s how DevOps works and that’s how security should work as well. You can’t break that workflow model just because your security tool.

    Shimel: Agreed. And you’re using the term “workflow model” and I think that’s a great descriptor of it, but, you know, a lot of goes back – and, again, part and parcel with DevOps – a lot of it goes back to the whole sort of Lean IT, based on Lean manufacturing. Sort of, it really is a factory pipeline kind of workflow model. And, when you think about it, the fact that, all this time, we’ve always kinda just stuck security on at the end, at the end of the packaging, so to speak, and then wondered why it didn’t work so great. It’s kinda obvious when you look at it that way, isn’t it?

    Golshan: No, absolutely. I mean, that’s a interesting analogy, right? If you think about it. If you have kind of a industrial environment and you have manufacturing, if you move from a manual workflows to robotics and automated work flows, the sort of security you apply is gonna be very different – how you look for bugs, how you look at, for example, applications. This is a good analogy because it’s a very indicative kind of model of what the current industry’s going through.

    Shimel: Mm-hmm.

    Golshan: So I’ve become a big believer that you can’t – this is where I think security goes wrong, trying to cover too many things at once. If you have a completely new emerging model, like containers and microservices and DevOps, it’s okay to say that our security or our approach deals just with that particular area. And, as small or as big as it may be, it’s important because it adds credibility that you actually understand what it fundamentally is built from and what it’s doing ’cause, if you say anything other than that – “Well, no, mine can cover network and end points and VMs and containers and Lambda.” – well, that’s just a lot of the reasons why the security industry got the bad rap it did. It’s just things fundamentally work different.

    Shimel: Agreed. Agreed. So, Ali, I think we’ve done a great job here, in the first ten minutes or so, of laying out these sort of macro-factors that are affecting not just the tech or even just the security industry, but the – it’s really affects the whole world because we’re all – every company’s a software company now. And so this really has worldwide bedrock kind of effect.

    But let’s turn back to StackRox, right? Full circle back to the beginning. So how are all of these things manifesting itself in –

    Golshan: Yeah, so the way we look at it is –

    Shimel: – in what became StackRox?

    Golshan: – is have one solution that has two modules to it that interoperate. We have the StackRox module that covers build and deployment and StackRox that covers runtime and production. And the way we look at this entire flow is very basic – on one side of the equation, you get to look at all the static and declarative information and you get to use it to say what things should be able to do, and then, any time you run it, be it in deployment, QA, or production, you get to learn from that application more and more and see what it actually can do.

    So our mission is really bridging what it should do versus what it can do. And the reason, for us, the automation part of it is a big part of it is, you know, as companies mature their overall workflow, they will have additional tools that are gonna be provided by their platform providers, by their cloud providers. As an example, you touched on vuln-scanning. Absolutely everybody should do vuln-scanning, but, more and more, we’re seeing it cloud providers and platform providers provide it, so the key there is not to provide vulnerability; it’s to be able to do vulnerability management.

    And the way we look at vulnerability management, it’s either being able to provide the customer the scanner or ingesting the data from all the scanners and then helping them prioritize it – “Where is this running in production? In production, what services are accessible by this?” So, if you have a low vulnerability that is running in a production environment, in a mission-critical app that is exposed to the Web, it should be valued higher than a high vulnerability that is running in dev in a closed-off EPC that has no access to any other services.

    So context is what helps you prioritize, but it also flows the other way. So, in runtime, if you’re doing detection, it’s important to understand, if an attacker, for example, shelved and broke into a – broke out of a container, grabbed some tokens or credentials, you wanna be able to go back and look at the, for example, the secrets that are being managed through the orchestrator and see where else those particular credentials or secrets are available, where else that deployment has happened.

    So, if you think about it, on the left side, you get a lot of really great insight. On the right side of the process, with the runtime and production side, you can make use of those to build very good boundaries and gating factors, and that naturally reduces your attack surface. So, as you feed this information back to the build and deployment, remove credentials or privileges or tools that are unnecessary, you’re inherently making your infrastructure and applications more secure. And, as a result, we have a much better landing zone for what we apply detection to. We don’t have to detect everything; we know where the weak points or where the gaps are in the system, to focus our detection.

    So that’s kinda how we think about the overall StackRox product. Give the customers, especially the build and deployment and dev teams, the right security, auditability, automation tools on the build and deployment side, and then give the right runtime detection, alerting, response, forensics, and enforcement on the runtime protection side, for security folks.

    Shimel: Excellent. And we’ve got a few minutes left here, Ali. I wanted to – for people who wanna get more information about StackRox, is it stackrox.com? S-T-A-C-K-R-O-X?

    Golshan: It is. Yeah. We have all of our information on our website and we actually offer free risk assessments using one module of our product, where a customer can just grab it and understand much better as to what they’re dealing with when it comes to their container infrastructure.

    Shimel: Okay. Let me come back one other thing to you and, you know, the proverbial “Let’s see your roadmap” slide, but I don’t really want you to show me your roadmap. But what do you think is next, not only for StackRox, but as a result of this macro environment that we’re living in? What’s the next shoe to drop here and how do StackRox sort of – I’m sorry about the noise there – how do StackRox respond?

    Golshan: Sure. I think it’s a couple of things. One is we’re seeing more maturity across this entire stack, so we’re seeing convergence around orchestrators; we’re seeing kind of the standards around runtime engines being built; we’re seeing how public cloud providers are consolidating certain services and making other services much easier to use. Perfect example’s all the efforts around, for example, Kubernetes, between AKS, between EKS and GKE.

    So, when we look at these trends, it shows a couple of different things. One, not only are the containers different, but the entire attack surface is going to look very different, so the way you approach this world is gonna look very different in the future. We’re not gonna make the same mistakes we did with traditional security, which is we’re not gonna make the host overly exposed with a lot of user accounts and credentials and privileges and tools and packages. So we think that that host will obstruct the way and the orchestrator will become kind of a new cloud operating system, with APIs.

    And, as a part of that, you’re gonna end up seeing a lot more complexity, so, as companies get more comfortable running applications as microservices, more containers, better grasp of the orchestrator, they’ll get more comfortable running more complex operations and applications on top of it. Naturally, that comes with maturity. And once you have that, you end up going from kind of what right now is, which is very early stages of this ecosystem, which I would consider to be low entropy, to a very high entropy environment you’re gonna end up seeing.

    As a result of that, the way we see security being built is, one, that’s why it’s very important to focus on automation, integration, and the workflow, but, secondly, we see security having to be built in cloud native as a form-factor-agnostic model, meaning leverage the existing frameworks to – if it’s collecting data, for example, from the system, use, for example, things like the extended Berkeley Packet Filter to use that. If you’re running on Amazon or Google, you can use things like Pub/Sub or Kinesis.

    So it’s building more modular, programmable, and pluggable architectures, versus building a binary that deploys in a container and then you replicate that container against a bunch of hosts. It’s about building a sensory model that can understand different environments and make sense of them, not single agents that unilaterally take action, because, as complexity, heterogeniality grows, naturally, there’s gonna be a lot more gray area, as far as use cases go. And that’s where you run into the old problem of alert fatigue, noise, false positives, and then, eventually, you end up with 50 security solutions.

    Shimel: Excellent. Excellent. Well, Ali, we’re about out of time, but I gotta tell you, man: I love this chat. This was great. I’d love to have you back on the –

    Golshan: Yeah, same here, Alan.

    Shimel: We’ll dive into some more of this stuff. Good stuff. Ali Golshan, cofounder and CEO of StackRox, here on DevOps Chat. This is Alan Shimel. You’ve just listened to another chat. Have a great day, everyone.

    — Alan Shimel

  • DevOps and the Dark Pools of Security Technical Debt

    DevOps and the Dark Pools of Security Technical Debt

    We live in a world of digital transformation where organizations across industries are embracing initiatives around automation, cloud deployments and containerization of applications—and for good reason. Enabling transformative technologies such as DevOps, for example, can deliver superior product differentiation, faster time to market for new applications and services and an improved overall customer experience. A win-win, right?

    Not so fast. Amidst the explosion of new technologies, tools and practices, many business leaders are jumping in head first without testing the security waters, so to speak.

    In their race to move fast, it’s all too common for organizations to quickly realize that they’ve landed in a deepening pool of security debt that hackers are eager to exploit. As evidenced by recent breaches resulting from misconfigured AWS security settings, leaked S3 buckets full of secrets, database security misconfigurations, ransomware and the growing interest in exploiting CI/CD tools for hijacking IT assets to mine cryptocurrency, attackers are increasingly targeting DevOps environments.

    Security and DevOps Teams Swimming in Different Pools

    When it comes to prioritizing efforts, devoting attention to the non-functional requirements of DevOps infrastructure security often takes a back seat to the more observable value that the output of these systems provide. This is where security technical debt is created.

    Often times it’s a shortcut in establishing a process around managing security configurations, which is the first step on the path to an “event.” Other times it’s aligning personal development processes to corporate IT security standards.

    No matter what the first step in the chain is, the accumulation of unaddressed issues, incomplete process and failure to attend and improve security posture is how companies and groups find themselves in security quicksand.

    How deep and how wide is the security debt pool for DevOps and digital transformation initiatives?  Numerous studies show that most organizations simply don’t know what their level of exposure is, but there is concern about the unmet need of consistent security posture. We know this because security and compliance consistently rank among the top three barriers for DevOps and cloud initiatives year over year as evidenced in Puppet’s “State of DevOps Report” and RightScale’s “State of Cloud Report.”

    According to the “CyberArk Global Advanced Threat Landscape Report 2018,” fewer than half of DevOps respondents said DevOps and security teams are well-integrated, and 41 percent said security teams are only brought in at the end of the development cycle. These two departments are clearly working in different silos.

    IT security professionals and developers often have different areas of responsibility and focus on the DevOps adoption path. Owing to variable technical capabilities and operational silos, roles and responsibilities can get confusing, and the resulting work may or may not get done—leaving technical debt.

    While DevOps teams typically take ownership over application testing, QA and remediation for security flaws in software, there remains a significant “gray area” that isn’t wholly within the DevOps pipeline flow. For example, managing the security “of the pipeline” and adherence to the corporate standards for login ins/2FA and privileged account management, management of access control to secrets and rotation for shared sensitive corporate assets such as databases can impact other teams. Who builds and maintains these core controls for which the business needs to attest regarding the security of these systems when service level responsibility or operational controls are outside the DevOps team’s controls?

    It’s around these gaps that security responsibilities and operational roles get murky, and the debt pools deepen while issues remain unaddressed. The security maturity of various DevOps tools and open source projects vary widely. Missing security features as well as misconfigurations are to blame for numerous security events at many organizations with new stack components at work.

    As Cyberark Labs research has shown, DevOps pipelines can be starting points for lateral movement to other IT systems, so designs of isolation and containment are often aspirational rather than the actual achieved outcome. And for DevOps teams under tight deadlines, there are only so many hours in the day, and only so many additional items that can be added to the burndown list before something has to push and isn’t addressed in any given release.

    Breaking Down the Silos to Stay Afloat

    When these issues hit a breaking point, we see one of two outcomes: security gets involved too late and “puts the brakes on,” or the team makes a premature push to production of a transformative initiative resulting in a security event when the security process simply isn’t ready.

    When integrated and working toward common KPIs and goals, security can provide the extra bandwidth, focus and expertise to avert issues when they are brought into the process in a consultative role early, so called “shifting left.” Security needs to be built into development pipeline processes, and as we’ve seen with breaches of late, security needs to be a “feature” of the pipeline itself, which is the new attack surface cited above.

    Likewise, security teams must embrace new ways of working in the “fast-flow” of modern software engineering. They need to leave behind alarmism and “all or nothing” mental models, which were the security modi operandi of prior eras.

    Business level visibility to KPIs of success in integrating DevSecOps across the organization is often the missing link.  Whether it is red teaming the DevOps pipeline to raise awareness of the latent issues, or measuring security feature delivery success, measurement creates shared values of burning down security tech debt. Whether it is the most common security events by incident (eg. github hardcoded passwords) or the lead time to security feature delivery by the team, accountability insure attention to security success.

    In the end, it’s critical that DevOps teams work with security to ensure that speed, velocity and resiliency aren’t sacrificed for security. Every company’s fortune rises with the new value being created by digital transformation, but it can also fall with the reputational damage of a breach. Organizations need consistently aligned security and risk postures across all these new tools and technologies to reduce the pernicious pools of technical security debt.

    — Elizabeth Lawler

  • DevOps Chat: DevSecOps with ShiftLeft CTO Chetan Conikee

    DevOps Chat: DevSecOps with ShiftLeft CTO Chetan Conikee

    Security in the DevOps process has gotten a lot of attention lately, as companies grapple with ways to include security further up the application development process. The industry is still very much a nascent space, as players remain engaged in the security-versus-speed conversation—one always is sacrificed for the other. ShiftLeft, however, doesn’t believe that’s the case.

    In this DevOps Chat, Chetan Conikee, CTO of ShiftLeft, discusses his company’s approach to DevSecOps and how it can help keep the security department from being the “department of ‘no.’”

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hey, everyone it’s Alan Shimel and I’m here for another DevOps Chat – DevSecOps related. I’m joined by Chetan Conikee, a CTO and cofounder of ShiftLeft. Chetan, I hope I didn’t mess your name up too bad.

    Chetan Conikee: Absolutely not, you did a great job. Thank you Alan, appreciate it.

    Shimel: Thank you. So Chetan, as I mentioned you’re the cofounder CTO of ShiftLeft – we’re going to talk a little bit more about ShiftLeft later in the show – but I wanted to start off talking about what is really one of the hottest topics in our world today and that is this whole whether you want to call it DevSecOps or DevOps and Security or Security Shifting Left – hence the name – talk to AppSec, some other people do – you know what about in your world? Are you seeing this kind of emphasis like never before on what some are calling DevSecOps?

    Conikee: Absolutely. Great question, Alan. Let me first start with the term shift left because it has two definite representations; one is a word and the other is a noun. Formerly ShiftLeft started as a way of doing things; meaning taking a practice and moving towards the left in order to reduce what they call as the meantime to repair. And that something could be testing or perhaps measurement of security as well.

    So when we adopted that name or changed the word to a noun we wanted to take this seriously; meaning how can we effectively build a product that can measure the security posture of another product at the left, informed from the right as well because often DevOps is nothing but a pipeline – an affective pipeline that is operating both at automation and high velocity. DevOps is all about taking code, building, packaging and thereafter deploying into your production ecosystem.

    But it doesn’t just end at deploying: it ends at measuring and feeding back how to optimize your systems thereafter. Often security does not fit well into this high velocity motion because most of the security products today are legacy or antiquated. These technologies like static analysis, dynamic analysis or perhaps a web application firewall finds it hard to fit into this streamline DevOps biplane. And I use that term emphasis with heart simply because these tools are producing outcomes and these outcomes are nothing but logs, information, alerts and all of this is sent back to humans in silos. And it is upon the human to figure out how to correlate this information and take affirmative action.

    So in essence this is impeding velocity rather than accelerating velocity, which DevOps is all about acceleration. As a consequence security gets left behind in organizations.

    Shimel: Can I tell you – I’m sorry – that was one of the best ones I’ve ever seen – one of the best definitions I’ve ever heard.

    Conikee: Thank you. Appreciate it. So we take this seriously as well because what we are trying to do is convert that definition into automaton, meaning how can you measure security thereafter quantify threats to vulnerabilities because often vulnerabilities are measures on the left, threats are measured on the right, so how can a human comprehend both these vectors, associate them and then address and fix these issues, because security is a huge concern these days.

    Shimel: Yeah. Agreed. So Chetan this brings up something I wanted to spend some time talking about here today and that is it’s a – not a survey but a study that ShiftLeft’s done – I think you guys had a partner on it – why don’t you share a little bit with our audience kind of just laying the groundwork there.

    Conikee: Absolutely. Today we are announcing the industry’s first real world benchmark of what we call a continuous application security solution. And in order to conduct this benchmark test we partnered and collaborated with Cobalt which is a leading provider of integration testing. They have some of the world’s best penetration testers working for them.

    So in order to conduct this benchmark test we submitted two instances of an application – a vulnerable application. By vulnerable I mean it essentially contains a certain set of vulnerabilities in it which we often test against in order to measure the efficacy of our platform.

    In this test we took that application and we deployed two instances in production. One instance was unpredicted: the typical instance that represents these vulnerabilities and have no protection applied to it. Whereas the other instance was protected by ShiftLeft’s one-time agent.

    So the premise of this test was to essentially let these penetration testers conduct tests against these two applications and thereby come back with results. And both these results had a certain degree of focus. The results were to identify what vulnerabilities can be declared by these pen testers upon the application that is not protected by ShiftLeft and thereafter subject the same set of tests on the application protected by ShiftLeft and come back to let us know whether they could penetrate that application or not, or perhaps could exploit that application or not.

    So this was a 14-day exercise and we defined the scope for this exercise – meaning what we essentially did was identified the most vulnerable or relevant applications called and racked by OWASP top ten and thereafter opened these applications to the Cobalt pen testers.

    Now as they began this exercise and Cobalt had hired essentially three white-hat hacking experts to attack both these applications. Cobalt was able to find and exploit all the six vulnerabilities in the unprotected application which is a validation for the fact that this was a beta application, it had the vulnerabilities and these exercises were successful from the pen tester’s perspective.

    Now they took that same exercise and applied it to the application protected by ShiftLeft and the outcome was they were not able to exploit any of those scoped vulnerabilities that we had described in the application.

    So the proof point for this exercise was to make ensure that ShiftLeft took the onus of protecting the application from any existential breaches or exploits applied by both potential white hats or black hats in the real world.

    Shimel: Got it. So fascinating stuff here. Talk to me now – so how – and forgive me and forgive our audience – we may not be knowledgeable about ShiftLeft – how does ShiftLeft in your company – not as the noun or verb here – how does what you do at ShiftLeft play into this Chetan?

    Conikee: Absolutely. Great question Alan. ShiftLeft is the industy’s first convert solution that seamlessly integrates into DevOps pipeline and protects the application by identifying its attack surface without reacting to threats. And let me explain what I just described in a moment.

    Often security solutions that have been thriving over the past few decades are built to essentially identify and understand exiting threats and thereafter protect the application. And by that I mean often when a threat is applied or an exploit is applied on the application that threat is published in public databases like missed CBEDBs and NBDBs.

    Then after all of these instruments that protect the application are validating every request applied on this application informed of the threat landscape. Now often threats have a lifecycle, meaning an attack or attacks an application by identifying potential breach paths. Now after they attack it is titled as an unknown vulnerability and when it is discovered it becomes a zero day vulnerability and thereafter when it’s published in a database it becomes a known vulnerability.

    Now note that often organizations are exposed in that journey from unknown to zero day known because it’s yet to be published. And for all these instruments that are protecting applications there is yet to update its policies to thereby protect the application.

    So we at ShiftLeft have devised a convert solution that first of all understands the applications evolving at that service. Now in the world of DevOps we deploy application multiple times a day, a week or a month. Now as the application evolves to serve a company’s business needs the application’s attack surface is evolving as well. It’s either heading in a positive drift or a negative drift. Often more times it exists or traverses in the negative side because engineers are coding at high velocity and when they code they don’t take security as a function into consideration.

    As a consequence attackers can figure out more creative ways to exploit the evolving attack surface of the application.

    So ShiftLeft has devised a technique that automatically processes byte code into the CI portion of the CICD of DevOps and extracts what we call is the attack surface of the application. Now from the attack surface we extract a policy: the policy that defines how to protect the application in one time because we do not want to feed these alerts back to humans and ask humans to correlate and address and prioritize these vulnerabilities.

    So in the spirit of respecting the fast motion of DevOps we extract the security profile from this attack surface and bootstrap an agent that is instrumented with the running application.

    Now the job of the agent is to act like one-fold full-time employee. The agent has taken the onus of identifying what is a potential imminent threat that is applied on the application by a black hat hacker and thereby understand better this particular track that’s taking this application into an inadmissible state because often attackers want to first take the application into a state that they wish to take it to and thereafter either exfiltrate data from the application or perhaps install some malicious payload on the host.

    So the job of the agent is to identify and segregate malicious intent and thereafter prevent that particular intent from manifesting.

    Shimel: Now Chetan I sit here listening to you – and keep in mind I have 20 years – 18 years in the ______ site space and I remember writing articles and papers on when is a zero day a zero day, when is it less than a zero day, you know, this whole discussion around what we can do to make our applications less vulnerable.

    And what amazes me is – I think we all agree, right, application security – at least for the last three to five years, maybe more – has really been at the forefront of cybersecurity research and so forth. And it’s only now that we’re starting to see solutions like ShiftLeft which are kind of not the same old same old taking a different view to an old problem or a different solution to an old problem.

    And yet, you know, I just got back from Black Hat – I’m not sure if you were there – we hear about one breach after the next and they’re still spinning money out of ATM machines. Are we finally on the right track to you think? Are we making progress?

    Conikee: Great question Alan. I would say that we have begun to make progress. You know, there is a journey, we have to embark on this journey and that journey means taking a practice and embedding into our typical pipeline – a pipeline of how we build applications, deploy applications and protect applications.

    So this is a beginning but a great beginning. But what has enabled this beginning is inspiration. You know often we learn from other systems and what I’m sensing is security is learning from DevOps. DevOps is all about automation, meaning before Amazon – the advent of Amazon or any public cloud like Azure or GC there was – the notion of provisioning a host took more than one step: we had to write a multitude of scripts in order to provision a simple physical appliance.

    And with the advent of companies like Chef Puppet and the public cloud they created this concept called this cloud formation; meaning how do you provision a virtual appliance in a matter of seconds and make it predictable thereafter? Because there is a system of record of your provisioning script and its evolution thereafter – that is one part. The other part is the predictability: how can you assess that particular script and understand its posture? Posture from a resource consumption perspective and resource provisioning perspective?

    Now if you take these two same principals to the antiquated world of security – and I use the term antiquated because in the past security was all about reacting to threats and alerts and then we had to deal with how to prioritize alerts because alerts can contain mixes of true positives, false negatives, false positives, et cetera. So how can we steal a page from the book of DevOps? How can we automate security? Meaning you’re creating code, from this code how can we automatically extract its security posture?

    Because in the world of security we are assessing code with three particular filters – or what they call SOSAs, sanitizers and syncs – SOSAs are where consumers interact with our application; sanitizers are where we are validating that input; and syncs are all communication points outbound from an application perspective.

    Often attackers are attempting to create new syncs after they successfully exploit the application. So how can we extract this particular fabric or script and thereafter take a security function into consideration?

    So this is our goal at ShiftLeft and we are hoping that this becomes a norm and other companies like us are adopting the same principal as well.

    Shimel: We can hope can’t we? But it’s interesting. You know I – what’s interesting Chetan is no developer raises their hand and says, “I want to develop insecure code.” No ops person says, “Boy, I want us to be the victim of a breach or have a security incident.” Nor does any security person say, “I hate the rest of the IT team so much that I hope they get hacked.” Right? None of this is true.

    But yet we have problems – and some of it granted is not just from a technology or toolset perspective, a lot of it frankly is cultural, right, as I’m sure you’ll agree as well. You know, and any time we could use tools that help foster culture, that help foster greater communication and teamwork – I mean I think we all benefit from it, right?

    And I hear a little bit with ShiftLeft, you know, this is something that I think is a byproduct if you will of your solution, is that true?

    Conikee: Absolutely. Well said, Alan. For the most part security should be an enabler rather than a department of no.

    Shimel: You’ve got it.

    Conikee: And security should – there should be a common lingua franca between various departments: meaning security, engineering, quality assurance and operations should speak the same language. The language should be comprehensible so that it becomes actionable.

    Shimel: Amen. Amen. You know, Chetan when we first started I think I mentioned to you the time goes so darn quickly.

    Conikee: Absolutely.

    Shimel: We’ve blown past our 15 minutes onto 20 here so we may have to call it an end but for people who may be interested in the recent study you did with Cobalt – and my friend Carolyn Long is a VP at Cobalt and she is one of the smartest people I know in security – where can they get more information?

    Conikee: Absolutely. Please visit blog.shifleft.io and on our blog we have a recently trending article that gives you the necessary detail about this exercise. And at the end of the article is the published benchmark’s test for those that want to delve into the detail – which I highly recommend – and thereafter visit www.shiftleft.io and – or contact us; we are happy to onboard your application and thereafter protect your application as well.

    Shimel: Fantastic. Chetan Conikee CTO co-founder ShiftLeft. Thanks for being our guest in this episode of DevOps chat and continued success with ShiftLeft – your success is all of our success.

    Conikee: Thank you Alan. Pleasure, and thank you for hosting us.

    Shimel: Okay. This is Alan Shimel for DevOps Chat. You’ve just listened to another DevOps Chat. Have a great day and see you soon everyone.

    — Alan Shimel