When the cloud first caught on, there was a problem with Shadow IT—developers spinning up instances in AWS without the IT team knowing they existed. Now with the ease of CI/DC automated deployments, the problem of Shadow Code—code being added to apps that did not go through the entire team process—has arisen.
In this DevOps Chats we speak with Elad Koren, VP of product at PerimeterX, about the Shadow Code issue and how his company can help.
As usual, the streaming audio is immediately below, followed by the transcript of our conversation.
Transcript
Alan Shimel: Hey, everyone, this is Alan Shimel, and you’re listening to DevOps Chats. Today’s DevOps Chats features Elad Koren, who is VP of product at PerimeterX, and Elad, welcome to DevOps Chats.
Elad Koren: Thanks, Alan. It’s an honor being here. Happy to join you.
Shimel: I appreciate you being here, and it’s our honor to have you. So, Elad, we’ve never—I don’t believe so—we’ve never interviewed or covered PerimeterX before on Security Boulevard or DevOps. And why don’t we start with, why don’t you give our audience a little background—what’s PerimeterX about?
Koren: Cool. I believe you’re right. PerimeterX has been around for a bit more than five years, and its main goal is to secure businesses, digital businesses across the web. It can be e-commerce, SaaS, and other online businesses, essentially providing solutions to the online digital challenges they have.
We started with our Bot Defender solution about four years ago, and this targets the bot problem around the web. I’m sure you’re well aware of it, being the security professional you are. About a year ago, we’ve also launched our Code Defender product, which looks at the client-side code and highlights vulnerabilities and issues around that. And we will, of course, develop some more in the future as part of our growth plans to continue in securing digital businesses around the world.
Shimel: Excellent. So, we’re gonna dive into that in just a second, but Elad, I also always like to ask people about their own personal journey. How did you come to be VP of Products at PerimeterX?
Koren: Wow. [Laughter] It was a long journey. I’m actually, I’ve actually been in security for more than 15 years. I started with the Israeli Intelligence and the IDF. Then moved to my startup where I took all the security and kind of compliance and fraud areas. From there, I moved on to RSA Security, so you’re probably familiar with RSA Security, now part of Dell. And after a short period in Payoneer as their Head of Compliance and Security, I found myself in PerimeterX, leading the product group and now the VP of Product. I’m very proud of what we do with really, a sense of accomplishment with what we’ve been doing so far.
Shimel: Excellent. Well, RSA Security was spun out of Dell now, right? Didn’t they do a PE deal a couple months ago, I thought?
Koren: Yeah, yeah, they had a discussion around that. It was actually interesting because RSA was one of the biggest assets EMC had before Dell acquired them. And once Dell stepped in, I think RSA wasn’t that interesting for them, so it was only a matter of time.
Shimel: Yeah, no, Dell seems to have a love/hate relationship with the security companies they acquire, but that’s for another podcast. Let’s stick to PerimeterX. [Laughter]
So, client-side code inspections and security—talk to us a little bit about what you’re doing there and then I wanna jump into this concept of shadow code that we spoke about off-mic.
Koren: Yeah. So, first of all, very important to understand, you know, the first question that comes to mind is, why are we talking about it on staging-devopsy.kinsta.cloud. So, one of PerimeterX’s top targets is to assist DevOps teams to do their job better. And the fact of the matter is that, after we’ve had our bot mitigation solution for quite a while and DevOps was one of our personas, we went out and asked our customers, what’s their next major problem.
And one of the things that came up for many, many customers was the fact that they have a challenge around their client side, okay? Developers are adding libraries—well, you know, it’s a common practice. However, some of these libraries, JavaScript or others, they’re not that secure, and they’re not that trustworthy. What we did at that point is kind of think of that problem and what it reminded of. And it kind of reminded us of the problem of the shadow IT that we’ve seen about 10 years ago or so, where employees would just come with their devices or do whatever they want with their bring your own device kind of concept, and companies started fighting that. But at the end of the day, they understood that they have no weight.
It’s kind of similar with code. Developers will find the best weight to add code, even if it’s a library that they can just take and adjust slightly, and even if it’s a GitHub repository that no one maintains, and these introduce significant risk.
Shimel: Yeah. Absolutely. You know, it’s interesting, I think no one in DevOps questions anymore when we talk about adding security to the mix. I mean, thank God. You know, a little bit of my own personal journey, right? The reason I got into DevOps is because I thought it was the greatest thing for security. I thought this, the way it came down that this whole DevSecOps movement and shifting left and making developers more aware of security—I envisioned that when I first found out about DevOps. And I was hoping it would give us another chance to do it right, to correct some of the mistakes that were kind of inherent in our whole model.
And I’m not saying it’s a panacea or it’s perfect now, but certainly, we can acknowledge that developers do care about—no one wants to develop insecure or crappy code. People have pride in what code they develop. Now, when they didn’t know—the more they know about security and the more tools we can give them that allow them to kind of do better, develop more secure code, the better off we are, I think.
But this issue of, as you’re calling it, shadow code, right, kinda growing ________ with shadow IT when the cloud first came.
Koren: Yeah.
Shimel: It’s a problem. And the thing about it is, again, developers don’t raise their hand and say, “I wanna sneak around. I wanna sneak a little code in the back door.”
Koren: Exactly.
Shimel: Right? They do it because they have time constraints, they wanna get it done, they want it to work faster, right?
Koren: They’re exactly the reasons that we’ve understood when we looked into it.
Shimel: Yeah.
Koren: And it makes sense. If you think about it, developers should be able to add whatever it is that they think can help them develop faster. I mean, it’s the business goal.
Shimel: Mm-hmm.
Koren: The main question that we asked is—well, how can you make it more streamlined, okay? How can you—and now, going one step backwards or sideways, there’s a disconnect, okay? The developers, as you said, they wanna make it a secure code, but the business owner or the person who manages the security posture of the application doesn’t necessarily know that somebody introduced a new vulnerability, because the developer thinks that this repository is safe. I mean, it should be, it’s in GitHub. [Laughter]
Shimel: Oh, no—agreed.
Koren: So, the thing is—and this is where we asked around, our customers, what are they doing to mitigate this type of risk and what are they finding out. And what we found out is, interestingly enough, you have a lot of performance monitoring tools, but you have zero security monitoring tools on client side in real time. You have static code analysis, you have a lot of other solutions trying to find the vulnerability before it goes out to production. But once it’s out there, nobody monitors.
And you have to keep in mind—I know you are keeping in mind, but our listeners—the fact that, well, when in run time, you can inject dynamic code, that changes. And if somebody else gets control over it, this is a problem.
Shimel: Yeah, yeah. Well, and there’s a couple, I think, other complicating factors, Elad, that I’d like you to address. And that is, you know, today, where does that client side code live? Is it in the container somewhere with the whole Kubernetes microservices mesh thing goin’ on?
Koren: Mm-hmm.
Shimel: Is it in the traditional hypervisor kind of environment, or maybe something like that or is it on bare metal and serverless or—you know, it’s a great time to be a developer, but it’s a very hard time to be a security person who’s trying to keep guardrails there for these people to do it as safely as possible, as securely as possible. How do you do that?
Koren: This is correct. You’re spot on. You’d expect that when a new code is introduced, the company or the developers would adopt it and manage it or maintain on their servers or on their microservers or their mesh. However, because they want to keep it updated, because they want to keep it as capable as possible, they sometimes take the dynamic library from their repository and, whenever there’s a CI/CD process going on, it pulls the library from there.
And that is where the shadow code becomes a problem. Because, unless you have the right processes around, how to make sure that this library is safe? How to make sure that this is, you’re not introducing anything malicious—and we have seen it happening in the past—this is where you have to address the problem. And you know, I can share some of the things that we’ve seen when we had the validation of the product before we launched it. We had a design partner, and they asked for something like this because their developers asked for the security team to allow them visibility into what’s happening on the client side. And we’ve actually—and it’s a big customer. I don’t want to name names, naturally, here, but—and you will share my surprise—when we found out, when we run on their client side that a very, very big vendor of there is using a jQuery from a non-trustworthy source.
Now, you’d expect something like this would never happen for a well-established company, but it does. And it’s surprising, but it does. And they weren’t the only ones.
Shimel: Hmm. Interesting. So, let’s now talk—so, I think we’ve defined the issue and the problem, right? How is PerimeterX helping this?
Koren: So, our approach when looking at this issue or this problem, naturally, we are a Dev first company. So, we came and asked the developers that we were working with—the DevOps, DevSecOps, and some of the risk analysts, some of the personas—what would be most beneficial. And what they said, and this is how, you know, I’m the VP of Products. I’m a Product Manager at heart. And customer input is the number one thing that I’m looking into when I’m building the products.
They said very clearly, if they can see, before moving from staging to dev to prod, what the code itself, when it’s pulled dynamically, is doing and have the visibility of what’s happening there in real-time on users—this is something that they don’t have today. Because that will introduce that level of visibility whenever something bad happens.
I don’t know if you’ve seen some of the Magecart attacks that have happened from third-party libraries. Again, don’t wanna name names, but if somebody looks online, they can find out. When we come in and we analyze in real-time on the client side, on every user going into the website, what the scripts are doing there, whether they send information to a bad domain, whether they try to take the information and store it in a cookie and then use it later on—we can quickly highlight the main risks and main problems and vulnerabilities, like I mentioned earlier with jQuery, so that the DevOps or DevSecOps can quickly raise a flag and say to the developer, “Listen, this script you’ve added introduces new vulnerability.” You can’t see that in the static code analysis, but you will see that on run time on the client side. And that is something revolutionary. We don’t have that today in anyone else’s solution out there.
Shimel: Hmm. Interesting. What—so, I always ask, and you’re the perfect person, VP of Product—now you’ve got a conundrum. You’ve got a really great security product here that’s gonna help developers with secure code. How do you—how do you sell it? How do you bring it into the organization? Do you talk to the security guy and say, “Hey, go talk to your developer team and tell them to use this?” Do you talk to your developers and say, “Tell your security team you wanna do this?” Do you gotta talk to the security team and the developer team? Whose budget does this come out of? Who ultimately is responsible for setting the policies and processes?
Koren: And this is—
Shimel: Is the solution hard to sell, maybe, though?
Koren: You’re correct. So, theoretically, it’s hard to sell. However, the main key here or the key differentiator is the fact that when we went in to look how we wanna sell this solution, before we even built it, we established the fact that it should fit into their processes. If it’s not fitting into their processes, if they have to start defining things like content security policy, if they have to start maintaining this on a day to day basis, it’s not going to work.
Solutions for DevOps or DevSecOps today are one, two, three. You have to have it in and you have to do it quickly and you have to have minimum manual effort, unless you definitely prove why it’s needed. And so, this is why we’ve added the fact that nobody needs to do the baselining on their own, we are doing that automatically with machine learning capabilities, what every script does, whether it’s ________ or not, and the fact that they can just have a small snippet on their client side and we’ll analyze whatever and flag in our portal or alerting system that they have, it actually fits into their CI/CD process, it fits into their day-to-day process. They can even have it as part of their regular security processes and have it fit into their SIM.
And this is where it really makes the difference, because you don’t have to do anything manually and you solve a problem and you fit into their processes, and security people are very happy about it. I can tell you that one of the customers that actually already bought this product was very happy with the fact that they have visibility to this area and they don’t have to maintain it on their own, because we are doing that, the product does that out of the box. So, this was a key component in our product.
Shimel: Huh. Excellent. Excellent, excellent. So—I told you, the time goes very fast. [Laughter]
Koren: [Laughter]
Shimel: We’re coming up on time, here. For people who are interested—let’s use our time wisely—how could they find out more information? What path do you suggest for them?
Koren: So, first of all, they can go out to our website and check out our solutions, both for bot mitigation as well as the Code Defender and other solutions that we have. I think reaching out to our team if you have any questions, if you have any concerns around the client side, I can tell you that even companies that thought everything’s fine were completely oblivious to threats and concerns around the client side that are out there and many people are not aware of. You can also reach out to me directly via LinkedIn or what have you and we can discuss.
I think in general, for DevOps and DevSecOps to be more aware of the dynamic nature of the code that is added to the client side and what to do. We actually have a blog post listing the things that are best to do if you want to make sure that it’s secure and you’re on top of things. So, it’s a good read as well. And that’s about it. So, we are here. We’re ready to help.
Shimel: Cool. Last question, just because you have to ask it because it’s out there—with everything going on in the world with the COVID and everything else, how is it affecting you guys and what are you seeing?
Koren: It’s actually amazing. We’ve seen some trends around the e-commerce landscape that we’ve actually had some blog posts on. And I think, you know, one of the things that kind of symbolizes these times, a few of the e-commerce sites that we are working with said that the traffic they experienced outgrows the Black Friday/Cyber Monday level of traffic.
I think it symbolizes kind of the boom of online life that we have. I’m sure you’re experiencing it as well. And I think the shift there is not going to go back to the previous levels. It’s definitely going to go down, but we are seeing so many trends, very interesting trends, and we are naturally adjusting what we are doing, but luckily and happily, we’re doing great these times and it’s—we’re very happy with what we have now.
Shimel: Well, I will tell you, for whatever reason, you know, and it may not all be good, because there is this whole, with the virus pandemic there’s a cyber pandemic of people trying to exploit. I mean, so, for instance, even here MediaOps where DevOps and Security Boulevard, our security site, used to be pretty even in terms of how many visitors—they both get 350 to 400,000 visitors, unique visitors.
Koren: Mm-hmm.
Shimel: All of a sudden, you know, Security Boulevard now this month is gonna be at 600,000.
Koren: Wow.
Shimel: Last month was 530,000. So, in the last two months since this COVID thing, you know, we’ve seen traffic on the Boulevard site—DevOps has stayed relatively steady. It hasn’t gone up, necessarily, it hasn’t gone down.
Koren: You know, one thing we’ve noticed, and I think it can also explain, I think it’s wider than that—attacks and in general fraud and security challenges became much more complicated, because the adversaries are out there and they are targeting a whole lot of other new sites that previously didn’t experience. We are seeing much higher levels of complexity in the attacks that we’ve previously seen on giants, now on even medium small—small- to medium-sized sites.
Shimel: Agreed. Crazy. Just crazy, right?
Koren: Yeah, indeed.
Shimel: You know, because that’s what you need when you have a virus pandemic, we need to have more cybercrime and that stuff to worry about.
Koren: Yeah. [Laughter]
Shimel: So, Elad, I want to thank you for joining us today on DevOps Chats. It’s been a pleasure.
Koren: Thanks, Alan.
Shimel: Be back on soon. Elad Koren from PerimeterX here on DevOps Chats, this is Alan Shimel and you’ve just listened to another DevOps Chats.
