Tag: devsecops

  • Infrastructure Access Management a Growing DevOps Priority

    Infrastructure Access Management a Growing DevOps Priority

    Organizations plan to address access management over the next 12 months, as the need to secure and streamline infrastructure-wide access controls serves as a prerequisite to other initiatives, like zero-trust.

    This was one of the key findings of a survey of 600 DevOps professionals conducted by Pollfish and sponsored by strongDM. The report also found that legacy access processes created severe team inefficiencies.

    These inefficiencies require intensive time and resources to fix and block agile development practices: Nearly nine in 10 organizations surveyed said they required two or more employees to review and approve access requests and that those could take days or weeks to fulfill.

    The survey also indicated that organizations continue to use access management practices that are not secure and that make it difficult to track and audit users and permissions of critical business systems.

    Tim Prendergast, CEO of strongDM, said as more jobs become technical, there’s a bigger need to deliver access to more people–and that can have a severe impact on a company’s ability to remain secure. He explained that when 65% of organizations are reporting their teams used shared logins—and over 40% used shared SSH keys—there’s virtually no way for you to know who is in your infrastructure or the havoc they may be wreaking.

    “This makes it difficult to pinpoint any leakage or loss because you have 20 copies of your house key floating around,” he said. “It’s an example of the trade-off most organizations make when it comes to speed and ease of access versus ensuring that access is secure.”

    Survey respondents said their biggest challenges were the time required to request and grant access (52%) and the task of assigning, rotating and tracking credentials (51%).

    Hurry Up and Wait

    “Using current approaches to access means you’re hiring these high-paid, technical resources and telling them to hurry up and wait,” Prendergast said.

    Nearly half (47%) of respondents said they struggled with onboarding employees and contractors and Prendergast pointed out that one in four organizations said simply getting approval for access required a process that involved four people.

    “Think about that—in 25% of organizations, you have technical resources basically twiddling their thumbs while they wait to get access to this database or to that Kubernetes cluster,” he said. “Now multiply that by however many databases, servers, employees and third-party vendors that you have. And that’s not even counting when new technologies like Kubernetes are added to your infrastructure. Eventually, even just the frustration of your team as they wait for access becomes a liability.”

    2022: A Year of Convergence

    Prendergast predicted 2022 will see DevOps and security converge beyond what we’ve already seen with DevSecOps, where it has been heavily focused on shifting left and bringing security into the development cycle earlier.

    “This convergence will be marked by new workflows, technologies and solutions that not just improve security, but that also improve the development cycle,” he said. “One great example is optimizing infrastructure access—when done right, you can improve your security posture with zero-trust methodologies while also making it easier for DevOps teams to access systems quickly and easily.”

    He added that two of the biggest workforce dynamics facing zero-trust are remote work and the Great Resignation.

    “You used to have this environment where you’d have to be physically present or on the VPN to have access—remote work broke that,” he explained. “And now you also have this large number of employees leaving their jobs. Do you know what systems they had access to? How do you know if all of that access has been turned off? What happens if they were using shared credentials?”

    Prendergast said that’s why addressing access is critical to meeting this challenge and getting to modern security—if you don’t know who has access to what or what they can do in each system, you can never get to zero-trust.

    “Organizations need to find a way to understand the relationship between each technologist and each technology and then be able to track and audit those relationships,” he said. “Until you do that, you’ll have a really hard time getting to zero-trust. These are the table stakes for modern security.”

  • EEEWW—Green iMessage Bubbles | ‘Great Resignation’ Redux | Faster Firefox 96

    EEEWW—Green iMessage Bubbles | ‘Great Resignation’ Redux | Faster Firefox 96

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: iMessage lock-in lessons, attrition-rate fixes, and more on Mozilla.

    1. ‘Never Date a Green Texter’

    First up this week: The weekend story getting the technorati chattering was about how Apple iMessage denizens pour scorn on Android users—identifying them by their “icky” green chat bubbles. An iPhone user’s message gets a “proper” blue background, y’see.

    Amusing though the story was, the writer seemed to be hell-bent on proving some sort of Apple C-suite-led “conspiracy to lock in.” The reality is far more mundane—but it teaches important lessons for DevOps.

    Analysis: Unintended consequences FTW

    Could your DevOps team make a trivial engineering decision that pays unexpected dividends? While lock-in might be distasteful, seemingly irrelevant design or technical choices can have huge ramifications with users.

    Tim Higgins: Why Apple’s iMessage Is Winning

    [The] pressure to be a part of the blue text group is the product of decisions by Apple executives starting years ago that have … helped to cement the iPhone’s dominance among young smartphone users. [It] encourages people to pay the premium … and remain loyal to its brand.
    …
    Teens and college students said they dread the ostracism that comes with a green text. The social pressure is palpable. … Jeremy Cangiano, who just finished up his MBA, … tried to cash in on it last year by selling his own merchandise that touted, “Never Date a Green Texter.”
    …
    The blue iMessage bubble was born out of a simple engineering need, according to Justin Santamaria, a former Apple engineer who worked on the original feature. … The idea that it would keep users locked in to using Apple devices wasn’t even part of the conversation at the time, he said.


    And then the other shoe dropped—via Ron Amadeo:

    Google took to Twitter this weekend to complain that iMessage is just too darn influential with today’s kids. … The company is hoping this public-shaming campaign will get Apple to change its mind. [But it] probably has the least credibility of any tech company [because it] has released 13 half-hearted messaging products since iMessage launched in 2011.
    …
    Google … seems to have learned nothing from its years of messaging failure. Today … the situation is an incredible mess, and no single Google product is as good as Hangouts was in 2015.


    Ouch. With a sociological perspective, here’s Christina “filmgirlcw” Warren:

    iMessage is incredibly popular with certain demographics. … It’s an in-group signifier—the way teens have always had in-group signifiers (when I was in high school, it was AIM, a Nokia, and wearing Abercrombie). … It’s always fascinating for me to go to other countries and see a ton of iPhone users not using iMessage but something else because that app has become the in-group signifier. … Every teen I know uses Discord more than they use almost anything else.
    …
    It isn’t about lock-in or regulation or technology. It’s about fashion.


    2. How’s your Attrition Rate?

    A study published this week says DevOps orgs and other internet companies rank third for attrition in the 38 industries surveyed. And if you lump that category together with “enterprise software,” it would be in clear first place.

    Analysis: We’re #1! Oh, wait …

    But some companies get this right—how can you be one of them? Culture, innovation, recognition and remote work. It’s definitely not only about pay.

    Donald Sull, Charles Sull and Ben Zweig report their fascinating study: Toxic Culture Is Driving the Great Resignation

    Between April and September 2021, more than 24 million American employees left their jobs. [But] while resignation rates are high on average, they are not uniform … from less than 2% to more than 30%. … More-innovative companies … are experiencing higher attrition rates than their more staid competitors.
    …
    Pay has only a moderate impact on employee turnover. … A toxic corporate culture is by far the strongest predictor of … attrition and is 10 times more important than compensation. … Job insecurity and reorganizations are important predictors of how employees rate a company’s overall culture. … But it is surprising that … the more positively employees talked about innovation at their company, the more likely they were to quit. … Employees are more likely to leave companies that fail to distinguish between high performers and laggards [or] that tolerate underperformance.
    …
    Our analysis identified … actions that managers can take … to reduce attrition. … Lateral career opportunities are 12 times more predictive of employee retention than promotions [and] 2.5 times more powerful [than] compensation. … Company-organized social events, including happy hours, team-building excursions [and] potluck dinners … are a key element of a healthy corporate culture. … Unsurprisingly, when employees discussed remote work options in more positive terms, they were less likely to quit.


    And the flipside of resignation? Hiring replacements. Let’s turn to the equilibrium of Nash Reilly:

    It’s only gotten harder in recent months [and] I think I know why. … You need to offer engineers some combination of these three things to pique their interest: Cool stuff to work on, smart people to work with [and] some degree of repeatability.
    …
    You can’t really prove to an engineer that you can give them things like good management, minimal churn, and decision making authority [so] let them talk to your employees. … Also, figure out a way to let your team “backchannel” to the candidate. [If] your product is not that cool … play up the things you can offer them: smart coworkers, and a stable working environment. … You can afford to be honest about what it is that you do offer to a candidate, [so] pitch them the truth: that you want to do good work, sustainably, for as long as you can—ideally with them on your team.


    Perhaps the smart money is already skating to where the puck will be? Here’s echelon:

    Engineering salaries are … one of the biggest expenses. … Companies love that they can now hire remotely and pay adjusted cost of living. They’ll hire as many workers in middle America as they can, and once enough of the workforce is concentrated outside of expensive cities, you’ll see them stop hiring in SF and NY entirely. [And] startup capital is already beginning to shift to … emerging markets. Employment will eventually head there too.
    …
    Make your money now while the iron is still hot.


    3. Say Something Nice About Firefox

    After accusing Mozilla of jumping the shark last week, perhaps I owe the Firefox folks a positive inclusion this time? The release notes for Firefox 96 certainly sound good.

    Analysis: Improved sound; better perf; stronger anti-CSRF

    Outside the Apple walled garden, Firefox is about the only non-Chromium browser. A browser monoculture can’t be a good thing for anyone—not DevOps nor users.

    Michael Gariffo: Firefox 96 update focuses on noise improvements, main thread efficiency

    On the desktop side of things, Mozilla’s latest release made “significant improvements” to the browser’s built-in [sound] noise suppression and auto-gain-control features. These improvements are aimed at web-based video chat users.
    …
    The second tentpole feature in version 96 focuses on reducing main-thread load, which should help the browser function better on older, slower, or congested systems. [And] a new enforcement setting for cookies … will, by default, apply the “Same-Site=lax” policy. Mozilla claims this will help better defend users against Cross-Site Request Forgery (CSRF) attacks.
    …
    Mozilla also acknowledged that it is working on correcting the performance of detached videos being played in full screen on macOS devices. … No timeframe was mentioned for when it might be available again.


    Okay. Sounds good. But not to Tetch:

    Never mind all that fancy stuff. … Any reduction in crashes?
    …
    FF crashes for me several times a month. … I always take the trouble to fill in the “What were you doing immediately before the crash?” section of the crash submission dialog, and never hear another thing.


    But more efficiency on the main thread? Tell me more. ksec:

    Such an important detail that deserves at least few more sentences of marketing message or explanation. But the whole release note just feels like they couldn’t be bothered anymore.


    The Moral of the Story: A weasel wishing Happy New Year to a chicken harbors no good intention.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Moritz Knöringer (via Unsplash)

  • Secure Software Summit: Exploring Secure Coding Best Practices

    Secure Software Summit: Exploring Secure Coding Best Practices

    In an era where software is dominating the world, the security and quality of code must remain a high priority. Delivering secure and reliable software at a rapid pace is crucial for most organizations today, but it’s not an easy task. What security best practices do organizations put in place to successfully produce secure software? How should DevOps teams incorporate updated security practices and tools throughout the software development lifecycle? At Secure Software Summit, taking place Jan. 27 at 9 a.m. PT, the world’s leading cybersecurity experts will come together to share and teach the latest methods and breakthroughs on secure coding and deployment practices.

    The virtual event features thought-provoking discussions, networking opportunities and educational sessions focused on:

    • Open source supply chain security
    • DevSecOps
    • Shifting security left
    • Security Chaos Engineering (SCE)
    • Security and compliance
    • Security metrics
    • Zero trust

    Secure Software features an outstanding lineup of innovators, practitioners and academics of secure software development, including:

    • Aaron Rinehart, co-founder and CTO at Verica
    • Dan Lorenc, founder and CEO at Chainguard, Inc.
    • Shinesa Cambric, principal program manager, Identity Signals and Protection Team at Microsoft
    • Vickie Li, developer evangelist at ShiftLeft
    • Fabian Yamaguchi, chief scientist at ShiftLeft
    • Jonathan Schneider, co-founder and CEO at Moderne, Inc.
    • Abhishek Arya, principal engineer and manager, Google Open Source Security Team at Google
    • Steve Springett, senior manager of product security at ServiceNow and chair of the OWASP CycloneDX Core Working Group
    • Jasmine Jackson, senior application security engineer at Disney
    • Suchakra Sharma, staff scientist at ShiftLeft
    • Harini Rangarajan, engineering manager, product security at Twilio
    • Mark Wireman, senior manager at Accenture
    • Shannon Lietz, vice president of Vulnerability Labs at Adobe
    • Vandana Verma, security relations leader at Snyk
    • Yash, head of security at Sendbird
    • Chetan Conikee, CTO at ShiftLeft
    • James Chiappetta, SVP of cybersecurity at Blackstone

    Attendees will have access to valuable resources available for download and will also receive a link to access the virtual environment on-demand. To see the full agenda and to register, please visit the Secure Software Summit website.

  • DevSecOps Will Cross the Chasm in 2022

    DevSecOps Will Cross the Chasm in 2022

    We’ve been talking about DevSecOps and shift-left security for years. Although this approach probably didn’t “cross the chasm” in 2021, we did see some very telling milestones. Cybersecurity VC funding surged to record heights with a focus on DevOps and cloud security and the need for DevSecOps became glaringly evident with the Log4j vulnerability. On a more personal note, our own open source security project Checkov has surpassed three million downloads.

    Much of this momentum was inevitable. Those in infrastructure and reliability circles have already been adopting and touting the benefits of shifting left and automating as much as possible, so in many ways, it’s expected that security would follow suit. Undoubtedly, the pandemic has also accelerated the need for developers to work more autonomously without having security personnel and processes acting as barriers to productivity and velocity.

    We’re confident that 2022 will see the silos between development and security teams continue to crumble as developer-led security practices become the norm for cloud-native organizations.

    DevSecOps finally crossing the chasm doesn’t mean that every enterprise and traditional organization will shift security left and adopt security best practices. It will, however, give them competitive advantages over those who don’t—both in decreased security costs and increased developer productivity and, thus, time-to-market.

    What else is in store for the future of DevSecOps?

    Rise of the DevSecOps Job Title

    Fewer than 5,000 people on LinkedIn currently have “DevSecOps” in their job title, yet there are over 20,000 current openings for DevSecOps roles. In 2022, expect to see more of those positions filled. This means security teams across industries will conduct fewer manual security audits, there will be a considerable influx of homegrown DevSecOps tooling and point solutions will likely begin to consolidate into single platforms. 

    Blurred Lines Between Application and Infrastructure Security

    Until recently, application security was a very well-defined (albeit fractured) space focusing on securing the custom code and open source packages that make up applications. However, with the rapid adoption of cloud-native applications, the lines between application and infrastructure security are blurring. We expect to see this trend continue as more engineers take on more infrastructure-related projects, vendors start catering to use cases outside of their core competencies (through acquisitions and in-house development) and the role of DevSecOps continues to expand within organizations.

    Infrastructure-as-Code: The Great Cloud Migration’s Next Chapter

    We’ve been talking about the great migration to the cloud for years. At this point, many companies are ready to move on to the next chapter: Infrastructure-as-code (IaC). As a result of this trend, DevSecOps will become much more important, as security needs to be baked earlier in development phases or risk being left behind. In addition, security teams will need to become more well-versed in development technologies and practices to provide the proper guidance for the new way applications are built and deployed. 

    More Software Supply Chain Attacks

    Hackers have been targeting retailers and security vendors for years, exploiting a minor weakness to gain access and move laterally into sensitive data. In the past year, however, software supply chain attacks stepped into the spotlight due to multiple supply chain attacks. We expect this trend to continue, which will, in turn, put more focus on securing supply chains. DevSecOps, which has focused mostly on tools and practices for securing the code and infrastructure, will expand to include the supply chain mechanism. 

    If 2021 was the year of hype for DevSecOps, we believe (and hope) that these best practices are embraced en masse across industries. The benefits of shift-left security are well documented: The number of high severity incidents is significantly reduced, the potential attack surface is minimized, compliance efforts are simplified and the time to remediation is lowered. Organizations also save money by catching misconfigurations and vulnerabilities earlier in the software development life cycle while at the same time gaining time back with tools, both open source and commercial, that are empowering developers to move fast and build applications that are more secure and reliable.

  • Best of 2021 – Hiring When You’re Not FAANG

    Best of 2021 – Hiring When You’re Not FAANG

    As we close out 2021, we at staging-devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the twentieth in our series of the Best of 2021.

    Most of us are not FAANG, and we shouldn’t act like we are. Instead of using that, I’m going to use the term we used to use to refer to them: “Web Monsters”, which isn’t 100% accurate, but includes more than just the five FAANG members (Facebook, Amazon, Apple, Netflix and Google) for this article. At some point, we are going to need a new term. Web Monsters doesn’t clearly include Apple or Citi, but FAANG clearly excludes Citi and eBay. But this blog is about something else, so please accept my apologies for that lack of clarity. Assume that ‘Web Monster’ means “Big, popular high-tech company that pays well and looks good on a résumé.”

    There are several angles to “Not Web Monsters” that we can address, but I’m mostly going to focus on DevOps hiring, with a nod to irrelevant sales/marketing techniques. Hiring is an easy target because Web Monsters offer resume-building, pay and benefits that are super attractive in the industry, so they can (right now, at least) reject 99% of applicants and still have a good selection of candidates to choose from. Most organizations cannot compete in even one of those areas, though a few can when it comes to salary and/or benefits.

    We in IT have long had a bad habit of asking our applicants to be more qualified than the average employee. It’s pretty ridiculous at times; things like asking for a candidate to have more years of experience with a technology than the technology itself has actually existed. What happens when you ask for the impossible? Honest, qualified applicants self-select out, and only those who don’t realize the technology hasn’t been around that long and liars end up applying. That’s just bad hiring policy, even if you are a Web Monster. But at smaller organizations where each hire has a larger impact on the organization, it can be devastating.

    It seems worse in DevOps. In fact, all of the hiring items listed here seem worse in DevOps because DevOps is actually a nightmare mish-mash of skill sets. No two people define the job the same because “the job” covers everything from cradle to grave for an application. There is no part of app dev that “DevOps” couldn’t touch. That’s problem number one. Define the responsibilities of the role clearly, don’t ask for everything that it could be. As a rule, an Ansible specialist is not going to be a DevSecOps expert, and you shouldn’t expect them to be. I think, at a minimum, we need to split hiring along my traditional emphasis lines… DevOps and DevOps. If you are writing code and using Checkmarx while GitLab is building the results and you configured it to roll out to test … you’re not a DevOps engineer, even though there’s a bit of ops in there. Your strengths need to be in development and app configuration. Your job description needs to focus on that.

    Could there be roles that need it all? Yes, particularly in the SRE or chaos engineering space; I could see that. But they are certainly not entry-level roles, or even close. Knowledge in the breadth of technologies required to touch the entire DevOps toolchain is developed over years and with a healthy curiosity. And, of course, the person you are hiring should have a similar skill set to those doing the job today. Which is a good sanity check against the “Throw in the kitchen sink” syndrome.

    Next up is take-home tests and shared coding during interviews. We know that you have a vested interest in landing the best people, but if you are doing coding exercises with people who are primarily DevOps, you are wasting everyone’s time. Most DevOps jobs are done with PowerScript and/or bash with YAML files. Other languages used occasionally have no more place in a core job interview than other occasional bits. “It’s about how you think” is a common refrain. I would argue that an assignment that takes an applicant hours to complete (or in several cases I know of, days) is not an efficient way to evaluate the way a prospective employee thinks. And, in at least one case, I’m pretty certain the interview solved an actual problem that employees didn’t—or couldn’t—solve. Ask the candidate questions about how they’d solve problem X. Ask them convoluted questions. They don’t need to get to code level to show you how they think.

    And finally, my nod to bad marketing: Unless you actually are doing the volume of one of the Web Monsters, the fact that Facebook is a customer of vendor X is totally irrelevant. In fact, like the government, most vendors can mark one or more of the Web Monsters as their customers, just because Web Monsters are so big they try a lot of different tools across many different teams. So having them as a customer should be irrelevant, but if it isn’t considered irrelevant, should be expected.

    The team you have today is rocking it. Hire more people like them. You know what works and what you need. Get back to customizing hiring processes to deliver more of what you need and stop over-asking. There are more great people out there looking for roles; don’t scare them off by asking for irrelevant experience. Stick to the core. What are they going to do daily? Require that, then probe for other useful items at interview time. And get more good peeps in the seats!

  • Best of 2021 – From Agile to DevOps to DevSecOps: The Next Evolution

    Best of 2021 – From Agile to DevOps to DevSecOps: The Next Evolution

    As we close out 2021, we at staging-devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the ninth in our series of the Best of 2021.

    As the workforce shifts to remote locations, networks are becoming more diffuse as the edge gets farther away. At the same time, the number of devices and applications on those networks continues to rise. Some estimates suggest that, by 2030, there will be 15 connected devices for every person on the planet.

    The rapid expansion of the remote workforce and their associated application needs has created a significant challenge for both application developers and the IT professionals tasked with deployment and maintenance of applications. Finding the right balance between speed of release and stability and reliability of applications – never an easy task – is paramount for the success of any organization.

    Agile and DevOps evolved to help organizations strike this balance, providing the best possible customer experience in the shortest realistic time frame. But have Agile and DevOps fulfilled that promise?

    The Promise of the Agile Manifesto

    While Agile as a methodology has been known and followed for nearly two decades, the ways in which organizations have applied Agile have been as varied as the organizations themselves.

    The Agile manifesto set out four basic values when it was first written in 2001:

    • Individuals and interactions take precedence over processes and tools.
    • Working software takes precedence over comprehensive documentation.
    • Customer collaboration takes precedence over contract negotiation.
    • Response to change takes precedence over mindlessly following plans.

    Agile processes are applied across a range of development activities, from development of standalone applications to development of cloud-native business applications. The Agile Manifesto also contains several more principles that further explain what the values mean when put in practice.

    These Agile principles can be condensed into a few areas of focus:

    Prioritize the customer over the product

    Agile development requires continuous customer engagement throughout the entire development process, with the end goal being frequent product delivery based on specific customer feedback. While this can present challenges for an Agile team, particularly when customers demand changes late in the development cycle, Agile maintains that the best way to elevate customer experience is to make the customer part of the development process.

    Trust Agile Teams to Self-Organize and Self-Manage

    Agile focuses on smaller teams that remain in constant communication with customers, as well as with each other. These teams should include both developers and those with knowledge of the larger business requirements being met through development. For these teams to work effectively, trust is key. Agile posits that self-organized teams, given leeway to pursue their projects in the way they see fit, result in the shortest development cycles.

    Keep it Simple

    Agile teams require simplicity. This correlates with Agile’s disdain for overuse of process. Agile seeks to minimize the total amount of work required, thus minimizing overall development time.

    Agility Leads to Agility

    Just as Agile teams continuously improve their product and the customer experience, they also constantly improve themselves. Constant evolution of teams and the processes they employ (remember, teams before processes) improves team agility itself.

    While the Agile values and principles seem simple, effective application of the values and principles is another story entirely. Indeed, at times, the implementation of Agile development has violated the very first value of the manifesto by focusing on which tools and processes should be used. But, in an environment where more users are clamoring for more products and services, implementation of Agile methodologies is not just important, it is imperative.

    Building on Agile with DevOps

    The precise meaning of the term itself and the scope of DevOps remain a source of debate. At a high level, DevOps is the marriage of the development cycle with post-development activities, such as deployment and maintenance.

    According to open source contributor Barbara Ericson of Cloud Defense, “Instead of IT operations and software development being siloed off from each other, DevOps breaks down the traditional boundaries that previously existed between them in order to achieve continuous integration and continuous delivery (CI/CD) of quality software features and applications to end users.”

    Just as with Agile, the effective implementation of DevOps requires cultivating a DevOps culture within the organization. However, Agile and DevOps are also not mutually exclusive. That is to say, DevOps is not merely a replacement for or an evolution of Agile. Although Agile and DevOps differ in their structures and approaches, they can work in unison to achieve the ultimate goals of speedy deployment and enhanced customer experience.

    While Agile and DevOps share common goals, they have not always agreed on how to achieve those goals. DevOps differs in many respects from Agile, but, at its best, DevOps applies Agile methodologies, along with lean manufacturing principles, to speed up software deployment.

    One area of particular tension between Agile and DevOps is that the latter relies heavily on tools; in particular, when it comes to the automation of testing and deployment processes. But DevOps can overcome the resistance of Agile developers to tool usage simply by applying Agile principles themselves.

    Effectively, DevOps proponents must convince Agile teams that dogmatic adherence to the underlying principles of Agile is actually inconsistent with Agile in the first place. Ironically, Agile developers who insist that the process is always bad actually violate Agile principles by refusing to acknowledge the benefits offered through change, another Agile principle.

    The challenge is to have the Agile development teams trust in the automation efforts of DevOps, while at the same time encouraging the DevOps team to consider the business goals of deployment rather than pursuing speed of deployment for its own sake.

    With constant communication between the Agile team and DevOps team (another Agile principle), development can achieve a degree of comfort with DevOps tasks and processes. This means that testing and deployment automation can proceed quickly, resulting in a decreased time to market, overall.

    Ultimately, Agile and DevOps can work together most effectively when they recall their shared goals: smaller change increments based on feedback resulting in faster delivery cycles.

    Continued Evolution: DevSecOps

    As networks expand and businesses continue to integrate cloud-based services (in many cases, building multi-cloud environments), security issues become more and more complex. One area in which Agile is lacking is in integrating security into the development process. Unfortunately, in an Agile environment, application security is often an issue addressed after development, rather than as part of the process. Enter the next iteration of DevOps: DevSecOps.

    DevSecOps revolves around three basic principles: ensuring data security while minimizing inconvenience for users in accessing data, using development tools that identify risks as early as possible in the development process and ensuring data encryption using technologies such as secure sockets layer (SSL) and virtual private networks (VPNs).

    While DevSecOps and Agile may also disagree about the priority of tools, security must become an integral part of any development project. This is why Agile, DevOps and DevSecOps must all work together to ensure not only rapid deployment but secure deployment.

    The Next Evolution

    Agile and DevOps were both important evolutions of the software development process, but DevSecOps is the next evolutionary step. Despite a few disparate characteristics, Agile, DevOps and DevSecOps can and should work in unison to achieve their shared goal: the best possible customer experience and short deployment cycles.

  • Only 30% of Orgs Fully Implement DevSecOps

    Only 30% of Orgs Fully Implement DevSecOps

    With the pressure to release more rapidly, security is shifting left within the continuous development pipeline at most organizations. This imperative is increasing with the rise of cyberattacks. Yet both a lack of training and poor visibility are stalling many DevSecOps rollouts, a recent CSA study found.

    DevSecOps is still a relatively new practice and has yet to reach maturity throughout most organizations. Though nearly all companies are on their way, only 30% of security professionals say they fully implement DevSecOps in practice today. That leaves most organizations in some state of DevSecOps ideation and planning.

    CSA recently released its Secure DevOps and Misconfigurations 2021 report which was commissioned by Trend Micro. The report found that misconfigurations commonly arise from default security settings. A lack of internal guidance and accessibility of security resources commonly hinder DevSecOps maturity, too.

    Below, I’ll review the other major takeaways from the report to see how they can inform organizations as they move to implement DevSecOps.

    State of DevSecOps Adoption

    DevSecOps is all about increasing the security around rapid deployment. It literally puts the security into DevOps by shifting security automation left and de-siloing the boundary between DevOps and IT security teams.

    While most engineers realize the benefits of DevSecOps, the state of each organization’s DevSecOps journey is very different. While an impressive 90% of organizations are in some phase of the journey toward DevSecOps, only 30% are implementing DevSecOps while 24% are in the planning phase, 18% are designing and 18% are still refining their DevSecOps strategy. Just over one in ten security professionals say their team has no plans to invest in DevSecOps at all.

    Looking to the future, 42% say they will implement full DevSecOps within the next 12 months. But due to its current nascent state, misconfigurations are still getting through. The primary reason cited for these misconfigurations was flawed or lacking internal guidance (33%).

    Many professionals say there is simply not enough training, support or internal knowledge around vulnerabilities and misconfigurations. Other top reasons for misconfigurations include insecure default settings (18%) and negligence (16%).

    Outside of misconfigurations, groups also report challenges with identity, authorization and access. Ensuring the correct privileges is vital to avoid escalation attacks, which are a prevalent issue. Between 2019 and 2020, 80% of companies experienced a data breach of some kind, many of which were due to misconfigured access controls. OWASP also reported that broken authorization is a top vulnerability for high-traffic web API endpoints.

    Common Threat Mitigation Practices

    To mitigate these threats, development and security teams must maintain best practices, yet certain issues prevent DevSecOps from emerging. For one, 60% of security professionals say their top challenge is insufficient visibility into security or compliance gaps—this is by far the most common challenge—and 11% also cite inconsistent cloud account onboarding. Finally, 10% say time-consuming, slow and/or insufficient architecture holds them back.

    One way to mitigate threats is to introduce routine security reviews more frequently. Yet it’s hard to set a definite benchmark here, as the frequency at which organizations review their cloud infrastructure for vulnerabilities or misconfigurations varies widely. Currently, 22% perform such security reviews daily, 22% monthly, 18% weekly and 23% quarterly, according to the survey. This rate will likely increase as DevSecOps becomes more commonplace and automated within the development life cycle.

    Staying up-to-date with modern security frameworks is another facet impacting DevSecOps adoption; and organizations tend to follow multiple frameworks to inform their security strategy. More than three-quarters (78%) follow the National Institute of Standards and Technology (NIST) guidelines, 67% follow the Center for Internet Security (CIS) benchmarks, 66% follow Cloud Security Alliance (CSA), 54% follow International Organization for Standardization (ISO) and 44% say they follow security recommendations from Amazon Web Services (AWS).

    For example, NIST, the government-operated standards-setting body, recently set cybersecurity guidelines that recommended the use of zero-trust and service mesh across departments. These security guidelines can be thought of as architectural benchmarks, especially for large, highly-regulated industries.

    DevSecOps Training Types

    Finally, investing in community resources and training is another way to increase DevSecOps awareness. Only half of survey respondents reported their DevSecOps best practices resources were moderately accessible—thus, the onus is on leaders to democratize such knowledge within their organization.

    The majority (81%) of respondents cited online articles and training as their primary form of learning about cloud security tools and vendors. Workshops, conferences and webinars follow closely. Organizations also adopt many internal knowledge-sharing methods in response to incidents. A full 85% said they conduct awareness training followed by table-top exercises (52%), attack simulations (45%) and protocol or response framework training (37%).

    Future DevSecOps In The Cloud

    It appears that most teams are still formulating their DevSecOps strategy. But looking to the future, 42% say they will implement full DevSecOps within the next 12 months. Simultaneously, more cloud traction is projected throughout the industry.

    Right now, 40% of organizations have between 41% to 99% of their workloads in the public cloud. And 55% of organizations will have between 41% to 99% of their workloads in the public cloud in the coming year. The type of workloads will also likely evolve in the coming year, as more organizations move to container platforms, Function-as-a-service and other serverless capabilities. To fully reap the benefits these new technologies promise, organizations must undoubtedly respond to a new class of cloud-native vulnerabilities.

    About The Report

    The Cloud Security Alliance (CSA) is a not-for-profit organization with a mission to widely promote best practices for ensuring cybersecurity in cloud computing and IT technologies. The Secure DevOps and Misconfigurations survey was conducted from July 2021 through September 2021 and collected over 900 responses from a global pool of IT security professionals working in various sectors and organizational sizes. For a full copy of the report, you can swap some personal information for a PDF download here.

  • Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users 

    Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users 

    As we close out 2021, we at staging-devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the third in our series of the Best of 2021.

    Linux does, occasionally, raise security concerns. While many users see it as the most secure, robust and versatile operating system available — that’s this writer’s opinion, as well — security precautions still have to be taken.

    A recent, widely publicized case illustrated this point; Linux creator himself, Linus Torvalds, warned against the use of the Linux 5.12 release. He described a “nasty bug,” and wrote that the situation is a “mess,” due to the use of swap files when adding Linux updates. This nasty bug, in fact, had the potential to destroy entire root directories.

    Some of the main takeaways following this “mess” include: tread very carefully when installing early Linux releases, especially those that involve swapping files instead of partitions, and especially, despite Linux’s well-known security advantages, avoid becoming complacent, because Linux security is not always foolproof.

    Hence, while the “state of Linux security today is quite good, and has evolved in a positive way with more visibility and security features built, like many operating systems, you must install, configure and manage it with security in mind; that is how cybercriminals take advantage, [via] the human touch,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic, a provider of privileged access management (PAM) solutions.

    A Patch for Nastiness

    As Torvalds noted a few weeks ago, “most people don’t use a swap file, but a separate swap partition and the bug in question really only happens when you have a regular file system, and put a file on as a swap.”

    “The bad news is that the reason we support swap files in the first place is that they do end up having some flexibility advantages, and so some people do use them for that reason. If so, do not use [release candidate] RC1,” Torvalds wrote. “Thus, the renaming of the tag.”

    After issuing the warning, Torvalds released a patch that he says prevents the bug from destroying swap file systems. However, it may have already been too late for early adopters of release 5.12. Ubuntu, a leading Linux distro, can swap files by default.

    “It is nasty bug if you are still using swap files,” Carson said. “If you do still use swap files, then you could be impacted, resulting in potential data loss or a corrupted system.”

    DevOps teams – or anyone else running Linux and installing patches, whether on multi-servers or on individual workstations – still need, of course, to follow strict best practices. “Like any operating system, security depends entirely on how you use, configure or manage the operating system,” Carson said. “Each new Linux update tries to improve security; however, to get the value, you must enable and configure it correctly.”

    Linux Goodness

    The fact that Torvalds was so forthcoming about the bug, as well as the level of transparency that the Linux kernel offers, also demonstrates one of the many reasons Linux remains popular. Given that the Linux kernel, in one variety or another, is used “not only in about 50% of the internet servers of the world, but also in a substantial part of all our smartphones, it is good to see this level of transparency at ‘root level,” said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), which providers cybersecurity and compliance software.

    “The security of Linux is based on its transparency; the ability to review the code of a distribution,” says Schrader. “Quite often forgotten is that transparency also involves talking about the mistakes, the errors, those nasty bugs.”

    Citing National Institute of Standards and Technology (NIST) vulnerability database statistics, Schrader described how, compared to the Windows family of desktop and server operating systems, for example, the Linux kernel shows better results for overall vulnerabilities. The number of vulnerabilities have also declined over the past four years, while Microsoft’s operating systems do not display the same trend, according to NIST’s national vulnerability database.

    Since Linux’s famous kernel is open source and transparent, it is possible to extrapolate that there are a greater number of potential vulnerability watchdogs compared to those monitoring vulnerabilities in closed systems. Some may argue that Microsoft has been, at times, less successful at detecting vulnerabilities and issuing much-needed patches.

    However, Linux users still must remain vigilant.

    “Still, for any of the Linux distributions, anyone using the early release candidates — RC1 in particular — should make sure that their own development or build process is undergoing change control, so that no mishaps will transfer the nasty bug into a production environment,”  said Schrader.

  • How Log4j Becomes a Serious DevOps Problem

    How Log4j Becomes a Serious DevOps Problem

    The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs. 

    Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

    Why DevOps Practitioners Should be Concerned

    The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

    In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

    However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

    Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

    Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

    Where Should Responsibility Lie?

    More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

    Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

    To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:

    • Creating and maintaining an SBOM 
    • Deploying automated tools that scan for discovered vulnerabilities
    • Automatically comparing the SBOM against vulnerabilities
    • Including DevSecOps processes within their CI/CD pipelines
    • Frequently testing applications for flaws
    • Coordinating with cybersecurity personnel when a flaw is discovered
    • Patching frequently

    With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.

     

     

  • Bridging the AppSec and DevOps Disconnect

    Bridging the AppSec and DevOps Disconnect

    Research estimates that cybercrime is going to cost the world $10.5 trillion annually by 2025, so it is no surprise that cybersecurity has become a top priority for business leaders. 

    Today, security teams are striving to harden their systems against cyberattacks and improve their resilience through more employee security training, incident response programs and the implementation of technology to monitor and detect threats. 

    Many organizations are also adopting a shift left security strategy, where security is being addressed earlier in the application development life cycle. These organizations are realizing that finding security issues in their software after it’s released is much riskier and more costly. After all, the longer a bug is in the wild, the longer an attacker has to find and exploit it. By implementing a shift left security strategy, the window of opportunity for an attacker decreases or can be completely removed if a security issue is addressed in the development stages, while also offering organizations significant security benefits and cost savings. 

    Shifting left is a security testing strategy where DevOps and security teams work in tandem to identify and address vulnerabilities during early development stages. This process significantly improves security as it means vulnerabilities are identified and mitigated before they are introduced to the enterprise environment.

    However, one of the biggest challenges organizations face when adopting a shift left security strategy (DevSecOps) is encouraging security teams and DevOps teams to work together.

    Encouraging AppSec and DevOps Collaboration

    Until recently, DevOps teams and AppSec teams were completely siloed. DevOps teams worked on accelerating the speed at which applications are released into the enterprise; application security teams only got involved in the later stages. However, this process has caused security concerns, particularly as more organizations move to agile development. 

    When security teams eventually got their hands on new applications and updates, security flaws were inevitably found that caused delays and friction with DevOps teams. 

    However, as organizations begin to embrace DevSecOps and shift left strategies, these two previously siloed teams are being forced to work together more closely. So, how can the friction be reduced and the disconnect bridged?

    Culturally, some ingrained attitudes and behaviors challenge the success of any DevSecOps efforts. Security teams have seen DevOps processes accelerate the speed at which software is delivered, but without security considerations, while DevOps teams experienced security slowing down processes and giving inconsistent results and feedback on security issues. Each party has their own manager to please; their own set of metrics that they’re measured against and a priority list as long as their arms already. Both teams follow different processes and, crucially, use different tools. DevOps can’t get around the security tool complexity and lack of integration with their existing toolset and security teams have no control over the CI pipeline to best implement security assurance.

    One of the best ways to overcome this friction is through better technology, process and culture that enables collaboration between teams. First, DevOps teams do care about security, but it might be lower on their priority list. Security teams must understand that DevOps teams care about code, quality and efficiency. They are not security experts and just because an organization is shifting left doesn’t mean they need to become one. Instead, organizations should implement tools for DevOps where security checks can be easily embedded.

    So, what can organizations do to improve the collaboration between AppSec and DevOps teams to achieve DevSecOps?

    DevSecOps is a culture and it requires a shift in thinking. Security teams and DevOps teams need to understand each other’s priorities and understand that by working together they are improving the overall security of the organization and decreasing the risk of cyberattacks. Does the DevOps team understand the potential risks a vulnerability could have on the organization overall? If not, security teams need to educate them.  

    Security automation tools offer great support when organizations are moving to DevSecOps, as they allow bugs and vulnerabilities to be flagged automatically as applications are being developed. This will allow them to be addressed before services go live; however, because they generate frequent alerts, this could add to DevOps’ already busy workloads. By deploying tools which can be synced with CI/CD pipelines and provide custom advisories so DevOps teams can address bugs themselves without having to ask AppSec teams, this can significantly speed up the process, reduce friction and ease workloads.