Author: Don Macvittie

  • DevSecOps Implementation: Patch Management

    DevSecOps Implementation: Patch Management

    Patch management has been a thorn in IT’s side for decades. While it is undeniably necessary for security and stability purposes, the staggering array of patches that are available make managing them effectively nearly a full-time job – or a job better left to users. Few IT professionals want to be the “patch person,” and yet, nearly all IT professionals want their systems protected by the latest patches. No one wants to find out an intrusion was easily preventable via patching. It goes without saying that the SolarWinds attack has left quite a few IT departments scrambling for an alternate solution.

    The world of patch management has gone through several evolutions that we won’t cover in much detail here. Instead, we’ll focus on what’s available, what you should look for and where the market is headed. Suffice it to say that patch management as a practice has grown from an individual applying patches to servers and server apps while employees were encouraged to patch their equipment, to complex systems that manage the patching themselves, with IT oversight. Currently, patch management is often bundled with endpoint management, simply because many of the patches are going to end points (user machines and devices).

    Out of this growth have come several market differentiators that you should be aware of when looking into a way to manage the avalanche of updates the average organization is seeing.

    • Centralized or de-centralized? Increasingly, patch management is run on servers/cloud and pushes changes out to those machines/devices that need them, or agents on each machine manage updating themselves. Which is best for a given organization really varies. If a lot of machines are running at the edge of their capability, an agent running updates in the background might be too much to ask. In that scenario, a server-managed install environment is probably better. Conversely, if a centralized location to run updates from is considered an architectural weakness (because it requires access to the update pushing engine), agents might be the better plan.
    • OS support. We’re way past the days when single-OS solutions were considered useful. List the operating systems used in your environment, and choose a product accordingly. Most products on the market support Windows/Mac/Linux, but there are variances. First off, make certain your chosen product supports all three. Then ask, “Which flavors of Linux?” Make certain the flavors running in-house are supported by the software.
    • App support. There are a lot of applications out there. Which are supported by the tool? How much leeway does the customer have for adding applications by pointing the tool at update locations? Does it support FOSS updates? There are a lot of those too, and which ones matter. Make certain the tools the organization uses are supported or can be added.
    • Many tools will scan machines for update issues. Understand what your chosen tools scan for, and how the scans impact performance on the target machines.
    • Updates are not 100% perfect. There will inevitably be problems. Make certain the tool can roll back changes. Updating 50 core infrastructure machines, only to find, hours or days later, that the update breaks something demands roll-back capability.
    • Inter-relationships. Updating a piece of software can require that another piece of software or a driver be updated, which can demand that UEFI/BIOS be updated. Or the CMOS on a RAID controller. Understand how cascading updates are handled by the tool, and if the cascading updates can be packaged, so that a failure automatically rolls back the updates applied thus far.
    • Approvals. Speaking of driver and BIOS updates, what tools are available to allow a human to approve changes? No one wants willy-nilly updates to the drivers on core systems – or even to drivers on user systems.
    • Security of the tool itself. We’re in perilous times where IT security is concerned, and frankly, an update command-and-control server is an attacker’s dream machine – take it over, put your back-door update into the mix, and tell it to update every machine it manages. It cannot be understated: an update management tool must be as protected as possible – both by the customer locking down the installation and by the developer making sure the code is solid.

    Once the system is chosen and worked into your DevSecOps environment, updates will be more responsive, which leaves one more thing to consider – does it feed into your agile tools? If the update management tool notifies a project management tool like JIRA when it updates a server, then if code on the server breaks, the developers aren’t in the dark about changes, and will have breadcrumbs to help them track down the source of the problem.

    Imagine if manual updates became a rarity instead of the norm. There is a cost associated with this process, but most of the products on the market can be had for a reasonable price – if you waste a couple hours a month (or for some products, a year) on updating, they’re paying for themselves. Of course, your time is a sunk cost, and the cost of these is an operational cost, so there is an evaluation your organization will have to undertake. Given how busy everyone in DevOps and DevSecOps is, it is likely worth the investment for all but over-staffed organizations. What? There are a few!

    Find what suits your organization, and put it to use. Faster updates, less time investment. The idea is sound, you just need to figure out what suits your needs. And keep rocking it. You are managing updates already, this is just a way to make your life easier. And we could all use a bit of easier in a busy, complex IT environment.

  • DevSecOps Implementation: It’s About the Data

    DevSecOps Implementation: It’s About the Data

    It seems we are constantly barraged with, “So-and-so announced that their systems had been breached and 80,000 user records were stolen,” notices. In fact, https://haveibeenpwned.com/ lists ten billion hacked accounts.

    That is no surprise. Databases exist to pool related data together. Even NoSQL type databases, which some saw as an opportunity for wildly disparate data to be gathered and analyzed, almost exclusively contain related data sets. We generally don’t want or need to correlate, say, the number of pirates in the world with annual global temperatures, for example.

    One would think, with decades of experience under our belts, we would have a good solution for protecting that data. If it was a simple problem, we would. It is not.

    Databases are the primary target of most attackers. After all, getting access to your user database gives them account names through which they can try and gain elevated privileges. Getting a large amount of user/credit card information gives them a revenue stream by selling them off (though it is now so common that prices on the black market have become depressed, and this isn’t really profitable, given the risk versus the reward, anymore).

    And databases must, by definition, have broad access. A database does not exist in a vacuum; it exists to serve an app. For server-based databases, it almost always exists to serve multiple apps. It must be on the corporate network (even if that network is a virtual private cloud), and it must … offer up data. It has users that have greater privileges than your average customer, and is notoriously difficult to protect with strong protections like encryption.

    Encryption poses problems at multiple levels. It is still a bit slow, for starters. This is the nature of the beast – if it takes X microseconds to respond to a request, encrypted data will take at a minimum X+decrypt time (regardless of where decryption occurs). If encryption is done by column, index fields cannot be encrypted (because “nZ19?SDF” is unlikely to be in the same sort order as “ford explorer”, for example) Which means some data is immediately excepted from encryption. If encryption is handled by the database system, then the time overhead exists, and the database system must send the resulting data over your network … unencrypted. I was involved in a couple of projects that required encrypted databases – the one I can vaguely talk about was putting information about your corporate network onto Android and Apple devices. Needless to say, since phones and tablets are misplaced quite often, and most have weaker security than your average laptop, we needed information about critical infrastructure to be encrypted. It was … painful. We only managed it successfully because corporate networks are finite resources, so the amount of data we needed to store was limited, and we could prove performance wasn’t going to be the issue.

    So what can you do? Continue to follow attempts to adequately encrypt corporate databases. Eventually, someone will crack this problem in a way that is performant, can be debugged and reliably protects the data.

    Until then, protect your databases. Use the tools we’ve talked about in this series to monitor and protect the entire network, knowing that the database is a high-profile target if someone gets in. And use the other security tools available. The major database vendors offer DBMS security tools to evaluate your current status. Some will flag weak passwords, others will monitor for injection attacks (as will development tools discussed in the first few installments of this series). Lock down interfaces to databases – by user, by IP, whatever makes the most secure, but still usable, environment for you. Set up monitors for behavior outside the bounds of normal behavior – a query of 250K records should raise a flag, for example, since the vast majority of apps will only do a subset that can be paginated. Sure, it will probably be a false positive caused by someone hand querying, but the one time it isn’t, you’ll be glad the monitor was there.

    And protect your database backups – be they actual backups or a dump to test. Fewer organizations are using real, unfiltered data for testing, but it crops up in most orgs at one point or another, as testers are trying to look at issues that fake/masked data can’t replicate. For actual backups, those must be encrypted. To lose your customers to a hacker because you had an unencrypted database dump lying around, and the attacker didn’t even have to attack the DB, is downright shameful, and happens too often. Along these same lines, and this should go without saying, but it’s happened enough that I feel compelled,  do not copy large chunks of your database out to an unsecured cloud storage bucket. Some vendors have addressed this issue by not allowing public unsecured buckets, but you can still do this, depending on the cloud you are using. So, don’t.

    The database security market will be best served by being rolled into SIEM and/or XDR. And while I’m in full-on pundit mode, for this statement, I’ll predict that is where it is headed. Database security is as much about overall security as it is the database, so overarching systems are the best place to detect and defend from. So watch for moves in that direction. A holistic view will serve you better in the long run.

    Meanwhile? Keep rocking it. Treat your data like gold, because to hackers, it really is. And, keep running the world’s business. Celebrate your successes as much – or more – than you learn from your failures.

  • DevSecOps Implementation: EDR/XDR

    DevSecOps Implementation: EDR/XDR

    We mentioned host intrusion detection and network intrusion detection in an earlier blog, and mentioned firewalls a couple of times in passing. Let’s delve a bit into the history to understand how these tools’ functionality has evolved over time.

    On the host side, we had virus detection, in one form or another, from early in the evolution of the Internet. The idea being that, if an antivirus program could scan your files, it could help protect you from malicious applications. Applied to an organization, if antivirus applications could scan all users’ desktops, it could help protect the organization. And it did kind of work. Most organizations were protected, at one point or another, by their antivirus software. Then, they realized the need to see more than what simply scanning files could show. A file that looked innocent on disk could assemble malicious code while executing, so runtime environments also needed to be watched. Generally speaking, the same antivirus companies led the charge into runtime protection. But there is more to protecting systems than simply looking at running applications. Scripts, even database entries, could be made malicious, and activity over time could indicate a bigger problem than a virus – an intrusion. This problem was the reason host intrusion detection was born. The idea that all of your system(s) needed protection all of the time created a mostly agent-based environment that monitored several aspects of the running system to watch for suspicious activity.

    On the network side, firewalls were implemented early in the growth of the Internet to allow limited access to corporate resources over the network. If no one ever needed to telnet to the machines on subnet X, the telnet port for subnet X could simply be blocked at the firewall, and no sleep need be lost over attempts to connect. Almost immediately after their inception, firewalls started logging attempts to access ports, and this information was made available to network administrators to review. This had an interesting impact on the management of hosts, because now, the firewall could inform network administrators if one of the internal hosts was constantly attempting to do something it shouldn’t be – possibly indicating an intrusion or a bad employee (more often than not, it turned out that undertrained, curious employees were just poking around). But firewalls – whether using whitelist or blacklist methodology – were fragile and/or interruptive. They also simply plugged holes in the perimeter without being proactive. That is when some bright people realized that, if you monitored streams running through the firewall, you could detect attacks and intrusions and offer admins proactive information about potential threats. That was the genesis of network intrusion detection (at least, as with all of these, as much of it as I can cram into a blog).

    But as time has moved on, there was a need for more. Network intrusion prevention was tried, but most organizations don’t want automated systems cutting off connections for fear of impacting real users. While it is still around, it hasn’t gained all that much traction. The next step was to “connect the dots,” so to speak. If the HIDS on host A was detecting a suspicious pattern of behavior, it only made sense to have the HIDS on the other hosts in the network take a look for similar behavior. Command and control of cross-HIDS installs is at the heart of endpoint detection and response (EDR), which gathers information from multiple HIDS agents and correlates it to get a more holistic view and alert on activity that a single host might not have noticed (a super simplified example: if the same user account tries to get super-user privileges with a single attempt on a dozen machines. HIDS might not raise a high-priority alert for a single attempt, but EDR sees twelve attempts on twelve different machines, not just the one on a given box, and can raise a high-priority alert).

    Extended detection and response (XDR) extends this information amalgamation and analysis to the network side, including attempts to gain access to the network (in our simple example above, imagine if the same user account tried to ssh into machines it doesn’t have network access to).

    As of this writing, EDR and XDR are using machine learning (ML) to perform faster processing of all the data they are seeing, and to limit false positives while raising critical alerts up to human operators. At this point in the ongoing feud between attackers and security personnel, I’ll go so far as to call it state-of-the-art. For now. Other attack vectors and XDR-defeating attacks will no doubt come along, and will have to be answered, but for now, if you have a publicly exposed network, it is the best you can get. Vendors have moved both EDR and XDR analysis to the cloud, ostensibly to improve analysis. It does reduce the amount of maintenance customers must perform, but cloud usage comes with an operational pricetag that prospects should understand before signing on the dotted line. Most mid-sized or larger organizations don’t have a storage or processing power shortage, so operational efficiency is what you are really paying for, in most cases. Make sure it’s worth the cost, and you have a plan if you lose access to that ML engine in the cloud.

    Meanwhile, keep kicking rear. Mark each day with, “At least we’re not SolarWinds,” and, remember: you’re doing your best to protect the org in a dynamic and hostile environment. Celebrate each day without a breach as a success – because someone did try.

  • DevSecOps Implementation: SIEM

    DevSecOps Implementation: SIEM

    The world is filled with events. Our inbox floods with events that marketers really want us to pay attention to, while news feeds flood us with events they’re trying to raise above the background noise, but then, the dog barking interrupts our consumption of that information. Our family is, meanwhile, texting us about events on the family level that may be related to events on the national or world stage … meanwhile, social media is full of garbage information about events that may or may not be real, and that you may or may not care about.

    We ignore the vast majority of these inputs and move along with our day. The inputs that require our attention usually, but not always, get it. Sometimes, we fail horribly at filtering, judging importance, or both, and it has life-impacting ramifications.

    This is not too far from the state of security event management when security information and event management (SIEM) was born. Disparate feeds were coming in from all over; analysts – be they dedicated security folks or systems administrators wearing an extra hat – had to guess which of dozens, hundreds or even thousands of events were a real threat to the environment.

    The first step to gaining control of that environment was standardization and aggregation. Get the events into one place, with similar information available for similar events. That was where SIEM came from.

    Since then, the number of events coming in has continued to rise. Companies that were seeing hundreds of events a day are now seeing hundreds an hour, and SIEM vendors had to keep up. The complexity of attacks, and the ability to detect them, has gone up, and SIEM vendors have had to keep up. Tools available for feeding the SIEM and for acting on the aggregated data have evolved … and SIEM vendors had to keep up.

    At this point in time, SIEM is best characterized as a data aggregation platform with limited intelligence that aids in filtering of irrelevant events. It’s akin to turning off social media and telling those close to you to call if needed; it filters out the worst noise and elevates the most important messages.

    Other tools can analyze their own data and make their own deductions, but SIEM is where you bundle all security-related events in one place, so it is the logical place to filter the noise and raise awareness. But it requires a high-volume, adaptable datastore. Vendors have reached that plateau.

    Once volume issues were mastered, SIEM vendors turned to helping filter the insane volume of security event noise. Yes, it is a security event if Bob logged in on a new workstation, but it probably isn’t a noteworthy security event. Unless Bob’s new workstation is in a country where the company doesn’t do business, or Bob is still logged in on his own workstation, blissfully unaware of the other login.

    So, rudimentary filters were applied to the event data in the SIEM. This got rid of a huge amount of noise. Next came filtering events that, logically, looked like notable security events, but just weren’t. Something was scanning ports on the firewall. An internal firewall. Sounds bad, unless the security tools’ logs showed that an authorized user kicked off the scan. Then, it’s a simple matter of asking the user in question, “Are you scanning X?” and moving along. It’d be better for the system to lower the importance of the event based on the fact that it is an authorized user doing the scanning, with corporate-approved tools. Then, an analyst may not need to get involved at all (my security friends are cringing because, literally, they trust no one, and would point out that authorized users can misuse tools, too – so I’ll note that here).

    But the event volume kept going up, and the analysts were even more swamped. Between adding new reporting apps (HIPS alone can add thousands of reporting points to a SIEM), and increases in events on existing monitored points, it was ugly.

    Enter AI. This is when machine learning (ML) started becoming the norm in SIEM. Now, we want the system to avoid using hard-and-fast rules that must be maintained, but to use knowledge it gains by watching the flow of data and an analyst’s resolution to selectively raise/lower the priority of events. Now, the events that actually bubble up to analysts are ones that the ML engine sees as fishy, and the number is greatly reduced. False positives still occur, but not nearly at the volume they did before, as the system learns how to detect them.

    But wait! There’s more!

    All that data in one place with ML engines crawling over it was just too much temptation for data scientists and deeply knowledgeable security personnel. So, the ability to plot a course through seemingly unconnected security events and determine that this or that was actually evidence of intruder or attacker activity spread across the system came next. This is where we are now – the ability to connect the dots on events that analysts (or the system) might have filtered out to find patterns that indicate intrusions.

    But it comes with a price. It’s not cheap to set up, it’s not easy to train, and maintenance of a system that takes inputs from everywhere and tries to correlate the resulting dataset is … work. We’re not talking a little bit of data here, and, like the environment it comes from, we’re not talking a little bit of complexity. Do you need it? I would say yes. Even if you only use SIEM as a big data store, it will help you perform post mortems when (not if) you have a breach. It is worth the effort, unless you are tiny or have no Internet presence to speak of.

    Keep rocking it! SEIM is another tool to consider as you move to DevSecOps dominance, but again, just a tool that requires you to keep the org safe.

  • DevSecOps Implementation: Intrusion Detection

    DevSecOps Implementation: Intrusion Detection

    Originally, this series was just going to be four articles on the DevSec side of DevSecOps. There are many reasons for this, but primarily because that side is cleaner. The other reason is that these topics are beyond the work we were doing at Accelerated Strategies Group.

    But we’ve had a number of requests to continue, and I’ve used many tools in the space, so on we go.

    Intrusion Detection

    Intrusion detection (IDS) started because attackers were getting through firewalls, and, once inside, seemed to have free rein. Intrusion detection wanted to offer another option for detecting and stopping attackers by watching things inside the firewall for indications your organization had been compromised. Relatively quickly, intrusion detection was supplemented by intrusion prevention (IPS). The idea of prevention was to proactively make changes based upon known issues to keep intruders out. Intrusion detection offered information about what paths attackers were exploiting, and intrusion prevention would plug them. Note that IDS and IPS are often further subdivided into host-based (HIDS) or network-based (NIDS) systems. Since their role – detection of intruders – is the same, for this overview we will discuss them together. A serious deep dive into IDS/IPS should look at HIDS and NIDS separately.

    In the DevSecOps space, some have folded these into security monitoring, but they are still performing the same basic tasks – just much better. In fact, if you found intrusion detection to be limited and clunky to work with because of rules maintenance and fingerprinting issues, it’s probably time to take another look. Here’s why:

    Problems With Traditional Intrusion Detection

    The problems with pure rules-based intrusion detection were two-fold.

    First, rules were hard and fast.

    “Look for X and Y and Z. If all are present, raise an incident.” This tended to result in only two types of rules – ones that were too narrow, and thus brittle, or ones that were too broad and that buried IT/security staff in alerts.

    Second, those rules were outdated before they hit the network.

    The old adage “You are always fighting the last war,” (meaning you are doing what worked the last time you won) is more true in IT security than almost anywhere else. For a rule or a fingerprint to be assembled, it has to have been seen. That means we were perpetually protecting against previously successful attacks and doing little against emerging threats. Security vendors took a lot of steps before coming to our current solution, which is a great step in the right direction.

    Attacks against IT infrastructure share a lot of commonalities. Purely rule-based systems counted on tribal knowledge to code those commonalities into rules. It was clunky and imperfect. Templates helped, by adding common items based upon the type of rule you were creating, but, like rules, they tended to be static and old news.

    Enter Machine Learning

    Bright people with an eye on the ever-changing environment knew that we needed something that could adapt to multiple attack forms/vectors and draw the type of conclusions that security analysts were tasked with. The ability to reduce the number of alerts that get raised is pretty close to miraculous. Anyone with a web presence sees thousands of security events a day, as attack bots and scanners automate chunks of the attack matrix. Some kind of solution to weed out the false information is nearly mandatory, as no organization has the manpower to monitor at those levels and reasonably follow up on threats.

    Machine learning (ML) makes connections between alerts – that might not be a big deal in isolation – and other events that, together, make a case for human intervention. Added to fingerprinting and rules, it is a powerful addition. While a successful attack would require going through rules and making certain you covered the behavior/changes found in the new attack, ML can take that information in and adapt automatically.

    This is cranking through volumes of data we couldn’t even imagine a decade ago, and finding anomalous behavior, correlating it and alerting where necessary. How much it reports up the chain is generally configurable, allowing an organization to tighten the flow of alerts that analysts have to deal with.

    Go With the Flow

    More recently, flow evaluation has become a positive step in the monitoring process. If this connection/user moves from A to B to C, it might be an attack. This allows for another way to rate suspicious activity and raise it to an analyst’s awareness.

    An increasing number of vendors are moving the analysis portion of these activities into their hosted environment. This is a “great strength/great weakness” solution – putting the data points gathered from your system into their cloud allows them to correlate events and improve all customers’ responses. The other side of the coin is that you are paying them operational costs each month, and for most vendors, you’re charged based on how much of the service you use. This has budget implications. It also assumes you have connectivity to their service. If you don’t, for any reason, no analysis will be done on collected data. A careful rollout and contingency planning are in order for this scenario to be successful.

    Integration with security information and event management (SIEM) systems is ubiquitous. We’ll talk about SIEM next; I just wanted to include here that IDS/IPS vendors are feeding SIEM.

    IDS isn’t perfect, and there are a ton of other things out there to help, but it’s a step in the process, one you should consider as you gear up your DevSecOps to professional levels. Some of you are using the tools, but don’t have them integrated into your DevOps systems. This is a nudge to do so. If resolution of an attack automatically creates tickets for Dev to make the code (or ops to make the config) more secure, it absolutely is a plus to the organization responding proactively.

    Keep kicking it. These are just tools, and they’re useless without you. Figure out what your organization needs, and update those needs regularly as the environment (both attackers and architecture) continually changes around you.

  • DevSecOps Implementation: Source Composition Analysis

    DevSecOps Implementation: Source Composition Analysis

    This is the second installment in this series on DevSecOps. Read the first installment, on Static Analysis, here.

    One of the better additions to security in recent years is source composition analysis (SCA). The purpose of SCA is to sit in the gap between static analysis and dynamic analysis to help you find issues introduced by included libraries, projects, etc. While effective, source composition analysis is only available on open source libraries and packages, which are the highest growth areas of external sources in any project. If we are purchasing a product to include in our application, we take our time and compare and contrast to make certain we have the right thing. If there is an open source option, we just drop it in. Particularly if the option is popular—little thought goes into using MySQL database in Python or its equivalent in other languages. If a Python project uses MySQL or MariaDB, it is needed, so it is grabbed and used.

    The growth of source composition analysis is a great boon to security. Having a tool that takes care of monitoring vulnerabilities in code that is included across the board, sometimes in very uncontrolled ways, means vulnerabilities are known before the app goes live or the new release is published. Security staff doesn’t have to try to ferret out what is included in projects across the org and then cross-reference those findings with the National Vulnerability Database (NVD) or project-specific lists. The tool takes a load off security’s shoulders and makes certain developers know the risks their open source software (OSS) usage represents. The best tools notify developers when including vulnerable open source and can be configured to require approval before a vulnerable OSS library/module can be included.

    With all of that said, let’s talk about what to look for. There are some similarities with our static analysis list, but some distinct differences also. Disclaimer: As mentioned in the Static Analysis blog, this is very high-level. The research we’ve done at Accelerated Strategies Group went much deeper than we can present in a series of blogs; this is just to help you understand the top-level issues and get started down the road to a solid DevSecOps plan:

    • Language Support: This goes beyond most users’ needs in static analysis. One of open source’s strengths is building off other projects. It is often the case that Python will call C modules and some Python projects even include Go libraries. This means your chosen development tools aren’t so much relevant for scanning as the development tools chosen by projects—and their included projects.
    • Depth of Checking: Which brings us to, “How deep does the tool go?” Nesting of open source libraries can be deep and complex, with the project an application uses directly just the tip of the ice-berg. Ideally, look for a tool that answers the, “Is this even ever called?” question. It’s a good start to alert on included OSS projects and it is far better to alert on ones where the vulnerable parts are called from somewhere in the hierarchy.
    • Where to Run: The market is slowly moving to support both cloud-based and local scans. Make certain your organization’s needs are met by the vendor’s offerings.
    • When to run: Some vendors bundle source composition analysis in with their static analysis IDE plugins; others have developers run the scan command line; others offer source composition analysis at the build step—when the app is together in one place; still others offer all of the above. Again, make certain the needs of your organization are met.
    • What Is Detected: As with most security tools, this is normally a base comprised of public sources (NVD, OWASP), with add-on detection from other sources. Know what those other sources are and what they offer: that is one major difference between most products.
    • Reporting: Make certain that vulnerability reporting has everything you need. Consider the possibility that it alerts on a vulnerable OSS library in source file insecure.go. All other uses of that vulnerability need to be visible in some way from that report. Removing the reference from insecure.go is useless if reallyinsecure.go also includes it and has an exception from management to keep the inclusion.
    • Reporting Integration: While scanning OSS usage and vulnerabilities the usage introduces is important, so also is the ability to see these results side by side with static, dynamic and even interactive scans so the overall security posture of the application can be evaluated.
    • Frequency Versus Cost: OSS vulnerabilities come in unpredictable patterns, with several dropped in a week after no important ones were seen the week before. You are going to want to run these scans frequently enough to feel comfortable with your security posture, but depending upon licensing, each run might cost your company money. Make sure you understand your needs and costs associated with meeting them.
    • Licensing: The newest feature to be added to these tools is the ability to list all licenses of all included libraries so that someone can review and see what risk is introduced to the organization from licensing. Many open source projects are licensed so that commercial use is limited; this tool puts all the various licenses together so an organization can evaluate litigation risk based upon license usage. The best tools allow you to also enter proprietary license information so that all licensing relevant to an application is reviewed together.

    These tools help improve security posture by automating what is an increasingly complex manual process and bringing in vulnerability sources that security (be they secure developers or dedicated security staff) would have to manually look for. They enable shifting OSS vulnerability resolution left, allowing your team to become more agile and proactive. Make certain the tool chosen is the right one for the organization.

    As I said in the last installment, you all are keeping the lights on in a truly difficult time, making a tough job look easy. Don’t let vulnerabilities that can be caught easily with these tools but might stay hidden without stop your great work. They are relatively easy to install and configure; once you have them, you have on-demand OSS vulnerability scanning and if you are like most organizations, OSS is a huge piece of your software infrastructure. Keep rocking it, and make sure you are covered for OSS vulnerabilities and licensing.

  • New Survey Highlights Codebase Size Problems

    New Survey Highlights Codebase Size Problems

    Sourcegraph and Dimensional Research have released a survey of developers at large organizations, which shows a massive growth in codebase size, number of repositories in use and complexity concerns.

    “The Emergence of Big Code – A 2020 Survey of Software Professionals” is definitely intriguing reading, and a few of the results are eye-opening—such as the difference between what development managers think developers are using to search code and what developers say they are actually using. But most of it simply validates what we all knew: Codebases are growing at an alarming rate, through a variety of growth vectors—support for new platforms, use of open source and internal development. Adding environment to code à la GitOps, systems reliability engineering and/or environment automation is adding a layer of completely new source and a layer of complexity not normally in the developer’s realm of immediate concern.

    I’ve been working with Accelerated Strategies Group over the last few months, and we’re looking at tools in this market. The ability to quickly get up to speed on large codebases and the ability to find a specific area in a codebase that is massive and/or spans repositories is a very real need that is not currently well-served. This piques my interest in what Sourcegraph brings to the table, but at the moment we’re focused on the survey, so perhaps in a future blog we’ll talk about Sourcegraph.

    While there is a lot packed into the survey, I think my favorite part is crammed in with other “What would you find useful?” results (the slide is below). Fifty-eight percent of respondents—all of whom are developers—chose, “Finding code by querying semantic relationships, not just matching a string.” As developers, these respondents no doubt are aware of the complexities of this type of feature, particularly in environments with a variety of programming languages and scripts (be they bash, SQL, whatever) all working together. This is huge, and first iterations would be shallow, but is a worthy goal. I’m intrigued to see how well SourceGraph et al solve it.

    Source: The Emergence of Big Code – A 2020 Survey of Software Professionals

    It is also intriguing to see more than half of respondents want the ability to explore ” … both known and unknown code with code intelligence that provides contextual understanding.” That, too, is a tall order. Code analysis has gotten better over time, but contextual analysis of large codebases is notoriously difficult. I’ll be intrigued in the near future to play with this concept and tools that offer it to see how reliable they are and how much human guidance is accepted and/or required to make it a reality.

    Code change velocity reporting and contextual information about code are both available and pretty solid from the majority of full-stack DevOps vendors. Perhaps that is why that particular response only netted 45% and 43% of responses, respectively—the other 50+% are already using them.

    We’re in an interesting time when consolidation appears to be on the upswing and the needs of developers and DevOps teams working in large codebases are becoming more clear and more complex at the same time. Over the next few years the needs reported in this survey will likely be replaced by more specific needs as these are fulfilled and find general use. The question of databases and database integration in large codebase migrations, for example, has these needs, but has other needs closer to the problem domain. As these general issues are resolved, expect your vendors to turn to those more specific, but every bit as complex, problems.

    And just keep rocking it. Find out what the tools and companies already in use at your org have added, and if those capabilities fit with your needs or can enhance your Dev/DevOps exercises. Keep the lights on, keep customers happy and watch for tools to help you make it even better.

  • The Use of Feature Flags in Code: Too Many Flags?

    The Use of Feature Flags in Code: Too Many Flags?

    It is interesting to watch the growth of an idea over time. The idea of using flags in code to do different things is as old as programming itself. Old-school configuration files used them, and they worked, so they were adapted.

    Thing is, given an idea that works, humanity inevitably takes it too far, and flags to control software behavior seem to have reached that point. If you are using one set of flags for A/B testing, another to segregate code in development at runtime, another to turn functionality on for some users but not others and yet another set of flags to hide data based upon user…You’re buried in unnecessary code branches. If your organization isn’t great about cleaning up after flags are no longer needed, you are building spaghetti code that future developers will not thank you for.

    The best answer to this growing burden is focus. What is the best use of feature flags/toggles in your environment? They certainly speed the delivery process by having the code there, in nightly builds, just not executed until the flag is flipped. This helps in rollout also by allowing operators to turn a feature on when it is ready by changing config rather than redeploying the application.

    For A/B testing, in many cases it makes more sense to handle switching at a load balancer, where again it is a config, but separate instances serve the UI in question. This, like all decisions regarding flags, should be based upon the volume of change. Changing the contents of a list is likely easier to test with flags, but large volume of change is probably better run separately and switched off of a load balancer.

    The same is true with the volume of change in source. A rollout might be the best option for handling large volumes of change, and feature flags may simply create more complexity.

    As with everything, approach the problem with a plan that works best, in this instance, in your environment. There is no denying that the ability to turn on a new feature and then turn it off if something goes wrong is appealing, just use the idea judiciously. And clean up after yourself. No shop that uses flagging extensively will say “yeah, we don’t clean it up,” because the mess of nested checks for flags would slow the code and make it passingly difficult to even read, let alone maintain, over time. So approach it from the beginning with “It is a tool, and when it no longer serves its intended purpose (the feature is fully live or abandoned), we remove it.”

    And keep rocking it. You are the engine of business, and this is what you do. Make it great and keep it running.

    — Don Macvittie

  • Oracle v. Google and Its Impact on APIs and Code

    Oracle v. Google and Its Impact on APIs and Code

    What, if any, impact will the Supreme Court’s rulings on Oracle v. Google have on the industry at large?

    TL/DR: It is unlikely to impact the vast majority of us, vendor and enterprise developer alike.

    Disclaimer: I am a technologist, not a lawyer. I have done some research, and talked to lawyers, but all that I say here is opinion—even my direct quotes of lawyers are their opinion. The Supreme Court can and has caught people off guard and taken cases in different directions than expected. Though I believe this path is the one it will follow, nothing is assured until the Court rules.

    Throughout this blog, references to The Case are actually using Harvard’s case document.

     

    Background

    On Nov. 15, the U.S. Supreme Court agreed to hear a dispute between Oracle and Google. The Oracle v. Google case has been going on for nearly a decade. And its roots go further back than that.

    The origins of the case go back to 2005, when Google and Sun Microsystems discussed the licensing of Java for Android. This will likely be important because it shows that Google knew it was someone else’s intellectual property (IP). More on this later.

    The crux of the case falls into three parts:

    1. Eight files that were decompiled and used. In one court case, these were found to never have made it into distribution and only used for testing (which limits liability).
    2. Two source files from Java used directly. These were found to be infringing, but damages were set to zero with consent of both legal teams.
    3. The part at issue: the copying of the entirety of Java’s interface from 37 modules. Most tech writers imply this as 37 APIs. It is not—it is hundreds of classes and thousands of methods. The 37 is the number of Java packages.

    The case has been through four court hearings/rulings so far. To sum up the relevant parts of these cases:

    1. The first case (split between a jury and district court, running at the same time) included patent violations. Those were defeated and Oracle has not appealed that defeat. On copyright infringement, the court found that Google did violate copyright on the 37 packages and the directly copied source code, but not on the eight decompiled files. The jury could not settle the issue of “fair use” of the 37 packages.
    2. The district court decided that the eight decompiled files and the copied source code for one function were violations, but remanded the rest back to the jury trial to settle the issue of fair use.
    3. In the end, the district court determined that APIs are not protected by copyright. Their decision involved two parts:
      1. That there is only one way to write a given API.
      2. The interface code and hierarchy can be replicated by third parties without violating copyright because it is an interface and not content.
    4. On appeal, the finding was reversed, and APIs were deemed copyrightable, sending the case back again.
    5. At this point, Google asked the U.S. Supreme Court to intervene. The court chose not to.
    6. Back at jury trial, the court found that while APIs may be copyrightable, Google’s use was fair use in the legal sense. This is the one topic the jury could not originally agree upon.
    7. Oracle appealed this decision, and the District Court ruled that it indeed was not fair use to copy the structure and exact interfaces.
      1. Where the jury and the district court appear to disagree is in the idea of APIs. Calling a publicly documented API is obviously fair use, and the jury latched onto that idea. The District Court appears to acknowledge the difference between calling an API and copying it.
    8. And after some tumultuous hearings and years in the court, the Supreme Court has agreed to weigh in on the issue.

    The real problem here is that copyright law is not a great fit for software. It never has been. If I rewrite your Java application in Python, using the same exact structure and naming, is that protected? All of the “words” have changed. It is akin to rewriting your Hindi book in English without a direct translation  … except that in software there is the added issue of it behaving exactly the same.

    But we can’t fix the misapplication of copyright law to software—only Congress can. So what to expect?

    A fair number of people, from the open source community and from businesses that see opportunity in a ruling that favors their world  view, are advocating that affording copyright protection to APIs is a horrible idea because it would hurt independent developers. This is not quite true. Independent developers are not generally copying Java. While this case does have bearing on coding practices and what developers can do, it does not have a huge impact on innovation, just on copying others’ work and how that is achieved—much like the “reverse engineering” court cases of the ’80s.

    There are good arguments on both sides of this case, but it seems to me that fair use will lose. They are public APIs in the sense that anyone can call them; indeed, some of these packages are so core to the system that you could argue every app must call them. But calling a function is not the same as re-creating it. In that sense, they are not in the public domain. Anyone can read a book, but that does not make it public domain.

    But no matter how this turns out, it is all good for the industry. This uncertainty has hung over the IT sector for nearly a decade. Just like when a new boss comes along and there is uncertainty in the department until everyone adjusts to the new demands/style of the boss, a solid ruling will allow settling to start. The industry can adapt once it knows what the rules are and we can move along.

    The real long-term answer is to have an equivalent to copyright and patent that is designed around the unique parts of computer code. Clearly, software is not the same beast as a book or magazine. The considerations are half-patent and half-copyright, and trying to bend those two to meet the needs of software has created some serious issues. Oracle quoted a 1995 case to showcase this problem—the company is quoting it to win the appeal, but it should be quoted as the reason the protections need to be designed for this medium that is not a publication and yet consists of words whose sequence matters: “Applying copyright law to computer programs is like assembling a jigsaw puzzle whose pieces do not quite fit.” (attributed by Oracle to the concurring opinion of J. Boudin in Lotus Dev. Corp. v. Borland Int’l, Inc., 1995).

    One interesting bit that the courts thus far have ignored, and I would love to see the Supreme Court take up, is this: If Google entered into licensing negotiations, the company knew it was Sun/Oracle’s IP. To then go and copy parts of it without licensing appears wrong on its face. It would be interesting to understand the process that ended in the copying and hear whether that bears upon the legal standing of the case. I can think of both malicious and innocent routes to get there; it would be interesting to hear testimony about motivations and thought processes.

    But even without delving into the licensing negotiations, this case stands to redefine copyright with regards to software. J. Michael Keyes, a partner at the international law firm Dorsey & Whitney, had this (and more) to say:

    “One of the issues that the court may decide is the scope of copyright protection for software. Google had early success below in convincing the trial judge that the snippets of copied code were not copyrightable. Google may urge to have the court take a look at this issue anew. Another huge implication is the scope of fair use. The jury found that Google’s use was “fair” and that it was not liable for infringement. The appellate court overturned that jury conclusion and found that Google’s use was not fair as a matter of law. The Supreme Court will likely wade into this thicket and provide guidance on the scope of fair use in the context of software code and what role an appellate court plays in reviewing jury decisions in that regard.”

    What we (all of us) do not want to do is confuse “good for the industry” with “legal standard.” It is not the court’s job to do what we think is best for the industry; it is the court’s job to determine applicable law and what that implies needs to happen. Presumably, the Supreme Court will do just that. And it may not please any of us. But if we think the current iteration of the law is flawed, that is something to take up with elected representatives, not ask a court to “fix” on the fly.

    For the vast majority of us, though, it impacts our development not at all. Using APIs or forking open source projects are all still perfectly acceptable, and no ruling by the Supreme Court is going to change that—because copying someone’s APIs is the issue here, not using them. And open source licenses allow for copying and improving projects. So carry on. Get the job done. And read about the Supreme Court’s final decision with a smile, because either way, the industry will adapt to the new normal and continue forward.

    — Don Macvittie

  • DataOps: DevOps Plus Big Data

    DataOps: DevOps Plus Big Data

    In traditional DevOps, there are the complimentary forms of development operations (that I call DEVops), and development operations (that I like to call devOPS). Between them they automate the toolchain and bring the people working on getting an application out to users onto the same team—not necessarily the same organizational team, but using their strengths in tandem to meet the needs of the business.

    DataOps is an interesting extension of DevOps. You still need all of the coding side to get data into the system and queries consistently maintained. You still need all of the operations side to get your database/noSQL/whatever up and running. In fact, while the development side in a DataOps environment might be (but isn’t necessarily) lighter, the operations side is almost always more complex. No matter the big data engine in use, it is a complex system in addition to other supported systems in a normal environment. My first installation of a big data environment (Cloudera, as it happens) was a weeks-long learning voyage. Only after I’d completed it did I use an automation tool (which is no longer available) to make it easy. My second round was hours … But it assumed the knowledge I had gained in the first attempt.

    The other bit of DevOps—ongoing monitoring and management—is also more complex in a big data environment, but we’ll come back to that in a moment.

    Once the normal DevOps systems are in place, ETL/data import tools will need to be supported also. The volume that these tools crunch through in a day makes their inclusion in DevOps in a data-heavy environment critical. If the data uptake is slow or the data itself inaccurate, there is an impact on the organization. This step also requires inclusion of data scientists, people who traditionally have not been pulled into the DevOps model. But they are the ones that can gauge the accuracy of data, and normally are responsible for data acquisition anyway.

    Which brings us to the monitoring and management stage of DataOps. Normally when we talk monitoring and management in DevOps, we are talking about tools that ultimately help with things like availability, responsiveness, auto-scaling and recovery. DataOps needs all of these things, like any other application does. And we’ve already mentioned that it has a complex environment embedded in your complex environment. But it also (some would argue more importantly) needs data monitoring. A knowledge of the state of the data is imperative. Whether from a piece of code gone wrong or data poisoning, monitoring the health of big data archives is essential to keep results from getting out of whack. This monitoring will again involve the people who know the data—data scientists by whatever title. This iterative process will need to run relatively constantly, checking on whatever makes the data valuable, normally using cross-field checks to verify that data hasn’t somehow been corrupted, or comparing trending averages to current average to monitor for sudden swings in results.

    Of course, for a system (and in research there are many) that spins up, finds results and spins down, the monitoring part is less essential, but for anything that keeps accessing and adding to the data pool, it is the key to a healthy application or application portfolio based upon the dataset.

    If all that sounds a bit intimidating, you’re not alone. The phrase “DataOps” has been around for a few years, and while data scientists use it, it hasn’t yet gained serious traction in the enterprise. Here’s how I would approach it as CIO: Given that DevOps is running on other projects, that is the low-hanging fruit. Given the rate of change for servers (as queries—particularly long-running ones—increase and decrease, systems needs change is one example), traditional DevOps is a greater benefit to big data than many other corporate systems. So roll out DevOps first without serious concern for DataOps beyond how DevOps impacts ETL—not ETL data quality, but ETL data availability/performance. Only when the system and the team are in place and the software/hardware/code part of the overall system is fully in a DevOps environment, start looking at ways to move data into it. Data monitoring will be the big one, but ETL under the initial round of DevOps implementation will have shown places it could be improved and ETL data integrity can be added, so improving those areas also. A measured, step-by-step approach that builds off of organizational experience is key here. Data monitoring and management will require different tools than normal DevOps monitoring and management, so having the data scientists look into available toolsets and how they mesh with existing tools early on will help with eventual implementation.

    Make no mistake, it is a big job, moving large data driven environments to DevOps. But the benefits are increased awareness of data quality and data quality issues, along with more responsive systems and teams that DevOps offers every business application. The amount of data available to any give organization is currently on a massive increase as IoT, open datasets, partnerships and ML drive greater pools. It is worth getting the automation and cooperation of DataOps into the mix as early as possible.

    — Don Macvittie