Tag: backup

  • US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)

  • Why Do You Need GitHub Backup?

    Why Do You Need GitHub Backup?

    You’ve probably heard the joke that there are two types of people in IT: Those who do backups and those who will start. Though it’s still valid, this joke has become less relevant to businesses and professionals. The IT industry has been increasing expenditures on security for years, and backup is a critical area. However, despite the growing awareness of the need for backups and the wide availability of modern backup solutions, the problem still exists. The number of security breaches is growing, and the topic of data security looks like an endless arms race. So what can organizations do?

    First, let’s analyze some data and trends. Year over year, the number of cyberattacks, especially ransomware, is increasing. The year 2020 saw attacks intensifying due to the pandemic and the sudden shift to remote work, for which, unfortunately, not everyone was prepared. Once organizations got better at enabling effective work-from-home models, the hope was that everything could settle into a new normal. Unfortunately, the new norm became these increased attacks.

    It’s a bit like the metaphysical yin and yang—two opposing but complementary forces. One drives the other. Suffice it to say that the estimated cost of the global damage caused by ransomware in 2021 is as high as $20B! And a ransomware attack occurs, on average, every 11 seconds around the world.

    According to an Identity Theft Resource Center (ITRC) report, by the end of September 2021, there were more incidents of this type than during all of 2020. The authors of the report emphasized that they found 26 cases where cloud databases were unsecured. As a result, hackers were able to access confidential data belonging to 99 million people.

    Git Threats: Are my Bitbucket/GitLab/GitHub Repositories Secure? 

    Back in May 2019, ransomware attacks impacted hundreds of repositories on GitHub, GitLab and Bitbucket. The attack wiped away all the data and left only one piece of information: The amount of ransom demanded and a method of payment. Eventually, most of the data was recovered but the cost and the amount of time spent recovering from the attack was enormous.

    According to the Cisco Benchmark Study, approximately 40% of companies experienced downtime longer than eight hours due to a major failure. And 39% reported that at least half of their systems had been affected by a serious breach. According to IBM, the total average cost of a data protection breach on a global basis is as much as $3.86M, and the highest ransom demanded by cybercriminals in 2020 was $15 million. These numbers are scary. On the other hand, in most cases of successful ransomware attacks, the ransom is ultimately not paid; even if organizations do not pay the ransom, the mere fact that they fell victim and experienced failure or downtime can be very expensive.

    GitHub as Backup Tool

    GitHub provides many useful security tools. However, we must be aware of two things: What does GitHub backup really mean and what are its features? And what responsibility do cloud services providers have versus users’ responsibility for security? Let’s start with the latter.

    Most cloud service providers (including Amazon, Microsoft, Google, IBM, Salesforce, GitHub or Atlassian) operate on the basis of the so-called shared responsibility model. Users of cloud services most often assume that providers are fully responsible for their protection. However, providers are primarily responsible only for ensuring the security and availability of infrastructure, software and access, period. Users are responsible for the security of the business data stored as part of these services. So users themselves have to take care of backup and disaster recovery plans and solutions. In short: Providers are responsible for the security of the cloud itself and users are responsible for data security in the cloud. And at this point, it is worth pointing out that, according to data from the UK’s CybSafe, in 2019, 90% of data breaches were caused by user error.

    It can happen to any of us—a ransomware attack, phishing, loss of access to a GitHub account or an entire computer or simply overwriting another developer’s work in a repository. It affects both individual users and large corporations. In July of 2019, Ubuntu Security reported that the credentials for a company-owned GitHub account were compromised. Hackers used these compromised credentials to create repositories, issues, pull requests, etc. The attack did not cause a major crash, but repairing the damage took some work and restoring some repositories or issue trackers to the previous state was difficult.

    There are some common features any good backup should have:

    • Automation (i.e. daily backups)
    • AES encryption with its own encryption key
    • Versioning
    • Long-term data retention
    • Disaster recovery process
    • Easy monitoring (audit logs, email notifications)
    • Central management
    • Multi-tenancy (to manage admins, privileges and roles)
    • Scalability

    Based on these criteria, GitHub (and other similar hosting services) cannot be considered a proper backup tool. GitHub is a great open source project, but its purpose is completely different.

    How to Back Up Manually

    Since the cloud service itself is not enough, how can you take care of GitHub backup? Well, you could do it on your own. And often, especially in small businesses, this is the preferred method. It may seem like a good idea at first. Knowing Git well, we can create appropriate scripts that create backups for us that give us full control over what is happening without having to pay anyone for it. But this is very risk and not profitable in the long run. Disk space costs money; our employees’ time to create scripts and perform the backup is another cost. And the most important thing, in this case, is that these scripts must be maintained and updated all the time, so they constantly generate additional costs. Any change in such a script may cause an error and, as a result, deprive an organization of a working backup. Of course you could also create an appropriate mechanism to test it, but that’s an additional cost. Manual GitHub backups do not end with writing a single script.

    I’ve experienced this myself. In one project, backups were generated using such a script and then kept on our own servers. It was cheaper this way. But we did not have any validation of the created backup. One day, while updating the backup scripts, we did a manual test and it turned out that we had a serious problem—the backup was created correctly but its name did not change, so we kept overwriting the same archive constantly! The effect? It turned out that we didn’t have any copies older than one month! Luckily for us, they weren’t needed because the systems and databases were working properly at that time and there weren’t any incidents, but it makes me cringe just thinking about “What if …?”

    All in all, manually writing backup scripts is not a very good idea for many reasons. The low cost of this solution is only theoretical; with time, the cost of maintaining this mechanism increases and, in practice, often only one or two people are responsible for it, which creates additional risk. Okay, you might have more control, but overall you are wasting time and money. The backup itself is also not enough, because you need a proper restoration solution for systems or data in the event of an incident. There’s also the question of how to create a backup of the script that creates the backup? Are you sure you cannot make better use of your employees’ working time?

    How Much Does it Cost?

    It is difficult to accurately measure this; it all depends on the level of complexity of systems, the number of repositories, the skills of your team, etc. However, you can estimate how much the lack of access to services or programmers’ downtime will cost. You can also calculate the development and maintenance costs of writing, testing, maintaining and updating your own backup scripts. The history of GitHub attacks and crashes shows that the data is usually recoverable, which is good news. On the other hand, the time when services are unavailable or when teams are unable to work can be very costly.

    Third-Party GitHub Backup Tools

    Another solution—and, in my opinion, a better one—is to use third-party backup software.

    In fact, in the GitHub documentation, it is recommended:

     “Backing up a repository: You can use the API or a third-party tool to back up your repository.”

    If you can use the GitHub API, why do you need a third-party GitHub backup tool? Well, first of all, such applications are created by experts in a given field who have knowledge, experience and stay up-to-date on the current security trends. Second, these solutions are ready and available right away; you don’t have to spend time and money reinventing the wheel. Yes, there is a cost, but in the long run (taking into account the factors described earlier) the cost usually turns out to be much lower than a bespoke solution. Outsourcing backup management responsibilities may be incredibly beneficial and will allow us to focus on developing our business instead of managing GitHub repository backup all over again.

    When it comes to backup, be smart. If external third-party tools allow us to develop our products faster and more efficiently, then we should not hesitate to use them. And never forget to implement proper backups and disaster recovery plans. Taking care of it today will help ensure a more secure future.

  • Fast Object Storage Key to Backup and DR Strategy

    Fast Object Storage Key to Backup and DR Strategy

    The massive shift to remote work necessitated by the COVID-19 pandemic kickstarted many companies’ digital transformation. While this transition had already begun for most organizations, the pandemic sped up the timeline. These events also had an impact on how storage and backup is done. Let’s look at why the right storage solution is key to your backup and disaster recovery strategy, especially when it comes to today’s work-from-home and hybrid scenarios.

    Hybrid Cloud and Storage

    While most companies have moved to the cloud in some form or another, many are relying on a hybrid approach. Enterprises may want to maintain their primary data storage on-premises for performance, security and compliance reasons. The public cloud, though, offers compelling services that can enhance this on-premises data.

    Using both the public cloud and on-premises storage allows for hybrid cloud use cases that are changing the way enterprise IT leaders think about data management and data protection. Enterprises are implementing hybrid cloud architectures to take advantage of cloud archive services, to maintain full copies of their on-premises data for disaster recovery, to protect against ransomware attacks, to take advantage of cloud service bursting and more.

    Businesses retain greater control over their private data by using hybrid cloud services. An organization can store sensitive data on a private cloud or local data center and, at the same time, make use of the robust computational resources of a managed public cloud. A hybrid cloud relies on a single plane of management, unlike a multi-cloud strategy that forces administrators to manage each cloud environment separately.

    Remote Work has Changed the Backup and Recovery Game

    Companies risk losing their most valuable assets – their data – if they lack strong backup and recovery infrastructure. Data loss affects businesses directly, when assets can no longer be leveraged, or indirectly, via loss of customer confidence and regulatory fines and penalties.

    The shift to remote work means that the way storage and backup is done has also changed. If the majority of a workforce is remote now, how do you back up laptops? How do you back up phones? Previously, employees would go to an office and their laptop and other devices would get backed up there – but now, that’s not the case. And with remote work in place for the long haul for many companies, backup methods have to change and adapt.

    Remote work opens up greater risks of information being stolen or lost. If an employee is at home, for instance, and they lose a file on their computer – and they’re used to having their computer backed up by the company – how do they restart that backup? If they have to go through IT, that seemingly simple act can take weeks.

    Self-Service Backup and Recovery

    Organizations need to have more flexibility – a way for individuals to perform backups in a self-service way without having to deal with centralized IT. In this way, people can protect their data themselves.

    With the right storage solution, authentication is built in; storage and backup are designed with security and data protection in mind. Unlike other storage solutions, object storage gives you your own keys and your own passwords, and you only see what you should. It’s very well protected. It’s also very siloed, so that employees can only do what the company wants them to be able to do, and they only have the minimum access they need. So, object storage can help organizations adopt a more fine-grained authentication and security posture; more so than with a typical file storage solution.

    In addition, having this kind of solution in place opens opportunities to provide your employees with the kind of self-service backup and recovery they need in a remote world where speed is of the essence and efficiency is front-and-center.

    Success in a Hybrid World

    The shift to the cloud is in full swing for many organizations, brought on by a variety of business factors and accelerated, in many instances, by the pandemic. It makes sense that the hybrid storage approach would become popular, as organizations naturally want to keep certain kinds of data close to the vest for security and/or compliance reasons. Performance is another consideration when it comes to where data should be stored.

    This gives organizations a lot to think and strategize about – including how to handle backup and recovery in a WFH world. Remote work increases security threats, so organizations need the option to empower employees to protect their own data. Object storage makes this possible with both speed and efficiency. There are many options available today for both cloud and storage configurations, so consider the above points when making decisions that will affect the security, compliance and performance needs of your organization.

  • Multi-Tiered Escalation Policies are Paramount

    Multi-Tiered Escalation Policies are Paramount

    As any team charged with keeping systems up and running will tell you, having one person on-call isn’t enough. What happens if your on-call engineer sleeps through his alert? What happens if his phone’s battery dies without him knowing, or if he gets an alert at an inconvenient time, like when stuck on a bus or in traffic? It will happen.

    In today’s always-on world, companies can’t afford for that to happen without a backup plan. One or more people should always be waiting in the wings, ready to spring into action if a primary on-call is unable to perform his or her duties to the best of his or her abilities at any given time.

    Engineers who perform backup duty don’t need to be as “on-call” as a primary engineer — no pagers needed here — but what backup engineers lack in readiness they can make up for in numbers. It often makes sense to have multiple backups or an entire team of backups.

    Operationally mature companies often group the currently-on-call engineer and all of his or her backups into an escalation policy. Escalation policies set the order in which each person is alerted and the delays between alerts. While there are many ways to organize these escalation policies, here are some of the patterns most often used:

    Primaries Aren’t Enough; Determine a Secondary, Too
    Most companies have a primary on-call engineer, usually determined by a calendar that designates when members of an ops team are on call.

    Operationally mature companies take their on-call practice one step further and supplement this rotation with a rotation of secondary on-call engineers. Why a secondary rotation instead of just paging the whole team when the primary falls through? If a large team is paged, you get a ‘tragedy of the commons’ effect— it’s urgent that somebody respond to the issue, but not clear how urgent the issue is that each individual person respond. Having a designated secondary removes that confusion.

    The secondary rotation is set up to shadow the primary rotation and generally has the same members as the primary rotation. The difference is that the order of the engineers in the secondary rotation is usually offset from that of primary rotation— if your primary and secondary are the same person, you’re going to have a bad time. For example, if a primary rotation contains Alex, Bob, then Charlie, then a secondary rotation may contain Bob, Charlie, then Alex in that order. Companies often find that if last week’s primary is this week’s secondary, the it helps make sure there’s somebody with recent context who can help if the primary runs into trouble. It is becoming more commonplace for organizations to have both a primary and secondary on-call rotation; but, in an environment where customers expect zero downtime, secondary rotations are no longer an optional tool.

    Enable a Strong Support System
    Larger companies and companies with significant numbers of operations tasks often deploy a separate support team that handles all basic operations tasks before the organization elevates the tasks to an on-call engineer. This team — the first tier support team — usually knows how to handle all of the smaller, repetitive problems that crop up often and have clear resolution procedures. These first-tier teams often offer their support to multiple engineering teams and are placed first in an escalation policy.

    If first-tier teams cannot resolve the issue, escalation policies typically dictate that primary then secondary on-call engineers must complete the task. However, there is one more support network operationally mature organizations must implement: management. Management is ultimately responsible for a company’s systems, and needs to be in the know when severe problems happen— both to hold their teams accountable and to muster additional support. At PagerDuty many of our EPs have a rotating manager schedule at the top of multiple escalation policies, and many of our customers use this pattern as well.

    Put Your Tiered Teams to Work
    In practice, an organization policy for a hypothetical “Database Ops” team might look like this:

    • Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
    • Assign the incident to the user who is on-call in the Primary DBA schedule
    • Assign the incident to the user who is on-call in the Secondary DBA schedule
    • Assign the incident to the team lead
    • Assign the incident to the dev manager

    This escalation policy would unfold over time, with timeouts of between 10 and 30 minutes before each subsequent escalation, depending on a team’s needs.

    Implementing an escalation policy that involves multiple tiers of resolution ensures that devops teams address all alerts — whether business-critical or minor — in a timely manner. Limited downtime is paramount in today’s fast-pace business environment. Customers expect their applications to run smoothly, or they expect to switch providers if devops teams cannot resolve issues fast enough. Organizations will be much less likely to see their processes fail, and much more likely to see happy customers, if they implement a multi-tiered escalation policy.