The first step is simply to merge my temporary-hack fixes (e.g., removing the I/O rate limiting during recovery operations) into the main Tarsnap server codebase. I lost about 3 hours to those.

The second step is to change the order of S3 GETs and adjust the parallelization of those; I think I can easily cut that from the ~20 hours it took down to ~5 hours.

The third step is to parallelize the third (replaying of log entries on a machine-specific basis) stage; that should cut from ~10 hours down to ~1 hour given a sufficiently hefty EC2 instance.

After that it's a question of profiling and experimenting. I'm sure the second stage can be sped up considerably, but I don't know exactly where the bottleneck is right now. I know the first stage can be sped up by pre-emptively "rebundling" the metadata from each S3 object so that I need fewer S3 GETs, but I'm not sure if that's necessarily the best option.

In the longer term, I'm reworking the entire back-end metadata store, so some of the above won't be relevant any more.

Will those changes also finally speed up client restores? The extremely slow restores are one of those things that remain terrifying as a customer.

We've had to consider moving to a different backup system for disaster recovery. Tarsnap covers the cases of fat-fingering and versioned backups nicely, but not disaster recovery of large-ish data sets.

That's a different issue, but related to the long-term back-end reworking. (It's not one back-end, it's several pieces of back-end, some of which are necessary for speeding up extracts and some of which aren't.)

The simplest mitigation to this is to store your primary systems in a different location to your backup. This way the likelihood of both your primaries and your backups becoming unavailable at the same time are significantly reduced.

I don't see how that mitigates the issue at all. The case I'm talking about is when your primaries totally bite the dust. It's obvious that a backup system shouldn't be in the same physical location as the primary system.

The issue I'm referring to isn't about lack of availability of tarsnap's backups, it's that it's very slow to restore backups if you need to do a complete restore (rather than just grab a few accidentally trashed files).

Would it be feasible to store snapshots of the metadata periodically as well, so you only have to replay mutations performed after the last snapshot in an emergency?

Yes, that's also something I'm looking into. But I want to have a decent "worst case scenario" recovery mechanism too.

Seems to me that keeping an EC2 instance spun up and idle might become a sensible approach for people who run downtime-sensitive apps with a single AWS region architecture. A lot of recent outages have been exacerbated for many people by the flood of control plane traffic and contention for new instances - at some stage paying for enough idle instances (which might be just one) spread over availability zones to allow you to at least have some chance of surviving an entire datacenter going dark has to be at least worth running the numbers on.

I also predict the control plane contention problem is going to get worse - I'm sure I'm not the only one working on a system to spin up and grab a few extra instances as soon as my monitoring detects the beginnings of a problem. It seems to be "the right thing" to do is start spinning up and configuring replacement instances immediately you suspect a problem - even if it turns out not to be a problem most of the time, when it _is_ a problem at least you're well ahead in the queue of the people who chose to wait 5 or 10 mins on the assumption that perhaps it's only a temporary network glitch.

(And there's a weird negative incentive for Amazon here. If they fail to address the poor heavy load performance of the control plane/management console/provisioning api, they may end up with more money in their pockets from people keeping spare idle instances or provisioning additional instances speculatively at the first sign of trouble.)

"I'm sure I'm not the only one working on a system to spin up and grab a few extra instances as soon as my monitoring detects the beginnings of a problem"

Evil idea of the day. Instead of actually spending money and buying possibly unneeded services from AWS, a somewhat less ethical person might choose to ddos the provisioning API at the first sign of trouble - then if it turns out that you _do_ need extra provisioning, you'll be first to know when the API becomes available (since you call the ddos off yourself when it suits you)…

It's a classic tragedy of the commons.

The normal response in this situation is to do away with the commons; a floating price for control plane requests, for example.

I don't think that would be very popular, though.

Like it or not, people designing for AWS must now assume that the control plane will simply be unavailable during partial outages.