This reminds me of something I've been wondering about since the Bitcoinica heist: how do people usually set up secure offline backups which can't be erased using the credentials on the backed-up server? I would probably do something with ssh authorized_keys if I had to make it from scratch, but are there obscure security/reliability risks, and tools which have already mitigated these risks for you?
The webserver has no credentials for accessing the backup server. Instead the backup server accesses the webserver.
This strategy places higher trust on the backup server, but the backup server is easier to defend -- it only needs connectivity to a small number of other IPs.
1. Don't make your backup server accessible on the public internet.
2. Don't allow shell access from any server that does have access to the public internet. When your web server gets hacked, you don't want your assailant to have the ability to shell around in your network.
3. If you need shell access from outside the network, have a host specifically for this purpose and disallow password authentication (.ssh/authorized_keys indeed)
4. Backup server is write-only. I don't have a hard-and-fast method for enforcing this, but a process (or kernel module?) that watches for incoming backups, moves them immediately, and prevents overwriting existing files seems simple enough.
The standard way used to be to use write-only media. For example, if you log to a server which writes the logs incrementally to a DVD writer, you can be fairly certain that the logs won't be erased...
That's only true if you have software that can mount arbitrary past sessions, which is rarely the case. When you put in a dvd, what gets mounted is the latest session -- which is supposed to also include all previous sessions, but doesn't have to.
One of the follow-up posts to this is going to be on using OSSEC-HIDS which will give you logfile chained checksums. It's not perfect, but again it's about achieving the most value for the least amount of effort.
My quick and dirty way to it is to run a cron on the backup server that chowns incoming files to another user (with a few refinements, like preventing exec, etc). But I'd definitely like something more solid.
You should look into a tool that stores meta information on the backup files, such as rdiff-backup. Manually restoring ownership/permissions from a backup is probably tiring.
