A desktop kernel wishlist - Big folder copy

Posted Nov 25, 2014 22:37 UTC (Tue) by nix (subscriber, #2304)
In reply to: A desktop kernel wishlist - Big folder copy by blujay
Parent article: A desktop kernel wishlist

OK, I'll lose the last 120s of unsaved work due to autosave kicking in :)

real-auto-save is... more than slightly risky. Not *everything* I touch is in git: I don't want absolutely everything really saved unless I ask for it.

I used obnam for a while, but its insistence on rolling its own data structures means that corruption is still a problem years on, and it's *fearfully* slow -- by which I mean that it took more than a week to back up a filesystem that took bup a day, and takes twenty hours or so to do the sort of routine backup run that I run every night (it takes bup an hour to do the filesystem tree walk and on the order of three minutes to do the backup).

I back up *everything* in bup. OS, virtual machine disk images (once a week, providing --smaller=500M to other runs), $HOME including things like build trees and object files from build runs, everything. The only stuff I exclude is things like DVD rips which I can just re-rip on disk failure and which never change nor benefit from deduplication.

The dedup really does save your bacon in a lot of areas: e.g. I can flip between git branches and completely ignore that this will lead to the working directory changing over and over again: the branches have been flipped to before, so they're already in the backup, so bup will deduplicate them almost entirely. Most of my virtual machine images disappear: they're neither compressed nor encrypted, so the binaries on them have largely been seen in other VM images, and vanish. Renames of huge filesystem trees just get deduplicated out of existence; huge swathes of inode changes caused by things like cp -al get almost entirely deduplicated (the ctime change is recorded, but the content gets deduplicated). And of course an ordinary cp -a won't use any more space on the backup either.

As for not being able to expire backups, y'know, that's not really a problem. As I mentioned above, even backing all that stuff up, I estimate that I'll run out of space on my backup device in ten to twelve *years*. By that point the disk will have died of old age -- and I rotate them every so often anyway for offsite backups, so it's no trouble wiping one and doing a new full backup every decade or so!

btw, expiry is being worked on: several promising approaches have been posted to the bup list of late. Personally I'd not expect you'd have to wait too long for it. I just don't think it's all that useful... your backups fill up so much more slowly than they ever did with rdiff-backup and the like.

to post comments

A desktop kernel wishlist - Big folder copy

Posted Nov 26, 2014 10:48 UTC (Wed) by jezuch (subscriber, #52988) [Link] (3 responses)

Speaking of bup... I agree that it's a marvelous piece of software, but the separate steps of indexing and actual backup are a hassle, and last time I investigated it it had problems with backing up metadata... so as a result of these two minor inconveniences and sheer inertia I'm still using my old set up ;) Has anything changed while I wasn't looking?

A desktop kernel wishlist - Big folder copy

Posted Nov 26, 2014 10:53 UTC (Wed) by mbunkus (subscriber, #87248) [Link]

I don't know what your exact problems were, but the README ( https://github.com/bup/bup/blob/master/README.md ) the following about meta data:

> 'bup save' and 'bup restore' have immature metadata support.
> On the plus side, they actually do have support now, but it's new, and not remotely as well tested as tar/rsync/whatever's. However, you have to start somewhere, and as of 0.25, we think it's ready for more general use. Please let us know if you have any trouble.
> Also, if any strip or graft-style options are specified to 'bup save', then no metadata will be written for the root directory. That's obviously less than ideal.

So if your previous experience was that meta data wasn't supported at all then it may be worth to take another look.

A desktop kernel wishlist - Big folder copy

Posted Nov 27, 2014 17:43 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

Metadata support appears to be more or less flawless by this point. Normal POSIX metadata works; xattrs work (though you probably want to use trunk bup for that if you're using a big-endian host); ACLs work. I'm not aware of any other metadata it might need to save :)

The index is, to me, a nice idea simply because it means that rather than having to support umpty million different ways to select and exclude files like most backup programs do, it only needs to support a few, you can pick files additively via multiple 'bup index' runs and then do a 'bup save' once you're happy with it. (And in the future, I can imagine building the index from things like a btrfs snapshot diff -- i.e., instantly -- which would let you do things like run a backup every minute if you wanted to!)

Currently the index's *implementation* is far from ideal, which makes it less useful than it might be -- each index update requires a lengthy merge step, so additive index updates aren't very tempting, and files that are deleted are marked as deleted in the index but never actually removed from it even after the backup is run. Work is underway to fix things on that front, by making the index sqlite-backed rather than its existing rather ugly kludge. But this has no effect on the content of the backup or on the backup run's speed, only on the output of 'bup save -v', which never stops reporting things as deleted. (I 'fix' this by just deleting the index every few weeks and letting the next 'bup index' run rebuild it. It means the next 'bup save' run is slower, because it has to rescan everything and determine that, oh, it's a duplicate, rather than skipping it as unchanged, but the resulting backup is exactly the same as it would have been in any case.)

A desktop kernel wishlist - Big folder copy

Posted Nov 28, 2014 12:13 UTC (Fri) by jezuch (subscriber, #52988) [Link]

Thank you all for your answers! I'll definitely be taking another look at bup :)