Compared to others

Conserve aims to fill a good niche of being simpler and safer than Duplicity or Obnam, but more friendly to cloud storage than rsync.

The great thing about backup programs is, to some extent, you don't have to pick just one: for safety, make multiple backups with different tools and approaches. So this document is not to knock the other tools or to persuade you not to use them, but rather to explain why it's worth writing Conserve too.

Duplicity

Duplicity addresses a similar case of backup to cloud or dumb servers, and uses librsync delta compression. But it fails several Conserve manifesto items, some of them by design:

• Verification and restoring a single file requires reading the whole archive.

  • Somewhat inherent in having a layer at which the whole backup is represented as one tar file, which does not support random access. This is perhaps not impossible to fix, if we change the format to add an index of blocks and locations within that block.
  • rdiff also done at whole-tar level so even having an index will not help with random access to anything but a full backup.

• I have experienced Duplicity bugs that make the archive unreadable and, although it is based on simpler underlying formats (rdiff, rdiffdir, tar, etc) it is still hard in practice to recover.

• Performance is unpredictable and sometimes bad.

  • Seems partly due to the caching layer? In Conserve we will not rely on a cache (there may be one later) but rather try to reduce the number of files that need to be seen at all for any operation.
  • For example a FUSE filesystem needs to list directories and retrieve individual files. Doing this on the Duplicity format would be unreasonably slow - at best, possible for modest-sized local backups with aggressive caching.

• For decent performance, requires sometimes doing a full backup; but in fact you may often have some files that never change and copying them all the way up seems redundant.

  • Conserve does still have a full-backup tier. It should need to be re-written less often, because there are multiple layers on top of it, not a single chain, therefore performance should degrade as something more like O(log n_backups) rather than O(n). Perhaps we can avoid that in Duplicity, by rearranging and garbage-collecting blocks in the archive? It is worth investigating though it seems at risk of complexity, or doing too many round trips, or avoiding the desirable property of not re-writing existing data.

• Until the first full backup completes, you can't restore anything?

  • But this could in principle be fixed in code without needing a format change: the data is all there and readable.

• Can resume backups but this seems particularly buggy?

  • This too is a code bug not a format bug, though I have a not-rationally-justified feeling the caching interacting with this is complicated.

There is, in fact, a page about replacing tar in Duplicity and Conserve should try to address all of them:

• Not indexed

  • Conserve proposes a two-level index within each band: one giving the first name within each block, and the other listing the filenames in the block. So to search for a filename in a band, we need to read the band index (fairly concise but proportional to the band size), which will identify the block in which it might be present; then that immediately says which data block to read. To find the most recent version of a filename across all bands, we'll have to search back down the stack, so typically repeating this at say four layers. For reading a single known filename we could perhaps add a Bloom filter per band.

• Not seek()able inside container: Because tar does not support encryption/compression on the inside of archives, tarballs nowadays are usually post-processed with gzip or similar. However, once compressed, the tar archive becomes opaque, and it is impossible to seek around inside. Thus if you want to extract the last file in a huge .tar.gz or .tar.gpg archive, it is necessary to read through all the prior data in the archive and discard it!

  • Addressed in Conserve by doing compression at the block level (say 10MB), and having pointers into that content: so to read the last file, you need to identify which block it's in, read and uncompress that whole block, and then pick out the relevant part.

• Doesn't support many filesystem features: Tar also doesn't have support for many features of modern filesystems including access control lists (ACLs), extended attributes (EAs), resource forks or other metadata (as on MacOS for instance), or encoding types for filenames (e.g. UTF-8 vs ascii). The format does not lend itself to extension so adding these features will probably not be easy.

  • Fairly easy to do in Conserve because we're not bound to the tar format, and it's easy to add new fields to the protobufs.

• General cruft: There are some legacy features of tar that make it a pain to work with.

  • Not a problem

• duplicity already doesn't use straight tar

  • Not a problem.

The proposal for replacing tar with a more block-oriented format is not dissimilar to the Conserve approach.

Obnam

• obnam format too complex? Otherwise looks pretty good.
• project was retired in 2017 https://blog.liw.fi/posts/2017/08/13/retiring_obnam/

TODO: More details.

Bup

https://github.com/apenwarr/bup/blob/master/DESIGN

Stores into git pack files but breaks files based on a rolling checksum.

[bup layers on the git format] in order to address two deficiencies in git when used for large backups, namely a) git bogs down and crashes if you give it really large files; b) git is too slow when you give it too many files; and c) git doesn't store detailed filesystem metadata.

TODO: More details.

• Will do well on large files with incremental changes by breaking them into a tree of content-addressed blocks, which is a bit hand-wavey in Conserve.
  • Running xdelta on huge files is too slow so we shouldn't count on it.
  • But breaking based on rolling checksums may be very reasonable.
• Blocks identified by SHA and collisions will be lost. (Which is incredibly unlikely but perhaps still not something you want in a backup system.)
• Huge numbers of packfiles with multiple layers of indexing on top to get decent performance. But apparently performance actually is decent with that?
• Separate "find files to back up" phase from actually copying them so that progress for the second part is accurate. Interesting idea.
• Does need a smart server -- though maybe that could be fixed without changing the format, or is not a compelling reason.
• Packfiles are entangled; no support for removing old backups; perhaps removing them will be risky?

So what is actually wrong or needs to be fixed relative to Bup, if anything?

rsync

As well as plain rsync, there are many systems built on rsync to pre-configure it for backups or to present a nicer interface. One example is Back in Time.

In some ways a very good option:

• Very fast at copying just the changes to modified files.
• Very good at pipelining to fill high-latency networks.
• Each file that's copied is immediately accessible, and backups can restore by copying the next file.
• Since the tree is stored unpacked, you can directly copy files or subtrees out, and can also use tools like diff and grep directly on the tree.
• If the archive is damaged, you can still copy out some parts of the tree.

But there are some drawbacks:

• rsync requires a smart server and can't be used with dumb cloud storage.
• Stores files unpacked, unencrypted, and uncompressed (unless, of course, this is done by the underlying filesystem)
• Can only represent things the destination filesystem can represent (usernames, permissions, etc) - however, some systems built on top of rsync such as Back in Time can store them externally
• The backup archive files will be writable (when the source files are writable) by the same user. So, it is fairly easy to accidentally and silently modify something in the archive.
• There is no overall stored checksum so no way to tell if the archive has been modified or damaged since the backup was made.

rsync has no built-in mechanism to keep multiple increments of a backup. The typical practice is to build a tree of hardlinks, and rsync can be configured to automatically break the hardlinks when the files change. This has some drawbacks:

• If a small change is made to a large file, only the delta will be sent across the network. However, it will use disk space for a new entire copy.
• It is not easy to see how much space each increment uses (though this can in principle be scripted.)
• One might like to backup to USB, Windows, or CIFS filesystems that don't support (or expose) hardlinks.

And not perfect at incremental operation:

• Recent versions don't require sending a whole file list before sending files but it does always start scanning the whole filesystem from scratch, so a short run may not make progress

Bacula

Bacula storage format seems very tape-oriented, with no random access.

Does have the interesting feature of recording inaccessible files in the backup and why they were inaccessible: permission denied, etc.

TODO: More study

Camlistore

Camlistore (https://camlistore.org/doc/overview):

Differences:

Conserve is focused only on storing filesystem trees, whereas Camlistore can store file trees and also unstructured data such as photos. Camlistore can extract data from social networks. You can look at Conserve as being less ambitious, or more focussed.

Camlistore indexes the contents of what it stores and can search them and Conserve probably would not.

Both want to offer a fuse and web interface to look at the archive.

Similarities:

• Storage, once written, is physically read-only until deleted.

• Targets cloud storage, assuming only GET, PUT, and LIST operations.

Borg

https://github.com/borgbackup/borg, https://borgbackup.readthedocs.io/en/stable/

Borg is a fork of Attic.

Borg's and Attic's charter and basic storage format is very similar to that of Conserve.

Notable differences:

• Borg has been around longer, has more contributors, and has been far more extensively exercised.

• Borg has some features that Conserve would like to have but does not yet, including:

  • Mountable as a filesystem
  • Web viewer

• Borg backups can be resumed if they are interrupted, although this seems to require later manual cleanup of the checkpoints. However, Borg loses its work if interrupted in the middle of a single large file, which Conserve should accommodate.

• Borg is in Python plus some C and Cython, and Conserve is in Rust:

  • Having written lots of Python, I (@sourcefrog) don't want to use it any more for production software that is performance-critical, reliable, a security boundary, parallel, or that deals with Unicode. Backup systems are all of these.
  • Rust is potentially faster for CPU-bound operations.
  • Python is basically able to only use a single CPU core while Conserve should be able to parallelize extensively for compression and IO. (But this is not implemented yet.)
  • Borg implements its own encryption format. I have considered encryption in a future Conserve format but might defer this to a lower-level filesystem.

• Borg implements its own encryption format. I have mixed feelings about doing this in Conserve rather than relying on an underlying encrypted filesystem.

• Borg relies on a local cache directory, without which it will not deduplicate and which can reported grow too big. In Duplicity the local cache seemed to be a source of fragility and potentially poor performance and there are indications it may be the same in Borg. In Conserve I would like to avoid relying on a cache for typical cases although obviously sometimes it can be a win (e.g., repeated interrogation of a remote archive.)

• Borg has its own protocol over SSH; for Conserve I would probably at first just use remote filesystem access.

• Borg stores (shared and exclusive) locks in the archive which may not work well on weakly consistent network or cloud filesystems.

• Borg chunks can be shared across multiple archives of different directories, and arbitrarily shared across different versions of those archives. I'm not sure yet how they manage garbage collection of chunks.

Duplicati

To judge from https://github.com/duplicati/duplicati/wiki/Developer-documentation the ideas are very similar, and as of 2017 it is much more complete.

Duplicati seems to rely on a local database (although with an option to recreate it), which I would like to avoid in Conserve because this has been a failure point in Duplicity.

I can't find enough details on the format to assess that.

TODO: More details, https://github.com/sourcefrog/conserve/issues/63