Duplicity: a backup and restore tool

I like Duplicity for backing up data on my home Linux kit because it meets certain criteria. Bear in mind that these may not be important to you!

1. Duplicity is open source.
2. It can be used on the command line, from the Linux shell. But for those that prefer it, there are GUI front-end options (eg, Deja Dup for GNOME).
3. It can take full and incremental backups.
4. Backup files are stored in tar gzip format. This was a plus for me. I wanted a format that could be relatively easily read by basic Linux tools that are always likely to be at hand.
5. Backups can be encrypted at rest (using GnuPG).
6. Duplicity can integrate with cloud storage – eg, S3 on AWS.

Reasons I wouldn’t use Duplicity for critical systems and data:

• It’s support for extended file attributes (xattr) like SELinux file contexts is limited. For those who’re interested, some discussions on this are below, but basically Duplicity appears to use an implementation of tar that isn’t capable of backing up xattr.
  • https://duplicity.gitlab.io/new_format.html
  • https://gitlab.com/duplicity/duplicity/-/issues/134

This is despite current versions of GNU tar being able to deal with extended file attributes like ACLs and SELinux file contexts via the ‘–xattr’ and ‘–selinux’ switches respectively.

I came up against this Duplicity limitation when restoring my laptop’s /home filesystem one day, losing SELinux contexts in the process. Thankfully I didn’t have any non-standard SELinux file contexts in place, so a simple ‘restorecon -Rv /home‘ got things back to normal.

I think it’s too risky to use Duplicity for backing up critical systems and data until it has robust support for extended file attributes and SELinux file contexts.

Installation on Fedora

Available from the default Fedora software repositories. Basically, install using:

$ sudo dnf install duplicity

See https://fedoramagazine.org/taking-smart-backups-duplicity for details of how to configure backups.

Installation on Ubuntu

$ sudo apt-get install duplicity

See https://help.ubuntu.com/community/DuplicityBackupHowto for details of how to configure backups.

Duplicity example

Here are some examples of using Duplicity on Fedora 34.

Dry-run to an external drive

This example was run as the root user (or it could be run using sudo) because the folders being used for this dry run were owned by different user accounts.

# duplicity --dry-run --name mylaptop_home /home file:///media/1tbusb/backups/mylaptop/home-backups/
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption: 
Cannot use empty passphrase with symmetric encryption!  Please try again.
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1627774508.08 (Sun Aug  1 01:35:08 2021)
EndTime 1627774539.34 (Sun Aug  1 01:35:39 2021)
ElapsedTime 31.26 (31.26 seconds)
SourceFiles 301977
SourceFileSize 88829478341 (82.7 GB)
NewFiles 160403
NewFileSize 88829478341 (82.7 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 160403
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
-------------------------------------------------

# 

I left my user errors in the transcript above, as it’s helpful to see how Duplicity responds in some error scenarios. In this case, me entering an empty passphrase when symmetric encryption will be used for the resulting backup files.

It all looks good, let’s proceed with the backup.

Actual backup to external drive

Again, as the root user (or via sudo):

# duplicity --name mylaptop_home /home file:///media/1tbusb/backups/mylaptop/home-backups/
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
First and second passphrases do not match!  Please try again.
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1627774633.42 (Sun Aug  1 01:37:13 2021)
EndTime 1627778822.78 (Sun Aug  1 02:47:02 2021)
ElapsedTime 4189.36 (1 hour 9 minutes 49.36 seconds)
SourceFiles 160401
SourceFileSize 88829623775 (82.7 GB)
NewFiles 160401
NewFileSize 88829623775 (82.7 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 160401
RawDeltaSize 88815026613 (82.7 GB)
TotalDestinationSizeChange 78745337285 (73.3 GB)
Errors 0
-------------------------------------------------

# 

Again my errors are left in. In this case, the second passphrase I entered didn’t match the first one, so I was prompted to re-enter it.

Warning! Make sure you record the passphrase in a safe secure place. If you lose the passphrase, you will not be able to decrypt the backup files to read back data or restore!!

Running an incremental backup

To only backup content that has changed since the last full backup, simply re-run the command exactly as we did above.

Duplicity detects the presence of a full backup at the specified target location, and performs an incremental backup instead.

To list what’s been backed up

To list the status of backup sets at the target backup location:

# duplicity collection-status file:///media/1tbusb/backups/mylaptop/home-backups/
Last full backup date: Sun Aug  1 01:36:59 2021
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/ca2e105abb0be7fd8378849b99843152

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Aug  1 01:36:59 2021
Chain end time: Tue Sep 14 15:07:02 2021
Number of contained backup sets: 2
Total number of contained volumes: 461
 Type of backup set:                            Time:      Num volumes:
                Full         Sun Aug  1 01:36:59 2021               376
         Incremental         Tue Sep 14 15:07:02 2021                85
-------------------------
No orphaned or incomplete backup sets found.
# 

Duplicity reports finding two backup sets at that location, one full, one incremental. The dates of the backups are listed (and yes, it took me a while to actually write and publish this post!).

Testing a restore

A backup is only as good as its ability to be restored. So it’s a healthy practice to test your backups by restoring from them periodically.

I tested the backup above by doing a full restore to a location different to the original files that were backed up:

# duplicity restore file:///media/1tbusb/backups/mylaptop/home-backups/ /home/restored/

I was prompted to enter the same GnuPG passphrase that I’d used when backing up the data. Once that was done, Duplicity went ahead and restored the data. Apart from the issue I encountered with losing SELinux file contexts, the restore was successful.

Learn more about Duplicity

As always, this only scratches the surface of what duplicity can do. To find out more:

$ man duplicity

or

$ duplicity --help

or track down the official Duplicity documentation for your favourite Linux distro.

And before using Duplicity, make sure you’re aware of what it can and cannot offer you. As an example, consider its limited capability with extended file attributes, and how this could affect you in the event of data loss.

Remember:

A backup is only as good as its ability to be restored.

(I don’t think I’m the first person to say that!)

Feel free to share your thoughts below. What backup method do you use? Do you think you’ll use Duplicity? Why or why not?