Duplicity: a backup and restore tool

Tux

I like Duplicity for backing up data on my home Linux kit because it meets certain criteria. Bear in mind that these may not be important to you!

  1. Duplicity is open source.
  2. It can be used on the command line, from the Linux shell. But for those that prefer it, there are GUI front-end options (eg, Deja Dup for GNOME).
  3. It can take full and incremental backups.
  4. Backup files are stored in tar gzip format. This was a plus for me. I wanted a format that could be relatively easily read by basic Linux tools that are always likely to be at hand.
  5. Backups can be encrypted at rest (using GnuPG).
  6. Duplicity can integrate with cloud storage – eg, S3 on AWS.

Reasons I wouldn’t use Duplicity for critical systems and data:

This is despite current versions of GNU tar being able to deal with extended file attributes like ACLs and SELinux file contexts via the ‘–xattr’ and ‘–selinux’ switches respectively.

I came up against this Duplicity limitation when restoring my laptop’s /home filesystem one day, losing SELinux contexts in the process. Thankfully I didn’t have any non-standard SELinux file contexts in place, so a simple ‘restorecon -Rv /home‘ got things back to normal.

I think it’s too risky to use Duplicity for backing up critical systems and data until it has robust support for extended file attributes and SELinux file contexts.

Installation on Fedora

Available from the default Fedora software repositories. Basically, install using:

$ sudo dnf install duplicity

See https://fedoramagazine.org/taking-smart-backups-duplicity for details of how to configure backups.

Installation on Ubuntu

$ sudo apt-get install duplicity

See https://help.ubuntu.com/community/DuplicityBackupHowto for details of how to configure backups.

Duplicity example

Here are some examples of using Duplicity on Fedora 34.

Dry-run to an external drive

This example was run as the root user (or it could be run using sudo) because the folders being used for this dry run were owned by different user accounts.

# duplicity --dry-run --name mylaptop_home /home file:///media/1tbusb/backups/mylaptop/home-backups/
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption: 
Cannot use empty passphrase with symmetric encryption!  Please try again.
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1627774508.08 (Sun Aug  1 01:35:08 2021)
EndTime 1627774539.34 (Sun Aug  1 01:35:39 2021)
ElapsedTime 31.26 (31.26 seconds)
SourceFiles 301977
SourceFileSize 88829478341 (82.7 GB)
NewFiles 160403
NewFileSize 88829478341 (82.7 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 160403
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
-------------------------------------------------

# 

I left my user errors in the transcript above, as it’s helpful to see how Duplicity responds in some error scenarios. In this case, me entering an empty passphrase when symmetric encryption will be used for the resulting backup files.

It all looks good, let’s proceed with the backup.

Actual backup to external drive

Again, as the root user (or via sudo):

# duplicity --name mylaptop_home /home file:///media/1tbusb/backups/mylaptop/home-backups/
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
First and second passphrases do not match!  Please try again.
GnuPG passphrase for decryption: 
Retype passphrase for decryption to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1627774633.42 (Sun Aug  1 01:37:13 2021)
EndTime 1627778822.78 (Sun Aug  1 02:47:02 2021)
ElapsedTime 4189.36 (1 hour 9 minutes 49.36 seconds)
SourceFiles 160401
SourceFileSize 88829623775 (82.7 GB)
NewFiles 160401
NewFileSize 88829623775 (82.7 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 160401
RawDeltaSize 88815026613 (82.7 GB)
TotalDestinationSizeChange 78745337285 (73.3 GB)
Errors 0
-------------------------------------------------

# 

Again my errors are left in. In this case, the second passphrase I entered didn’t match the first one, so I was prompted to re-enter it.

Warning! Make sure you record the passphrase in a safe secure place. If you lose the passphrase, you will not be able to decrypt the backup files to read back data or restore!!

Running an incremental backup

To only backup content that has changed since the last full backup, simply re-run the command exactly as we did above.

Duplicity detects the presence of a full backup at the specified target location, and performs an incremental backup instead.

To list what’s been backed up

To list the status of backup sets at the target backup location:

# duplicity collection-status file:///media/1tbusb/backups/mylaptop/home-backups/
Last full backup date: Sun Aug  1 01:36:59 2021
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/ca2e105abb0be7fd8378849b99843152

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Aug  1 01:36:59 2021
Chain end time: Tue Sep 14 15:07:02 2021
Number of contained backup sets: 2
Total number of contained volumes: 461
 Type of backup set:                            Time:      Num volumes:
                Full         Sun Aug  1 01:36:59 2021               376
         Incremental         Tue Sep 14 15:07:02 2021                85
-------------------------
No orphaned or incomplete backup sets found.
# 

Duplicity reports finding two backup sets at that location, one full, one incremental. The dates of the backups are listed (and yes, it took me a while to actually write and publish this post!).

Testing a restore

A backup is only as good as its ability to be restored. So it’s a healthy practice to test your backups by restoring from them periodically.

I tested the backup above by doing a full restore to a location different to the original files that were backed up:

# duplicity restore file:///media/1tbusb/backups/mylaptop/home-backups/ /home/restored/

I was prompted to enter the same GnuPG passphrase that I’d used when backing up the data. Once that was done, Duplicity went ahead and restored the data. Apart from the issue I encountered with losing SELinux file contexts, the restore was successful.

Learn more about Duplicity

As always, this only scratches the surface of what duplicity can do. To find out more:

$ man duplicity

or

$ duplicity --help

or track down the official Duplicity documentation for your favourite Linux distro.

And before using Duplicity, make sure you’re aware of what it can and cannot offer you. As an example, consider its limited capability with extended file attributes, and how this could affect you in the event of data loss.

Remember:

A backup is only as good as its ability to be restored.

(I don’t think I’m the first person to say that!)

Feel free to share your thoughts below. What backup method do you use? Do you think you’ll use Duplicity? Why or why not?

5 thoughts on “Duplicity: a backup and restore tool

  1. You ‘n that durn Command Line! 😉 Just kidding…you professionals need it. I may test Duplicity anyway, just to see if the GUI front-end works wid Cinnamon & KDE. I’ve used Clonezilla for years as a full backup & restore for my Linux OSes ‘n some of their tests. Have never bothered wid incremental backups for either Linux or Windows…didn’t like them for some reason that I have now forgot?! 😉 Have tested Clonezilla on Windows also and it worked. However, I use Windows Backup & Restore app (Windows 7) on my primary Win machine. I do keep all my Data separate from my Win OSes…use a Data SSD that I manually backup to two other exterior drives weekly…a Samsung Portable SSD T7 & a SanDisk Extreme Portable SSD v2.

    “A backup is only as good as its ability to be restored.” Yep! 😉 Usually test the backup image for a new Linux OS I might want to save, but Clonezilla has been reliable for me. Main Windows gets backed-up every few day, to one SSD, then a few more days to another SSD just in case one backup ever fails.

    Great tip/s…as usual, Thanks!

    Liked by 1 person

    1. Hehe – yep! I ❤️ command line!

      Let me know what you think of Duplicity on KDE. I’m curious. I like Clonezilla too. It’s rock solid.

      Incremental backups can be a pain to restore from depending on how the tool implements them. Their only advantage is really the ability to backup things that have changed since the last full backup – so saving on time and space. But if one has time and space, my preference would be to go full each time. Much simpler!

      Thank you for your input in this discussion!

      Like

      1. “Incremental backups can be a pain to restore from depending on how the tool implements them.”

        Now I remember why I never liked incremental backups…could never get it restored right. Let business deal wid that stuff! 😉

        I backing up my Fedora 36 Cinnamon SPIN right now (Clonezilla), as I’m prepping to test Duplicity…and to also see if the GUI front-end works wid Cinnamon on it. Not sure if I’ll get to testing a KDE Distro, but will let you know how this test goes.

        Liked by 1 person

      2. Looks like it is for just Data backups, i.e. a full bare-metal restore of OS, settings, apps, data, etc. didn’t seem to be offered during first test. At least the “Backups” app installed by dnfdragora, the GUI front-end deja-dup for Duplicity, didn’t seem to offer that.

        Was the easiest Data backup to install & use that I recall ever testing. Backed up the Home folder to a test 120 GB SSD, and then restored the Home folder to a 62 GB USB…fast. Really FAST! Worth testing & tinkering wid some more, for sure.

        Most normal Linux users (not talking about the ‘Technical people‘ such as Developers, Programmers, IT Specialists, Maintainers, etc.) seem unable to figure out how to use Clonezilla, and I’ve been looking for an easier full bare-metal restore version to do a post on…that is at least as reliable as Clonezilla.

        Liked by 1 person

      3. Thank you Karmi, that’s very useful feedback. It makes sense that the tool wouldn’t offer bare-metal recovery. I wouldn’t recommend it for a full system backup and restore, given it’s inability to read extended file attributes for a start. But yes, as a plain data backup and restore it seems to work really well.

        I agree with you, and haven’t come across anything as versatile as Clonezilla – but also agree, it is not the most user-friendly! Best of luck hunting.

        Liked by 1 person

Leave a Reply to Karmi Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: