TheRosiek.com Random tech notes and tutorials

Duplicity Install and Backup Samples

Duplicity is a backup tool that works off of rsync and rdiff libraries to copy only changes to a backup location. It can use compression and encryption tools on the data and also has the ability to save to Amazon's S3 service. More details can be found here.

• Installation on OpenBSD 4.4
• Installation on OpenBSD 4.6
• Installation on Debian Lenny 5.0
• Sample Backup Scripts

Installation on OpenBSD 4.4

The 4.4 version was the most difficult to get working since the majority of the issues came from the given OpenBSD libraries. Even installing the Duplicity port from the packages didn't function right.

First we need to add a few packages. You can use the pkg_add function with whatever mirror to obtain the following, some depend on others so there will be others in the file install list:

• python-2.5.2p4
• py-boto-1.3
• gpgme-1.1.5
• librsync-0.9.7
• ncftp-3.2.1

When the main Python package is installed, it will ask you to create a few symbolic links, so create those.

Version 4.4 needs a separate Python XML package to work properly. If it's not installed, you'll get a series of errors when trying to send data to S3; I believe the XML error is when it tries to read the response. Something like this will error out:

To avoid that, a separate Python XML package needs to be downloaded and installed:

Now we can install Duplicity.

If you run the Duplicity jobs as root in a cron job, there is something about OpenBSD (I'm sure a security issue) that causes it to fail. I would get the output below in my log only when it ran as a cron job:

Odd that it can't read the temporary folder that it created. Changing the folder location also did not work. The solution is to create a separate user for only backups. The can be an issue if you have files that cannot be read by all users and need backup, but I found in my case this worked for the specific files that needed to be saved.

Make sure to add the new user to the deny list in SSH with DenyUsers dpbackup in the file /etc/ssh/sshd_config; there isn't any reason for it to log in.

Now su as this new user. A GPG key needs to be created so that the compressed backups can be encrypted and signed. This way no one else that may have access to our S3 account (Amazon employees) can read the data.

There will be a series of questions, most of the defaults are fine.

• Choose option 1 for DSA and Elgamal (the default)
• Choose the default key size of 2048
• Leave the default that the key will not expire, option 0
• Enter a User ID, Email address, and comment for the key.
• Type O for OK to accept.
• Enter a long passphrase for the key and allow it to be generated. I usually do at least 20 characters since the password will just sit in a script anyway.

Move the keys to some other safe place so that they can't be lost. No key means the backups are worthless. Typically a second backup source is a good idea.

See sample scripts below for backup jobs.

Installation on OpenBSD 4.6

First we need to add a few packages. You can use the pkg_add function with whatever mirror to obtain the following, some depend on others so there will be others in the file install list:

• python-2.5.4p1
• py-xml-0.8.4p8
• py-boto-1.7a
• gpgme-1.1.5p0
• librsync-0.9.7p0
• ncftp-3.2.2

When the main Python package is installed, it will ask you to create a few symbolic links, so create those.

Now we can install Duplicity.

If you run the Duplicity jobs as root in a cron job, there is something about OpenBSD (I'm sure a security issue) that causes it to fail. I would get the output below in my log only when it ran as a cron job:

Odd that it can't read the temporary folder that it created. Changing the folder location also did not work. The solution is to create a separate user for only backups. The can be an issue if you have files that cannot be read by all users and need backup, but I found in my case this worked for the specific files that needed to be saved.

Make sure to add the new user to the deny list in SSH with DenyUsers dpbackup in the file /etc/ssh/sshd_config; there isn't any reason for it to log in.

Now su as this new user. A GPG key needs to be created so that the compressed backups can be encrypted and signed. This way no one else that may have access to our S3 account (Amazon employees) can read the data.

There will be a series of questions, most of the defaults are fine.

• Choose option 1 for DSA and Elgamal (the default)
• Choose the default key size of 2048
• Leave the default that the key will not expire, option 0
• Enter a User ID, Email address, and comment for the key.
• Type O for OK to accept.
• Enter a long passphrase for the key and allow it to be generated. I usually do at least 20 characters since the password will just sit in a script anyway.

Move the keys to some other safe place so that they can't be lost. No key means the backups are worthless. Typically a second backup source is a good idea.

See sample scripts below for backup jobs.

Installation on Debian Lenny 5.0

The Debian install is a little bit simpler and can run the backup job as root inside cron. Get some install packages first:

Install Duplicity:

Creating a user is optional, but good security practice for it not to be root.

Make sure to add the new user to the deny list in SSH with DenyUsers dpbackup in the file /etc/ssh/sshd_config; there isn't any reason for it to log in.

Now su as this new user. A GPG key needs to be created so that the compressed backups can be encrypted and signed. This way no one else that may have access to our S3 account (Amazon employees) can read the data.

There will be a series of questions, most of the defaults are fine.

• Choose option 1 for DSA and Elgamal (the default)
• Choose the default key size of 2048
• Leave the default that the key will not expire, option 0
• Enter a User ID, Email address, and comment for the key.
• Type O for OK to accept.
• Enter a long passphrase for the key and allow it to be generated. I usually do at least 20 characters since the password will just sit in a script anyway.

Move the keys to some other safe place so that they can't be lost. No key means the backups are worthless. Typically a second backup source is a good idea.

See sample scripts below for backup jobs.

Sample Backup Scripts

The first portion of the script defines the variables we'll need to use. The AWS keys are defined for you when you sign up for S3. Passphrase is the GPG passphrase set on the key generated from gpg --gen-key. The S3 bucket should be fairly unique, so I use the host name of the server. The others are pretty obvious but will be explained later.

Just some sample backup methods for MySQL or Subversion if needed.

This is only necessary on OpenBSD since it's a security feature. We open it up now from 128 and close it back down later.

Most of these options can be read in the man page of Duplicity, and there are many more to choose from. Basically this backup is going to do a full backup ever 7 days (from the $FULL_IF_OLDER_THAN variable), and use encryption with the highest bzip compression, before sending it to S3. It will write a fresh backup log to the defined file, which we'll email out later.

This line just gives us some space in the log file; really it's just for email formatting.

This command will check how many full backup sets are already on S3, and remove any more than what is defined in KEEP_MAX_SETS.

Again, for formatting purposes.

This command lists out the current files in our backup set so they can be reviewed in the email, making sure everything is working out it should.

Now we can mail out the log file. The -s flag is for the subject line, and the TO_EMAIL is defined in our variables. We're just writing the log file as the body of the email.

Since we exported the keys and passphrases, we want to make sure we don't leave those around any longer than we have to; set them null.

Just a little clean up so we don't waste space.

This is for OpenBSD only. Since we opened the open file limit up at the beginning of the script, close it back down.

End it.