TKLBAM-FAQ

Frequently Asked Questions

Author: Liraz Siri <liraz@turnkeylinux.org>
Date: 2013-09-05
Manual section:7
Manual group:backup

GENERAL QUESTIONS

Is TKLBAM open source?

Yes, TKLBAM is licensed under the GPL3. You don't have to care about free software ideology to appreciate the advantages. Any code running on your server doing something as critical as encrypted backups should be available for peer review and modification.

Where can I use TKLBAM?

On any system descended from a TurnKey Linux installation, regardless of hardware or location. Storing backups to Amazon S3 is easiest because authentication and key management are automatic. You just need to run:

tklbam-backup

But you can also backup to any storage target supported by TKLBAM's back-end Duplicity including the local filesystem, NFS, Rsync, SSH, FTP, WebDAV, Rackspace CloudFiles and even IMAP.

The local filesystem is one of the easier storage targets to use because you don't need to mess around with authentication credentials.

So assuming you want to store your backup at /mnt/otherdisk:

tklbam-backup --address file:///mnt/otherdisk/tklbam/backup
tklbam-escrow /mnt/otherdisk/tklbam/key

And restore like this:

tklbam-restore --address file:///mnt/otherdisk/tklbam/backup \
               --keyfile=/mnt/otherdisk/tklbam/key

Not as easy as the Hub-enabled "automatic" mode, but still vastly easier than your conventional backup process. The disadvantage is that you won't be able to restore/test your backup in the cloud, or from a VM running in another office branch (for example). Also keep in mind that a physical hard disk, even a RAID array, provides much much lower data reliability compared with Amazon S3.

For this reason we recommend users use local backups to supplement cloud backups (e.g., providing fast local access).

Why can't I backup a non-TurnKey Linux system with TKLBAM?

TKLBAM is short for 'TurnKey Linux' Backup and Migration. It's designed specifically for TurnKey Linux and depends on many system-level details that don't necessarily apply to other Linux distributions (e.g., installation method, versioning signatures, etc.).

In the future, we may figure out how to extend the design to support additional operating systems, but it's not trivial and we don't have a timeline on when, or even if, that will happen.

In the meantime, if you really want to use TKLBAM, consider virtualization-based workarounds. For example, if you install a TurnKey Linux VM on top of a Windows Server installation, you could use TKLBAM to backup anything that goes into the TurnKey Linux VM.

Which TurnKey appliances are supported?

With a few exceptions, all TurnKey appliances images from the 2009.02 release batch onwards will work with TKLBAM.

Unfortunately, at this time PostgreSQL based appliances (PostgreSQL, LAPP, OpenBravo) are not yet supported.

PostgreSQL support is in the works but it's not ready yet.

Which databases are supported?

MySQL and PostgreSQL.

PRICING QUESTIONS

How much does this cost?

TKLBAM (and the TurnKey Hub) are currently free for private beta users. Amazon S3 storage fees are around $0.15/GB per month. Full details of Amazon S3 pricing can be found here.

You can use simulation mode to calculate how much uncompressed data TKLBAM is going to store in a full backup:

$ tklbam-backup --simulate
CREATING /TKLBAM
FULL UNCOMPRESSED FOOTPRINT: 148.30 MB in 6186 files

In practice, the actual footprint of a full backup will usually be smaller due to compression, but this depends on the type of data being compressed (e.g., text compresses very well, video very poorly).

By default, a full backup is performed if one month has passed since the last full backup. In between, incremental backups will be performed which only record changes since the last backup. The full backup frequency can be customized. See this manual page for details.

The Hub says my backup costs $0.00, what am I really paying?

If you notice $0.00 in the backups console, there's no need to open a support request. It's not a bug. At 15 cents per gigabyte, if you have just a few megabytes of data Amazon doesn't charge you anything.

Backups start from about 100KB for a freshly installed TurnKey appliance. Remember, TKLBAM only saves changes you've made since the appliance was installed.

In fact, a significant number of users are being charged less than 1 cent a month.

USAGE QUESTIONS

How does TKLBAM know what to backup on my system?

Every TurnKey appliance that TKLBAM supports has a corresponding backup profile, which is downloaded from the Hub the first time you backup an appliance. When required the profile can be updated on demand (e.g., if we need to fix the profile)

The profile is stored in /var/lib/tklbam/profile and contains the following text files:

  1. dirindex.conf: a list of directories to check for changes by default. This list does not include any files or directories maintained by the package management system.
  2. dirindex: appliance installation state - filesystem index
  3. packages: appliance installation state - list of packages

Users can override which files and directories are checked for changes by configuring overrides (See below).

Why is an override called an override?

Because it "overrides" the default backup configuration in the appliance profile.

How do I remove a file or directory from being included in my backup?

By adding a negative override to /etc/tklbam/overrides:

echo -/var/www/*/logs >> /etc/tklbam/overrides

You can also specify overrides on the command line at backup time:

tklbam-backup -- -/var/www/*/logs

How do I add a directory to my backup?

By adding an override to /etc/tklbam/overrides:

echo /mnt/images >> /etc/tklbam/overrides

Or on the command line:

tklbam-backup /var/www/*/logs

Make sure you understand the implications of doing this. For example, if you add a directory handled by package management this may break package management on the system you restore to.

How do I exclude a database or table from my backup?

By adding a negative database override to /etc/tklbam/overrides:

# exclude drupal5 database
echo -mysql:drupal5 >> /etc/tklbam/overrides

# exclude sessions table in drupal6 database
echo -mysql:drupal6/sessions >> /etc/tklbam/overrides

Or on the command line:

tklbam-backup -- -mysql:drupal6/page_cache

By default ALL databases are backed up so adding a negative database override excludes only that database or table from the backup.

Excluding a table only excludes its data. The schema is still backed up as long as the database is included.

Specifying a positive database override changes the default behavior so that only the database or table specified in the override is included in the backup.

You can mix positive overrides with negative overrides.

Can I have multiple TKLBAM backups on a single system?

Yes. For example, let's say your default TKLBAM backup is several gigabytes in size and you'd like to create a lighter 100 MB backup that will be updated more frequently and take less time to update/restore.

cp -a /etc/tklbam /etc/tklbam.light

echo -/var/www/*/logs >> /etc/tklbam.light/overrides echo -/home/liraz/bigfiles >> /etc/tklbam.light/overrides echo -mysql:mydatabase/bigtable >> /etc/tklbam.light/overrides

export TKLBAM_CONF=/etc/tklbam.light

mkdir /var/lib/tklbam.light export TKLBAM_REGISTRY=/var/lib/tklbam.light

tklbam-init tklbam-backup

For convenience you may want to create a script that sets the TKLBAM_REGISTRY and TKLBAM_CONF environment variables:

cat > /usr/local/bin/tklbam-backup-light << EOF
#!/bin/bash
export TKLBAM_CONF=/etc/tklbam.light
export TKLBAM_REGISTRY=/var/lib/tklbam.light

tklbam-backup
EOF

chmod +x /usr/local/bin/tklbam-backup-light

Can I use TKLBAM to only backup a single directory?

Yes. There are a couple of ways to do this:

  1. The hard way: Configure TKLBAM overrides to create a system-level backup that only backs up a single directory. Tweak the configuration, test it with tklbam-backup --simulate, rinse repeat until your backup doesn't include anything you don't want.

  2. The easy way: Use the --raw-upload option. This dashes TKLBAM's brains out so instead of creating a system level backup it just backs up the directory you specify. In other words, --raw-upload turns TKLBAM into a directory-level backup tool rather than a system-level backup tool.

    For example, let's say you have a collection of big files that you don't want to include in your system backup because you don't want to bloat your backup.

    So you configure an overrides to exclude that directory from your default backup and create another TKLBAM backup just for the big files:

    echo -/home/liraz/bigfiles >> /etc/tklbam/overrides
    
    export TKLBAM_REGISTRY=/var/lib/tklbam.bigfiles tklbam-backup
    --raw-upload=/home/liraz/bigfiles
    

    Keep if you use the --raw-upload option you'll also need to use the --raw-download option when you restore:

    tklbam-restore --raw-download=/home/liraz/bigfiles <your-backup-id>
    

    If you don't use the raw-download option, TKLBAM will assume you are trying to restore a system-level backup and you'll probably get an error.

What's the difference between a full backup and an incremental backup?

A full backup is a backup that can be restored independently of any other backup. An incremental backup links with the last backup before it and only includes changes made since.

Backup chains are links of backup sessions which start with a full backup, and then a series of incremental backups each recording only the changes made since the backup before it. Incremental backups are useful because they are fast and efficient.

Restoring an incremental backup requires retrieving the volumes of all backup sessions made before it, up to and including the full backup that started the chain. The longer the backup chain, the more time it will take to restore.

How often does a does a full backup happen, how can I configure this?

By default, a full backup will happen if the last full backup is older than 30 days. Between full backups, all backup sessions are incremental.

We recommend enabling the daily backup cron job so that daily incremental backups happen automatically:

chmod +x /etc/cron.daily/tklbam-backup

You can override the default by setting the full-backup parameter in the tklbam configuration:

# create a full backup every 14 days
echo full-backup 14D >> /etc/tklbam/conf

I forgot my passphrase, and I "lost" my escrow key. Can you help me?

Sorry, if your server is gone (e.g., terminated EC2 instance) nobody can help you. Next time either save an escrow key somewhere[s] safe or don't set a passphrase.

Don't misunderstand, we'd love to help if we could, but we can't. The encryption key for your backup was generated locally on your server not ours. We designed passphrase protection to use special cryptographic countermeasures to make typical cracking techniques (e.g., dictionary attacks) very difficult even for someone with access to massive amounts of computer resources.

Note, if the system you backed up is still available, just log into it as root and change the passphrase (you don't need to know the old passphrase):

tklbam-passphrase

AMAZON S3 QUESTIONS

Do I have to use Amazon S3 for storage?

No. You can dump the raw TKLBAM backup to a directory and then store it anyhow you want.

For example here's how we'd a system backup into a simple unencrypted tarball:

cd /tmp
mkdir mybackup
tklbam-backup --dump=mybackup/
tar jcvf mybackup.tar.bz2 mybackup/

And later restore it like this:

cd /tmp
tar jxvf mybackup.tar.bz2
tklbam-restore mybackup/

The --dump option bypasses Duplicity, which usually create a series of encrypted archive files that can be incrementally updated. These archive files are stored by default in the Amazon S3 storage cloud but you can override this with the --address option and specify any storage back-end supported by Duplicity (e.g., local directory, rsync over ssh, ftp, sftp, etc).

Duplicity can be forced by adding the --address option when you backup and restore but make sure you know what you're doing because doing this complicates usage.

The Hub normally helps you manage your backups metadata when it auto-configures the storage address. If you specify a manual address you need to manage storage locations, encryption keys and authentication credentials by hand.

Many things can go wrong so please be careful and always rehearse a restore.

Why can't I access TKLBAM storage buckets with other Amazon S3 tools?

TKLBAM doesn't store it's data in generic S3 buckets, but in an isolated TKLBAM-specific area on S3. This means generic S3 tools such as the AWS management console, or S3Fox will not be able to access the storage buckets in which TKLBAM backup volumes reside.

What are the advantages of isolating TKLBAM Amazon S3 storage?

  1. Easier sign up process. Users don't need to know anything about S3 API keys or understand the implications of giving them to us.
  2. Security: you don't need to give us access to your generic S3 account. If someone compromises your regular AWS API Key they still can't get to your encrypted backup volumes and say... delete them.
  3. Cost transparency: TKLBAM related storage charges show up separately from your generic S3 storage.

What happens if my payment method to Amazon is invalidated?

Amazon supports payment by credit card and bank account. We recommend heavy users add a bank account as their payment method, as it's usually more permanent than a credit card.

In any case, if your payment method is invalidated (e.g., cancelled or expired credit card), billing will fail and Amazon will attempt to contact you (e.g., by e-mail) to provide a new, valid payment method.

FAULT TOLERANCE FOR THE PARANOID IT GUY

Is the Hub TKLBAM's central point of failure?

Yes and no. On one hand, much of the streamlined usability of TKLBAM depends on the availability of the Hub. On the other hand, we designed TKLBAM to degrade gracefully if the Hub ever goes down (it shouldn't!).

As we scale the Hub we will gradually add capacity and build in additional layers of fault tolerance.

We have monitoring in place which alerts us immediately if anything unexpected happens.

If the Hub goes down, will my backup cron jobs still work?

Yes. Backups which have already been configured will continue to work normally. If TKLBAM can't reach the Hub it just uses the locally cached profile and S3 address.

If my connection to the Hub goes down, can I still restore?

Yes - manually. It just won't be as easy. You'll need to do a couple of steps by hand:

  1. transfer the escrow key to the restore target.

    This means you'll need to have stored the escrow key somewhere safe or be able to create it on the backed up machine.

  2. specify the S3 address and the key manually when you restore.

    For more details see the tklbam-restore documentation.

If the Hub goes down, can I still create a new backup?

Yes - but only manually. Just remember the Hub won't know anything about these backups so you'll have manage keys and authentication credentials by hand.

SEE ALSO

tklbam (8)