There are various ways to backup data on a computer, and there are various tools that help you with this task. In this blog post, we are going to take a look at a command-line backup tool called restic.
restic ¶
restic is an easy, fast, efficient, and secure backup utility. It is free and licensed under the BSD 2-Clause License.
It is actively developed on GitHub: https://github.com/restic/restic
restic is written in Go, and it runs on Linux, Windows, macOS, FreeBSD, and OpenBSD.
You can download the binary from the release page: https://github.com/restic/restic/releases
Or you install the application with a package manager. Visit the installation documentation page to find
the installation steps for your operating system.
For this blog post, I worked on Windows 10 with restic 0.9.6.
I downloaded the zip file from the release page, unzipped it, and copied the exe file it into an arbitrary folder.
On Windows, you could copy the exe into the %SystemRoot%\System32 folder, and then it's available in the path automatically.
Or you add the path to your restic folder to the %PATH% environment variable.
restic supports out of the box many different backends:
- Local directory
- SFTP
- HTTP REST server
- AWS S3 (or self hosted Minio Server)
- OpenStack Swift
- BackBlaze B2
- Microsoft Azure Blob Storage
- Google Cloud Storage
If your preferred backend is not listed here, you can couple restic with rclone and use all the supported service of this tool. Visit the restic documentation to learn more about rclone support. And visit the rclone homepage to see a list of all supported backends.
restic is not a trivial file copy program. It works a little bit differently. It is based around the concept of repositories and snapshots. When you backup something, it is stored in a snapshot, which is stored in a repository.
The data in such a repository is always encrypted with AES. This is especially important when you back up sensitive data, and you send the backups to a server which is not under your control, for example to one of the cloud providers.
Initialize Repository ¶
Before we can backup something, we have to initialize the repository. For this example, I connected an external hard disk (z:),
and there I created an empty folder backup.
Run the following command to initialize the repository.
restic init --repo z:/backup
restic now asks you for the password. You have to remember this password. If you lose it, you can no longer access the backup!
enter password for new repository:
enter password again:
created restic repository 5db7847a29 at z:/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
Backup ¶
The repository is created, and we can start backing up data. Each time you run a backup, restic creates a snapshot in the repository. To back up one or multiple folders or files, enter this command
restic backup -r z:/backup c:/work d:/another_work_folder
-r is the short form of --repo and tells restic which repository to use.
At the end of the command, you list files and directories you want to backup. In this example
I backup two folders, which are located on two different local disks.
restic asks for your password and backs up the data. For demo purposes, I created one text file in each folder so restic had to transfer two files, which it tells me in the summary output.
enter password for repository:
repository 5db7847a opened successfully, password is correct
created new cache in C:\Users\sr\AppData\Local\restic
Files: 2 new, 0 changed, 0 unmodified
Dirs: 2 new, 0 changed, 0 unmodified
Added to the repo: 1.179 KiB
processed 2 files, 20 B in 0:00
snapshot 1ba98702 saved
Rerun the same command and see one of the benefits of restic. It created another snapshot, but it did not copy any files because there were no changes. This saves space on the backup drive, and it saves time and bandwidth when you back up over a network.
enter password for repository:
repository 5db7847a opened successfully, password is correct
Files: 0 new, 0 changed, 2 unmodified
Dirs: 0 new, 0 changed, 2 unmodified
Added to the repo: 0 B
processed 2 files, 20 B in 0:00
snapshot 47a15bab saved
The backup command also supports command-line options to exclude and include files based on patterns. See the official documentation to dig deeper into this topic.
A tool like this is ideal for an automated backup script. The problem is that restic always requires a password to access the repository and to encrypt the data. The secure way would be that you always enter the password manually, which is not an option when you want to install a scheduled job that runs unattended.
There are different ways to specify the password and pass it to restic.
You can set the environment variable RESTIC_PASSWORD, or you can store the password in a file
and then specify the path to that file with the option --password-file or the environment variable RESTIC_PASSWORD_FILE.
The third option is to call a program when a password is needed with the option --password-command or the environment variable RESTIC_PASSWORD_COMMAND.
Different backends require different environment variables. On this page you find all supported environment variables, and this page tells you how to configure the supported backends.
Restore ¶
The restore command restores files and folders. restic does not restore the files into the original path.
You have to specify a target location where restic restores the files.
restic restore 47a15bab -r z:/backup --target c:/restore
Instead of specifying the snapshot ID, you can also use the keyword latest to restore the latest backup.
restic restore latest -r z:/backup --target c:/restore
Like the backup command, restore supports filter options, if you want to restore just a host or a path or just a couple of files.
The following example restore just the file workfile.txt
restic restore latest -r z:/backup --target c:/restore --include workfile.txt
On supported operating systems, you can mount the repositories with FUSE and then browse them like a regular folder. This is not supported on Windows.
If you want to learn more about the restore command, visit the documentation page.
Manage Snapshots ¶
List ¶
We called the backup command twice, so there are now two snapshots in our backup repository.
To list them, we use the snapshots command.
$ restic snapshots -r z:/backup
enter password for repository:
repository 5db7847a opened successfully, password is correct
ID Time Host Tags Paths
-----------------------------------------------------------------------------
1ba98702 2020-04-01 16:35:38 rog D:\another_work_folder
c:\work
47a15bab 2020-04-01 16:38:55 rog D:\another_work_folder
c:\work
-----------------------------------------------------------------------------
2 snapshots
If you have many snapshots, you can filter them by path and host, and you can group them. See the documentation page for more information.
Diff ¶
The diff command compares two snapshots with each other
$ restic diff -r z:/backup 1ba98702 47a15bab
enter password for repository:
repository 5db7847a opened successfully, password is correct
comparing snapshot 1ba98702 to 47a15bab:
Files: 0 new, 0 removed, 0 changed
Dirs: 0 new, 0 removed
Others: 0 new, 0 removed
Data Blobs: 0 new, 0 removed
Tree Blobs: 0 new, 0 removed
Added: 0 B
Removed: 0 B
Check Integrity ¶
From time to time, you should check the integrity of the repository. The data in the repository is encrypted but the whole repository
is just a collection of files and folders, and somebody could overwrite or delete a file. The check
command verifies the repository and tells you if everything is okay or not.
restic check -r z:/backup
repository 5db7847a opened successfully, password is correct
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
no errors were found
Cleanup ¶
To remove a snapshot restic provides two commands forget and prune. The forget command only deletes the snapshot from
the index. The snapshot is no longer visible when you run the snapshots command, but the data is still there. To actually remove the data, you have to call the prune command.
restic forget -r z:/backup 1ba98702
restic prune -r z:/backup
Because you often use these two commands in tandem, you can also add the --prune option to the forget command.
This command has the same effect as the two commands above
restic forget --prune -r z:/backup 1ba98702
The documentation
recommends to run a check after prune.
Deleting snapshots manually is cumbersome, and it's easy to delete the wrong snapshot accidentally. For these reasons, restic provides a way to delete snapshots based on a policy. A policy could be to keep the last 3 snapshots or to keep snapshots of the last 20 days.
The forget command supports various keep-* parameters.
For example to delete all but the last snapshot you can execute this command
restic forget --prune --keep-last 1 -r z:/backup
If you want to keep the snapshots of the last 7 days, you issue this command. If there are multiple snapshots in a day, restic removes all but the last one for that day.
restic forget --prune --keep-daily 7 -r z:/backup
For example, you backup 6 times a day for 10 days. You now have 60 snapshots. After you run this command, there are 7 snapshots left in the repository. One snapshot for each of the last 7 days.
You can even combine multiple policies.
restic forget --prune --keep-daily 7 --keep-monthly 12 --keep-yearly 3 -r z:/backup
With this command forget keeps one snapshot for each of the last 7 days, always the last snapshot of a month
for the last 12 months and always the last snapshot of a year for the last 3 years.
See the documentation for more example and a more in-depth description of the options.
Rest Server ¶
Backup and restore works, but there is one issue we have to keep in mind. restic always requires a password to access the repository and to encrypt the backup. When you install a scheduled job, you have to store this password somewhere, as mentioned before, you can store it in an environment variable or a file. The problem is that when an attacker gains control over your computer, he could also figure out this password. He then could not only delete the data on your computer but also all the snapshots in the restic repository.
If that is a concern, you should consider installing a restic REST server. This server has a unique feature called
append-only mode. When the server runs in this mode, you can backup your data normally but the server refuses to execute the forget
command, even when an attacker has access to your restic password. Your backups are secure and can't be deleted,
as long as your backup server is secure.
In this section, I'm showing you how to install this REST server on a virtual private server. I rented a Debian 10 VPS from Hetzner (referral link), using the smallest configuration for €2.68 / per month with a 20 GB SSD for this example installation.
Install ¶
First, we download the server application from the GitHub release page. The server, like the client, is written in Go and consists of just one binary.
mkdir /opt/restic
mkdir /opt/restic/data/
cd /opt/restic
wget https://github.com/restic/rest-server/releases/download/v0.10.0/rest-server_0.10.0_linux_amd64.tar.gz
tar xzf rest-server_0.10.0_linux_amd64.tar.gz
mv rest-server_0.10.0_linux_amd64\rest-server .
rm -fr rest-server_0.10.0_linux_amd64
chmod 744 rest-server
Next, we create a password. This step is optional, and you can run the server without authentication. The password we create here is just for authenticating to the rest server. This password has nothing to do with the repository password. The repository password is still required but will always be set from the backup client.
For creating the password, we need the htpasswd program, which is part of the Apache utils package.
apt install apache2-utils
Then we create a password and encrypt it with bcrypt. Choose an arbitrary username. The password file needs to be located in the data directory.
In our installation this is /opt/restic/data
cd /opt/restic/data
htpasswd -B -c .htpasswd backup_user
I don't want to run the rest-server with the root user, so I create a new user here, and change ownership of the program directory.
adduser --system --shell /bin/false --gecos 'Restic Rest Server user' --group --disabled-password restic
chown restic:restic -R /opt/restic
Now we install the service in systemd so that the rest-server will be started when the server starts.
cd /opt/restic
nano rest-server.service
Paste the following code into the file, save and close it.
[Unit]
Description=Rest Server
After=syslog.target
After=network.target
[Service]
Type=simple
User=restic
Group=restic
ExecStart=/opt/restic/rest-server --append-only --path /opt/restic/data
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
The important flag here is --append-only. This is the flag we talked about before to prevent somebody from executing the forget command.
ln -s /opt/restic/rest-server.service /lib/systemd/system/rest-server.service
systemctl daemon-reload
systemctl start rest-server
systemctl enable rest-server
Check if the service runs properly
systemctl status rest-server
The rest-server runs by default on port 8000. Make sure that you open this port in the firewall, or change the port
with the --listen option
/opt/restic/rest-server --listen ":55443" --append-only --path /opt/restic/data
Backup ¶
As with any backend, we first have to initialize the repository. Instead of a path, we now have to specify the URL to the rest-server. Because we also configured username and password, we have to pass this information as well to the tool.
restic init -r rest:http://backup_user:password@95.217.184.211:8000/
If that command is successful, you can now backup to your own restic rest-server
restic backup -r rest:http://backup_user:password@95.217.184.211:8000/ c:/work d:/another_work_folder
Now let's see what happens when we try to delete the snapshot.
restic snapshots -r rest:http://backup_user:password@95.217.184.211:8000/
restic forget -r rest:http://backup_user:password@95.217.184.211:8000/ 075b5b93
enter password for repository:
repository f03e5278 opened successfully, password is correct
Remove(<snapshot/075b5b931d>) returned error, retrying after 552.330144ms: blob not removed, server response: 403 Forbidden (403)
Although the password is correct, you get a bunch of error messages, thanks to the --append-only flag of the server.
Even when an attacker gains access to your restic password, he can't delete the backups.
The question now is what if you want to delete old snapshots because you only have limited disk space on your VPS.
The repository the rest-server creates is a regular repository, and you can access it with the restic command-line tool locally.
Open an SSH connection to your VPS, and install the restic command-line tool.
apt install restic
Now you can run the forget command. Enter the correct password, and your snapshot is gone.
restic forget --prune -r /opt/restic/data 075b5b93
TLS ¶
So far we used HTTP for the transfer, this is not a problem for the backup because that is already encrypted on the client-side, but we sent our HTTP username and password in plaintext over the Internet.
We can solve that by installing a TLS certificate. We are going to create a Let's Encrypt certificate, and
for that, you need to have a domain name. Add an A record with the IP address of your server.
For this example I use the domain backup.rasc.ch
Install certbot and request a TLS certificate
apt-get install certbot
certbot certonly --standalone
When successful, cerbot should have created the key and certificate in the /etc/letsencrypt folder.
In my example, the TLS certificate is stored in /etc/letsencrypt/live/backup.rasc.ch/fullchain.pem
and the private key is located here /etc/letsencrypt/live/backup.rasc.ch/privkey.pem.
Open the service file.
nano /opt/restic/rest-server.service
Add the required TLS options
ExecStart=/opt/restic/rest-server --tls \
--tls-cert /etc/letsencrypt/live/backup.rasc.ch/fullchain.pem \
--tls-key /etc/letsencrypt/live/backup.rasc.ch/privkey.pem \
--append-only --path /opt/restic/data
Restart the rest-server
systemctl daemon-reload
systemctl restart rest-server
With this configuration in place, you now have to use HTTPS from the client-side
restic snapshots -r rest:https://backup_user:password@backup.rasc.ch:8000/
If you want to learn more about the restic rest-server check out the project page:
https://github.com/restic/rest-server