header image

Code Snippets

Linux Incremental Backups To AWS S3 Using Restic

| Last updated: 
#!/usr/bin/env bash
set -euo pipefail

backup_name=<name of backup>

export RESTIC_PASSWORD=<long secure password to encrypt backups>
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/<aws bucket name>/$backup_name"

export AWS_ACCESS_KEY_ID=<aws access key id>
export AWS_SECRET_ACCESS_KEY=<aws secret access key>
export AWS_DEFAULT_REGION=<aws bucket region>

failure_email_address=failure@example.com
gmail_email_address=user@example.com
gmail_password=gmailapppassword # Google app password, see https://support.google.com/accounts/answer/185833?hl=en

file_list=(
  "/home/<user>"
)

file_exclude_list=(
  "**/.bun/**"
  "**/.bundle/**"
  "**/.cache/**"
  "**/.docker/**"
  "**/.gem/**"
  "**/.npm/**"
  "**/.pulumi/**"
  "**/.rbenv/**"
  "**/.rvm/**"
  "**/.sst/**"
  "**/.terraform/**"
  "**/.tfenv/**"
  "**/.vscode-remote-containers/**"
  "**/.vscode-remote/**"
  "**/.vscode-server/**"
  "**/.vscode/**"
  "**/.yarn/**"
  "**/.Trash/**"
  "**/Downloads/**"
  "**/OneDrive/**"
  "**/bundle/gems/**"
  "**/node_modules/**"
  "**/ruby/gems/**"
  "**/storage/**"
  "**/tmp/**"
  "**/vendor/gems/**"
)


function init {
  echo "[$(date +"%T")] Initialising backup repository..."
  restic init
}

function backup_filesystem {
  echo "[$(date +"%T")] Backing up $backup_name..."

  restic --verbose backup \
    --files-from <(printf '%s\n' "${file_list[@]}") \
    --exclude-file <(printf '%s\n' "${file_exclude_list[@]}") \
    --exclude-if-present .resticignore

  local backup_exit_code=$?
  if [[ $backup_exit_code != 0 ]]; then
    return $backup_exit_code
  fi

  echo "[$(date +"%T")] Removing old backups..."

  restic --verbose forget \
    --keep-daily 7 \
    --keep-weekly 5 \
    --keep-monthly 12 \
    --keep-yearly 10
}

function notify_failure {
  local backup_exit_code=$1

  echo "[$(date +"%T")] Notifying of failure via gmail..."

  failure_message_path=$(mktemp)
  echo -e "Subject: Backup of $backup_name failed\n\n[$(date +"%T")] Backup of $backup_name failed with exit code '$backup_exit_code" > "$failure_message_path"
  curl --ssl-reqd \
    --url 'smtps://smtp.gmail.com:465' \
    --user "$gmail_email_address:$gmail_password" \
    --mail-from "$gmail_email_address" \
    --mail-rcpt "$failure_email_address" \
    --upload-file "$failure_message_path"
}

function backup {
  backup_filesystem
  local backup_exit_code=$?

  if [[ $backup_exit_code == 0 ]] ; then
    echo "[$(date +"%T")] Backup of $backup_name succeeded"
  else
    echo "[$(date +"%T")] Backup of $backup_name failed"
    notify_failure $backup_exit_code
  fi

  echo "[$(date +"%T")] Complete"
}

if [[ ! $(type -t "$1") == function ]]; then
  echo "Invalid command entered"
  exit 1
fi

TIMEFORMAT="Task completed in %3lR"
time "${@:-default}"

Bash script to perform incremental encrypted backups of Linux files to AWS S3 using the awesome restic backup tool.

Outline of Script

Features:

  • Creates incremental encrypted backups using cron
  • Backup rotation to keep storage cost controlled
  • Send an email via Gmail on backup failure

The script can be used for remote backups to AWS S3 as part of a backup and restore strategy with backup rotation scheme.

Steps

  • Install restic v0.12.1+ and ensure it is available to root user
  • Add the script as a file and make sure the script is executable (chmod +x <path-to-script>)
  • Create a private AWS S3 bucket for storing the backups in
  • Create an AWS IAM user with the minimum privileges required for managing the backups in the AWS S3 bucket, e.g.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>/<backup name>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>"
            ]
        }
    ]
}
  • Generate access credentials for the AWS IAM user (AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY)
  • Go through the script and fill in the variables, e.g.
    • <name of backup>
    • <long secure password to encrypt backups>
    • <aws bucket name>
    • <aws access key id>
    • ... others
  • Update the file_list and file_exclude_list arrays to select the files for backup. A few typical patterns are already in the file_exclude_list, but to reduce storage costs this array will likely need updating to be specific to the system being backed up
  • Check the backup rotation values set on the restic forget command are suitable and update if required
  • Run the script once with the init argument to initialise the backup repository, e.g. /root/restic-backup init
  • Run the script once with the backup argument to perform the initial backup, e.g. /root/restic-backup backup
  • Configure the crontab for the root user to run the script periodically, crontab.guru can help with this, e.g. 
0 6 * * * /root/restic-backup backup >> /root/restic-backup.log 2>&1
  • Check the backup logs to ensure the script runs correctly (tail -f /root/restic-backup.log)

Notes

  • Sensitive values such as RESTIC_PASSWORD, AWS_SECRET_ACCESS_KEY, & gmail_password are stored in plain text in the script. It may be best to edit how the script stores or fetches the sensitive values if the script can be accessed by other users, to make it more secure.
Back to all snippets