How to check your backups are running.

Almost everyone who runs a website or a server has a backup. Far fewer know that it is still running. A backup is a scheduled job, and like every scheduled job it can stop without a sound: a changed password, a full disk, a moved folder, a server rebuilt without its crontab. The usual way of finding out is trying to restore, which is the worst possible moment.

This guide explains how to monitor backups so a missed one is reported the next morning, and what monitoring alone cannot tell you.

Why backups fail silently

A backup does its work in the background, at night, and produces no visible result when it goes well. That makes success and failure look identical from the outside. Typical causes of a stopped backup are:

• The schedule was lost. A server rebuild, migration or restore left the cron entry behind.
• Authentication expired. A password, API key or token used to reach the database or the storage destination was rotated or revoked.
• The destination is full. Storage quota reached, so new backups are refused while old ones remain.
• The source moved. The database was renamed, the files were relocated or a new site was added that the job does not know about.
• The job hangs. A lock from a previous run is never released, so every later run waits or exits at once.
• It succeeds with nothing in it. The job exits cleanly but writes an empty or truncated file.

Most of these leave no trace except a gap in the backups, which nobody looks at.

Monitor the backup with a check-in

The dependable way to know a backup ran is to have it report in. Create a heartbeat check on the backup’s schedule and have the job request its address after it succeeds. If a night’s backup does not happen, or fails partway, the check-in does not arrive and you are alerted. The idea is explained in heartbeat monitoring explained.

For a script or cron entry

The check-in only runs if backup.sh exits successfully. For this to mean something, make sure the script itself stops on the first error. In a shell script, begin with set -euo pipefail so a failed dump cannot be followed by a “successful” upload.

With time and failure reporting

This reports a failure the moment it happens, and records how long each backup takes. A backup that normally takes ten minutes and suddenly takes two, or two hours, is worth looking at even though it “succeeded”.

For Laravel and similar tools

If backups are scheduled in the application, for example with a package that runs from the Laravel scheduler, attach the check-in to the scheduled task with thenPing and pingOnFailure. The Laravel scheduler monitoring guide shows how, and why you also want a heartbeat for the scheduler itself.

Make the check-in mean something

A check-in proves the job reached its last line. It does not prove the backup is good. Make the script verify its own work before it checks in:

• Check the file exists and is not tiny. Fail if the dump is missing or smaller than a sensible minimum.
• Check the upload. Confirm the file arrived at the destination, with a matching size or checksum.
• Check the exit status of every step, not just the last one.
• Check the date. The newest backup in the destination should be from today.

Test restoring, not just backing up

Monitoring tells you a backup ran. Only a restore tells you it works. Plan to restore from a recent backup into a scratch environment on a regular schedule, perhaps monthly, and confirm that the data is complete and the application starts. If you can automate it, do, and have the restore test check in to its own monitor.

Follow the habit many sysadmins swear by, which is to keep copies in more than one place: the working data, a backup on the same provider for quick recovery, and a copy somewhere else entirely in case the first two are lost together.

Set the schedule and grace period carefully

• Match the check’s schedule to the backup’s real one. A nightly backup is a daily check, not an hourly one.
• Make the grace period longer than the slowest backup you have ever had, including growth. Backups get slower as data grows.
• Give every backup its own check: database, files and each site separately. A single shared check hides which one failed.
• Pause the check during planned maintenance, and resume it afterwards.

What to do when the alert comes

1. Run the backup by hand and read the output. The failure is usually obvious: a permission error, a full disk or an authentication failure.
2. Check the destination for the most recent good backup, so you know how much data is currently at risk.
3. Fix the cause and run the backup again, and wait for the recovery notice.
4. Add a verification step to the script if the failure could have gone unseen another way.

Start free

Backups are the best first thing to put under cron job monitoring. Every Pingvella account has three heartbeat checks free, and alerts that reach you by email, Slack, Discord, Microsoft Teams, Telegram or webhook. For jobs in general, see how to monitor cron jobs.

Common questions

There are more answers on the FAQ page.