Test your damn backup scripts.
-
Here's a tip: your email provider (Gmail, Fastmail, ProtonMail, etc) probably has a way to send SMTP email with an app password.
Use a tool like postfix or (my favorite recently) msmtp to send yourself an email **every time your backup completes**. Then, test what happens if the backup doesn't complete. If your mobile provider has an email->SMS bridge, you can even send yourself a text message.
Will you ever know if your backup doesn't work? Figure that out BEFORE you deploy. While it's fresh.
@vkc I find using runitor to be really helpful. It works with healthchecks.io which will notify me if a check wasn't started, didn't complete, or failed.
-
Here's a tip: your email provider (Gmail, Fastmail, ProtonMail, etc) probably has a way to send SMTP email with an app password.
Use a tool like postfix or (my favorite recently) msmtp to send yourself an email **every time your backup completes**. Then, test what happens if the backup doesn't complete. If your mobile provider has an email->SMS bridge, you can even send yourself a text message.
Will you ever know if your backup doesn't work? Figure that out BEFORE you deploy. While it's fresh.
If you're doing a pull style backup, where your backup target pulls the files from the main machine, test what happens if the host's directory got encrypted, unmounted, or deleted.
This is a gotcha I've seen hundreds of times before. "We back everything up!" Sure, but when they encrypted your /opt/appdir/ and you pulled those files, guess what you just did?!?
What if the drive got borked on the host machine... did you just delete your backups?
Test that crap!
-
@vkc Also test your restores.
@NormanDunbar yes yes yes yes yes!
-
@cadusilva I like ntfy but I always worry what happens when it fails. I've used it alongside email for a while!
-
@cadusilva I like ntfy but I always worry what happens when it fails. I've used it alongside email for a while!
@vkc your post made me think about it. It wouldn't hurt to configure another trigger for SMTP like you've said.
-
If you're doing a pull style backup, where your backup target pulls the files from the main machine, test what happens if the host's directory got encrypted, unmounted, or deleted.
This is a gotcha I've seen hundreds of times before. "We back everything up!" Sure, but when they encrypted your /opt/appdir/ and you pulled those files, guess what you just did?!?
What if the drive got borked on the host machine... did you just delete your backups?
Test that crap!
@vkc is it really a backup if you don't keep multiple versions?
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc A wise mentor once said to me, while I was setting up the companies backup solution. if you remember nothing else remember this:
"Data redundancy implies Data Inconsistency"
"If you are not constantly thinking about this fact, don't even bother."
He then helped me implement a "Network Recycle Bin" whose absence was the root cause of our emergency backup scheme implementation.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
Reminds me of a phone conference I was once in with a tape drive vendor. A colleague of mine was asking questions.
Colleague: Have none of your other customers experienced these read performance problems?
Vendor: Your use case is a bit atypical.
Colleague: What exactly is atypical about our use case?
Vendor: We can tell from the logs that you have had a lot of read activity on these drives. Most of our customers mainly use the drives for writing to tapes. -
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc You just reminded me to check my BorgBackup mount script. It worked but I forgot sudo the first time so I added a root check to it.
-
Here's a tip: your email provider (Gmail, Fastmail, ProtonMail, etc) probably has a way to send SMTP email with an app password.
Use a tool like postfix or (my favorite recently) msmtp to send yourself an email **every time your backup completes**. Then, test what happens if the backup doesn't complete. If your mobile provider has an email->SMS bridge, you can even send yourself a text message.
Will you ever know if your backup doesn't work? Figure that out BEFORE you deploy. While it's fresh.
Back in the 90’s I was managing some servers and had Perl scripts running cron jobs. When the jobs were done… they would send me an email with the logging output of the job.
One day I realized that I had not seen an email from a specific job/server for a few days. (failure mode, not getting the email and not realizing it)
Oops.
So… I wrote a Perl module that the jobs were “registered” with… and that became the dashboad of my system.
Instead of me having to remember the email.
-
If you're doing a pull style backup, where your backup target pulls the files from the main machine, test what happens if the host's directory got encrypted, unmounted, or deleted.
This is a gotcha I've seen hundreds of times before. "We back everything up!" Sure, but when they encrypted your /opt/appdir/ and you pulled those files, guess what you just did?!?
What if the drive got borked on the host machine... did you just delete your backups?
Test that crap!
@vkc Do you have pointers to some generic documentation?
(Because I'd say it should be a mandatory chapter of every software doc.)
Like how do I test restoring my phone backup without having another phone?
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc For me running backup is a manual process - where I check my cloud and external backup every time and I try to remember to change password for the encryption once a month
A few years back I had it running as a cron job, with an email notification - but after a while I usually get infected with notification blindness and suddenly I realize - often when adding new things to the backup script or tidying up - why haven't I heard anything from the server for weeks ? But I've been incredible lucky in the past, only my desktop system drives seems to die on me.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc Corollary: There's only one reason to make a backup: it allows you to do a restore. What, you don't have a workflow or a plan for doing a restore?
You should probably do that.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc wait, you guys make backups?
-
Here's a tip: your email provider (Gmail, Fastmail, ProtonMail, etc) probably has a way to send SMTP email with an app password.
Use a tool like postfix or (my favorite recently) msmtp to send yourself an email **every time your backup completes**. Then, test what happens if the backup doesn't complete. If your mobile provider has an email->SMS bridge, you can even send yourself a text message.
Will you ever know if your backup doesn't work? Figure that out BEFORE you deploy. While it's fresh.
@vkc this is so right! I send myself a message via a ntfy script when each backup finishes and it alerted me to the backups having stopped. I can’t remember what caused it, but it was a very easy fix and I have since had to use it… life saver!
-
"Schrödinger backup" is absolutely a thing
@vkc @baardhaveland you beat me to it.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc you're not my mum, you can't make me!
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
why do you need backups when you have snapshots, were the last words the product owner said before the great night
-
@vkc is it really a backup if you don't keep multiple versions?
@Eggfreckles @vkc I use an rsync option to make incremental backups using the previous successful backup as reference. Functionally each backup is a full backup, but they share unmodified files. This allows me to keep several weeks worth of nightly backups. I should perhaps add a check for suddenly exploding disk use as a possible indicator for trouble.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
TickIT auditor's story.
Asked about backups, was told they were done nightly to disc on rolling 7 days[0] and the backup discs stored in a cupboard down the hall[1].
How and when are they tested? Blank looks.
Walk through reloading backups.
Data on every disc corrupted. Panic ensued.[0] Not enough.
[1] Not far enough away in case of fire, flood, plague of locusts or whatever.
