Test your damn backup scripts.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc You just reminded me to check my BorgBackup mount script. It worked but I forgot sudo the first time so I added a root check to it.
-
Here's a tip: your email provider (Gmail, Fastmail, ProtonMail, etc) probably has a way to send SMTP email with an app password.
Use a tool like postfix or (my favorite recently) msmtp to send yourself an email **every time your backup completes**. Then, test what happens if the backup doesn't complete. If your mobile provider has an email->SMS bridge, you can even send yourself a text message.
Will you ever know if your backup doesn't work? Figure that out BEFORE you deploy. While it's fresh.
Back in the 90’s I was managing some servers and had Perl scripts running cron jobs. When the jobs were done… they would send me an email with the logging output of the job.
One day I realized that I had not seen an email from a specific job/server for a few days. (failure mode, not getting the email and not realizing it)
Oops.
So… I wrote a Perl module that the jobs were “registered” with… and that became the dashboad of my system.
Instead of me having to remember the email.
-
If you're doing a pull style backup, where your backup target pulls the files from the main machine, test what happens if the host's directory got encrypted, unmounted, or deleted.
This is a gotcha I've seen hundreds of times before. "We back everything up!" Sure, but when they encrypted your /opt/appdir/ and you pulled those files, guess what you just did?!?
What if the drive got borked on the host machine... did you just delete your backups?
Test that crap!
@vkc Do you have pointers to some generic documentation?
(Because I'd say it should be a mandatory chapter of every software doc.)
Like how do I test restoring my phone backup without having another phone?
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc For me running backup is a manual process - where I check my cloud and external backup every time and I try to remember to change password for the encryption once a month
A few years back I had it running as a cron job, with an email notification - but after a while I usually get infected with notification blindness and suddenly I realize - often when adding new things to the backup script or tidying up - why haven't I heard anything from the server for weeks ? But I've been incredible lucky in the past, only my desktop system drives seems to die on me.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc Corollary: There's only one reason to make a backup: it allows you to do a restore. What, you don't have a workflow or a plan for doing a restore?
You should probably do that.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc wait, you guys make backups?
-
Here's a tip: your email provider (Gmail, Fastmail, ProtonMail, etc) probably has a way to send SMTP email with an app password.
Use a tool like postfix or (my favorite recently) msmtp to send yourself an email **every time your backup completes**. Then, test what happens if the backup doesn't complete. If your mobile provider has an email->SMS bridge, you can even send yourself a text message.
Will you ever know if your backup doesn't work? Figure that out BEFORE you deploy. While it's fresh.
@vkc this is so right! I send myself a message via a ntfy script when each backup finishes and it alerted me to the backups having stopped. I can’t remember what caused it, but it was a very easy fix and I have since had to use it… life saver!
-
"Schrödinger backup" is absolutely a thing
@vkc @baardhaveland you beat me to it.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc you're not my mum, you can't make me!
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
why do you need backups when you have snapshots, were the last words the product owner said before the great night
-
@vkc is it really a backup if you don't keep multiple versions?
@Eggfreckles @vkc I use an rsync option to make incremental backups using the previous successful backup as reference. Functionally each backup is a full backup, but they share unmodified files. This allows me to keep several weeks worth of nightly backups. I should perhaps add a check for suddenly exploding disk use as a possible indicator for trouble.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
TickIT auditor's story.
Asked about backups, was told they were done nightly to disc on rolling 7 days[0] and the backup discs stored in a cupboard down the hall[1].
How and when are they tested? Blank looks.
Walk through reloading backups.
Data on every disc corrupted. Panic ensued.[0] Not enough.
[1] Not far enough away in case of fire, flood, plague of locusts or whatever. -
If you're doing a pull style backup, where your backup target pulls the files from the main machine, test what happens if the host's directory got encrypted, unmounted, or deleted.
This is a gotcha I've seen hundreds of times before. "We back everything up!" Sure, but when they encrypted your /opt/appdir/ and you pulled those files, guess what you just did?!?
What if the drive got borked on the host machine... did you just delete your backups?
Test that crap!
@vkc noob question: isn't a backup just a copy of your files somewhere else? what happens when you purposefully delete a file? does it get deleted on the backup too? what if you delete a file accidentally?
or are backups strictly for hardware failure, not for human error?
(i am doing manual backups (a little less regularily than i'd like) and have to use a tool to review all changes between my current files and the prev. backup to be sure because i don't understand automatic backups) :c
-
@Eggfreckles @vkc I use an rsync option to make incremental backups using the previous successful backup as reference. Functionally each backup is a full backup, but they share unmodified files. This allows me to keep several weeks worth of nightly backups. I should perhaps add a check for suddenly exploding disk use as a possible indicator for trouble.
@wfk @vkc I used to do something similar and then I switched to BORG. https://www.borgbackup.org/
I still need to spot test my backups occasionally.
-
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
the truth:
"...If you aren't testing your backups, you don't have backups..." -
Test your damn backup scripts.
Don't just assume everything is working. Unplug servers and test it then. How will you know it's broken?
If you aren't testing your backups, you don't have backups.
@vkc and the restores
and the data access post restore and that your restore process will be timely enough for your business critical processes -
@vkc noob question: isn't a backup just a copy of your files somewhere else? what happens when you purposefully delete a file? does it get deleted on the backup too? what if you delete a file accidentally?
or are backups strictly for hardware failure, not for human error?
(i am doing manual backups (a little less regularily than i'd like) and have to use a tool to review all changes between my current files and the prev. backup to be sure because i don't understand automatic backups) :c
-
@vkc noob question: isn't a backup just a copy of your files somewhere else? what happens when you purposefully delete a file? does it get deleted on the backup too? what if you delete a file accidentally?
or are backups strictly for hardware failure, not for human error?
(i am doing manual backups (a little less regularily than i'd like) and have to use a tool to review all changes between my current files and the prev. backup to be sure because i don't understand automatic backups) :c
@CIMB4 a reasonable way to manage this sort of thing is what we call "snapshots": points in time.
I maintain tiered snapshots on my server at home- a once-per-month snapshot that lasts a year, a once-per-day snapshot that lasts three months, and a once-per-hour snapshot that lasts two weeks.
This protects both against human error ("oops, I didn't mean to delete that"), and malice ("lets encrypt all your files"). If the snapshots sync with a remote system, it can handle hardware failure too.
-
@CIMB4 a reasonable way to manage this sort of thing is what we call "snapshots": points in time.
I maintain tiered snapshots on my server at home- a once-per-month snapshot that lasts a year, a once-per-day snapshot that lasts three months, and a once-per-hour snapshot that lasts two weeks.
This protects both against human error ("oops, I didn't mean to delete that"), and malice ("lets encrypt all your files"). If the snapshots sync with a remote system, it can handle hardware failure too.
@CIMB4 the benefit to snapshots is that if you accidentally delete a file, you're covered. But if you purposefully delete a file, it'll *eventually* disappear from the snapshots too as they age out.
This gives you the maximum flexibility when it comes to managing your backups. I routinely mirror my snapshots to an offsite location, and I test recovering them regularly.
A popular filesystem which makes this easier is ZFS. But it can be as simple as copies of files on a big hard drive.
-
Z zeitverschreib@freundica.de shared this topic
