The Sunny Gardener - August 2023
Test Your Backups
As I mentioned last month, I moved to a new backup strategy, using pgBackRest to backup the database, along with scripts to backup media and various config files.
I had tested the backup and restore process and those tests had worked, however after running this new setup for a week or two, I figured I ought to do a test restore of the actual backups that I was taking from the live environment.
Lo and behold, attempting to restore the most recent backup failed. Not great!
I quickly made a manual backup and spent the next couple of days trying to identify what was wrong. As it turned out, the version of pgBackRest that I had started with had added some new options recently, which I had used according to their quick start guide, and those new options had introduced new bugs. So it goes.
Ultimately, even with the bugs, the backups I had taken so far were still usable, but required manually restoring multiple recent backups in the correct order.
Fortunately, the recently introduced bugs had already been fixed. I upgraded pgBackRest and tested each of the backups that were taken for the next week. It does appear that the issue has been resolved.
Really Though, Test Your Backups
Sadly, a week later, several other servers learned a much harsher lesson.
The servers musician.social, firefish.lgbt, and outdoors.lgbt were completely erased due to a configuration error.
The admin had tested the backup and restore process, but not restoring the actual backups that were being taken automatically. When attempting to restore the erased servers from their backups, they discovered that their backups didn’t actually include all of the data needed to restore them.
Bad news everyone. It is with immense regret that I write to inform you we have suffered a total loss of data for firefish.lgbt, musician.social, and outdoors.lgbt.
The admin noted that they would not be re-starting outdoors.lgbt or firefish.lgbt.
musician.social has been recreated from scratch, and is back online again on a different host. If you were previously following any users of musician.social, you may need to unfollow/re-follow them in order for the link between your accounts to work again, assuming that account has been recreated.
One of the affected accounts was @radiofreefedi@musician.social of the Radio Free Fedi project, a streaming radio station playing music from across the fediverse, check it out if you haven’t already!
Summary
The most recent new user wave is trailing off with new signups roughly balancing out the folks that stop using their account after a month or two, keep the monthly active user count relatively stable.
Hosting Costs
No big changes in costs this month. Database backups cost a bit more than usual because I used up a bunch of extra bandwidth testing restoration.
Expand text version
| sunny.garden | 2023-09-01 | |
|---|---|---|
| Main Server | 4-8GB-160GB | -$36 |
| Database Backups | -$12 | |
| Media Storage | 159GB/mo. | -$6 |
| Media Bandwidth | Free | $0 |
| Media Backups | 13GB/mo. | -$1 |
| Domain Name | $35/yr | -$3 |
| Email Server | -$5 | |
| Donations One-Time | $0 | |
| Donations Recurring | $45 | |
| Monthly Totals | ||
| Previous Balance | $174 | |
| Total Donations | $45 | |
| Total Expenses | -$63 | |
| Balance | $156 |
Canadian Dollars
Thank you, to everyone that has donated on ko-fi! There’s no obligation or expectation that you donate in order to use the server, and I’m able to cover the costs myself if needed.
– Brook