Where did everything go?!

We've been having trouble with the server for some time, and I arranged a new server and was set to start the migration today.

Unfortunately, the datacenter hosting the old server is doing some (unannounced - or at least unannounced to me) scheduled maintenance that is taking much, much longer than expected and right now I can't get to the old server to migrate things to the new server.

I'm currently loading up the last full off-site backup (three weeks old, sigh) and will bring that up if the maintenance takes three weeks itself, like it did with the datacenter fire last year.

It will take a little while to set up, as we've jumped forward three full versions of Linux and I have to manually tweak a bunch of stuff, but the databases are already copied from the backup.

That's the worst case, but tech support have been telling me the server will be back online in about an hour for the past six hours, so I'm starting to expect a worst case scenario here.

Update 2022-12-03 12:00-ish UTC: Main files are synced from backup. Remaining files (archives, logs and so on) are copying now. I need to get a few hours sleep before I tackle bringing sites back online.

We did have automated daily offsite backups - but that backup server turns out to have a corrupt ZFS volume and I can't get the backups back. So we have to go back to the most recent manual offsite backup from November 10. Tech support are still working on bringing the original server back online but I don't have an ETA.

Update 2022-12-03 13:00-ish UTC: Networking problems on the planned new server. Fortunately I have yet another server. And now two copies of that manual backup.

Update 2022-12-03 14:00-ish UTC: Running a data check and index repair on the database backup on the second new server. All looks good so far. I really do need to get some sleep now. Maybe when I wake up the server will be back. If not, I'll continue getting the backup server up and running.

— Pixy