POPBOB.WTF OUTAGE POST MORTEM TL;DR: VPS provider shat the bed, it took 1 day to have an emergency backup restored (by the provider) + 1 day to get the VM working again, data after December 7 is gone, a daily backup system was actually installed to prevent me having to leave data integrity up to pure luck like an idiot
On December 8th, 2025, at an unknown time between 4:07am EST (latest known timestamp for content that existed before the outage) and 7:30am EST (when the outage was first noticed), the VPS provider that hosted the servers for all popbob.wtf content experienced a catastrophic failure. Specifically, an array of NVMe drives (one of which held all user data for popbob.wtf) failed, leading to erratic behaviour that eventually led to all of the VMs assigned to that array shutting down.
The normally active system for detecting outages was not functional due to the system hosting it being restarted and not properly connecting its container to the network, leading to it being unable to send out notifications that the VPS was down.
Attempts were made to restart the server and return it to a normal state, however they went nowhere and the server was temporarily disabled by the VPS provider by 11:30am EST. Communications with their support revealed that an emergency backup was taken for most of the affected VPSes, which was to be restored within the next 24 hours. Whether the popbob.wtf VM was included in the backup and/or when the emergency backup was taken was not disclosed.
The following day, at 5:19am EST the server was rolled back by the VPS to its current state at (what I can assume to be) 6am on December 7th, 2025. All data before December 7th 2025, 6am has been effectively lost, leaving a period of ~1 day completely unaccounted for. Content after this date may appear on other instances or indexers, but will functionally not exist on this server. The rollback was not communicated to me and remained unnoticed until 5:11pm at December 9th, 2025.
Following the discovery, an operation to restore the server and its containers to a healthy state began, as well as the installation of an automated daily backup system for user data to multiple sources, to prevent any future catastrophic failures. I apologize for both the downtime and the lack of preparedness for such an incident.
Users that have an account on the ATproto PDS are advised to both add a self-controlled rotation key to their identity document (in case of a much longer outage or shutdown) and to set up regular backups of their account repository, using tools such as PDS MOOver or ATbackup. You should also download your Fedi data from time to time, but you can't really use it for anything other than importing your follows and mutes so ¯\_(ツ)_/¯.