← Go back to Tor Project status

Tor Weather data loss

October 27, 2023 at 6:23 AM UTC


Resolved after 300h 5m of downtime November 8, 2023 at 6:29 PM UTC

The Tor Weather service broke during an upgrade to Debian Bookworm. The database was destroyed (but eventually recovered from backups) and the service was down for about a week.

Specifically, all changes made after 2023-10-27 06:23:11 UTC have been lost. The service itself went offline some time before 2023-10-31 20:33UTC, therefore about 4 days of data was lost. The data loss was due to a flaw in the bookworm upgrade procedure, since then corrected, combined with expiration of the continuous backups which would normally have allowed full recovery.

If you have registered or made any changes on Tor Weather after October 27th, you will need to redo those changes.

The service was restored on November 8th (2023-11-08 18:29UTC) from the October 27th backup. So the service was offline for a little over 8 days, from October 31th to November 8th.

A more detailed discussion and post-mortem can be found in issue 41388, including a full timeline of the incident, heroic deleted file recovery procedures, and future improvements to our systems that should keep this specific problem from happening again.

It is not in our habit to lose data, and we apologize profusely for this mishap.

Last updated: November 8, 2023 at 11:03 PM UTC