https://status.torproject.org/affected/torproject.org/ Incident history github.com/cstate en 2023-02-04T01:57:31+00:00 2023-02-04T01:57:31+00:00 https://status.torproject.org/issues/2023-02-04-hetzner-outage/ Sat, 04 Feb 2023 01:57:31 +0000 https://status.torproject.org/issues/2023-02-04-hetzner-outage/ 2023-02-04 03:33:46 +0000 We are witnessing a large outage at our main service provider, Hetzner. According to the information we have gathered so far, four switches (4!) have failed and that affects four (yes, again, 4!) of the servers in the 8-node Ganeti cluster. Affected servers: check-01 chives colchicifolium cupani fsn-node-01 fsn-node-02 fsn-node-04 fsn-node-07 gitlab-02 gnt-fsn henryi loghost01 majus materculae media-01 onionoo-backend-01 onionoo-backend-02 onionoo-frontend-02 perdulce polyanthum relay-01 static-master-fsn staticiforme submit-01 tbb-nightlies-master weather-01 Upstream resolved the situation after a few hours of downtime. <p>We are witnessing a large outage at our main service provider, Hetzner. According to the information we have gathered so far, four switches (4!) have failed and that affects four (yes, again, 4!) of the servers in the 8-node Ganeti cluster.</p> <p>Affected servers:</p> <ul> <li>check-01</li> <li>chives</li> <li>colchicifolium</li> <li>cupani</li> <li>fsn-node-01</li> <li>fsn-node-02</li> <li>fsn-node-04</li> <li>fsn-node-07</li> <li>gitlab-02</li> <li>gnt-fsn</li> <li>henryi</li> <li>loghost01</li> <li>majus</li> <li>materculae</li> <li>media-01</li> <li>onionoo-backend-01</li> <li>onionoo-backend-02</li> <li>onionoo-frontend-02</li> <li>perdulce</li> <li>polyanthum</li> <li>relay-01</li> <li>static-master-fsn</li> <li>staticiforme</li> <li>submit-01</li> <li>tbb-nightlies-master</li> <li>weather-01</li> </ul> <p>Upstream resolved the situation after a few hours of downtime. According to Hetzner it was &ldquo;a short disruption of a line card in one of our access routers&rdquo;. Details of the incident are available in <a href="https://gitlab.torproject.org/tpo/tpa/team/-/issues/41057">tpo/tpa/team#41057</a>.</p> https://status.torproject.org/issues/2022-01-25-routing-issues/ Tue, 25 Jan 2022 06:00:00 +0000 https://status.torproject.org/issues/2022-01-25-routing-issues/ 2022-01-27 18:58:00 +0000 We are experiencing intermittent network outages that typically resolve themselves within a few hours. Preliminary investigations seem to point at routing issues at Hetzner, but we have yet to get a solid diagnostic. We&rsquo;re following that issue in issue 40601. <p>We are experiencing intermittent network outages that typically resolve themselves within a few hours. Preliminary investigations seem to point at routing issues at Hetzner, but we have yet to get a solid diagnostic. We&rsquo;re following that issue in <a href="https://gitlab.torproject.org/tpo/tpa/team/-/issues/40601">issue 40601</a>.</p>