outage at main provider

We are witnessing a large outage at our main service provider, Hetzner. According to the information we have gathered so far, four switches (4!) have failed and that affects four (yes, again, 4!) of the servers in the 8-node Ganeti cluster.

Affected servers:

• check-01
• chives
• colchicifolium
• cupani
• fsn-node-01
• fsn-node-02
• fsn-node-04
• fsn-node-07
• gitlab-02
• gnt-fsn
• henryi
• loghost01
• majus
• materculae
• media-01
• onionoo-backend-01
• onionoo-backend-02
• onionoo-frontend-02
• perdulce
• polyanthum
• relay-01
• static-master-fsn
• staticiforme
• submit-01
• tbb-nightlies-master
• weather-01

Upstream resolved the situation after a few hours of downtime. According to Hetzner it was “a short disruption of a line card in one of our access routers”. Details of the incident are available in tpo/tpa/team#41057.