Fibercut in DC2

[Admin]

Hello!

We had a fibercut earlier today in the DC2 part of our Milan datacenter deployment.
It was repaired in about 1 hour, but after a short period online, the switches failed and we have someone in the field trying to either repair or change them.
Our main billing panel database is hosted there so we are unable to send notifications to the affected customers (the DC2 IWStack xen deployments only).

There is no dataloss, the problem is only in the networking.
We have an ETA of under 2 hours at this time (subject to change, in case the issue is not actually in the switches, but maybe a bad intermittent connection in the fiber some place)



Update:

The fibers are rechecked. 144 were cut through a passage, many other people in the area are affected. It is possible that the repair was not done correctly.

[Admin]

All services have been restored.

We re-routed some services in dc1 too to enhance redundancy which required some rewiring and as a result there were a couple of minutes of network downtime for many nodes in DC1.
We will be drawing more conclusions and think of compensation next few days.

We are very sorry for this incident.