Fibercut in DC2

Post Reply
Admin
Site Admin
Posts: 490
Joined: Wed Jul 25, 2012 10:54 pm

Fibercut in DC2

Post by Admin » Tue Mar 10, 2015 7:39 pm

Hello!

We had a fibercut earlier today in the DC2 part of our Milan datacenter deployment.
It was repaired in about 1 hour, but after a short period online, the switches failed and we have someone in the field trying to either repair or change them.
Our main billing panel database is hosted there so we are unable to send notifications to the affected customers (the DC2 IWStack xen deployments only).

There is no dataloss, the problem is only in the networking.
We have an ETA of under 2 hours at this time (subject to change, in case the issue is not actually in the switches, but maybe a bad intermittent connection in the fiber some place)



Update:

The fibers are rechecked. 144 were cut through a passage, many other people in the area are affected. It is possible that the repair was not done correctly.

Admin
Site Admin
Posts: 490
Joined: Wed Jul 25, 2012 10:54 pm

Re: Fibercut in DC2

Post by Admin » Wed Mar 11, 2015 12:19 am

All services have been restored.

We re-routed some services in dc1 too to enhance redundancy which required some rewiring and as a result there were a couple of minutes of network downtime for many nodes in DC1.
We will be drawing more conclusions and think of compensation next few days.

We are very sorry for this incident.

Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests