Prometeus - Your 'net consultants !

We are experiencing issues with this node, they might be hardware related.
More info as it becomes available.

For some 80 minutes, our services in milano were disrupted by a powerful attack, measuring in excess of 80 gbps.
They are back up now, but we are not sure if the attacker will not change strategy again, so further disruptions may happen still.

We believe we have found the issue, as the fc cards were overwhelmed by the number of commands. This cause them to stop accepting new to clear the backlog which inceased the iowait to the point the orchestrator considered the node dead starting the VMs on another which had the same issue in turn, br...

After a lot of investigations we believe there is a problem with the fiberchannel links, one of them is probably causing some issues affecting the others or the switches. When the nodes lose connectivity, they are automatically rebooted to protect the data. A lot of such events at the same time crea...

We are experiencing some reboots among the Milano IWStack clusters.
While the orchestrator tries to cope restarting VMs on other nodes, due to the big number of events, the restarts take a long time, up to 30-50 minutes.
We are investigating the issue.

Follow up:

Due to last issue, the dc people changed the power feeds and should have not cut the power again, but they did :(
We are restoring the services again...

Hello!

One of the xen clusters in DC2 experienced some issues, we corrected them and restarting the services.

After some time without issues, I think the event was cleared up.
Thank you for your understanding.

The servers are online, however, we are investigating some scattered reports about certain instances not starting.
This affects only iwstack customers with dallas services.
Since we see many instances running, we presume there should be very few people affected.

Latest reply we got from the Datacenter:

01/20/2016 03:01 Support Staff 2 (Incero):
Machines should be online now or have ipmi online for you to check. We are still working onsite with the electrical team.

We are working to restore the service.