Prometeus - Your 'net consultants !

Hello!

At about 7:30 Italy time, both our datacenters in milan were hit by a powerfailure which lasted over an hour.
When power came back, many services restarted, some 25%. Others we were quickly able to restore, some other 25%.
At this time, most services are restored with the following exceptions:
1. IWStack master and parts of the KVM zone, the Xen zone was up and running shortly after the power was restored. This means many KVM VMs in milano are now working but, as with the Xen zone fully working, cannot be controlled from the interface as the master is still down.
2. Quite a few KVM VMs are not working, mainly because ISOs were left mounted and the VMs booted from them, or the NFS storage for the ISOs was not working at the time they went online and the VMs with mounted ISOs, did not start. This is also affecting the KVM IWStack zone on a large scale. In the case of non IWStack (solus) KVM, you should use the solusvm panel to unmount the iso and restart the vm. If it does not work, try again later when there will be less people doing the same thing. This is not possible for iwstack, unfortunately, as the master is not available, but if it will be and your VM is not up, you will have to do a similar thing.
3. Some Shared hosting nodes are not fully up. they will be fixed after IWstack.
4. Pm14 has a hardware issue. It will be fixed at the end.

All other services are supposedly working. If you are not on one of the affected services described above, please open a ticket to look into your specific case. DO NOT do it if you know you are on a KVM node and you did not check the ISO situation and the console first. Also, if you are on pm14, unless we reported it fixed and your vm is not starting. Also, in the KVM IWStack zone, please check the iso situation after you see the iwstack interface (the real one, not the billing panel interface) is up but your vm is not and you tried to start it without success.

Update:

IWStack is up. There are glitches still such as stuck snapshots and broken virtual networks. The stuck snapshots will be cleared by us, the stuck networks should be cleaned by following this procedure:
1. close all VMs using the stuck network.
2. issue a restart with clean up option.
3. wait a few minutes.
4. restart one vm and if it works, restart the others, if it does not work, issue another clean up restart and if it still does not work, the last chance before opening a ticket is deleting it and creating it anew.

If your regular KVM (not the cloud ones) is not up or reachable, please login to your solusvm panel and check if you have no mounted ISO, if you do, unmount it and restart. It may also need some fsck. In some rare circumstances, this might also apply to the cloud KVMs.
If your KVM keeps restarting, we would like to know in a ticket to investigate further.

There are still issues with legacy shared hosting and myoffload server.
All other issues , including fixing the pm14 broken disk should be dealt with at this time.
A huge thank you for bearing with us in this difficult time.
We will be designing some contingency plan for similar incidents, this is the first time it happens in our milano datacenters.