Prometeus - Your 'net consultants !

We are investigating an issue with libvirt not communicating with the orchestrator.
The problem is intermittent and it appears it only affects ubuntu guests, which might indicate an issue with an OS update breaking some functionality. This should not normally happen, as the OS is isolated, perhaps the emulated bios has some incompatibility, but, since it did not change on our side, it must be an ubuntu update.
It also only affects KVM not Xen.
This seems to be a complex issue which requires a lot of troubleshooting.
While this happens:
1. Ubuntu instances are crashing from time to time;
2. They get stuck at grub kernel selection needing manual input to start.

If you are affected and your instance is NOT ubuntu, please open a ticket.
Otherwise, check this space for further updates.

We isolated mainly the ubuntu 14.04 instances (the bulk of the problematic instances) to some 2 nodes and they seem stable for now.
This is a very weird bug, we are still investigating, but, so far, it seems like an Ubuntu issue.
The large number of instances crashing and rebooting affected some other functionality too. Hopefully that will not happen now since they are isolated, and will also stay stable until we have a definitive answer and a solution if it depends on us in any way.
We are also updating libvirt on the nodes we vacate which will hopefully solve other, older, bugs.

After careful testing, found an update which does not conflict with Ubuntu 14.04 anymore and for the past few days things were stable.