Prometeus - Your 'net consultants !

Hello! All nodes in NL for IWStack are at or close to capacity, therefore, restarting VMs after a crash or shutdown may not be successful as there will not be enough memory to run. We are adding new capacity, however, since the nodes there are local storage only, this will not solve the issue with o...

Server is up, but the raid rebuild keeps failing with a disk dropping even after being changed. We may need to replace the node.

Hello!

pm14 locked up with raid issues, we are trying to recover it.

Hello!

The server has been recovered completely as far as we can tell.

Hello!

PM73 crashed this morning and is not coming back, we are investigating the possible causes, looks like hardware failure.
Since we do not know what is wrong yet, there is no ETA. We will update this when we have more data.

At this time the service works.
We had some cache issues affecting the storage.
If you experience FS corruption, please shutdown the vm and do a FS check at restart just to make sure. We do not see data corruption, but it is likely the cache delay might have created some small problems.

It appears that there were multiple problems, among them a network and a caching one. The network problem has been fixed, the cache is bypassed for now.
The service is restored, but we experience a bit of io problems due to the lack of caching.

The service is restored, however, we are still investigating the causes. An aparent slowdown of the network might have caused the issue, but there are no clues on why it happened yet.

We are aware of the problems with the IWStack infrastructure.
The cause seems to be a cascade failure and, for now, we do not know what is the reason for it. The problem is ongoing, we will keep updates here.

Some servers have been restored. This is a complicated manual operation, if you really need your data, please open a ticket.