Search found 476 matches

by Admin
Sat Feb 11, 2017 7:13 am
Forum: Technical difficulties
Topic: NL IWStack capacity issues
Replies: 0
Views: 94299

NL IWStack capacity issues

Hello! All nodes in NL for IWStack are at or close to capacity, therefore, restarting VMs after a crash or shutdown may not be successful as there will not be enough memory to run. We are adding new capacity, however, since the nodes there are local storage only, this will not solve the issue with o...
by Admin
Wed Nov 30, 2016 6:19 pm
Forum: Technical difficulties
Topic: pm14 raid issues
Replies: 1
Views: 42644

Re: pm14 raid issues

Server is up, but the raid rebuild keeps failing with a disk dropping even after being changed. We may need to replace the node.
by Admin
Wed Nov 30, 2016 10:11 am
Forum: Technical difficulties
Topic: pm14 raid issues
Replies: 1
Views: 42644

pm14 raid issues

Hello!

pm14 locked up with raid issues, we are trying to recover it.
by Admin
Tue Nov 01, 2016 10:19 am
Forum: Technical difficulties
Topic: PM73 failure
Replies: 1
Views: 44320

Re: PM73 failure

Hello!

The server has been recovered completely as far as we can tell.
by Admin
Mon Oct 31, 2016 11:22 am
Forum: Technical difficulties
Topic: PM73 failure
Replies: 1
Views: 44320

PM73 failure

Hello!

PM73 crashed this morning and is not coming back, we are investigating the possible causes, looks like hardware failure.
Since we do not know what is wrong yet, there is no ETA. We will update this when we have more data.
by Admin
Tue Oct 11, 2016 5:49 am
Forum: Technical difficulties
Topic: IWStack outage
Replies: 3
Views: 58022

Re: IWStack outage

At this time the service works.
We had some cache issues affecting the storage.
If you experience FS corruption, please shutdown the vm and do a FS check at restart just to make sure. We do not see data corruption, but it is likely the cache delay might have created some small problems.
by Admin
Tue Oct 11, 2016 5:20 am
Forum: Technical difficulties
Topic: IWStack outage
Replies: 3
Views: 58022

Re: IWStack outage

It appears that there were multiple problems, among them a network and a caching one. The network problem has been fixed, the cache is bypassed for now.
The service is restored, but we experience a bit of io problems due to the lack of caching.
by Admin
Tue Oct 11, 2016 4:21 am
Forum: Technical difficulties
Topic: IWStack outage
Replies: 3
Views: 58022

Re: IWStack outage

The service is restored, however, we are still investigating the causes. An aparent slowdown of the network might have caused the issue, but there are no clues on why it happened yet.
by Admin
Tue Oct 11, 2016 2:40 am
Forum: Technical difficulties
Topic: IWStack outage
Replies: 3
Views: 58022

IWStack outage

We are aware of the problems with the IWStack infrastructure.
The cause seems to be a cascade failure and, for now, we do not know what is the reason for it. The problem is ongoing, we will keep updates here.
by Admin
Mon Sep 19, 2016 11:27 pm
Forum: Technical difficulties
Topic: pm28 complete failure
Replies: 2
Views: 41085

Re: pm28 complete failure

Some servers have been restored. This is a complicated manual operation, if you really need your data, please open a ticket.