Hello,

 

We are aware that there are persistent outages at the moment. We are aware this is annoying and are working on a solution. It's just as annoying for me as it is for the rest of you.

 

As of a few minutes ago, we have put further automatic technical measures to detect and attempt to automatically fix certain issues.

 

For the technically minded -

There is a particularly annoying fatal bug right now with the frontend load balancers going down every few days, I've had a look into what's causing this and the problem is in no way immediately obvious. It looks as though it is some very bizarre attack that confuses nginx's internal connection management, or perhaps a flaw in our in-house DDoS prevention modules that causes a deadlock in some fashion - that someone is exploiting to their advantage, or is innocently failing.

Until we pinpoint the bug, we've now put into place a suite of tests that check if our servers are responding the way they should be, and if they aren't the appropriate server will be automatically restarted and any caches cleared.

Last night our script automatically detected a fault and made sure to prevent any downtime. So hopefully we have a working workaround for this bug for now.

I've also pinpointed this bug to an odd locking issue with our webserver, specifically the SSL automatic provisioning module we use. We have reported this bug and we are working to debug it.