Umbraco Cloud Incident closed

Since Wednesday February 27th, 23:24 CET and until March 1st, 19:00 CET we have had multiple incidents taking customers Umbraco Cloud sites offline several times.

What did we do?

In short we stopped all other activities and everyone at Umbraco HQ joined in either directly contributing to solve the problem, or to get information out to the affected customers and helping them where possible.

Additionally we raised suppliers and external consultants to help ensure that no stones were left unturned in the process of solving the problems.

A maintenance window was announced with short notice in order to allow us to do what was necessary in order to bring stability to the system.

Current status

Overall and since March 1st, 19:00 CET the system has been performing well and as prior to the incident, except for a few individual sites that has not been coming back online as expected. We have been working on those and they are now back in normal operations again.

We have seen individual sites being restarted due to balancing of the system. This is part of a normal situation.

Monitoring is heightened and we continue our extended monitoring and support throughout the weekend.

What’s next

Now the analysis will begin and we will be presenting a post-mortem in due course.

We will follow the status of the system closely hence we have increased our monitoring. This includes staff on duty around the clock.

Should you experience problems with your site, please reach out to our support at contact@umbraco.com. It is manned around the clock.

We apologize for the problems this incident have caused for you and your customers.

Best regards - The Umbraco HQ team