Partial DNS outage for *
Incident Report for Servebolt

There was a partial DNS outage for domain that lasted for approximately 2 hours, May 30th.

This affected anyone using a DNSSEC enabled DNS resolver, while trying to access services under the domain. This may have included SSH/SFTP access, PHPMyAdmin access and Test domains.

Some production customer traffic was impacted. Production domains with CNAME references may have experienced unavailability during the time of the event. For those customers using Cloudflare, we were able to reroute impacted traffic.

It is possible that email deliverability in the period was reduced to some destinations, during the partial outage. We do not believe that to have been significant.

*Root cause
The problem was a DNSSEC related configuration change done by our DNS provider for It that was executed in the wrong order, causing DNSSEC enabled DNS resolvers to discard responses.

*Preventive measures
To prevent this from happening ever again, Servebolt is moving the DNS zone to Cloudflare.

We apologize for any disruptions and inconvenience that it may have caused for you, your team or your clients.

Kind regards,
Servebolt Operations Team

Posted May 31, 2019 - 12:28 CEST

This incident has been resolved.
Posted May 30, 2019 - 22:04 CEST
A fix has been implemented and we are monitoring the results.
Posted May 30, 2019 - 16:41 CEST
We have identified a problem relating to DNSSEC. This primarily impacts access to test domains, as well as access to SSH, SFTP and phpmyadmin for anything under the domain for customers using DNSSEC enabled DNS resolvers.
Posted May 30, 2019 - 14:51 CEST
We are currently investigating this issue.
Posted May 30, 2019 - 14:44 CEST
This incident affected: Servebolt Cloud OSL and Servebolt Cloud AMS.