Partial DNS outage for *.raskesider.no
Incident Report for Servebolt
Postmortem

There was a partial DNS outage for domain raskesider.no that lasted for approximately 2 hours, May 30th.

This affected anyone using a DNSSEC enabled DNS resolver, while trying to access services under the raskesider.no domain. This may have included SSH/SFTP access, PHPMyAdmin access and Test domains.

Some production customer traffic was impacted. Production domains with CNAME references may have experienced unavailability during the time of the event. For those customers using Cloudflare, we were able to reroute impacted traffic.

It is possible that email deliverability in the period was reduced to some destinations, during the partial outage. We do not believe that to have been significant.

*Root cause
*
The problem was a DNSSEC related configuration change done by our DNS provider for raskesider.no. It that was executed in the wrong order, causing DNSSEC enabled DNS resolvers to discard responses.

*Preventive measures
*
To prevent this from happening ever again, Servebolt is moving the raskesider.no DNS zone to Cloudflare.

We apologize for any disruptions and inconvenience that it may have caused for you, your team or your clients.

Kind regards,
Servebolt Operations Team

Posted May 31, 2019 - 12:28 CEST

Resolved
This incident has been resolved.
Posted May 30, 2019 - 22:04 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 30, 2019 - 16:41 CEST
Identified
We have identified a problem relating to DNSSEC. This primarily impacts access to test domains, as well as access to SSH, SFTP and phpmyadmin for anything under the raskesider.no domain for customers using DNSSEC enabled DNS resolvers.
Posted May 30, 2019 - 14:51 CEST
Investigating
We are currently investigating this issue.
Posted May 30, 2019 - 14:44 CEST
This incident affected: Servebolt Cloud OSL and Servebolt Cloud AMS.