Induced Outage Post-mortem

When we get more details we will add, but here is a short report on what happened to today.

Some time earlier today, the all the DNS servers at went belly up.  This affected many domains, thousands, perhaps even millions around the net. is not the host for service, but was the DNS host for the domain.  That was a mistake and we apologize for not correcting this setup earlier. Generally, I move DNS service for any domains I care about to a more reliable infrastructure, where we have complete control. We hadn't got around to that for the domain and it was still running off the default but historically unreliable DNS at

While is still down as I write this, and countless domains registered through them are still down, we found a backdoor that allowed us tocorrect the the DNS, repointing it to our reliable DNS hosting - so such a failure can never impact again. domains have been down for hours, all over the net. I suspect this will have a huge fallout for - I know I've about had it with them and will look to move my many domains off to someone else. If anyone has suggestions for good registrars, let me know - I mean the main reason we still use is the "devil-you-know" and I don't want to move to some other provider that is even worse.

So in short, is back up and we're sorry for the outage that, while not the fault of the servers or network, it was a result of our failure to setup DNS for on a proper infrastructure.

Update: Actual outage appears to have been roughly 90 minutes - going offline for most people around 6:30am pacific time and coming back online (after moving DNS off at about 8am. It took a little while to figure out what was wrong and then a little while longer to find the backdoor that allowed us in to switch the DNS over. 

0 Responses to “ Induced Outage Post-mortem”