12 September 2012

about this week's GoDaddy outage

It is not clear if a cabal of Anonymous hackers, or simple network administration issues, caused the GoDaddy outage of Monday past. I guess it does not really matter

What really would have hurt is if the root domain server constellation had been compromised, to well and truly take down the internet. A Domain Registrar sends along updates to those root servers periodically, and GoDaddy's outage, from the extent of our involvement with them, simply impaired our ability to renew domains, and set new nameservers (NS records). As we had no urgent renewals pending, that is to say, not at all

We do not rely on GoDaddy for DNS services, and really, never have relied on them for production purposes. For PMman and for our ISP and COLO services, we run three geographically diverse nameservers for most of our purposes. We also run a few others for customers' needs (PTR records for a couple of datacenters we are in, testing, demonstration units)

The true 'masters' of our externally visible DNS servers are simply not accessible from the public internet. We push out updates to our public nameservers by cryptographically protected rdnc transactions. Those transactions are logged, and the information causing a given RDNC transaction are created by queries into a local database with a custom written LAMP control interface based on the FOSS tools that are in a stock CentOS install. Compared to manually editing zone files, checking variants in and out of a version control system, and so forth, this more readily provides us with scalability, traceability and auditability. Why, I caught a piece of lint in a zone file just last week, reading the overnight error report emails

We also retrieve the state of the generated zone files at the client public nameservers, and check them for consistency and coherency, essentially after each update, to prevent errors from propagating. ACLs, transaction logging and other checks provide more tracability, and we closed the mouse hole that that 'lint' crept in through in short order

As a result of the GoDaddy outage, a couple of our 'alumni' tech support folks who have moved on in their careers to other employment, gave us a call Tuesday, because they remembered how paranoid I am on making sure DNS is available. I appreciate the calls, and we've some new customers as a result

People have strong opinions about GoDaddy, sometimes for reasons of political correctness; I like them, by and large, because they provide a workmanlike product for a price that is hard to beat. They sure beat the heck out of the old Network Solutions rates. I have something like 500 domains that I administer and renew and most are there, although some are at other registrars for both historical and other reasons

And while Danica Patrick is not my cup of tea, she is not hard on the eyes, either