Thursday, 6 February 2014

No, Google wasn't down yesterday ...

Yesterday afternoon, Google services died on me. Gmail went away, docs went offline. Everything was out, even the apps status page that tells you if there’s an outage.

I immediately tweeted to see if anyone was seeing this, and got back responses whichshowed it was out for everyone on campus, though no one offcampus seemed to be seeing problems.

It was, to quote a colleague, as if the aliens had landed.

Now Google was not out, and was not feeling indisposed. What had happened was more interesting.

A router between here and Googleland had decided it was over forwarding packets to Google and had decided to send them someplace else. For some reason monitoring did not pick up on the error and fail over to a secondary route (or router).

Why this happened is still under investigation.

However this does neatly highlight a problem with outsourcing everything to the cloud. Not only do the cloud providers have to do their thing, the infrastructure also has to - it’s no good if your cloud provider never goes down if you infrastructure does.

Which on a sparsely populated continent on the bottom of the world increasingly means infrastructure and links to overseas data centres.

Bascally, even if we can minimize latency, adjacency still counts, or to put it more simply, using more adjacent services ie those on the same continent reduces the admittedly small risk of the internet going away, if only for a few minutes …

Written with StackEdit.

No comments: