Spain and Portugal were plunged into an unprecedented power blackout on April 28, the largest outage on the Iberian Peninsula in living memory. But while remote team members based there returned to pen and paper for the day, it was business as usual at our Spanish data center: ES-MAD1.
We’re pleased to report that there was no visible interruption in data center operations, and customers were able to continue operating their infrastructure.
How did we manage this?
In the initial aftermath of the outage, we experienced some reduced redundancy as our operational incident management procedure kicked into play and on-site power automatically transferred to back-up generators. Our data centers are designed to be self-sufficient, even in rare scenarios like these, and we remained vigilant – constantly monitoring the situation, and had an emergency plan ready for potential worst case scenarios
During the remaining hours of the blackout, two IP transit connections and two internal core networks were up and running. For a short period, we noticed occasional downtime with some of our multiple external network connections – this was due to some providers’ equipment rebooting because of the sudden outage.
You can see from the data relayed from one of our Internet Exchange peering links just how sudden and unprecedented the outage was. You can also see when the IX peering went down for a few minutes, but also how quickly this came back online. It’s worth noting that our multiple ISP policy and redundant core network connections to other zones means that even one transit going down doesn’t affect availability.
Preparation pays off
“This outage was a once-in-a-generation event but I’m pleased to report that there was no impact to customers using our Spanish data center. Ensuring customer data is completely secure, and that their mission-critical services can run uninterrupted is part of our DNA. We prepare relentlessly for events like this, and our procedures and iron-clad infrastructure have proven immensely robust, even in the midst of a major international power outage.” – Teemu Toivanen, Head of Support, UpCloud.
