[RESOLVED] Outage due to service provider (Amazon Web Services) connectivity issue

We are currently experiencing a system wide outage due to connectivity issues at our service provider, Amazon Web Services.

We are attempting to route around the connectivity issue.

Updates to follow.

Update 11:58am:

Current status from http://status.aws.amazon.com/

We are continuing to attempt to route around the network issues.

Update 5:00pm:

The majority of systems are operational again. Only a small number of merchants with dedicated SSL certificates are currently affected. We are working on routing the remaining merchants around the connectivity issue.

Update 6:16pm

All systems are fully operational. 

Outage Details

Late this morning Amazon Web Services suffered a major outage affecting several critical systems within their infrastructure. Although we run several levels of redundancy, the outage impacted too many systems for the redundancy to be effective. During the day we worked to route around the problem but were prevented from doing so due to each of the strategies we have in place unable to be implemented due to maintenance functionality within the AWS console not working.

By mid afternoon the root cause of the AWS service had been addressed, but the flow on effect continued to render systems inoperative. We were able to progressively bring systems online as the afternoon progressed.

Despite this outage we are still on track to achieve greater than 99.9% uptime for May. Prior to this month we had achieved greater than 99.99% uptime consistently for quite a few months. We will apply what we have learned from this incident to further improve our systems so that future outages have even less impact.