Why did AWS Support fail with US-EAST-1 again? (2024)

Amazon has blamed a “subsystem responsible for capacity management for AWS Lambda” for an AWS outage in its US-EAST-1 region that took down over 100 services for approximately four hours late on June 13.

Despite recent architectural changes to improve AWS Support resilience AWS said that “customers may also have experienced issues when attempting to initiate a Call or Chat to AWS Support” during the incident.

The AWS outage took place from 11:49am PDT to 15:37pm PDT.

Given its scale, few companies have the ability to affect as many customers in one fell swoop as AWS does; the incident affected FIFA, Fox News, and the McDonalds app amongst many hundreds of others.

Some 104 AWS services including AWS Account Management, CloudWatch, Glue, Fargate, Secrets Manager and more saw “increased error rates and latencies” and AWS said that “customers may have experienced authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS…”

The root cause of the issue was rapidly resolved and no cloud hyperscaler (or indeed on-premises data centre) can avoid sporadic issues. The impact on AWS Support that may leave the hyperscaler with some questions to answer, however, after promised changes to AWS Support infrastructure.

In late 2021 AWS promised to build a “new support system architecture that actively runs across multiple AWS regions” amid criticism of US-EAST-1 (AWS’s most fragile region) being a single point of failure for AWS Support when severe outages like one in December 2021 happen.

This issue was flagged in Gartner’s 2022 Magic Quadrant for Cloud Infrastructure and Platform Services, which pointed to AWS’s “regional dependencies and communication” as cause for some concern.

Join peers following The Stack on LinkedIn


As Gartner’s analysts put it in October:AWS’s operational incident of 7 December 2021 revealed some multiregion dependencies on the internal AWS network, which is hosted in US-EAST-1. Because US-EAST-1 also hosts support ticketing for North America, AWS customers also had difficulty communicating with technical support during the incident…

Yet in a note published on August 1, 2022 AWS said it had launched a new “AWS Support Center console URL… [that] ensures you can always contact AWS Support via the AWS Support Center Console… built using the latest architecture standards for high availability and region redundancy.”

Why this rebuild did not resolve the perennial issue of US-EAST-1 outages taking out AWS Support with them remains an open question.

The Stack has put it to AWS and will update this if we get an answer.

See also: Azure mauled over fresh cross-tenant vuln

Why did AWS Support fail with US-EAST-1 again? (2024)

FAQs

What caused AWS to go down? ›

AWS outage causes: Internal issues

As AWS describes it, this “triggered an unexpected behavior from a large number of clients inside the internal network”. Basically, AWS unintentionally triggered a Distributed Denial of Service (or DDoS attack) on their own internal network.

Has AWS ever had a region failure? ›

September 20, 2015: Amazon's DynamoDB Outage

In September 2015, AWS's US-East region experienced a brief network disruption impacting its NoSQL database service, DynamoDB. As a result, some of the internet's biggest sites and cloud services were made unavailable.

What is the root cause of AWS Lambda outage? ›

AWS narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including those using an API Gateway) and indirectly through other AWS services.

What is AWS US East 1? ›

US-EAST-1 is Amazon's oldest region, dating its genesis all the way to 2006. In theory, it is the cloud colossus's most resilient region, as it boasts six availability zones and 10 local zones.

Why is AWS slowing down? ›

The law of large numbers will naturally slow down its growth rate over time, but AWS has felt headwinds from optimization efforts and variable declines amongst customers. Again, this is a good thing - it means AWS can participate in the downside and the upside as consumption patterns change.

Why there is no AWS region in Russia? ›

No major cloud service provider is willing to take the risk of opening a region in Russia because of security reasons. There's no way anyone can guarantee that the russian security services will not simply barge in at gunpoint and tamper with or try to hook into the platform infrastructure directly.

Which AWS region has the most outages? ›

0:00:00. N. Virginia emerged as the most unreliable AWS Region, topping the list with a total of 23 partial outages in 2022, which collectively lasted for 61 hours and 7 minutes.

What happens if a region goes down in AWS? ›

If one availability zone fails, your application can continue to run in another availability zone without any interruption. However, if the entire region fails, your application will fail.

Why is AWS Lambda so slow? ›

To speed up AWS Lambda performance, focus on reducing cold starts. Cold starts can slow things down by making functions take longer to start running. When you minimize cold starts, you ensure your functions are ready immediately.

Why not to use Lambda AWS? ›

Limited Control. With AWS Lambda, control is another issue that deters teams. Since Lambda functions run on Amazon Machine Instances, they use industry-standard tools for development. Teams do not have the freedom to custom install packages or software on the running environment.

Why is it called AWS Lambda? ›

If you're unfamiliar with Lambda, it's what has been termed a "serverless" computing service from AWS, which uses events to drive resource allocation and computation. Lambda is named after functions from lambda calculus and programming. Those functions act as a good analogy for the service.

Is Cloudfront in US-East-1? ›

Cloud front is a global CDN but it is managed from the global region of us-east-1.

Does the US military use AWS? ›

Leading government agencies around the world use AWS to lower costs, become more agile, and innovate faster. The Joint Warfighting Cloud Capability (JWCC) contract enables AWS to continue providing Department of Defense (DoD) customers with secure, reliable, and mission-critical cloud services.

Does the IRS use AWS? ›

The GSS AWS boundary contains all servers and devices that are directly managed by the IRS. This includes the operating system, middleware, and database management systems hosting IRS applications.

Why is AWS laying off? ›

In explaining why AWS had decided to make the staff cuts, a spokesperson stated: “We've identified a few targeted areas of the organization we need to streamline. These decisions are difficult but necessary as we continue to invest, hire, and optimize resources to deliver innovation for our customers.

What happens if the AWS is down? ›

Aside from Amazon and thousands of other AWS web sites going down, there would be a mad scramble to restore the servers. It's much more likely that one data center would go down, so web sites that have replicated across multiple data centers would not be affected much, if at all.

How much did the AWS outage cost? ›

Amazon Web Services, March 2017, $150 – $160 million

According to a report by Axios, the AWS downtime could have incurred losses of $150 and $160 million for the S&P 500 and financial services companies affected.

Would Amazon have survived without AWS? ›

So to give you a direct answer, no Amazon wouldn't be profitable as of now without AWS. it took Amazon more than 14 years—58 quarters after its May 1997 initial public offering—to make, cumulatively, as much profit as it produced in the latest quarter alone.

Top Articles
Latest Posts
Article information

Author: Arline Emard IV

Last Updated:

Views: 6613

Rating: 4.1 / 5 (52 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Arline Emard IV

Birthday: 1996-07-10

Address: 8912 Hintz Shore, West Louie, AZ 69363-0747

Phone: +13454700762376

Job: Administration Technician

Hobby: Paintball, Horseback riding, Cycling, Running, Macrame, Playing musical instruments, Soapmaking

Introduction: My name is Arline Emard IV, I am a cheerful, gorgeous, colorful, joyous, excited, super, inquisitive person who loves writing and wants to share my knowledge and understanding with you.