Issues Related to Current AWS Outage

Incident Report for ActiveProspect

Postmortem

Yesterday (Dec 7) around 7:30 am CT, Amazon's cloud services experienced a significant outage in their US-EAST-1 region of service. The outage was not completely resolved until around 2:30 pm later in the day. As a result of this, ActiveProspect's TrustedForm product was also affected since it uses many of the services in the affected region. The TrustedForm team monitored the situation as best as we could and relayed information from our AWS contacts as soon as it became available.

Having had time to assess the event, we believe the impact to us – and more importantly, to you – is minimal. We saw an uptick in storage latency and and increase in 404s and 50x errors for claiming. We believe, however, that there was little impact to the capture of publisher Certs and the data which makes them up.

We have re-processed claims from that time period to ensure they are properly archived and available to you. Note: this will not affect your billing.

Resiliency is important to us and we're proud that at a time when many websites and services were offline across the web, ours was mostly operating as designed. Nevertheless, we are always looking at ways to make our systems less susceptible to outages.

Thank you for your patience during this event and for trusting ActiveProspect with your business. If you have any questions regarding yesterday's outage, please reach out to your representative.

Sincerely,
The TrustedForm Team

Posted Dec 08, 2021 - 12:51 CST

Resolved

According to Amazon, they've addressed the outage. All our services should be fully operational now.

"We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. We are seeing improvement in availability across most AWS services. All services are now independently working through service-by-service recovery. We continue to work toward full recovery for all impacted AWS Services and API operations. In order to expedite overall recovery, we have temporarily disabled Event Deliveries for Amazon EventBridge in the US-EAST-1 Region. These events will still be received & accepted, and queued for later delivery."

We will do our own investigation to determine what impact this had on the TrustedForm services, address any missing claims, and provide an update our findings as soon as we can.

Thank you for your patience throughout today outage,
The TrustedForm Team

Posted Dec 07, 2021 - 17:01 CST

Update

No changes in status. No updates from AWS.

Posted Dec 07, 2021 - 15:54 CST

Update

No changes in status.

Update from Amazon:

[12:34 PM PST] We continue to experience increased API error rates for multiple AWS Services in the US-EAST-1 Region. The root cause of this issue is an impairment of several network devices. We continue to work toward mitigation, and are actively working on a number of different mitigation and resolution actions. While we have observed some early signs of recovery, we do not have an ETA for full recovery. For customers experiencing issues signing-in to the AWS Management Console in US-EAST-1, we recommend retrying using a separate Management Console endpoint (such as https://us-west-2.console.aws.amazon.com/). Additionally, if you are attempting to login using root login credentials you may be unable to do so, even via console endpoints not in US-EAST-1. If you are impacted by this, we recommend using IAM Users or Roles for authentication. We will continue to provide updates here as we have more information to share.

Posted Dec 07, 2021 - 14:47 CST

Update

AWS is still reporting issues with US-EAST-1, however we are seeing greatly increased performances compared to earlier. We will continue to monitor a post messages once per hour.

From AWS's status page:

[11:26 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time. Root logins for consoles in all AWS regions are affected by this issue, however customers can login to consoles other than US-EAST-1 by using an IAM role for authentication.

Posted Dec 07, 2021 - 13:47 CST

Update

We are continuing to monitor the AWS issues with US-EAST-1. We have seen an overall improvement, but they are still showing degraded services. You can see the specific services affected at https://status.aws.amazon.com/.

Posted Dec 07, 2021 - 12:44 CST

Update

Latest from AWS:

We are working on multiple parallel paths to mitigating this issue in the US-EAST-1 Region. We are starting to see early signs of recovery, but we are not out of the woods yet. We will provide another update shortly on our path to recovery.

Posted Dec 07, 2021 - 11:55 CST

Update

Update: We are seeing some of AWS's services get restored. We will continue to monitor the outage and update you as the situation changes.

From earlier.

AWS services for "us-east-1" are currently experiencing an outage. TrustedForm uses this region for many of its services and is therefore affected. We are seeing issues in the following areas:

- Timeouts in certificate creation and data capture
- Slowdowns on claiming and extending certs

Posted Dec 07, 2021 - 11:26 CST

Monitoring

AWS services for "us-east-1" are currently experiencing an outage. TrustedForm uses this region for many of its services and is therefore affected. We are seeing issues in the following areas:

- Timeouts in certificate creation and data capture
- Slowdowns on claiming and extending certs

We are monitoring this outage closely and will provide regular updates (every 30 minutes) whether or not that situation changes.

Posted Dec 07, 2021 - 10:49 CST

This incident affected: TrustedForm Application, TrustedForm Certify, and TrustedForm Facebook.