TrustedForm Claim API Issues
Incident Report for ActiveProspect

This evening TrustedForm had an outage that affected certificate generation. Certificates generated between approximately 4pm central and 6pm central cannot be claimed. The data collected on the webforms for certificates was not successfully written to our database.

We were running what was meant to be a limited test as part of an ongoing TrustedForm replatforming, but human error resulted in a misconfiguration which caused this problem. We take this very seriously.

Please see the below FAQ.

Can the certificates generated during this incident be claimed?
No. The certificates generated during this time period were not properly recorded by our systems. The certificate URLs you have captured with leads generated during this time period cannot be claimed.

Why did this happen?
In short: human error. We ran what was meant to be a limited test of a new data store which we are implementing as part of our replatforming effort. This data store is faster, more resilient, and able to handle huge bursts of traffic. When we ran our test, we made a configuration error which caused this outage.

What are we doing to ensure this won’t happen again?
We are taking a variety of steps to ensure this won’t happen again. Firstly, we have identified the failure in our process which was meant to protect against this kind of error. We have corrected that failure. Secondly, we are adjusting our alerting to ensure that if it does happen again, we can catch it more quickly. Thirdly, we are replatforming the system to reduce complexity and increase our ability to more effectively control changes such as this one. Lastly, we are moving to a cloud provider which will allow us to implement the necessary changes.

Who is affected by this incident?
All TrustedForm customers are affected, with the exception of those using only TrustedForm for Facebook. That solution was not affected by this incident.

What TrustedForm operations are affected?
The claim API, the ping API, and the check-in API on certificates that were issued during this time period will be unsuccessful.

What will the TrustedForm claim API return for affected certificates?
The TrustedForm claim API will return an HTTP 404 (not found) for certificates generated during this time period. A limited number of calls to the claim API returned HTTP 500 (internal server error) during the incident.

Was TrustedForm for Facebook affected?
No. TrustedForm for Facebook uses a separate infrastructure and was not impacted by the changes made for this test.

Posted Jun 17, 2020 - 20:40 CDT

We have confirmed that our systems are fully functional and claims are working as expected. We plan to update customers further with the status and our plans to prevent future issues.
Posted Jun 17, 2020 - 19:31 CDT
We have confirmed an update has resolved the original issue and claims are successful. Certificate recordings that were started prior to our fix will be unsuccessful in claiming, but all certificates issued after our fix are performing as expected.
Posted Jun 17, 2020 - 17:50 CDT
We are still experiencing some missing certificates despite our reverted changes. We are investigating the cause.
Posted Jun 17, 2020 - 17:31 CDT
A fix has been implemented and we have confirmed that certificates are being stored. We will continue to monitor to ensure consistency across our servers. Claims for this period will be unsuccessful.
Posted Jun 17, 2020 - 17:07 CDT
We have identified that the certificates that are returning a "Not Found" are unavailable and are not being stored properly. We are working to resolve the issue as quickly as possible to restore functionality.
Posted Jun 17, 2020 - 17:06 CDT
New unclaimed TrustedForm Certificates are currently being displayed as "Not Found" due to a configuration change. We are investigating the impact.
Posted Jun 17, 2020 - 16:56 CDT
This incident affected: TrustedForm Application.