This evening TrustedForm had an outage that affected certificate generation. Certificates generated between approximately 4pm central and 6pm central cannot be claimed. The data collected on the webforms for certificates was not successfully written to our database.

We were running what was meant to be a limited test as part of an ongoing TrustedForm replatforming, but human error resulted in a misconfiguration which caused this problem. We take this very seriously.

Please see the below FAQ.

Can the certificates generated during this incident be claimed?
No. The certificates generated during this time period were not properly recorded by our systems. The certificate URLs you have captured with leads generated during this time period cannot be claimed.

Why did this happen?
In short: human error. We ran what was meant to be a limited test of a new data store which we are implementing as part of our replatforming effort. This data store is faster, more resilient, and able to handle huge bursts of traffic. When we ran our test, we made a configuration error which caused this outage.

What are we doing to ensure this won’t happen again?
We are taking a variety of steps to ensure this won’t happen again. Firstly, we have identified the failure in our process which was meant to protect against this kind of error. We have corrected that failure. Secondly, we are adjusting our alerting to ensure that if it does happen again, we can catch it more quickly. Thirdly, we are replatforming the system to reduce complexity and increase our ability to more effectively control changes such as this one. Lastly, we are moving to a cloud provider which will allow us to implement the necessary changes.

Who is affected by this incident?
All TrustedForm customers are affected, with the exception of those using only TrustedForm for Facebook. That solution was not affected by this incident.

What TrustedForm operations are affected?
The claim API, the ping API, and the check-in API on certificates that were issued during this time period will be unsuccessful.

What will the TrustedForm claim API return for affected certificates?
The TrustedForm claim API will return an HTTP 404 (not found) for certificates generated during this time period. A limited number of calls to the claim API returned HTTP 500 (internal server error) during the incident.

Was TrustedForm for Facebook affected?
No. TrustedForm for Facebook uses a separate infrastructure and was not impacted by the changes made for this test.