Beginning early Monday morning on October 19 at approximately 3:30am (CT), TrustedForm experienced a degradation in services. During this period, the ability to create claims was impacted. This degradation in services continued until we resolved the issue that night (Oct 19) around 11:15pm (CT).
As soon as we resolved the issue affecting our systems, all applications and services quickly returned to normal.
During a one hour period between 1:55pm (CT) and 2:55pm (CT) and a 20 minute period between 5:30pm (CT) and 5:50pm (CT) on October 19, our services degraded to the point where we were unable to capture certificate and event data from pages running the TrustedForm script. Certificate information from this period will either be incomplete or missing entirely.
Our engineering staff has resolved both incidents, which were the consequence of the same issue: a customer using our product in an unorthodox way. This problem had a cascading effect across the TrustedForm services, requiring more processing time to retrieve data from our database and presenting it to the requester.
We are now monitoring this sort of usage, and considering putting limits in place that would prevent it.
Last Tuesday (Oct 20), we reprocessed those claims which came through the LeadConduit system during this time. If you used the API, our system will have no record of those claims submitted. There was a small number of claims that could not be resubmitted.
As stressful as this was for our team, we know these sorts of incidents are equally stressful for you. It's for this very reason that we take these matters seriously, continually looking for ways we can improve stability in our applications, and ensuring that when problems do arise that we communicate them to you as soon as we are able.
We apologize for last week’s disruption and the delay in the postmortem as we isolated the cause, and we are grateful for your patience.