On the evening of Monday November 30th, TrustedForm encountered a problem that prevented new Certificates from being stored in the database. Each impacted lead had what appeared to be a valid Certificate URL, however the associated Certificate was not stored in the database, and consequently could not be claimed. The symptom of this outage was that an attempt to claim an impacted certificate would have resulted in a "Not Found" message (HTTP 404).
There are two periods during which Certificate storage was impacted: 8:04pm central to 8:40pm. And again from 10:56pm to 12:36pm. Certificates generated during those times are lost and cannot be recovered.
What follows is a summary of the technical problem we encountered. The root cause of the outage was heavy database load resulting from a one-off query being run to service a customer support request. This condition uncovered a bug in the way TrustedForm handles performance loss at the database layer. Normally, high database load should not result in a failure to write records. Once our team realized that there was a reduction in the number of Certificates being generated, the query was stopped and the condition resolved.
Our highest priority is ensuring a stable and bug free system. The underlying bug has been identified and we are currently working on a resolution. In the meantime we're monitoring database utilization closely so we can intervene on anomalous spikes. Typically, the database runs with very low resource utilization so we're not expecting any further trouble.
We take these kinds of problems very seriously. Please accept our sincere apologies for this outage, and let us know at support@activeprospect.com if there is any further information or help we can provide.