LeadConduit Issues
Incident Report for ActiveProspect
Postmortem

Today around 16:00 UTC we became aware of a database corruption caused by a bug in LeadConduit. As we worked to resolve the problem, we inadvertently caused additional corruption and were forced to perform a partial database restore from a daily database backup taken this morning around 13:00 UTC.

As a result, any configuration changes you made in LeadConduit today between 13:00 and 19:40 UTC (8AM - 2:40PM central) were lost to the database restore. Your leads and statistics were not affected.

In the history of the company, this may be the first time we've had to restore a database from backup. However unlikely it is to be required, we always keep onsite and offsite database backups, so we can respond quickly to these sorts problems.

We take this sort of problem very seriously. Our number one rule is to be careful stewards of our customers' data. We're working on solving the bug that caused the original database corruption. We have identified and have made a specific process adjustment that will prevent additional database corruption from occurring during this kind of repair, should it become necessary, in the future.

Please accept our deepest apologies for the trouble we caused for you today.

Posted Aug 21, 2014 - 15:49 CDT

Resolved
We have identified and resolved the issue with real-time lead handling and the UI issues with LeadConduit.

Inbound real-time leads should be handled appropriately. We are monitoring LeadConduit closely and will replay spooled leads shortly.
Posted Aug 21, 2014 - 14:41 CDT
Investigating
We are investigating an issue with LeadConduit. The UI is currently down and all inbound leads are currently being spooled.

Our team is investigating and we will update as we know more.
Posted Aug 21, 2014 - 14:21 CDT