Elevated Errors
Incident Report for ActiveProspect
Postmortem

At approximately 9:28am central time, our database cluster suffered the loss of its primary member. Normally during this condition, the cluster automatically fails over to one of the several surviving members. However, due to a recent infrastructure change the failover process failed unexpectedly.

After diagnosing the problem, our team set out to fix the infrastructure change that was preventing failover from working properly. At approximately 10:25am central time, the database cluster properly failed over to one of the surviving nodes.

During this outage SuppressionList API calls and TrustedForm claim API calls were failing. If you are directly calling the TrustedForm Claim API (rather than using LeadConduit to do so), you should retry your claim calls that resulted in error.

LeadConduit was still handling leads correctly but lead events were a few minutes delayed making it into the database. LeadConduit Classic was receiving leads correctly, but some were processed in a delayed fashion.

There should be no data loss from this outage. If you have any questions or concerns, please reach out to us at support@activeprospect.com

Posted Jan 17, 2018 - 10:54 CST

Resolved
We have resolved the outage and are working on a full post-mortem.
Posted Jan 17, 2018 - 10:28 CST
Monitoring
The issue was traced to a database node failure. Normally this situation is handled with automatic failover. In this case, however, the failover didn't work properly. We're still investigating why the failover didn't work and are planning to take the appropriate steps to prevent this condition from occurring again
Posted Jan 17, 2018 - 10:28 CST
Update
We have identified the issue and the team is working towards a fix. We hope to have full service restored shortly.

Currently, LeadConduit lead handling is not affected, but LeadConduit Classic leads are being spooled and filtered leads are not delivering.
Posted Jan 17, 2018 - 09:44 CST
Identified
We are currently investigating errors across our LeadConduit, SuppressionList and TrustedForm products. We will update with more information as it becomes available.
Posted Jan 17, 2018 - 09:34 CST