All leads spooled during the outage earlier have now been replayed, and LeadConduit Classic continues to run normally. Database deletion jobs have been paused until we finish investigating the exact cause of today's problem.
We apologize for today's problems. Maintaining uninterrupted service is our number-one priority, and we'll be looking at everything we can do to prevent this from happening again.
Posted Feb 02, 2016 - 16:50 CST
Monitoring
The root cause of this outage has been resolved. LeadConduit is now running at full capacity. We are replaying all spooled leads now.
Our team traced the root cause to a large database deletion that caused a lock in our database cluster. We rolled the offending transaction back, but the rollback took an inordinate amount of time to complete. Once it completed, the database cluster was still non-responsive. After running diagnostics, we determined that a restart of the database master server was necessary to restore service. Once that was complete, the database cluster came back online.
We are currently monitoring the database for additional signs of duress, but don't expect an issue going forward.
Posted Feb 02, 2016 - 14:55 CST
Update
We are still working to resolve the issue. We don't currently have an ETA, but will update again shortly. Leads are still being spooled and will be replayed after the system has been brought back online.
Posted Feb 02, 2016 - 14:23 CST
Identified
The root cause of the problem has been identified as a database resource issue. We're doing everything we can to correct the problem. Incoming leads are still being saved and will be processed as soon as possible once the system has recovered.
Posted Feb 02, 2016 - 13:13 CST
Investigating
We are investigating a Partial Outage of LeadConduit Classic. The UI is currently down and realtime delivery is affected, but we are spooling leads so inbound data is not lost.