SuppressionList, BatchRobot, ActiveProspect ID outages

Incident Report for ActiveProspect

Resolved

At 12:18 PM central time one of our application servers began alerting on low memory. At that time it was determined that we needed to update a software package and reboot. After both were done, the affected services failed to start normally. The root cause was determined to be an incompatibility between a library and the new software package. This was an unexpected outcome due to the fact that the software package update was only a new patch version.

Our team worked to find the root cause and fix it. Services were back online around 1:26 PM. This issue would have been avoided had we properly verified correct functionality of services on one server before applying the new software and rebooting the other. We are revising our maintenance guidelines accordingly.

Posted Jun 01, 2015 - 14:42 CDT