January 2017

LeadConduit Performance Issues
All services are back to normal and stable.
Jan 3, 13:43-16:26 CST

December 2016

Service restored following a database cluster failure
Tonight at 1:24am central the primary member of our database cluster failed. TrustedForm, SuppressionList, and the LeadConduit Classic filter functionality were all impacted by the failure and were unavailable until the cluster member was restored around 2:15am. Normally in a failure scenario such as this one, one of the other two members of the database cluster will be elected as the primary member and our apps will continue operating correctly after a brief period. That didn't happen tonight. We're still investigating why this didn't happen. In the meantime, all services are up and operating correctly.
Dec 24, 02:24 CST

November 2016

No incidents reported for this month.

October 2016

No incidents reported for this month.

September 2016

TrustedForm: Expanded Video Support
While deploying TrustedForm's new expanded video support, we discovered an issue with IE 6/7/8/9 that caused claims to error. Claims were successfully stored, however, the http api call returned an error.
Sep 30
[Scheduled] LeadConduit Classic & BatchRobot database udpate
The update has been completed successfully. All services are back to normal; leads received by LeadConduit Classic during the maintenance window have been processed.
Sep 13, 21:01-21:26 CST

August 2016

No incidents reported for this month.

July 2016

No incidents reported for this month.

June 2016

TrustedForm claim problems
A delay was detected in the time between when certificates were issued and when they became available. No data was lost. You may re-claim any certificates previously came back as errors.
Jun 24, 11:27-11:52 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files between 1:08am and 1:16am CDT. The issue has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. Undelivered files will be included in the next delivery.
Jun 18, 01:19 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files between 12:14am and 1:22am CDT. The issue has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. Undelivered files will be included in the next delivery.
Jun 3, 01:28 CST
TrustedForm Database Node Issues
TrustedForm continues to operate normally.
Jun 2, 09:19-12:23 CST

May 2016

[Scheduled] LeadConduit Classic database maintenance
All maintenance is finished, and all spooled leads have been replayed. LeadConduit Classic and BatchRobot are now back to full, normal operation.
May 28, 21:52-22:21 CST
LeadConduit Classic database problem
All leads that were not correctly queued since last night at approximately 10PM CT have now been queued and will be delivered as normal. If you see leads that arrived between that time and this correction that appear to be "stuck" or have been mishandled, please contact support@activeprospect.com. Our sincere apologies for the inconvenience.
May 26, 11:50-13:00 CST
[Scheduled] LeadConduit Classic database maintenance
All spooled leads have been processed and LeadConduit Classic is operating normally.
May 25, 21:00-22:04 CST
BatchRobot database maintenance
The database upgrade affecting BatchRobot has been resolved.
May 25, 21:23-21:41 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files between 1:45am - 2:02am CT. The issue (background processes not working as expected) has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. Undelivered files will be included in the next delivery. We are continuing to investigate why these processes are experiencing failures.
May 17, 02:04 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files between 12:45 - 1:00am CT. The issue (background processes not working as expected) has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. Undelivered files will be included in the next delivery.
May 16, 01:03 CST
LeadConduit Classic database issue
The issue has been resolved, and all spooled leads have been processed. Our database monitors have been updated to better account for the condition that triggered this incident.
May 9, 14:30-14:54 CST

April 2016

[Scheduled] LeadConduit Classic database maintenance
All spooled leads have been replayed and the system is back to normal.
Apr 24, 21:32-21:42 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files between 8:55pm and 11:15pm CDT. The issue has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. Undelivered files will be included in the next delivery.
Apr 16, 23:38 CST
Lead Handling Down
We have successfully rolled back and the system has stabilized. We will continue to investigate what led to this release issue and will work to mitigate future outages.
Apr 15, 10:57-11:21 CST
Database outage
Starting around 6:45am central time, one of our database nodes became unresponsive. Normally this situation is handled with automatic failover. In this case, however, the failover didn't work properly. We were able to restore service by bringing the errant database node back online. We're still investigating why the failover didn't work and are planning to take the appropriate steps to prevent this condition from occurring again.
Apr 7, 08:04 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files overnight. The issue (background processes not working as expected) has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. Undelivered files will be included in the next delivery.
Apr 5, 09:40 CST

March 2016

No incidents reported for this month.

February 2016

[Scheduled] LeadConduit Classic URL Change
The scheduled maintenance has been completed.
Feb 29, 21:00-21:24 CST
[Scheduled] LeadConduit Classic & BatchRobot database update
The database maintenance is complete, the applications are running normally and all spooled leads have been processed.
Feb 18, 22:01-23:08 CST
TrustedForm
From 7:51 AM CT to about 8:12 AM CT TrustedForm was listed as undergoing a Major Outage. We were able to track this to an issue with load on the app and resolve it fairly quickly. This issue did not affect issuing of certs, but did affect cert claiming. No certs were lost during this outage.
Feb 12, 08:26 CST
TrustedForm Claims
TrustedForm continues to operate normally.
Feb 4, 10:30-17:13 CST
Network problems affecting LeadConduit, LeadConduit Classic, BatchRobot
This incident has been resolved.
Feb 2, 23:57 - Feb 3, 00:27 CST
LeadConduit Classic
All leads spooled during the outage earlier have now been replayed, and LeadConduit Classic continues to run normally. Database deletion jobs have been paused until we finish investigating the exact cause of today's problem. We apologize for today's problems. Maintaining uninterrupted service is our number-one priority, and we'll be looking at everything we can do to prevent this from happening again.
Feb 2, 12:44-16:50 CST

January 2016

[Scheduled] Leadconduit Classic, BatchRobot, SuppressionList, and ActiveProspectID server updates
The scheduled maintenance has been completed.
Jan 27, 11:00-11:39 CST
TrustedForm
The system continues to operate normally.
Jan 26, 09:15-21:36 CST
TrustedForm Claims
We have fixed the issue. The root cause was a denial of service. We've blocked the offending traffic. The outage caused the loss of certificates generated between the hours of 7:45am and 2:45pm central today. Additionally, any unclaimed certificates from the last 3 days can no longer be claimed. This is unacceptable for us and we have a list of items to change in the coming days to prevent this from happening again. Among them are more monitoring to help us identify low-level anomalies in our database cluster and more stringent traffic control. Please accept our sincere apologies for this outage.
Jan 26, 14:05-15:28 CST
BatchRobot issue
BatchRobot was inaccessible from approximately 2:15 PM CST until 2:45 PM CST due to high load on the database. Deliveries during this time were sent after the load subsided.
Jan 14, 15:36 CST
LeadConduit Classic issue
Load problems with LeadConduit's master database this afternoon caused problems accessing the system for approximately 30 minutes. Lead processing was also delayed, and a small number of leads had to be "spooled" for later processing due to this issue. The system is back to normal, and those leads have now been replayed and processed as usual.
Jan 14, 15:12 CST

December 2015

[Scheduled] ActiveProspect ID server update
The scheduled maintenance has been completed.
Dec 15, 13:15-13:30 CST
TrustedForm
We've resolved this incident, and are taking steps to resolve it in the long term.
Dec 11, 09:54-14:05 CST
[Scheduled] Suppressionlist server update
The scheduled maintenance has been completed.
Dec 8, 09:30-09:45 CST
TrustedForm Certificate Source Information
We've resolved an issue where a small number of TrustedForm Certificates contained incorrect source information.
Dec 7, 17:39 CST
[Scheduled] LeadConduit Classic upgrade
The scheduled maintenance has been completed.
Dec 7, 10:00-10:30 CST
TrustedForm: New Certificates Not Captured
On the evening of Monday November 30th, TrustedForm encountered a problem that prevented new Certificates from being stored in the database. Each impacted lead had what appeared to be a valid Certificate URL, however the associated Certificate was not stored in the database, and consequently could not be claimed. The symptom of this outage was that an attempt to claim an impacted certificate would have resulted in a "Not Found" message (HTTP 404). There are two periods during which Certificate storage was impacted: 8:04pm central to 8:40pm. And again from 10:56pm to 12:36pm. Certificates generated during those times are lost and cannot be recovered. What follows is a summary of the technical problem we encountered. The root cause of the outage was heavy database load resulting from a one-off query being run to service a customer support request. This condition uncovered a bug in the way TrustedForm handles performance loss at the database layer. Normally, high database load should not result in a failure to write records. Once our team realized that there was a reduction in the number of Certificates being generated, the query was stopped and the condition resolved. Our highest priority is ensuring a stable and bug free system. The underlying bug has been identified and we are currently working on a resolution. In the meantime we're monitoring database utilization closely so we can intervene on anomalous spikes. Typically, the database runs with very low resource utilization so we're not expecting any further trouble. We take these kinds of problems very seriously. Please accept our sincere apologies for this outage, and let us know at support@activeprospect.com if there is any further information or help we can provide.
Dec 1, 14:30 CST

November 2015

[Scheduled] LeadConduit Classic server update
The scheduled maintenance has been completed.
Nov 30, 14:00-14:53 CST
[Scheduled] BatchRobot BugFix
The BatchRobot BugFix is complete and no user-facing issues are expected. If you notice issues with a BatchRobot delivery that you suspect may be related to this change, please email support@activeprospect.com.
Nov 17, 15:00-15:06 CST
LeadConduit Classic Filtering
At approximately 4:00 PM CT on 11/09/2015, we updated the filtering functionality in LeadConduit Classic, inadvertently causing a change in the way it works, and causing some lead delivery problems. Full resolution was made today at approximately 9:00 AM CT. A change in one of the libraries we use caused the filter behavior to change. When a filter blocked a lead, it resulted in an 'error' outcome rather than a 'failure' outcome. The issue was not uncovered until approximately 8:30 AM CT on 11/10/2015, at which time we were able to identify the issue and patch the code. If you use filtered deliveries, the impact to you depends on the configuration of the affected destination's "on delivery error" setting under the multi-destination behavior. If it's set to “do not continue to the next destination, retry later”, then the process is halted and retried a few minutes later. If the error occurs again, then the process repeats until the error resolves. If it's set to the default, "continue to next destination, retry later", then that destination will be skipped and the remainder of the destinations will be processed. Later, that destination will be retried. If the error occurs again, then the destination is retried until the error resolves. If your deliveries have alerts that disable destinations after a certain number of consecutive errors, then this issue could have also triggered that. We apologize for the inconvenience, and are continuing to look into what happened and the oversight related to this release in an effort to prevent similar situations in the future.
Nov 10, 10:21 CST

October 2015

No incidents reported for this month.

September 2015

No incidents reported for this month.

August 2015

LeadConduit Classic UI Outage
LeadConduit continues to operate normally.
Aug 27, 07:45-09:44 CST
EULA Updated and Testing Needed for New VideoReplay Feature
We've updated our End User License Agreement and are preparing to roll out TrustedForm VideoReplay in 4 - 6 weeks. If you use TrustedForm, please visit this page to learn how to test it as soon as possible: http://activeprospect.com/video-replay-faq/. If you have vendors who use TrustedForm, please share the page with them.
Aug 7, 16:38 CST
[Scheduled] Database upgrade
The database upgrade has been completed successfully.
Aug 3, 21:00 - Aug 4, 00:43 CST

July 2015

No incidents reported for this month.

June 2015

SuppressionList, BatchRobot, ActiveProspect ID outages
At 12:18 PM central time one of our application servers began alerting on low memory. At that time it was determined that we needed to update a software package and reboot. After both were done, the affected services failed to start normally. The root cause was determined to be an incompatibility between a library and the new software package. This was an unexpected outcome due to the fact that the software package update was only a new patch version. Our team worked to find the root cause and fix it. Services were back online around 1:26 PM. This issue would have been avoided had we properly verified correct functionality of services on one server before applying the new software and rebooting the other. We are revising our maintenance guidelines accordingly.
Jun 1, 14:42 CST

May 2015

No incidents reported for this month.

April 2015

No incidents reported for this month.

March 2015

Trustedform and SuppressionList outage
Between 9:50pm and 10:30pm CDT, our data store was unresponsive. During this time, access to trustedform and suppressionlist were unavailable.
Mar 15, 22:53 CST
[Scheduled] LeadConduit Classic server maintenance
All leads captured during the maintenance window have now been processed.
Mar 11, 22:30-23:34 CST
LeadConduit database under heavy load
The backlog of leads has now been completely replayed. All LeadConduit services are back to normal. A postmortem is to follow.
Mar 10, 20:25-21:56 CST
BatchRobot delayed deliveries
At 2015-03-05 06:50 UTC, our provider SoftLayer performed maintenance which prevented scheduled BatchRobot deliveries from occurring. Affected deliveries will include undelivered leads in the next scheduled delivery, and can be manually triggered to deliver sooner.
Mar 5, 10:30 CST

February 2015

No incidents reported for this month.

January 2015

[Scheduled] LeadConduit server maintenance
This maintenance has been successfully completed. The UI is back online, leads delayed during the downtime have been processed, and incoming leads are being handled normally again.
Jan 29, 21:01-21:50 CST
BatchRobot and LeadConduit outage
At 2015-01-23 03:25 UTC Our provider SoftLayer experienced a network partition which impacted connectivity to BatchRobot and LeadConduit Classic. Approximately 10 minutes later, the network began operating properly again. During that time, lead submissions would not have succeeded because our servers could not be contacted.
Jan 22, 22:54 CST

December 2014

No incidents reported for this month.

November 2014

No incidents reported for this month.

October 2014

LeadConduit Filter outage
This incident has been resolved.
Oct 15, 15:24 CST
Intermittent LeadConduit outages
The problem has been identified and resolved. One of our servers experienced an out-of-memory condition. After resolving the condition, the service is fully operational again.
Oct 1, 12:07-12:14 CST

September 2014

TrustedForm degraded performance
TrustedForm is now responding normally. Our team continues to look into the root cause of the slowness.
Sep 2, 10:56-11:31 CST

August 2014

LeadConduit Issues
We have identified and resolved the issue with real-time lead handling and the UI issues with LeadConduit. Inbound real-time leads should be handled appropriately. We are monitoring LeadConduit closely and will replay spooled leads shortly.
Aug 21, 14:21-14:41 CST
BatchRobot Delivery SFTP Failures
This morning there was a BatchRobot SFTP delivery problem reported. The problem has been investigated, fixed, and put into current operational usage. If any customers are missing any scheduled deliveries that take advantage of SFTP it is advised to redeliver the failed scheduled deliveries at this time.
Aug 18, 12:59 CST
DNS trouble for custom adapters
Connection tests from geolocated servers have determined that the DNS changes have has propagated to a large number of servers across the Internet.
Aug 14, 20:01-21:32 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files overnight. The issue (background processes not working as expected) has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI. We are aware of intermittent issues regarding file generation and delivery and continue to research long term solutions to these issues.
Aug 12, 10:27 CST
BatchRobot File Generation and Delivery Issues
BatchRobot experienced issues generating and delivering some files overnight. The issue (background processes not working as expected) has been resolved and failed files can be regenerated and redelivered from within the BatchRobot UI.
Aug 11, 09:03 CST
BatchRobot File Generation Errors
The issue with generating and delivering files in BatchRobot has been identified and resolved. Some backend processes were not responding as expected and were restarted. Previously failed files can be regenerated and redelivered from within the BatchRobot UI
Aug 8, 09:44-10:39 CST
[Scheduled] BatchRobot database maintenance
This took somewhat longer than planned, but BatchRobot is back up online and fully functional.
Aug 6, 21:00-23:09 CST
SuppressionList database outage
After running a database maintenance job this issue has been resolved. We will follow up shortly with a software release which will prevent this issue from occurring again.
Aug 6, 22:02-23:07 CST
TrustedForm degraded performance
At approximately 11:50am CDT, the TrustedForm application experienced degraded performance, where api calls and web requests might have been delayed and possibly timed out. During this time, certificate issuing requests were not impacted. This issue was addressed by 11:55am CDT.
Aug 6, 12:12 CST

July 2014

Legal Terms Updated
We have updated our legal terms, effective July 9, 2014. You can review the new ActiveProspect Terms of Service here: http://activeprospect.com/legal/tos. These terms apply to all ActiveProspect services. They also incorporate our "Acceptable Use Policy," which was previously provided in a separate document. Additionally, we have implemented a TrustedForm End User License Agreement for publishers using the TrustedForm Script on their websites. This agreement was previously called the TrustedForm Terms of Service or, alternatively, the TrustedForm Publisher Terms. It has been revised and updated and is available for review at http://activeprospect.com/legal/trustedform-eula. Unless you have executed a separate written agreement with ActiveProspect, your continued use of the ActiveProspect services or the TrustedForm Script represents your consent to the revised ActiveProspect Terms of Service and TrustedForm EULA referenced above.
Jul 10, 15:23 CST

June 2014

Lead handling issue
The last of the spooled leads have now been processed.
Jun 30, 12:07-22:16 CST
[Scheduled] Trustedform deployment
The scheduled maintenance has been completed.
Jun 10, 09:00-23:45 CST

May 2014

[Scheduled] LeadConduit database maintenance
The scheduled maintenance has been completed.
May 21, 21:54-22:16 CST

April 2014

"Heartbleed" Security Alert
Last Monday, April 7, internet security researchers reported the discovery of Heartbleed, an internet-wide vulnerability in the encryption protocol known as Secure Sockets Layer (SSL). We took immediate corrective actions Monday, and by Wednesday we completed all the fixes necessary to ensure the security and continuous operation of all ActiveProspect services. Those measures included: * Updating affected libraries to patched versions * Re-keying of all SSL certificates * New passwords for all system level accounts * New keys for all system-level services that grant access There is no evidence of intrusion or unauthorized access to client data. As an additional precaution, we’re asking all ActiveProspect users to immediately change their passwords and API keys. Each password and API key will need to be reset separately for LeadConduit and ActiveProspect ID. If you have any questions about managing your accounts, please contact customer support.
Apr 11
New ActiveProspect Status Page
We have implemented a new system status page: http://status.activeprospect.com, which consolidates key system information into a single view. On this new page, you can clearly see current status of all the systems in the ActiveProspect platform, as well as historical up-time data for each. The status page also will provide outage incident reports and announcements of upcoming planned maintenance, such as have previously been communicated on this blogs. You can subscribe to notifications of new incidents and announcements via email, SMS, and RSS.
Apr 10