tag:status.activeprospect.com,2005:/historyActiveProspect Status - Incident History2024-03-29T00:23:23-05:00ActiveProspecttag:status.activeprospect.com,2005:Incident/203911542024-03-28T16:58:51-05:002024-03-28T16:58:51-05:00Degraded performance for TrustedForm Certify on several sites<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>16:58</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved by rolling TrustedForm Script back to 1.9.4. We will continue investigation on issues caused by the latest version of the script.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>16:29</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>16:16</var> CDT</small><br><strong>Update</strong> - Rolling back TrustedForm Script to 1.9.4</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>15:25</var> CDT</small><br><strong>Investigating</strong> - We've received issues from 3 clients that the TrustedForm Script has not been working properly since around 3p CT on March 27th. Currently investigating and assessing impact.</p>tag:status.activeprospect.com,2005:Incident/203909682024-03-28T15:05:53-05:002024-03-28T15:05:53-05:00Incorrect outcomes for Trestle integration<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>15:05</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>14:51</var> CDT</small><br><strong>Monitoring</strong> - There was a problem with an updated version of the LeadConduit integration with Trestle, which incorrectly parsed the response from that system. Between about 2:25 and 2:50 (CT), those events in LeadConduit were incorrectly marked as "failure", though the standard filter rule steps in place in most flows did not halt processing based on that. The change has been reverted.</p>tag:status.activeprospect.com,2005:Incident/196720102024-01-10T13:00:00-06:002024-01-10T16:27:34-06:00LeadConduit Lead Export Degraded Service<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>13:00</var> CST</small><br><strong>Resolved</strong> - This afternoon from 1:04pm to 1:48pm CST LeadConduit had a faulty deployment that resulted in failed LC - Lead Exports jobs. This has been corrected and service restored. We are determining what steps we can take to handle this type of event in the future. Customers who experienced a failed export will need to retry the failed job. <br /><br />We know leads are important to the success of your business, and we take this matter very seriously. We apologize for the inconvenience this has caused you. If you have any questions or concerns, please don't hesitate to reach out to our support team (support@activeprospect.com). <br /><br />Sincerely, <br />The ActiveProspect Team</p>tag:status.activeprospect.com,2005:Incident/193687902023-12-07T11:32:12-06:002023-12-07T11:32:12-06:00LeadConduit Degraded Performance<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:32</var> CST</small><br><strong>Resolved</strong> - This morning (December 7, 2023) between 9:38am CT and 9:43am CT, LeadConduit received an influx of traffic 5x greater than normal. At the same time, our hosting provider (AWS) experienced issues with the Availability Zone (AZ) we are hosted in. The result of this was an inability for AWS to scale up our servers to handle the increased traffic. This resulted in a 5 minute period of degraded performance for our customers, specifically some leads during that time period were never received by our system.<br /><br />We are determining what steps we can take to handle this type of event in the future. In the meantime, if you are aware of leads that were not received and processed, we recommend re-submitting them.<br /><br />We know leads are important to the success of your business, and we take this matter very seriously. We apologize for the inconvenience this has caused you. If you have any questions or concerns, please don't hesitate to reach out to our support team (support@activeprospect.com).<br /><br />Sincerely,<br />The LeadConduit Team</p>tag:status.activeprospect.com,2005:Incident/191534572023-11-17T13:13:03-06:002023-11-17T13:13:03-06:00TrustedForm integration issue<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>13:13</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>12:38</var> CST</small><br><strong>Monitoring</strong> - The issue has been fixed and we are monitoring performance.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>12:12</var> CST</small><br><strong>Identified</strong> - Some customers using LeadConduit to access TrustedForm "v4" functionality may see an elevated error rate due to misconfiguration. We are working to rectify the issue as quickly as possible.</p>tag:status.activeprospect.com,2005:Incident/190769442023-11-08T09:30:00-06:002023-11-09T13:49:44-06:00Degraded Performance: TrustedForm Certify SDK<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:30</var> CST</small><br><strong>Resolved</strong> - There was a potential issue in a newly deployed version 1.9.6 of the TrustedForm Certify SDK, deployed Nov 8 at 9:36 AM CST. Some pages that use the SDK were reporting not having the certificate URL added to the form. The changes were rolled back to version 1.9.4 on Nov 8 at 1:52 PM CST.</p>tag:status.activeprospect.com,2005:Incident/186991032023-10-05T12:19:06-05:002023-10-05T12:19:06-05:00Service disruption for Facebook<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>12:19</var> CDT</small><br><strong>Resolved</strong> - The Facebook service has been processing data normally since 10:30am CT. This incident is resolved.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>11:09</var> CDT</small><br><strong>Monitoring</strong> - A fix has been successfully deployed, and our Facebook service is now operating normally while we are continuously monitoring the results.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>10:42</var> CDT</small><br><strong>Identified</strong> - Our technical team has successfully identified the issue that was affecting our services and we are working on a fix that is going to be implemented.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>08:57</var> CDT</small><br><strong>Investigating</strong> - We are currently experiencing a disruption with our Facebook service, which may affect its availability. Our technical team is actively investigating the issue to identify and resolve the root cause.</p>tag:status.activeprospect.com,2005:Incident/186235822023-09-30T09:00:16-05:002023-09-30T09:00:16-05:00LeadsBridge Database Maintenance<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>09:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>08:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>11:25</var> CDT</small><br><strong>Scheduled</strong> - We will be performing a scheduled upgrade of our database to enhance performance and security. During this upgrade, our LeadsBridge services may experience a brief interruption lasting up to 1 hour. We apologize for any inconvenience this may cause and appreciate your understanding as we work to improve our infrastructure.</p>tag:status.activeprospect.com,2005:Incident/181089712023-08-10T16:27:03-05:002023-08-10T16:27:03-05:00Increase in Error rates for retaining TrustedForm certificates<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>16:27</var> CDT</small><br><strong>Resolved</strong> - TrustedForm retain/claims errors returned to normal at approximately 3:15pm CDT and have remained normal since. Retain/claim requests that resulted in a HTTP 5xx error should be retired to ensure that the TrustedForm certificate is retained.<br /><br />We appreciate your patience as we worked to resolve these errors.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>15:23</var> CDT</small><br><strong>Monitoring</strong> - The increased error rates have been resolved and we are monitoring retain/claim requests.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>14:46</var> CDT</small><br><strong>Identified</strong> - We have identified an issue that is causing increased error responses when retaining/claiming TrustedForm certificates. We are currently working to resolve the errors. <br /><br />There is no identified impact on the creation of TrustedForm certificates.</p>tag:status.activeprospect.com,2005:Incident/178451242023-07-13T17:02:08-05:002023-07-13T17:02:08-05:00Delay in Certificate archiving<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>17:02</var> CDT</small><br><strong>Resolved</strong> - The number of certificates waiting to be extended continues to drop. We will continue to monitor for any further unexpected delays in extending, retaining and auto retaining certificates.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>12:12</var> CDT</small><br><strong>Monitoring</strong> - We are continuing to monitor a delay in extending certificate retain windows, however archiving and auto retaining certificate times have returned to normal.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>10:07</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an issue that may cause a delay in certificate archiving and auto retaining certificates.</p>tag:status.activeprospect.com,2005:Incident/178573552023-07-13T10:00:00-05:002023-07-14T14:50:29-05:00Degraded Performance: TrustedForm Certify SDK<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>10:00</var> CDT</small><br><strong>Resolved</strong> - An issue was discovered in a newly deployed version (1.9.0) of the TrustedForm Certify SDK. Some pages that use the SDK inside an IFrame were not generating certificates. These changes were rolled back to version 1.8.39 at 10:07AM on July 13. 1.9.0 was deployed at 2:48PM CDT on July 12.</p>tag:status.activeprospect.com,2005:Incident/175640942023-06-13T18:02:26-05:002023-07-14T14:46:54-05:00Infrastructure Provider Degradation<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>18:02</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:49</var> CDT</small><br><strong>Monitoring</strong> - Our services are resuming normal functionality. We are working internally to ensure everything is in order. We have no reason to believe any data was lost.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:06</var> CDT</small><br><strong>Investigating</strong> - We are aware of an issue with our Infrastructure Provider that started around 2pm CT. Our services are functional at the moment, some functionality may be delayed. We are working to mitigate the impact.</p>tag:status.activeprospect.com,2005:Incident/173531022023-05-24T11:58:04-05:002023-05-24T11:58:04-05:00Missing Lead Reporting Information<p><small>May <var data-var='date'>24</var>, <var data-var='time'>11:58</var> CDT</small><br><strong>Resolved</strong> - Indexing of today's data has caught up to current, and new leads are once again visible immediately. If your account continues to show a delay or missing data, please contact AP support. We apologize again for the inconvenience.</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>11:36</var> CDT</small><br><strong>Monitoring</strong> - The issue with our search provider has been addressed and data for the past few hours' leads is being indexed as quickly as possible. We'll mark this issue resolved when that process has caught up and real-time data is being processed normally again.</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>10:58</var> CDT</small><br><strong>Identified</strong> - We've identified the issue with our search system service provider and are continuing to work to resolve it. All leads are being handled normally, despite recent ones not yet being visible in the LeadConduit UI.</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>09:34</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an issue where lead data is not displaying on reporting pages. The leads themselves are still being captured and stored, but the reporting of that information is not working correctly. We have narrowed down where the problem lies and will resolve the incident as soon as possible.<br /><br />We apologize for the inconvenience this is causing you,<br />The LeadConduit team</p>tag:status.activeprospect.com,2005:Incident/172299152023-05-12T16:57:48-05:002023-05-12T16:57:48-05:00Home and Lead Events pages showing incorrect data<p><small>May <var data-var='date'>12</var>, <var data-var='time'>16:57</var> CDT</small><br><strong>Resolved</strong> - The data propagation necessary for resolving this incident has been resolved sooner than anticipated. Your "Home" and "Lead Events" pages should be back to normal. As always, if you see anything out of the ordinary, please contact Support.<br /><br />The LeadConduit Team</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>11:00</var> CDT</small><br><strong>Monitoring</strong> - We upgraded our search database this morning. As a result of this, you will notice some data isn't showing as normal on your "Home" and "Lead Events" page. This data will be propagated over the next couple of days. None of your leads have been affected or lost.<br /><br />We apologize for the inconvenience.</p>tag:status.activeprospect.com,2005:Incident/168692422023-04-17T18:02:19-05:002023-04-17T18:02:19-05:00Planned TrustedForm Database Maintenance<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>18:02</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>16:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>14</var>, <var data-var='time'>15:49</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing planned maintenance during this time.</p>tag:status.activeprospect.com,2005:Incident/167419512023-04-04T18:00:26-05:002023-04-04T18:00:26-05:00TrustedForm Database Maintenance<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>18:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>17:17</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.activeprospect.com,2005:Incident/167019502023-04-03T17:16:13-05:002023-04-03T17:16:13-05:00TrustedForm Database Maintenance<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>17:16</var> CDT</small><br><strong>Completed</strong> - We have completed the necessary steps for today's database migration. We will be finishing in a follow-up maintenance tomorrow during the same time period.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>16:01</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>16:56</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing a scheduled maintenance on our TrustedForm database during this time.</p>tag:status.activeprospect.com,2005:Incident/166984102023-03-30T14:56:21-05:002023-03-30T14:56:21-05:00TrustedForm Certificate Issuer Incident<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>14:56</var> CDT</small><br><strong>Resolved</strong> - TrustedForm has been processing normally since 9a CDT.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>10:10</var> CDT</small><br><strong>Monitoring</strong> - TrustedForm is now operating normally. We will continue to monitor it and investigate the impact of this incident.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>09:05</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an ongoing issue that is negatively impacting our TrustedForm application's ability to create new certificates.</p>tag:status.activeprospect.com,2005:Incident/166323212023-03-25T09:23:25-05:002023-03-25T09:23:25-05:00LeadsBridge Database Maintenance<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>09:23</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>08:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>10:22</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.activeprospect.com,2005:Incident/165975742023-03-21T18:23:34-05:002023-03-21T18:23:34-05:00LeadConduit volume caps not being reset<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>18:23</var> CDT</small><br><strong>Resolved</strong> - The fix for this issue has been deployed and verified.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>17:11</var> CDT</small><br><strong>Identified</strong> - We've identified a problem with some volume caps not being reset correctly since the database maintenance yesterday. A fix is being deployed now.</p>tag:status.activeprospect.com,2005:Incident/165495872023-03-20T19:00:17-05:002023-03-20T19:00:17-05:00LeadConduit Database Maintenance<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>19:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>18:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>16:38</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance on our LeadConduit database during this time. We do anticipate a brief period of network connectivity loss during the failover event.</p>tag:status.activeprospect.com,2005:Incident/152044692022-12-14T16:24:30-06:002022-12-14T16:24:30-06:00TrustedForm failing to claim some recent certificates<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>16:24</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>16:23</var> CST</small><br><strong>Update</strong> - Issue is now resolved. There were some issues claiming masked certs between 3:10p until 3:50p Central time. No data was lost. We are investigating to determine if any claims need to be retried.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>15:45</var> CST</small><br><strong>Update</strong> - TrustedForm is failing to claim recent certificates which include typing speed, keys per minute, and words per minute data.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>15:42</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.activeprospect.com,2005:Incident/138817442022-11-28T13:41:46-06:002022-11-28T13:41:46-06:00LeadConduit's Blacklist Alliance integration failing<p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>13:41</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>13:25</var> CST</small><br><strong>Identified</strong> - The LeadConduit integration with Blacklist Alliance started failing today due to an incorrect API key. This has been updated and the new configuration is being deployed in production now.</p>tag:status.activeprospect.com,2005:Incident/112360992022-09-23T14:44:09-05:002022-09-23T14:44:09-05:00TrustedForm Certificates display blank page<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>14:44</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved. No data was lost and all impacted claim archives will be fixed in the upcoming week. Thank you again for your patience.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>12:57</var> CDT</small><br><strong>Monitoring</strong> - We have deployed an update to TrustedForm and are monitoring to ensure that certificates are correctly displayed. We are currently investigating archives that may display a blank page and are working on fixing them. No certificate data was lost. We appreciate your patience during this event.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>12:07</var> CDT</small><br><strong>Update</strong> - A fix has been coded for viewing Certificates immediately after they are claimed. It is being deployed shortly and should be fixed within the hour.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>11:33</var> CDT</small><br><strong>Update</strong> - We've identified the root cause and are working on a correction.<br /><br />Claim archives (generated roughly 15 mins after a claim request is made) will now display correctly. We will fix the archives that were stored with blank Certificates. Those started at 8:45a CT and ended at 11:20a CT.<br /><br />The Certificate will still display blank until the archive is created. This is a temporary view and no data has been lost. We will have this fixed today.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>11:06</var> CDT</small><br><strong>Update</strong> - After investigation we've determined that no data has been lost. We are continuing to investigate the cause of the issue.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>10:50</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an issue where unclaimed and newly claimed TrustedForm certificates are displaying an empty page when viewed.</p>tag:status.activeprospect.com,2005:Incident/112218982022-09-22T11:30:43-05:002022-09-22T11:30:43-05:00Minor Lead Processing Delay<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>11:30</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>11:29</var> CDT</small><br><strong>Update</strong> - The 3rd party we use which was experiencing intermittent errors early this morning has resolved their issues, and our processes have returned to a nominal state.<br /><br />Thank you for your patience,<br />The LeadConduit Team</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>11:27</var> CDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>10:27</var> CDT</small><br><strong>Investigating</strong> - We are currently experiencing intermittent errors with one of our 3rd party services. Because of this, customers may experience minor delays while processing leads, or see discrepancies in our graphs and reports. We are monitoring the situation closely to ensure services remain available and no data is lost. We will continue to provide updates as the situation changes or new information is available.<br /><br />Thank you,<br />The LeadConduit Team</p>