Postmortem -
Read details
Nov 2, 17:52 PST
Resolved -
After an extended period of monitoring, this incident is now considered resolved. Our efforts do not stop here. Cornerstone teams continue to work closely with Spamhaus and our customers to strengthen email hygiene practices, while also enhancing our email infrastructure and application safeguards to help prevent similar incidents in the future. A detailed Root Cause Analysis (RCA) will be shared within 7 to 10 business days. For additional details, please refer to the related Known Issue article: ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Delivery-failures-for-UK-Swimlanes ).
Oct 22, 12:41 PDT
Update -
Email delivery has remained stable since October 10, with no new delivery failures related to the Spamhaus blacklisting. All emails continue to be successfully delivered, and we are maintaining close monitoring to ensure ongoing stability.
Oct 20, 21:27 PDT
Update -
Our monitoring confirms that email delivery has remained fully stable since the IP configuration change on October 10th. No new delivery failures have been observed due to the Spamhaus blacklisting, and all emails are being successfully delivered.
Oct 13, 04:19 PDT
Update -
Our monitoring confirms that email delivery has remained fully stable since the IP configuration change on October 10th. No new delivery failures have been observed due to the Spamhaus blacklisting, and all emails are being successfully delivered.
Everything continues to look good at this time. We are continuing to monitor the system and will provide another update within the next 24 hours as we complete the 48-hour observation window.
Oct 12, 04:04 PDT
Update -
Email delivery remains stable following the IP configuration change on October 10th. No new delivery failures have been detected, and all emails continue to be successfully delivered.
We will continue to monitor the situation closely and provide an update after 24 hours.
Oct 12, 04:03 PDT
Update -
Our monitoring confirms that email delivery has remained fully stable since the IP configuration change on October 10th. No new delivery failures have been observed due to Spamhaus blacklisting, and emails are being successfully delivered.
Everything continues to look good at this time. We will continue to monitor until the 48-hour window is complete. We will update you again in the next 24 hours.
Oct 11, 04:01 PDT
Monitoring -
The CSOD Technology Team has successfully updated our email configuration to route traffic through an IP address in good standing. Beginning at 1200 AM Pacific Time on October 10th, UK Swimlane customers should no longer experience issues with email delivery due to Spamhaus blacklisting. All emails from Cornerstone are now expected to be delivered successfully.
We will continue to closely monitor the situation for the next 48 hours before closing this incident.
Background:
The email delivery disruption affecting UK Swimlane customers was due to our primary email IP being blacklisted by Spamhaus, starting at approximately 3:30 PM Pacific Time on October 6th. During this period, some customers may have experienced intermittent email delivery failures, depending on their organization’s IT security policies.
We appreciate your patience and will provide further updates as needed.
Oct 10, 03:57 PDT
Update -
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
Oct 10, 03:47 PDT
Update -
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
Oct 10, 01:41 PDT
Update -
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
Oct 9, 23:11 PDT
Update -
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
Oct 9, 21:06 PDT
Update -
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
Oct 9, 19:11 PDT
Update -
Communication with Spamhaus is ongoing as we await confirmation of the delisting request. We will continue to share updates on a regular basis.
Oct 9, 16:02 PDT
Update -
Communication with Spamhaus is ongoing as we await confirmation of the delisting request. We will continue to share updates on a regular basis.
Oct 9, 13:31 PDT
Update -
We’re actively engaging with SpamHaus and still waiting for their reply regarding our delisting request. The next update will be provided in 4 hours or sooner if any feedback from SpamHouse is available before that time.
Oct 9, 09:12 PDT
Update -
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. The next update will be provided in 4 hours or sooner if any feedback from SpamHouse is available before that time.
Oct 9, 05:03 PDT
Update -
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
Oct 9, 02:45 PDT
Update -
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
Oct 9, 00:01 PDT
Update -
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
Oct 8, 21:39 PDT
Update -
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
Oct 8, 19:41 PDT
Update -
We are in ongoing communication with SpamHaus and are currently awaiting their response to our delisting request. We’ll continue to provide regular updates until the issue is fully resolved.
Oct 8, 17:40 PDT
Update -
We continue to communicate with SpamHaus and are awaiting a response to our delisting request. Regular updates will be shared until this is resolved.
Oct 8, 15:44 PDT
Update -
We are in ongoing communication with Spamhaus and awaiting confirmation of our delisting request. We will continue to provide regular updates until this is resolved.
Oct 8, 13:33 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 8, 10:00 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 8, 08:03 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 8, 05:47 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 8, 02:07 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 8, 00:07 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 7, 22:01 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 7, 20:08 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 7, 18:00 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 7, 16:01 PDT
Update -
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
Oct 7, 13:56 PDT
Update -
We are still awaiting a response from Spamhaus regarding the delisting request and will provide an update as soon as we receive any information.
Oct 7, 12:00 PDT
Update -
We are currently awaiting a response from Spamhaus regarding the delisting request and will provide an update as soon as we receive any information.
Oct 7, 07:55 PDT
Update -
Cornerstone IPs remain listed on the Spamhaus blocklist, which means email deliveries may continue to be impacted, depending on the client’s Domain Reputation Checker policies for Spamhaus.
We are still awaiting further updates from Spamhaus regarding the delisting request.
Oct 7, 04:45 PDT
Update -
We are currently awaiting a response from Spamhaus regarding the delisting request and will provide an update as soon as we receive any information.
Oct 7, 02:35 PDT
Update -
We are continuing to work on the fix for this issue.
The next update will follow within 2 hours or as soon as we receive further information.
Oct 7, 00:36 PDT
Update -
Email delivery success rate is improving. We will update you further in the next two hours.
Oct 6, 22:31 PDT
Identified -
We have identified the cause of the issue and have implemented internal corrective measures. We are working to restore normal operations. We will update you further in the next two hours.
Oct 6, 20:39 PDT
Update -
We are continuing to investigate this issue as our top priority and will provide regular updates until it is fully resolved. Thank you for your continued patience and understanding.
Oct 6, 18:35 PDT
Investigating -
Cornerstone UK email sending IPs have been temporarily blacklisted by Spamhaus, a leading global email reputation authority, due to poor email hygiene detected in email activity originating from certain customer accounts.
During this time, emails may fail to deliver. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
Oct 6, 16:49 PDT