On February 19th, 2025, Cornerstone engineers were alerted by internal monitoring tools to email delivery issues affecting clients hosted on US SL1, US SL2, US SL3, and US SL5 PRD AWS. The disruption was caused by emails being sent without our IP address included in the SPF record of the sender's domain, leading to mail delivery failures. As a result, our primary email IP was blocked by real-time blacklists (RBLs) due to authenticated emails being sent without proper SPF record inclusion. This blockage contributed to the email delivery issues. To resolve the situation, we coordinated with RBL organizations to have our IP delisted and ensured its reinstatement in the rotation. Services are now functioning as expected. To prevent similar incidents in the future, we are implementing several actions to improve email hygiene and system reliability. We remain committed to providing reliable service and appreciate your understanding.
Posted Mar 06, 2025 - 04:26 PST
Resolved
This incident has been resolved.
Posted Feb 23, 2025 - 08:06 PST
Monitoring
Email queues are processing normally. We are continuing to closely monitor activity for any changes.
Posted Feb 22, 2025 - 22:19 PST
Update
We are treating this as a top priority and actively working on mitigating the delays.
We've noticed that delays may be occurring for some customers who have not safelisted our IP addresses in their SPF (Sender Policy Framework) records. We recommend adding our IPs to ensure reliable email delivery.
Please follow these steps to safelist our IPs:
(1) Download the Customer Email Configuration document available via the 'General Minimum Requirements' page on Online Help (https://help.csod.com/help/csod_0/Content/General_Minimum_Requirements.htm). You can access it directly by expanding the 'Client Email Configuration' section. (2) Share this document with your IT department to update your SPF record accordingly.
We will continue to post updates as they become available.
Posted Feb 22, 2025 - 13:13 PST
Update
Our teams are continuing to investigate the issue and we will update here as soon as we have identified a root cause.
Posted Feb 21, 2025 - 19:12 PST
Investigating
This swimlane is experiencing some delays in processing emails. These emails are still processing and are not stuck, but are processing at a slower rate than expected. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
Posted Feb 21, 2025 - 17:09 PST
This incident affected: US SL3 (AWS) (Uptime, Response Time), US SL5 (AWS) (Uptime, Response Time), US SL1 (AWS) (Uptime, Response Time), and US SL2 (AWS) (Uptime, Response Time).