Service Disruption - Error logging in to US SL1, SL2, SL3 and SL5 portals

Incident Report for Cornerstone

Postmortem

Incident Summary:
On November 27th, users accessing the US PROD environment experienced 504 Gateway Errors, resulting in intermittent inability to access platform features. The issue affected all clients in the region during the incident window.

Root Cause:
The errors were caused by traffic from the CDN being temporarily blocked, which prevented requests from reaching the platform. This disruption was related to network configuration that impacted traffic flow to the US PROD environment.

Corrective Action:
A network configuration adjustment was applied to restore CDN traffic to the platform, which immediately resolved the errors. Platform performance and availability were verified, confirming full resolution of the incident.

Preventive Measures:
To reduce the likelihood of similar incidents in the future:

Cornerstone is reviewing network configurations and implementing safeguards to prevent traffic blockage.
Internal processes for network changes are being strengthened to minimize impact on production traffic.

Posted Dec 03, 2025 - 15:51 PST

Resolved

The CSOD Technology Team observed a service disruption on US SL1, SL2, SL3 and SL5. The problem began on 27 Nov 5:20 AM Pacific Time and service was restored at 6:26 AM Pacific Time. During this time, clients with portals on these swimlanes were not able to access the application. Clients should now be able to access the portals successfully. RCA will be provided in 7-10 business days.

Posted Nov 27, 2025 - 07:43 PST

Monitoring

The CSOD Technology Team observed a service disruption on US swim lanes (SL1, SL2, SL3 and SL5). The problem began on 27 Nov 5:20 AM Pacific Time and service have been restored now. Clients should now be able to access the portals successfully now. We are actively monitoring the system performance before we confirm the incident to be resolved.

Posted Nov 27, 2025 - 06:40 PST

Investigating

US swim lanes (SL1, SL2, SL3 and SL5) are experiencing a service disruption. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Posted Nov 27, 2025 - 05:38 PST

This incident affected: US SL3 (AWS) (Uptime), US SL5 (AWS) (Uptime), US SL1 (AWS) (Uptime), and US SL2 (AWS) (Uptime).