Unable to access reports in Reporting 2.0 (US Swimlanes Production Environment)

Incident Report for Cornerstone

Postmortem

Incident Summary:
Starting April 13th, 2026, intermittent errors were observed in reporting functionality across US Production swimlanes (SL1, SL2, SL3, and SL5).

Root Cause Analysis (RCA):
The issue was caused by the auto-scaling mechanism. Newly provisioned nodes were not fully ready to handle reporting requests, leading to processing delays and request timeouts.

Corrective Action:
Auto-scaling (scale-out) has been temporarily paused. The system is currently operating on a fixed, increased server capacity without scale-in or scale-out to ensure stability.

Preventive Actions:

  • Capacity has been proactively increased to handle expected load without relying on auto-scaling.
  • Additional nodes are being utilized with auto-scaling disabled to maintain consistent performance.
  • Engineering teams are working on enhancements to improve node readiness and ensure seamless scaling in the future.
Posted Apr 28, 2026 - 10:14 PDT

Resolved

After a period of monitoring and no recurrences, we are considering this resolved.
Posted Apr 13, 2026 - 11:40 PDT

Monitoring

Our Engineering teams have identified the issue and deployed a fix to address it. We will continue to closely monitor the situation to ensure stability.
Further updates will be shared through the Known Issue article and Statuspage. Thank you for your understanding and patience.
Posted Apr 13, 2026 - 07:43 PDT

Investigating

US swimlanes are currently experiencing functional issues, including difficulties accessing reports. Clients with portals in this swimlane may encounter delays or disruptions while performing certain tasks.

This issue is our top priority, and our teams are actively working toward a resolution. We appreciate your patience and recommend checking back periodically for updates as they become available.
Posted Apr 13, 2026 - 07:34 PDT
This incident affected: US SL3 (AWS) (Response Time), US SL5 (AWS) (Response Time), US SL1 (AWS) (Response Time), and US SL2 (AWS) (Response Time).