On November 21st, 2022 Cornerstone engineers were alerted by internal monitoring tools for latency. Clients hosted in LAX SL1/2/3/5 Prod swimlane experienced latency for Reporting 2.0.
Cornerstone engineers were engaged for investigation and troubleshooting, and issues pointed to network disruption on the underlying hardware which host the services. The issue was resolved by migrating the service to a different cluster with stable hardware.
To mitigate this situation from occurring in the future and to reduce the risk of a future impact Cornerstone is actively working internally to review the incident with the vendor. Internal teams are working with our third-party vendors to establish the issue and conducting a thorough investigation and creating remediation plans to avoid the recurrence of this incident and resolved this permanently.
Posted Dec 12, 2022 - 00:32 PST
Resolved
This incident has been resolved.
Posted Nov 21, 2022 - 09:40 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 21, 2022 - 09:26 PST
Identified
This swimlane is experiencing some functional issues with Reporting 2.0. Clients with portals in this swimlane may experience delays or issues executing some tasks. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
Posted Nov 21, 2022 - 09:03 PST
This incident affected: LAX-SL1 (Response Time), LAX-SL3 (Response Time), and LAX-SL5 (Response Time).