On the evening of Nov 8th 2020 Pacific Time, Cornerstone Engineers were alerted by internal monitoring tools of database service failures in our LAX data centre affecting a limited number of clients.
The latency occurred before US business hours on Nov 9th and again later during the same US business day. Following our escalation process, multiple teams were engaged to investigate, and we then reached to our 3rd party storage vendor after an issue was identified with our storage hardware.
Engineers addressed the issue with the affected clients and paused feeds and job queues to alleviate stress on the remaining hardware while the bad hardware was being replaced.
Replacement was successful during US business day of 9th Nov and issue was fully resolved.
To mitigate this situation from occurring in the future and to reduce the risk of a future impact, Cornerstone is working with the hardware vendor on enhanced escalation and improved procedures to avoid any further impact
Posted Nov 10, 2020 - 22:42 PST
Resolved
The CSOD Technology Team observed delays in processing background or queued tasks on this swim lane. The problem began at 8:11 AM Pacific Time and was resolved at 3:15 PM Pacific Time. During this time, background tasks may have taken longer than normal to complete.
Posted Nov 09, 2020 - 15:29 PST
Investigating
This swimlane is experiencing some delays in processing background or queued tasks . During this time, background tasks may take longer than normal to complete. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.