Major Issue - Service Disruption - US Swimlane

Issue Summary:

‌

On Monday, starting at 5:40 AM PST, customers using basic (username/password) authentication were unable to initiate new sessions.

SSO-based login workflow and existing active user sessions didn’t observe errors in the authentication workflow, however the elevated errors and latencies were observed across multiple functionalities including Edge, Reporting, Recruiting and Performance Reviews etc.

By 8:00 AM PST, the reported issue was resolved confirming all the impacted functionalities to be working as expected.

‌

Root Cause:
The issue occurred due to the backend authentication service was unable to complete the login process for new sessions. The authentication service and the subsequent dependent services failed to initialize due to a missing component due its dependency on a third-party plugin . The service was attempting to download the plugin dynamically at runtime, and the source was either unavailable or blocked, leading to a cascade failure in the authentication path for new logins via username/password. Poor logging and missing observability further delayed diagnosis and resolution.

‌

Corrective Action:

• The plugin was downloaded and hosted locally to eliminate runtime dependency on external sources.
• Service was restarted with the locally available plugin, restoring normal login functionality.

‌

Preventive Actions:

1. Audit and catalog all third-party runtime dependencies used across the environment.
2. Introduce local caching or bundling mechanisms for essential third-party libraries/plugins.
3. Improve logging and observability for critical services to ensure faster root cause identification.
4. Update operational runbooks and team training to include this class of failures.
5. Implement automated checks to validate the availability of external dependencies during build/deploy phases.