Issue summary:

A code release aimed at correcting a bug in logged-in user session timing inadvertently impacted the Splash platform which led to a system-wide outage.

Issue timeframe:

January 17, 2024 4:46pm ET to January 17, 2024 4:59pm ET (7 mins)

Sequence of events:

• January 17, 4:46pm ET Code change released.
• January 17, 4:47pm ET First report of customer impact received by support.
• January 17, 4:56pm ET Code change revert initiated.
• January 17, 4:59pm ET Code change revert completed.

Root cause:

A fix was deployed to resolve a bug that was causing interruptions to the logged-in user experience. Prior testing in release environments failed to catch the issue before Production deployment due to different downstream dependencies. The code that was released caused the site to be inaccessible, requiring the code change to be reverted.

Steps to prevent recurrence:

• Splash will implement additional QA and regression testing controls to ensure more rigorous checking before any code releases.