Issue summary:

Splash experienced major issues populating inline and modal-contained registration forms within the UI, impacting guests' registration ability.

Issue timeframe:

January 14th, 2025, 1:48 PM EST to 3:19 PM EST (1 hour, 31 minutes)

‌

Sequence of events:

• January 14th, 1:48 PM EST - Splash implemented changes to our third-party Security system.
• January 14th, 2:01 PM EST - Internal notifications indicated issues with the applied changes.
• January 14th, 2:22 PM EST - First customer-reported issues were alerted to Support; Internal validation initiated.
• January 14th, 2:45 PM EST - Issue replicated and validated; Engineering response team activated.
• January 14th, 2:52 PM EST - System impact confirmed and the investigation continued.
• January 14th, 2:55 PM EST - Script adjustment identified as the root cause; Hotfix validated and script adjustment reverted.
• January 14th, 3:01 PM EST - Incident adjusted to Identified; Engineering and internal Response team continue to stress-test for further validation.
• January 14th, 3:03 PM EST - Incident adjusted to Monitoring; No further reports of impact received.
• January 14th, 3:19 PM EST - Incident marked as Resolved
  

Root cause:

A script introduced to our advanced Security system, resulted in an unexpected mismatch between wildcard request origins and security-specific cookies, blocking the registration form population within the Splash UI.

Steps to prevent recurrence:

• Splash to implement improvements to existing internal alert systems.
• Splash to increase QA for security-specific implementations, including but not limited to, replication and validation outside of production environments.
• Splash to request improvements from third-party security systems and available documentation for explicit indication of impact between such changes and options impacted.