Splash Is Experiencing Site-Wide Downtime

Incident Report for Splash

Postmortem

Issue summary:

Several large event actions impacted the Splash platform which led to a system-wide outage.

Issue timeframe:

January 23, 2024 12:10 ET to January 23, 2024 12:17 ET (7 mins)

Sequence of events:

January 23, 12:10 ET A large email send and a large contact import were started.
January 23, 12:12 ET First report of customer impact received by support.
January 23, 12:15 ET Database scaled down after completion of email send and contact import.
January 23, 12:17 ET All access was restored.

Root cause:

Due to several customers performing large actions in Splash that exceeded our native limits, Splash became inaccessible. After these actions were completed, access to Splash was restored.

Steps to prevent recurrence:

Splash is investigating various restrictions in our system scaling to better process actions performed in Splash.

Posted Jan 30, 2024 - 13:27 UTC

Resolved

This incident has been resolved.

Posted Jan 23, 2024 - 18:10 UTC

Monitoring

Access to Splash has been restored and we are monitoring.

Posted Jan 23, 2024 - 17:24 UTC

Investigating

Splash is presently experiencing a site-wide issue and is unavailable. We are investigating the underlying cause of this issue.

Posted Jan 23, 2024 - 17:19 UTC

This incident affected: Logged In Experience (Event Page Design (CMS), Guest List Management (RSVP & Ticketed), Event Creation, Team Management, Analytics & Reporting, Email Design, User Login, Event Settings).