Issue summary:

The platform experienced brief inaccessibility, displaying a 505 (or similar) errors due to the database being under high traffic load and increased activity.

Issue timeframe:

February 7th, 2024 10:37 AM EST - February 7th, 2024 10:58 AM EST (21 minutes)

‌

Sequence of events:

• February 7th, 10:37 AM EST System down alert fires; investigation started.
• February 7th, 10:45 AM EST Determined that database connections started to spike at approximately 10:30 AM EST.
• February 7th, 10:50 AM EST System impact identified, and root cause traced to an abnormally elevated concurrence of email sends, API calls, and increased platform activity.
• February 7th, 10:52 AM EST Source(s) identified to be causing high load and breaching autoscaling expectations.
• February 7th, 10:55 AM EST Platform regained responsiveness and access was reinstated.
• February 7th, 10:58 AM EST Platform confirmed fully restored.

‌

Root cause:

An abnormally large concurrence of email sends, API calls, and general platform activity resulted in ​​high database load. Connection and volume management processes failed to adequately manage the thread consumption, leading to temporary platform access unavailability.

‌

Steps to prevent recurrence:

• Autoscaling and volume management processes will be optimized to appropriately handle similar scenarios.