During this day, Monday, October 20th, between 07:11 AM (UTC) and approximately 10:30 AM (UTC), we observed a significant increase in error rates and latencies on the platform, which affected some of our clients deployed in the Amazon AWS us-east-1 region. This was due to a regional failure in their services that impacted several services dependent on DynamoDB. The root cause reported by AWS for this initial outage was an underlying DNS resolution issue. AWS implemented mitigations quickly, achieving full resolution of the DNS problem by 10:30 AM (UTC), with most core service operations returning to normal. For the Modyo clients affected by the incident, recovery was noted starting around 08:35 AM (UTC).
Despite the DNS mitigation, stability issues continued, though this time without affecting the Modyo platform, as they involved failures in launching new EC2 instances and elevated SQS polling delays for Lambda. Subsequently, starting at 2:14 PM (UTC), an escalation of network connectivity issues and significant errors was confirmed across multiple services, once again including DynamoDB, SQS, and Amazon Connect. The latest official AWS update (at 3:04 PM UTC) identified that the root cause of this second, more extensive outage stems from an issue within the internal EC2 network. AWS continues to actively work on investigating and mitigating this deeper cause.
It is important to emphasize that, despite the severity of these outages in the us-east-1 region, Modyo did not register a direct or critical operational impact during the second part of the incident. This is because the services affected in the second phase of the incident (DynamoDB, SQS, etc.) are not part of our critical infrastructure or are not used in the main path of our applications. In the case of Modyo Connect's managed infrastructure services, the impact could vary from client to client due to the services being utilized that might be affected.
We will continue to monitor the incident in case any action is required on our part.
Best regards,
José Antonio Silva
CTO, Modyo