Everything is operating optimally, and node connection status alerts are working normally again.
Posted Aug 07, 2021 - 06:34 CDT
The underlying issue is now resolved - we are bringing services back online and monitoring alerts and notifications.
Posted Aug 07, 2021 - 01:30 CDT
AWS has reported improvements but we are still seeing issues and monitoring for further improvement.
Posted Aug 06, 2021 - 23:22 CDT
We are continuing to work on a fix for this issue.
Posted Aug 06, 2021 - 21:49 CDT
AWS has identified an issue and is working to resolve. We are monitoring and working to clear disconnect alerts to minimize notification once the issue is resolved.
Posted Aug 06, 2021 - 20:57 CDT
It appears AWS is experiencing issues that are impacting part of our control plane leading to the false alarms about nodes disconnecting. Out of an abundance of caution we are taking steps to prevent any changes from being pushed down to nodes.
Posted Aug 06, 2021 - 20:26 CDT
We are continuing to investigate. At this point it appears these disconnects are false alarms due to a database issue.
Posted Aug 06, 2021 - 20:05 CDT
We are currently investigating an issue with the control plane reporting multiple nodes disconnecting and then reconnecting.
Posted Aug 06, 2021 - 19:55 CDT
This incident affected: Node (Cloud Controller) and Management (Management Portal).