Resolved
Service has been restored. The Elasticsearch cluster in westeurope-1 has stabilized, and search, aggregation, and data-modeling indexing are operating normally.
Monitoring
The Elasticsearch cluster in westeurope-1 became overloaded and entered a red state. This degraded search and data-modeling indexing. search and aggregate requests could be slow or fail, while new instances and index changes could take longer to appear.
Root cause: A rollout restart of pg3 overloaded the Elasticsearch cluster.
Fixing status: The incident is now in Monitoring. Engineering team paused some background synchronizers to reduce load, allowed Elasticsearch to stabilize, and then started re-enabling the synchronizers. The fix was marked as applied.
Investigating
We are currently investigating performance degradation and high latency affecting search and data modeling services in the westeurope-1 region.
Impact: Search and aggregate requests may experience high latency or temporary failures. Newly created instances or index updates may take longer than expected to appear.
Status: Our engineering team is currently scaling down background syncing processes to reduce load and restore cluster stability. Further updates will be posted as soon as they are available.
