Summary: We experienced an issue affecting the Property Graph search service in the West Europe region. The Elasticsearch cluster became unstable after receiving unusually large synchronization requests, which caused circuit-breaker errors.
Steps taken: We stabilized the Elasticsearch cluster, then re-enabled background synchronization in controlled stages while monitoring cluster health, CPU usage, and indexing rates.
Resolved status: The service has stabilized and normal functionality has been restored. Background synchronization has resumed, and we will continue monitoring to ensure it remains healthy.
Resolved
Summary: We experienced an issue affecting the Property Graph search service in the West Europe region. The Elasticsearch cluster became unstable after receiving unusually large synchronization requests, which caused circuit-breaker errors.
Steps taken: We stabilized the Elasticsearch cluster, then re-enabled background synchronization in controlled stages while monitoring cluster health, CPU usage, and indexing rates.
Resolved status: The service has stabilized and normal functionality has been restored. Background synchronization has resumed, and we will continue monitoring to ensure it remains healthy.
Monitoring
We have implemented mitigation steps for the issue impacting Data Model (DM) Search and aggregation queries in the westeurope-1 region. The service has stabilized, and we are currently monitoring the system to ensure full recovery.
Search and aggregation requests for data models in westeurope-1 should now complete successfully without HTTP 503 errors. Customers may still see slight delays in data freshness as background sync operations catch up, but normal functionality has been largely restored. Core data storage and other regions continue to operate normally.
Our engineering team has successfully stabilized the underlying Elasticsearch cluster and carefully re-enabled background sync operations in controlled stages. We will continue to closely monitor cluster health and performance metrics to ensure the service remains stable.
Investigating
Summary:
We are currently investigating an issue impacting Data Model (DM) Search and aggregation queries in the westeurope-1 region.
Customer Impact:
Requests to search or aggregate data models in westeurope-1 may fail with HTTP 503 errors or return incomplete/delayed results. Core data storage and other regions are operating normally.
Current Actions:
Our engineering team is actively working on cluster recovery and load mitigation, including throttling background sync operations to restore search performance.
Next Update:
We are monitoring recovery progress and will provide another update within 60 minutes or as soon as significant updates are available.
