Problem: The Propertygraph Elasticsearch cluster in westeurope-1 experienced degradation, temporarily entering yellow and red states.
Impact: Customers with projects in westeurope-1 may experience delays in newly created instances appearing in search results, along with elevated latency for search and aggregate queries affecting data modeling indexing (datamodelstorage) Core data storage is unaffected.
Root Cause: A syncer update triggered concurrent restarts across syncer deployments simultaneously, overwhelming the search master nodes with high-volume reads. In addition, an search pod encountered a PVC mount failure during rolling updates.
Status & Mitigation: Mitigation steps have been applied. Our engineering team scaled down the syncers to allow the cluster to recover to a healthy green state, subsequently re-enabled the synchronizers in stages, and resolved the PVC volume mount issue. The cluster is stabilizing, and our team is actively monitoring system performance and recovery.
Monitoring
Problem: The Propertygraph Elasticsearch cluster in westeurope-1 experienced degradation, temporarily entering yellow and red states.
Impact: Customers with projects in westeurope-1 may experience delays in newly created instances appearing in search results, along with elevated latency for search and aggregate queries affecting data modeling indexing (datamodelstorage) Core data storage is unaffected.
Root Cause: A syncer update triggered concurrent restarts across syncer deployments simultaneously, overwhelming the search master nodes with high-volume reads. In addition, an search pod encountered a PVC mount failure during rolling updates.
Status & Mitigation: Mitigation steps have been applied. Our engineering team scaled down the syncers to allow the cluster to recover to a healthy green state, subsequently re-enabled the synchronizers in stages, and resolved the PVC volume mount issue. The cluster is stabilizing, and our team is actively monitoring system performance and recovery.
Investigating
We are currently experiencing performance degradation affecting search and data modeling indexing services in the westeurope-1 region.
Impact: Search and aggregate queries may experience elevated latency, and newly created instances or index updates may take longer to appear in search results. Core data storage is unaffected.
Our engineering team is actively investigating the issue and working on mitigation steps to stabilize the service. We will provide updates as more information becomes available
