• Problem: The production propertygraph-search Elasticsearch cluster in westeurope-1 went red for roughly 10 minutes.
• Root cause: Propertygraph syncer activity and cluster topology created excessive master-coordination overhead. PVC mount failures were also observed on Elasticsearch master-data pods, but were not confirmed as the original cause.
• Resolution: Engineering Team scaled down the propertygraph syncers to recover cluster health, then restored every propertygraph-sync-v3 deployment to one replica in stages. They also reduced node/master-coordination overhead and reconfigured all 15 master-data nodes with 19 GB heaps; the cluster reached zero unassigned shards and was considered stable.
Resolved
• Problem: The production propertygraph-search Elasticsearch cluster in westeurope-1 went red for roughly 10 minutes.
• Root cause: Propertygraph syncer activity and cluster topology created excessive master-coordination overhead. PVC mount failures were also observed on Elasticsearch master-data pods, but were not confirmed as the original cause.
• Resolution: Engineering Team scaled down the propertygraph syncers to recover cluster health, then restored every propertygraph-sync-v3 deployment to one replica in stages. They also reduced node/master-coordination overhead and reconfigured all 15 master-data nodes with 19 GB heaps; the cluster reached zero unassigned shards and was considered stable.
Monitoring
The engineering team is investigating and fixing issues with the Data Modelling search and aggregate APIs in the westeurope-1 cluster. Customer impact includes increased error rates on DM APIs and in the data exploration app in Fusion, and delays before new data is available in search results.
Monitoring
The engineering team is investigating and fixing issues with the Data Modelling search and aggregate APIs in the westeurope-1 cluster. Customer impact includes increased error rates on DM APIs and in the data exploration app in Fusion, and delays before new data is available in search results.
Monitoring
The engineering team is investigating and fixing issues with the Data Modelling search and aggregate APIs in the westeurope-1 cluster. Customer impact includes increased error rates on DM APIs and in the data exploration app in Fusion, and delays before new data is available in search results.
