File upload propagation delays (isUploaded flag) in the az-eastus-1 cluster have been fully resolved. The root cause was an Azure Event Grid outage in the East US region (05:00 UTC June 9 – 06:30 UTC June 10, 2026), which prevented blob storage notifications from reaching our internal file processing pipeline. Propagation lag has now stabilized at normal, near-zero levels. All affected workflows, including CDF Function deployments and file-dependent processes, should be fully operational. We will continue to monitor system metrics to ensure stability.
Resolved
File upload propagation delays (isUploaded flag) in the az-eastus-1 cluster have been fully resolved. The root cause was an Azure Event Grid outage in the East US region (05:00 UTC June 9 – 06:30 UTC June 10, 2026), which prevented blob storage notifications from reaching our internal file processing pipeline. Propagation lag has now stabilized at normal, near-zero levels. All affected workflows, including CDF Function deployments and file-dependent processes, should be fully operational. We will continue to monitor system metrics to ensure stability.
Identified
We have identified a significant delay in the propagation of the isUploaded flag for files uploaded in the az-eastus-1 cluster. Files may take over 2 hours to be marked as uploaded after a successful upload.
Root cause identified: At least one project in the az-eastus-1 cluster is being rate limited by the DMS (Data Modeling Service), which is causing the file storage subscriber to back up and delay processing of upload notifications.
Our engineering team has applied a fix to increase the DMS write concurrency limit for the affected project, and the change has been deployed. We are actively monitoring the situation to confirm recovery.
We will provide further updates as the situation progresses.
