We have identified a significant delay in the propagation of the isUploaded flag for files uploaded in the az-eastus-1 cluster. Files may take over 2 hours to be marked as uploaded after a successful upload.
Root cause identified: At least one project in the az-eastus-1 cluster is being rate limited by the DMS (Data Modeling Service), which is causing the file storage subscriber to back up and delay processing of upload notifications.
Our engineering team has applied a fix to increase the DMS write concurrency limit for the affected project, and the change has been deployed. We are actively monitoring the situation to confirm recovery.
We will provide further updates as the situation progresses.
Identified
We have identified a significant delay in the propagation of the isUploaded flag for files uploaded in the az-eastus-1 cluster. Files may take over 2 hours to be marked as uploaded after a successful upload.
Root cause identified: At least one project in the az-eastus-1 cluster is being rate limited by the DMS (Data Modeling Service), which is causing the file storage subscriber to back up and delay processing of upload notifications.
Our engineering team has applied a fix to increase the DMS write concurrency limit for the affected project, and the change has been deployed. We are actively monitoring the situation to confirm recovery.
We will provide further updates as the situation progresses.
