Problems with timeseries and sequences in EUR-N1
Resolved

A regression in datastore configuration during the addition of additional checksum/error detection resulted in a temporary loss of resiliency.The regression has been identified and forward-fixed to retain additional error detection.

Tue, Mar 28, 2023, 12:37 PM
(2 years ago)
·
Affected components

No components marked as affected

Updates

Resolved

A regression in datastore configuration during the addition of additional checksum/error detection resulted in a temporary loss of resiliency.The regression has been identified and forward-fixed to retain additional error detection.

Tue, Mar 28, 2023, 12:37 PM

Monitoring

The API is responding normally, and we are continuing to investigate the root causes.

Mon, Mar 27, 2023, 02:12 PM(22 hours earlier)

Monitoring

Data is fully replicated.The API is responding normally, and we will continue to monitor through the weekend.

Sat, Mar 25, 2023, 12:02 PM(2 days earlier)

Monitoring

Restoration of full replication level is proceeding, and expected to complete with the next 24 hours.The API is responding normally, and we continue to monitor.

Fri, Mar 24, 2023, 04:00 PM(20 hours earlier)

Monitoring

We have identified a faulty storage node, and are migrating data away from it. Until migration is complete, data is below normal replication levels.The API is functioning normally again.

Fri, Mar 24, 2023, 12:42 PM(3 hours earlier)

Investigating

The cluster error rates have returned to normal levels, but we continue to investigate the source of the problem.

Fri, Mar 24, 2023, 11:47 AM(55 minutes earlier)

Investigating

Error rates are declining, but we are continuing to investigate.

Fri, Mar 24, 2023, 11:11 AM(35 minutes earlier)

Investigating

We are continuing to investigate the cause of the issue and recover alternatives.We are working to make the most recent backup available as a fallback.

Fri, Mar 24, 2023, 10:35 AM(36 minutes earlier)

Investigating

The timeseries and sequences datastore replication process began experiencing errors at 9:07 UTC, and API error rates for timeseries and sequences increased significantly.

This will be visible in services and applications reading and writing to timeseries and sequences, including Fusion.

Fri, Mar 24, 2023, 10:09 AM(25 minutes earlier)

Investigating

We are continuing to investigate this issue.

Fri, Mar 24, 2023, 09:54 AM(14 minutes earlier)

Investigating

We are currently investigating the issue.

Fri, Mar 24, 2023, 09:53 AM
Powered by