Update - Data is fully replicated. The API is responding normally, and we will continue to monitor through the weekend.
Mar 25, 2023 - 13:02 CET
Update - Restoration of full replication level is proceeding, and expected to complete with the next 24 hours. The API is responding normally, and we continue to monitor.
Mar 24, 2023 - 17:00 CET
Monitoring - We have identified a faulty storage node, and are migrating data away from it. Until migration is complete, data is below normal replication levels. The API is functioning normally again.
Mar 24, 2023 - 13:42 CET
Update - The cluster error rates have returned to normal levels, but we continue to investigate the source of the problem.
Mar 24, 2023 - 12:47 CET
Update - Error rates are declining, but we are continuing to investigate.
Mar 24, 2023 - 12:11 CET
Update - We are continuing to investigate the cause of the issue and recover alternatives. We are working to make the most recent backup available as a fallback.
Mar 24, 2023 - 11:35 CET
Update - The timeseries and sequences datastore replication process began experiencing errors at 9:07 UTC, and API error rates for timeseries and sequences increased significantly.
This will be visible in services and applications reading and writing to timeseries and sequences, including Fusion.
Mar 24, 2023 - 11:09 CET
Update - We are continuing to investigate this issue.
Mar 24, 2023 - 10:54 CET
Investigating - We are currently investigating the issue.
Mar 24, 2023 - 10:53 CET
Cognite will be shutting down API Key / non-OIDC based authentication on 31 March 2023
Cognite plans to shut down legacy-auth at 12:00 CET on 31 March 2023. This should come as no surprise to anyone as we have been preparing for this for nearly two years time. There has been ongoing work to migrate customers from legacy-auth to OIDC. See product doc on Intro to Authentication: https://docs.cognite.com/cdf/learn/cdf_auth/cdf_auth_intro/
Why are we disabling legacy-authentication?
Cognite moved to OIDC (OpenID Connect) authentication for many reasons. One very important reason is that OIDC provides an easier to ensure a high level of security for our customers and for Cognite.
Cognite announced the deprecation of its support for legacy authentication in March 2021, following which, extended support for this authentication method was granted until March 31st 2023 to ensure that customers were able to fully migrate without disruption to critical services. Cognite chose to implement support for OIDC to ensure compliance with the latest and most secure standards of authentication available, and discontinuation of legacy authentication serves to ensure customers remain secure.
Additional Information If you need help or encounter ‘issues’, please log a ticket at support@cognite.com
When Cognite disables legacy authentication on 31 March, users will no longer be able to use API keys to log on to CDF without the use of an external IdP provider such as Azure Active Directory. There will be no change to the URL used to log in to applications or access the API. Posted on
Mar 22, 2023 - 13:37 CET
Resolved -
This incident is resolved and the time series and sequences services are now operating at normal performance levels and with the resiliency that the services were designed for.
Mar 20, 09:30 CET
Update -
The engineering team is still not able to lift the rate limits that has been configured to protect the backend storage during the recovery and optimization work that is ongoing. Users will see 429 response codes from the API when rate limits kick in. Users will see higher rates of 429s between 4.00 am and 10:00 am UTC due to work ongoing to improve redundancy.
Mar 18, 08:37 CET
Update -
The engineering team is still working on stabilizing the performance of the timeseries and sequences service. There has been adjustments to the rate limiting in place to reduce the load on the system. End users will get a 429 response code to their requests if their request rate exceeds the rate limits. We are considering further relaxing rate limits, and a new update will be made here if and when this happens.
Mar 17, 13:51 CET
Update -
The engineering team is still working on stabilizing the performance of the timeseries and sequences service. There is rate limiting in place to reduce the load on the system. End users will get a 429 response code to their requests if their request rate exceeds the rate limits. We are adding more resources to the backend systems, but we are not able to lift the rate limits before the processing of the backlog is complete. It will still be a few hours.
Mar 17, 09:14 CET
Update -
The storage backend has now recovered completely and is running with the desired number of replicas. Risk for dataloss is no longer a concern in this incident. There is a processing backlog that now needs to be addressed. Cognite engineering is working on an ussue with query performance degradation related to high load.
Mar 16, 09:51 CET
Update -
The engineering team has fixed the problems related to the replication in the backend database. We are currently running with a normal level of resiliency. But we have still not lifted the rate limiting as we want to observe the system for a while longer before opening up for full load on the system.
Mar 16, 07:03 CET
Update -
The engineering team is still working on resolving this incident. We have had two low-level storage failures in the storage backend. There is redundancy in the system, but not all replicas are fully operational Cognite is now bringing up a restore cluster to mitigate the chances of data loss. Incoming traffic is still being rate limited to protect the service and the storage backend from too high a load during the work on containing and eradicating the incident.
Mar 15, 18:16 CET
Update -
The engineering team is continuing to investigate how to improve the performance of the backend for time series and sequences. To prevent data loss, the team has configured rate limiting for the services. Users will see 429 https responses if these new rate limits are exceeded.
Mar 15, 15:10 CET
Identified -
Cognite Engineering is working on an incident where the backend datastores for timeseries and sequences have performance problems that results in a need to throttle incoming load and from time to time return 5xx responses due to system overload. The engineering team is working on improving the storage system's performance. A new update will be posted when end users experience is believed to change.
Mar 15, 13:56 CET