Cognite engineering has concluded that the incident was fully mitigated at 6:00 am UTC on January 20th. No data loss is expected as a result of this incident, but operations like deletes, updates and creates will have had noticeable delays in the 12 hour period from when this incident started 18:00 UTC January 19th.
Resolved
Cognite engineering has concluded that the incident was fully mitigated at 6:00 am UTC on January 20th. No data loss is expected as a result of this incident, but operations like deletes, updates and creates will have had noticeable delays in the 12 hour period from when this incident started 18:00 UTC January 19th.
Monitoring
The engineering teams have deployed changes that have improved the situation. The engineering team continues to monitor the situation to be able to take additional actions if the changes deployed this far is not enough to resolve the incident. The end users may experience 429 response codes while the system recovers.
Identified
Cognite Engineering is currently investigating how to resolve an incident where updates, creates and deletes are delayed significantly. Most Cognite APIs have methods that are served via a search index, and the ongoing incident is preventing indexing from happening in a timely manner. End users will not experience elevated error rates, but will observe that it takes a long time for CUD operations to be reflected when accessing resource types through Search or List methods of the APIs.