The fix that was deployed on 18:40 UTC on March 17th has proven to mitigate this incident completely.
The fix ensures that both RAW and Transformations are running healthy, and the Incident team has identified several additional preventive measures in place to avoid this incident from happening again. The work on these are currently ongoing.
As a result, this incident will now be closed.
Resolved
The fix that was deployed on 18:40 UTC on March 17th has proven to mitigate this incident completely.
The fix ensures that both RAW and Transformations are running healthy, and the Incident team has identified several additional preventive measures in place to avoid this incident from happening again. The work on these are currently ongoing.
As a result, this incident will now be closed.
Monitoring
Cognite Engineering has deployed a fix correcting the software issue causing this RAW outage. The service has been running without error since 18:40 UTC. Cognite is currently monitoring the performance of both RAW and Transformations, and will conclude whether there are additional actions required to resolve this incident during business hours tomorrow. The monitoring will continue through the night.
Investigating
Cognite Engineering is continuint to investigate the RAW outage in the EUR-W1 cluster. The root cause is still not found, and the service continues to be unavailable. As a result of this outage, also the Transformations service reports a high failure rate. The impact on customers will be higher latency in data incoming and also data missing in applications and scripts accessing data from RAW.