The incident has been resolved and all systems are operating normally again. Some DOIs registered or updated since about 18:00 GMT on July 23 might not have been updated properly. We are working on this and hope to have this resolved by 16:00 PM GMT tomorrow.
The incident was triggered by very unusual activity by a member, trying to update more than a million DOIs in a single morning.
Posted Jul 24, 2019 - 19:35 UTC
Monitoring
We have manually removed a large number of queued jobs from our backlog and are monitoring the situation. There maybe still some delay to those DOI's registered today.
Posted Jul 24, 2019 - 17:44 UTC
Investigating
We are currently experiencing an increased load on our core services, which is causing a delay to complete background queued jobs such as indexing of DOI's within search results or those available in the API. No data loss is occurring, just processing time to be fully realised across all our services is affected.
Posted Jul 24, 2019 - 14:56 UTC
This incident affected: API Servers (REST API) and Frontend Applications (Fabrica, Search).