GRACC Down
Incident Report for OSG Consortium
Postmortem

The GRACC database (OpenSearch) relies on a CEPH backend store. The CEPH store was unreachable due to a misconfiguration of a node within the data center. Once the node was configured correctly, GRACC recovered on it’s own.

Posted Jul 09, 2024 - 17:29 UTC

Resolved
The underlying storage system utilized by GRACC malfunctioned and caused GRACC to fail. After the configuration was fixed, GRACC immediately recovered on it's own. No data loss was experienced.
Posted Jul 08, 2024 - 13:52 UTC
Investigating
The GRACC accounting system is down. We are investigating the issue.
Posted Jul 07, 2024 - 13:27 UTC
This incident affected: Accounting (GRACC Frontend, GRACC Backend, GRACC APEL Reporting).