GRACC Down

Incident Report for OSG Consortium

Postmortem

The GRACC database (OpenSearch) relies on a CEPH backend store. The CEPH store was unreachable due to a misconfiguration of a node within the data center. Once the node was configured correctly, GRACC recovered on it’s own.

Posted Jul 09, 2024 - 17:29 UTC

Resolved

The underlying storage system utilized by GRACC malfunctioned and caused GRACC to fail. After the configuration was fixed, GRACC immediately recovered on it's own. No data loss was experienced.

Posted Jul 08, 2024 - 13:52 UTC

Investigating

The GRACC accounting system is down. We are investigating the issue.

Posted Jul 07, 2024 - 13:27 UTC

This incident affected: Accounting (GRACC Frontend, GRACC Backend, GRACC APEL Reporting).