Slow performance impacting OSDF origins on ap1, ap40

Incident Report for OSG Consortium

Resolved

Performance of the OSDF origins should be back to normal.
Posted Jan 13, 2025 - 19:12 UTC

Identified

Performance of the file systems that serve the OSDF origins for ap1 and ap40 has been slowly improving.
We are hoping to resolve the underlying cause in the coming days.
If this issue is significantly impacting your work, please let us know and we can provide further guidance.
Posted Jan 09, 2025 - 17:20 UTC

Investigating

We've confirmed several reports of slow performance of the OSDF origins for ap1 and ap40, located at `/path-facility/data/` and `/ospool/ap40/data/`, respectively.
We are actively investigating the cause, which at this time we believe is a consequence of our recovery efforts to restore the data from other directories affected by https://status.osg-htc.org/incidents/xcqqhm91pg61 .

Users may encounter slow file transfers to/from the OSDF origins, commands that query files in those directories may be slow or hang up entirely, and jobs may go on hold for file transfer failures.
Unfortunately there is not a good workaround at this time.
We ask that users be patient while we investigate and resolve this issue.
Posted Jan 07, 2025 - 19:27 UTC
This incident affected: OSPool (AP 40) and PATh Facility (AP 1 Origin).