Data from AP40 and PATh Facility origins currently inaccessible
Incident Report for OSG Consortium
Update
While existing data in the OSDF data locations /ospool/ap40/data, /path-facility/data, /path-facility/projects remains inaccessible, we have created a new temporary data store to serve data for individual directories in:
- /ospool/ap40/data
- /path-facility/data
Please contact us at support@osg-htc.org if you are not able to access your data folder.

Initially, there is no data inside these directories but it is usable for jobs immediately. Once we are able to bring back the original data store in 2-3 weeks, we will provide a mechanism for users to consolidate their data back into one location.
Posted Dec 06, 2024 - 17:40 UTC
Identified
We have identified the issue that was causing problems for the OSDF data locations /ospool/ap40/data, /path-facility/data, and /path-facility/projects on Thursday, Nov 21.

While there has been no data loss, any existing data in /ospool/ap40/data, /path-facility/data and /path-facility/projects is currently inaccessible and will remain so for 2-3 weeks as we work to recover data from other portions of the same underlying data store.

An alternative data store for using OSDF will be created and shared tomorrow or Wednesday, in order to enable the use of these directories in the meantime. A process for integrating the two data stores will be communicated at a later time.
Posted Nov 26, 2024 - 00:08 UTC
Investigating
The filesystem backing the following data spaces is down:

/ospool/ap40/data

/path-facility/data

/path-facility/projects

These are the data stores that are accessible through AP40 and Path Facility data origins. At this point, data in any of these paths is not accessible from the path on the Access Point or through an osdf:/// URL in an HTCondor job.
Posted Nov 21, 2024 - 22:38 UTC
This incident affects: PATh Facility (AP 1, AP 1 Origin) and OSPool (AP 40).