Identified - We have identified the issue that was causing problems for the OSDF data locations /ospool/ap40/data, /path-facility/data, and /path-facility/projects on Thursday, Nov 21.
While there has been no data loss, any existing data in /ospool/ap40/data, /path-facility/data and /path-facility/projects is currently inaccessible and will remain so for 2-3 weeks as we work to recover data from other portions of the same underlying data store.
An alternative data store for using OSDF will be created and shared tomorrow or Wednesday, in order to enable the use of these directories in the meantime. A process for integrating the two data stores will be communicated at a later time.
Nov 26, 2024 - 00:08 UTC
While there has been no data loss, any existing data in /ospool/ap40/data, /path-facility/data and /path-facility/projects is currently inaccessible and will remain so for 2-3 weeks as we work to recover data from other portions of the same underlying data store.
An alternative data store for using OSDF will be created and shared tomorrow or Wednesday, in order to enable the use of these directories in the meantime. A process for integrating the two data stores will be communicated at a later time.
Nov 26, 2024 - 00:08 UTC
Investigating - The filesystem backing the following data spaces is down:
/ospool/ap40/data
/path-facility/data
/path-facility/projects
These are the data stores that are accessible through AP40 and Path Facility data origins. At this point, data in any of these paths is not accessible from the path on the Access Point or through an osdf:/// URL in an HTCondor job.
Nov 21, 2024 - 22:38 UTC
/ospool/ap40/data
/path-facility/data
/path-facility/projects
These are the data stores that are accessible through AP40 and Path Facility data origins. At this point, data in any of these paths is not accessible from the path on the Access Point or through an osdf:/// URL in an HTCondor job.
Nov 21, 2024 - 22:38 UTC
OSPool
Major Outage
AP 20
?
Operational
AP 21
?
Operational
AP 23
?
Operational
AP 40
?
Major Outage
Jupyter Notebooks
?
Operational
OSPool GlidenWMS Frontend
?
Operational
OSPool Central Managers / Collectors
Operational
Open Science Data Federation
Operational
StashCache Redirector
Operational
CVMFS Synchronization
Operational
Data Federation Accounting Service
?
Operational
Caches
?
Operational
Pelican Director
?
Operational
Pelican Registry
?
Operational
Hosted CEs
Operational
Hosted CE Infrastructure
?
Operational
Message Bus
?
Operational
GlideinWMS Factory
?
Operational
OASIS
?
Operational
Network Monitoring Pipeline
?
Operational
Software Repositories
Operational
Yum Repos
?
Operational
GridCF Repo
?
Operational
OSG Hub
?
Operational
Accounting
Operational
GRACC Frontend
?
Operational
GRACC Backend
?
Operational
GRACC APEL Reporting
?
Operational
Websites
Operational
Display
?
Operational
Main Website
?
Operational
DNS
?
Operational
OSGConnect Website
?
Operational
Topology
?
Operational
Hosted Submit
Operational
Hosted Submit Infrastructure
?
Operational
Hosted GlideinWMS
Operational
IGWN GWMS Frontend
Operational
JLAB GWMS Frontend
Operational
GLUEX GWMS Frontend
Operational
UCSD CMS GWMS Frontend
Operational
UCSD CMS VO Collector
Operational
Kubernetes Infrastructure
Operational
Tiger
Operational
River
Operational
Tempest
Operational
PATh Facility
Major Outage
AP 1
?
Degraded Performance
AP 1 Origin
?
Major Outage
Collaborations
Operational
AP 23
?
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance