Cooling failure at data center hosting some of OSG / PATh services
Incident Report for OSG Consortium
Resolved
This incident has been resolved.
Posted Mar 08, 2024 - 15:09 UTC
Monitoring
Cooling has been restored and all user facing services should fully functional. If you experience any issues, please reach out to support@osg-htc.org for assistance.
Posted Mar 05, 2024 - 16:20 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 04, 2024 - 21:15 UTC
Update
Though the underlying cooling issue has not be rectified, the OSG team has been able to bring back up ap20.uc.osg-htc.org and ap21.uc.osg-htc.org. Though they are back online, we they may need to be turned off in the future if the cooling situation changes.

The Yum repository has been restored by transferring operations to another OSG hosting provider.
Posted Mar 04, 2024 - 20:55 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 04, 2024 - 14:56 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 04, 2024 - 13:51 UTC
Identified
The cooling infrastructure at the University of Chicago data center has failed and required turning off all of the OSG / PATh assets hosted there. This includes OSPool login hosts ap20.uc.osg-htc.org, ap21.uc.osg-htc.org, ap22.uc.osg-htc.org, and ap23.uc.osg-htc.org, which will not be available until the servers have been turned back on. We have no indications on when the cooling will be restored, therefore we have no estimate on when the services will be available.
Posted Mar 02, 2024 - 16:21 UTC
This incident affected: Software Repositories (Yum Repos, GridCF Repo, OSG Hub), Collaborations (AP 23), Kubernetes Infrastructure (River, Tempest), and OSPool (AP 21, AP 20).