Production Factory Outage

Incident Report for OSG Consortium

Resolved

This incident has been resolved.

Posted Jul 21, 2023 - 16:52 UTC

Monitoring

The factory is back up and we're monitoring it.

Posted Jul 20, 2023 - 21:50 UTC

Identified

The factory got into a weird state after a disk hit 100% full yesterday July 19 around 1 pm PT Pacific and has been unable to submit new glidiens since then. We are currently working on recovering it.

As a reminder, while the factory is down, the impact is:
Pools that get capacity using the factory (including the OSPool) will continue to work with the currently allocated capacity, although that capacity will decrease over time until the Factory service is restored. Users may notice fewer running jobs over time, and contributors may notice no new incoming glidein jobs

Posted Jul 20, 2023 - 18:34 UTC

This incident affected: GlideinWMS Factory.