Kubernetes Networking Issue

Incident Report for OSG Consortium

Resolved

This incident has been resolved.
Posted Jul 22, 2025 - 21:38 UTC

Monitoring

We have implemented additional fixes to services and they are currently stable. We will continue to closely monitor for any further issues.
Posted Jul 22, 2025 - 19:54 UTC

Investigating

The system issues were not fully resolved, and multiple services are affected. We are actively working to resolve this.
Posted Jul 22, 2025 - 18:34 UTC

Monitoring

We have implemented a fix and are monitoring the services.
Posted Jul 22, 2025 - 16:46 UTC

Update

User facing services impacted by this networking issue include core OSDF services and the OSPool Notebook service. Starting around 3am CDT, jobs reading or writing data using an OSDF URL will have gone on hold.
Posted Jul 22, 2025 - 14:26 UTC

Investigating

We are currently investigating a Kubernetes networking issue affecting multiple services.
Posted Jul 22, 2025 - 13:35 UTC
This incident affected: Kubernetes Infrastructure (Tiger), Open Science Data Federation (Pelican Director, Pelican Registry), and OSPool (OSPool Central Managers / Collectors).