The Case of the Missing Connectivity

Last Tuesday, our production API pods went silent. Not crashed, not OOMKilled - just couldn’t talk to the database anymore.

The Symptom

$ kubectl exec api-pod-7d8f9 -- curl db-service:5432
curl: (7) Failed to connect to db-service port 5432: Connection refused

But the pods were running. They were healthy. They just couldn’t reach each other.

Turns out someone had applied a NetworkPolicy during the morning deployment that blocked ALL ingress traffic with no exceptions.

We needed to allow database connections from API pods with proper egress rules.

Always test NetworkPolicies in staging first. And never apply “deny-all” policies in production without a rollback plan.

Debugging time: 2 hours | Root cause: Overzealous security policy