fs-prod-cs-tool2 NotReady — registry egress interruption

Incident inc-4821 · owner Network Operations — Nordic Federated Bank. Focus mode widens the reading canvas for evidence review.

Back to workspace Evidence

Analysis package

Verification agent confidence, supporting evidence and rejected hypotheses.

Read-only Agent OS

No shell execution
No cluster-admin
No secret reads
No database writes
No firewall changes
No autonomous remediation

Root cause

Verification agent confidence score

88%

Registry egress traffic from fs-prod-cs-tool2 is being reset mid-transfer, most likely by SSL inspection on the outbound path introduced in change CHG-20482. Container image layers for the CNI plugin cannot complete, so the container runtime network never becomes ready and the node reports NotReady.

Supporting evidence

4 verified classes

verifiedImage pull attempts reset by peer at consistent layer boundaries (9/9 failures, external registry only).
verifiedCalico CNI pod stuck in ContainerCreating; FailedCreatePodSandBox repeats every ~35 seconds.
verifiedRegistry connectivity probe fails on outbound TCP 443 after 5-8 seconds while TLS handshake succeeds.
verifiedExisting workloads already scheduled on the node remain healthy — no compute, memory or disk pressure.

Rejected hypotheses

3 ruled out

rejectedNode resource exhaustion — no pressure conditions and utilisation under 50%.
rejectedKubelet certificate expiry — certificate valid until 2026-12-03.
rejectedCNI configuration drift — configuration matches healthy peers byte for byte.

Recommendation

Requires human execution — the Agent OS never remediates

Validate outbound TCP 443 connectivity from the node subnet to the registry egress range and confirm SSL-inspection exclusions cover registry.corp.internal and the upstream mirror. Re-run the image pull after the exclusion is verified.