The issue has been fixed.

Post Mortem:

At 15:38 UTC, the Edge router er1.cgn1 was being drained for maintenance work. At 15:40 UTC, we noticed that there was packet loss when traffic was being routed via spine1.cgn1 and er2.cgn1 towards the internet. The loss started after er2.cgn1, pointing to a possible issue either within the router or towards the upstream links. At the time of issue, we were suspecting an issue on er1.cgn1 where traffic was inadvertibly routed from the upstream to the router, although we advertised it not to do so. After further investigation, it turned out to be an issue on er2.cgn1 itself where traffic from unnumbered interfaces was discarded.

Our dataplane requires unnumbered interfaces to have a Loopback interface assigned explicitly, which was not the case for one of the connections towards the spines.

At 15:46 UTC, a fix has been rolled out accordingly and measurements in the automation have been taken to ensure that this specific error does not happen again.

The fix shows positive results. We will share more details about the root cause once we have finalised the fix.

The issue was identified and a fix has been rolled out. We are monitoring the results.

We are investigating partial packet loss on TCP streams under certain routing conditions in CGN1.

We are investigating the issue