We have been having problems with some connections using the GRE/IPIP tunnel since about 2021. These failures are intermittent packet loss or no communication at all. And we have received multiple reports of this problem on CommunitySlack. These problems are most likely due to a software bug in our edge router, and we know the following:
- Confirmed on some JuniperNetworks routers.
- Only GRE/IPIP tunnel interfaces that are required to be running NetFlow (jFlow).
- Not confirmed for EtherIP tunnels or direct connections.
- It cannot be recovered by deleting NetFlow or tunnel settings or by restarting the process, and requires a router restart to recover.
- I have tried several software versions and none have been corrected.
- Some TCP communications are receiving packets with damaged payloads, and BGP sessions can be established (BGP control packets arrive normally), but other communications are not possible at all.
We procure our network devices inexpensively through second-hand or online auctions, so vendor technical support is not available. Therefore, no progress has been made in fixing the bug, but we have identified this issue in our testing and in your reports. Stopping all NetFlow will not cause problems, but we are unable to do this. Because NetFlow is essential for DDoS protection and peering.
Routers dedicated to GRE/IPIP tunnels will be installed in Tokyo and Osaka (POP03 and POP52). This is a temporary solution and has the following limitations:
- Full routes are not supported; only default routes or partial routes (peering routes) are supported for BGP connections. This is due to the RIB/FIB performance of the router.
- Due to the installation of one unit each in Tokyo and Osaka, redundant connections using GRE/IP-IP tunnels in the same area are not supported. However, redundant configurations using GRE/IP-IP tunnels and EtherIP tunnels or other methods are supported.
- For those already using GRE/IPIP tunnels on other routers, we will accommodate changes to dedicated routers upon request.
Our organization recommends the EtherIP connection for tunnel connections. Please consider using EtherIP tunnels for new connections.