unbounded-net : issue timeout AKS control plane routing

[cello86]

During the installation of a Flex Node on a private AKS cluster, we encountered timeout issues toward the control plane, which hosts the private link on the same subnet as the AKS nodes.

Jul 16 16:55:00 kubelet[175]: E0716 16:55:00.235062 175 status_manager.go:1045] "Failed to get status for pod" err="Get "https://xxx.privatelink.westeurope.azmk8s.io:443/api/v1/namespaces/unbounded-net/pods/unbounded-net-kube-proxy-onprem-pero-site-hf57m\": dial tcp 10.6411.13:443: i/o timeout" podUID="d1fdb0d3-728e-475d-a108-16ea4da3be89" pod="unbounded-net/unbounded-net-kube-proxy-onprem-a-site-hf57m"

The network is 10.64.11.0/25. After the unbounded-net-cni container starts up, we see that a route is created that redirects traffic to the unbounded0 interface. As a result, the traffic bypasses the physical interface.

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.96.127.1     0.0.0.0         UG        0 0          0 enp5s0
10.64.11.0      0.0.0.0         255.255.255.128 U         0 0          0 unbounded0
10.96.127.0     0.0.0.0         255.255.255.0   U         0 0          0 enp5s0
10.160.127.0    0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
172.16.100.0    0.0.0.0         255.255.252.0   U         0 0          0 unbounded0
172.16.200.0    0.0.0.0         255.255.252.0   U         0 0          0 unbounded0
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
192.168.0.0     0.0.0.0         255.255.192.0   U         0 0          0 unbounded0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0

We have created both the Site and SitePeering resources, but the issue persists even after rebooting the node.

apiVersion: net.unbounded-cloud.io/v1alpha1
kind: Site
metadata:
  name: azure-cloud-site
  namespace: unbounded-net
spec:
  tunnelProtocol: VXLAN
  nodeCidrs:
    - 10.64.11.0/25   
  podCidrAssignments:           
    - cidrBlocks:
        - 192.168.0.0/18     
---
apiVersion: net.unbounded-cloud.io/v1alpha1
kind: GatewayPool
metadata:
  name: azure-gateway-pool
  namespace: unbounded-net
spec:
  type: Internal
  tunnelProtocol: VXLAN
  nodeSelector:
    kubernetes.io/os: linux
---
apiVersion: net.unbounded-cloud.io/v1alpha1
kind: Site
metadata:
  name: onprem-a-site
  namespace: unbounded-net
spec:
  tunnelProtocol: VXLAN
  nodeCidrs:
    - 10.96.127.0/24              
  podCidrAssignments:
    - cidrBlocks:
        - 172.16.100.0/22
---
apiVersion: net.unbounded-cloud.io/v1alpha1
kind: GatewayPool
metadata:
  name: onprem-a-gateway-pool
  namespace: unbounded-net
spec:
  type: Internal
  tunnelProtocol: VXLAN
  nodeSelector:           
    location: a
---
apiVersion: net.unbounded-cloud.io/v1alpha1
kind: Site
metadata:
  name: onprem-b-site
  namespace: unbounded-net
spec:
  tunnelProtocol: VXLAN
  nodeCidrs:
    - 10.160.127.0/24            
  podCidrAssignments:
    - cidrBlocks:
        - 172.16.200.0/22         
---
apiVersion: net.unbounded-cloud.io/v1alpha1
kind: GatewayPool
metadata:
  name: onprem-b-gateway-pool
  namespace: unbounded-net
spec:
  type: Internal
  tunnelProtocol: VXLAN
  nodeSelector:
    location: b

In this scenario the unbounded-net-node created wrong routes because it used the nodeCidrs value present into the site CRD config and created the routes for the VXLAN traffic but it's not correct.

10.64.11.0      0.0.0.0         255.255.255.128 U         0 0          0 unbounded0
10.96.127.0     0.0.0.0         255.255.255.0   U         0 0          0 enp5s0
10.160.127.0    0.0.0.0         255.255.255.0   U         0 0          0 unbounded0

Marcello

[phealy]

As you've said this is a small cluster, you shouldn't need GatewayPools, and I suspect their inclusion is where the routing issue is coming in. Please try your config with the following:

• The 3 site objects defined as you have above
• No GatewayPools, SiteGatewayPoolAssociations, or GatewayPoolPeerings
• The SitePeering as below:

apiVersion: net.unbounded-cloud.io/v1alpha1
kind: SitePeering
metadata:
  name: peering1
spec:
  sites:
    - azure-cloud-site
    - onprem-a-site
    - onprem-b-site
  meshNodes: true
  tunnelProtocol: VXLAN

Let me know what your route table looks like after this? If you can post the output of the following commands while exec'd into an unbounded-net-node pod, it would be helpful:

ip -4 route show
unroute

[cello86]

Patrick W. Healy (@phealy) I removed all old resources:

# kubectl delete gatewaypoolassignments --all
# kubectl delete gatewaypoolpeerings --all
# kubectl delete gatewaypools --all
gatewaypool.net.unbounded-cloud.io "azure-gateway-pool" deleted
gatewaypool.net.unbounded-cloud.io "onprem-a-gateway-pool" deleted
gatewaypool.net.unbounded-cloud.io "onprem-a-gateway-pool" deleted

I updated the sie peering and restarted:

# kubectl apply -f site-peering.yaml
sitepeering.net.unbounded-cloud.io/expressroute unchanged
# kubectl rollout restart daemonset -n unbounded-net unbounded-net-node
# kubectl get daemonset -n unbounded-net
NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                              AGE
unbounded-net-kube-proxy-azure-cloud-site     0         0         0       0            0           net.unbounded-cloud.io/kube-proxy=managed,net.unbounded-cloud.io/site=azure-cloud-site     16d
unbounded-net-kube-proxy-onprem-a-site     2         2         1       2            1           net.unbounded-cloud.io/kube-proxy=managed,net.unbounded-cloud.io/site=onprem-a-site     16d
unbounded-net-kube-proxy-onprem-b-site   0         0         0       0            0           net.unbounded-cloud.io/kube-proxy=managed,net.unbounded-cloud.io/site=onprem-b-site   16d
unbounded-net-node                            6         6         5       6            5           <none>     

I reset the node and recreated the agent

# crictl exec -it 66a0d5a219d5e ip -4 route show
default via 10.96.127.1 dev enp5s0 proto static
10.96.127.0/24 dev enp5s0 proto kernel scope link src 10.96.127.46
172.16.100.0/22 dev unbounded0 proto static scope link src 172.16.101.1 metric 1 mtu 1280
192.168.0.0/24 dev unbounded0 proto static scope link src 172.16.101.1 metric 1 mtu 1280
192.168.1.0/24 dev unbounded0 proto static scope link src 172.16.101.1 metric 1 mtu 1280
192.168.2.0/24 dev unbounded0 proto static scope link src 172.16.101.1 metric 1 mtu 1280
192.168.3.0/24 dev unbounded0 proto static scope link src 172.16.101.1 metric 1 mtu 1280

# crictl exec -it 66a0d5a219d5e unroute
10.64.11.18/32 via 10.64.11.18 dev vxlan0 proto VXLAN node aks-system-31744701-vmss000006 vni 1 mtu 1280 healthy
10.64.11.20/32 via 10.64.11.20 dev vxlan0 proto VXLAN node aks-infra-36765633-vmss000000 vni 1 mtu 1280 healthy
10.64.11.5/32 via 10.64.11.5 dev vxlan0 proto VXLAN node aks-dev-94474965-vmss000000 vni 1 mtu 1280 healthy
10.64.11.7/32 via 10.64.11.7 dev vxlan0 proto VXLAN node aks-system-31744701-vmss000000 vni 1 mtu 1280 healthy
10.96.127.45/32 via 10.96.127.45 dev vxlan0 proto VXLAN node dev-aks-unbounded-pr01.fineco.it vni 1 mtu 1280 healthy
172.16.100.0/24 via 10.96.127.45 dev vxlan0 proto VXLAN node dev-aks-unbounded-pr01.fineco.it vni 1 mtu 1280 healthy
192.168.0.0/24 via 10.64.11.7 dev vxlan0 proto VXLAN node aks-system-31744701-vmss000000 vni 1 mtu 1280 healthy
192.168.1.0/24 via 10.64.11.18 dev vxlan0 proto VXLAN node aks-system-31744701-vmss000006 vni 1 mtu 1280 healthy
192.168.2.0/24 via 10.64.11.5 dev vxlan0 proto VXLAN node aks-dev-94474965-vmss000000 vni 1 mtu 1280 healthy
192.168.3.0/24 via 10.64.11.20 dev vxlan0 proto VXLAN node aks-infra-36765633-vmss000000 vni 1 mtu 1280 healthy

Marcello

[phealy]

That looks like the bad routes are gone now - do you concur?

[cello86]

Patrick W. Healy (@phealy) I tested the procedure with other nodes and on the secondary site and the problem didn't occur:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.160.127.1    0.0.0.0         UG        0 0          0 enp5s0
10.160.127.0    0.0.0.0         255.255.255.0   U         0 0          0 enp5s0
172.16.100.0    0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
172.16.101.0    0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
172.16.200.0    0.0.0.0         255.255.252.0   U         0 0          0 unbounded0
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 unbounded0

# kubectl get site -A
NAME                 NODE CIDRS            POD CIDR ASSIGNMENTS                   NODES   SLICES   AGE
azure-cloud-site     ["10.64.20.0/25"]     [{"cidrBlocks":["192.168.0.0/18"]}]    4       1        21d
onprem-pero-site     ["10.96.127.0/24"]    [{"cidrBlocks":["172.16.100.0/22"]}]   2       1        21d
onprem-savona-site   ["10.160.127.0/24"]   [{"cidrBlocks":["172.16.200.0/22"]}]   2       1        21d

Could the problem be related to the orphan gateway pools?

Thanks,
Marcello