Re: [Networking] ERSPAN decapsulation drops DHCP unicast packets

[Volodymyr Litovka]

Hi Bagas,

this tested on:

- 5.19.0-42 on Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
  -- this is host hardware
- 6.2.0-32 on Virtio network device (under KVM 6.2 on host hardware above)
- 6.5.0-060500rc7 on Virtio network device (under KVM 6.2 on host 
hardware above)

Result is the same for all cases.

Thank you.

On 8/27/23 04:07, Bagas Sanjaya wrote:
> On Sat, Aug 26, 2023 at 09:55:30PM +0200, Volodymyr Litovka wrote:
> > Hi colleagues,
> >
> > I'm trying to catch and process (in 3rd party analytics app) DHCP packets
> > from ERSPAN session, but cannot do this due to absence of DHCP unicast
> > packets after decapsulation.
> >
> > The model is pretty simple: there is PHY interface (enp2s0) which receive
> > ERSPAN traffic and erspan-type interface to get decapsulated packets
> > (inspan, created using command "ip link add inspan type erspan seq key 10
> > local 10.171.165.65 erspan_ver 1", where 10.171.165.65 is ERSPAN target).
> > Then I'm going to rewrite headers in the proper ways (nftable's netdev
> > family) and forward packets to the pool of workers.
> >
> > Having this, I'm expecting everything, which is encapsulated inside ERSPAN,
> > on 'inspan' interface. And there is _almost_ everything except DHCP unicast
> > packets - tcpdump shows about 1kps on this interface of decapsulated
> > packets, but no DHCP unicast (see below traces).
> >
> > To avoid any interactions, I removed and disabled everything that can catch
> > DHCP in userspace - systemd-networkd, netplan, dhcp-client. There is no DHCP
> > server and ifupdown - for test purposes, I'm bringing networking manually.
> > Apparmor disabled as well. Kernel (Linux 5.19.0-42-generic
> > #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC) compiled without CONFIG_IP_PNP
> > (according to /boot/config-5.19.0-42-generic). Nothing in userspace listens
> > on UDP/68 and UDP/67:
> Can you reproduce this on latest mainline?
>
> > # netstat -tunlpa
> > Active Internet connections (servers and established)
> > Proto Recv-Q Send-Q Local Address           Foreign Address
> > State       PID/Program name
> > tcp        0      0 0.0.0.0:22 0.0.0.0:*               LISTEN      544/sshd:
> > /usr/sbin
> > tcp6       0      0 :::22 :::*                    LISTEN      544/sshd:
> > /usr/sbin
> >
> > I have no ideas, why this is happening. Decapsulation itself works, but
> > particular kind of packets get lost.
> >
> > I will appreciate if anyone can help me understand where is the bug - in my
> > configuration or somewhere inside the kernel?
> >
> > Evidence of traffic presence/absence is below.
> >
> > Thank you.
> >
> > Encapsulated ERSPAN session (udp and port 67/68) contains lot of different
> > kinds of DHCP packets:
> >
> > # tcpdump -s0 -w- -i enp2s0 'proto gre and ether[73:1]=17 and
> > (ether[84:2]=67 or ether[84:2]=68)' | tshark -r- -l
> >   [ ... ]
> >      7   0.001942  0.0.0.0 → 255.255.255.255 DHCP 392 DHCP Discover -
> > Transaction ID 0x25c096fc
> >      8   0.003432  z.z.z.z → a.a.a.a         DHCP 418 DHCP ACK      -
> > Transaction ID 0x5515126a
> >      9   0.005170  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
> > Transaction ID 0xa7b7
> >     10   0.005171  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
> > Transaction ID 0xa7b7
> >     11   0.015399  n.n.n.n → z.z.z.z         DHCP 690 DHCP Request  -
> > Transaction ID 0x54955233
> >     12   0.025537  z.z.z.z → n.n.n.n         DHCP 420 DHCP ACK      -
> > Transaction ID 0x54955233
> >     13   0.030313  z.z.z.z → m.m.m.m         DHCP 413 DHCP Offer    -
> > Transaction ID 0xa7b7
> >
> > but decapsulated traffic (which I'm seeing on inspan interface) contains
> > just the following:
> >
> > # tcpdump -i inspan 'port 67 or port 68'
> > listening on inspan, link-type EN10MB (Ethernet), snapshot length 262144
> > bytes
> > 17:23:36.540721 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> > Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> > 17:23:39.760036 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> > Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> > 17:23:44.135711 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> > Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> > 17:23:52.008504 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> > Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> What hardware?
--
Volodymyr Litovka
  "Vision without Execution is Hallucination." -- Thomas Edison