Heisenbug: no DHCP packets received by dnsmasq (dracut-ng test suite)

[Benjamin Drung]

Hi everyone,

I ran into a heisenbug where the culprit is not clear to me. It could be
the linux kernel, QEMU, or dnsmasq. We want to switch from
isc-dhcp-server to dnsmasq in the dracut-ng test suite. I prepared this
change in https://github.com/dracut-ng/dracut-ng/pull/2271 but test 60
fails on most distribution.

How to reproduce
================

Clone dracut-ng and include the dnsmasq patch from
https://github.com/dracut-ng/dracut-ng/pull/2271
Then run `test/test.sh debian:sid 60`. You can replace `debian:sid` by
other containers.

On all containers tested so far (Fedora, Debian, OpenSUSE, etc) except
for the Ubuntu containers the dracut-ng test 60 fails. See
https://github.com/dracut-ng/dracut-ng/issues/2283 for the tested
containers.

Test 60 setup
=============

In a podman container start a QEMU VM with "-netdev socket" or
"-netdev dgram" as server. This server runs dnsmasq. The test has
multiple client tests. Each client test start a QEMU VM as client
connecting the netdev to the server. The same server is used for all
client tests. The client will try to get a DHCP lease from dnsmasq.

Heisenbug symptoms
==================

The third (when using "-netdev socket") or the fifth (when using 
"-netdev dgram") client test hangs. The client sends DHCP discover
packets but dnsmasq on the server never receives them.

When I start to debug this behavior by running tcpdump on the server,
the DHCP discover packets are reaching dnsmasq and the test succeeds.

Running only the failing client test will make the client test succeed.

Dracut-ng bug: https://github.com/dracut-ng/dracut-ng/issues/2283

-- 
Benjamin Drung
Debian & Ubuntu Developer