Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[Dragos Tatulea]

On 18.09.26 08:06, jiabin deng wrote:
> Hi Dragos,
> 
> Thank you for pointing us to this fix.
> 
> Our original test used Ubuntu kernel 7.0.0-30-generic, based on
> v7.0.12, and did not include that change. We have now repeated the
> test using Ubuntu's mainline build of Linux 7.2.6
> (7.2.6-070206-generic), with its in-tree mlx5_core driver.
> 
> I checked the v7.2.6 source: it contains the small-frame guard from
> commit e2466392a0b8496000e12181cb1ee1535eb0da25  In
> mlx5e_handle_rx_cqe_mpwrq_shampo(), frames with
> cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN set match = false and
> flush = true [2].
> 
> The zero-filled corruption ending at the next 64-byte boundary in
> GPU memory still reproduces with this kernel.
> 
> We kept the same sender and receiver data-checking logic, with
> tcp-data-split on and rx-gro-hw on. One additional environment
> change is that the NVIDIA GPU driver was updated from 595.71.05 to
> 595.91.07. The NIC firmware remains 32.43.2400 (MT_0000001117).
> 
> We repeated the same eight message sizes, from 32 KiB to 4 MiB,
> with 50 iterations per size:
> 
> - All 400 iterations completed.
> - 105 iterations passed; 295 contained corrupted data.
> - 21,599 corrupted ranges were recorded, all zero-filled and
>   ending at the next 64-byte boundary in GPU memory.
> - For each size from 256 KiB through 4 MiB, all 50 iterations
>   contained corruption.
> - The full-message snapshots taken before and after
>   cuFlushGPUDirectRDMAWrites() were identical in all iterations.
> 
> For example, in the first 32 KiB iteration, an 8-byte corrupted
> range starts at offset 2147495416 bytes from the beginning of the
> GPU buffer. This offset is 56 bytes into a 64-byte-aligned block,
> so the eight zero bytes end exactly at the next 64-byte boundary.
> 
> These results come from the reproducer's runtime byte comparisons.
> 
> Could you suggest the next targeted checks or additional debug
> information that would help narrow this down?
> 
Did you also try CPU buffers? From the NIC perspective it shouldn't
be any different.

Could you share a program + reproduction steps?

Thanks,
Dragos