Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools
From: Luigi Rizzo
Date: Fri Nov 06 2026 - 14:03:29 EST
On Fri, Nov 6, 2026 at 7:43 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote:
>
> On Fri, Nov 06, 2026 at 10:17:38AM -0800, Naman Gulati wrote:
...
> > This problem extends to generally reducing IOMMU overhead in
> > networking - taking the dma map/unmap cost out of the hot-path and
> > avoiding IOTLB thrashing.
>
> Sure, that makes sense, but just having a pool of entirely normal
> pre-dma-mapped memory is quite a different problem from what launched
> this thread...
Since I launched the thread, I guess I should comment...
I wanted to address multiple problems, in random order:
- IOTLB thrashing
- better security than DMA-FQ
- reduce dma map/unmap costs (both on the host and guest)
- reduce/avoid bounce buffer overhead in Confidential Computing and
overcommitted VMs
- not just for networking, also for disks and accelerators etc.
Now it turns out that one way to address all of them was to
- create a special memory allocator, which distributed ordinary memory but
with additional metadata that make it easier to implement large
iommu mappings,
track lifetime etc.
- hook this allocator into relevant kernel subsystems so that consumers
could use it with little or no changes.
As it happens, the first version I posted last month was not terribly clean,
and possibly not terribly well presented, and had some redundancy that
was unnecessary.
The next version (hopefully next week) is hopefully a lot cleaner,
and i think the best witness is how small is the change to pagepool,
basically "if the device has opted in, ask for that special memory"
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index b775b6305fb78..e12ba47a20e43 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
#include <net/page_pool/helpers.h>
#include <net/page_pool/memory_provider.h>
#include <net/xdp.h>
+#include <linux/pisa.h>
#include <linux/dma-direction.h>
#include <linux/dma-mapping.h>
@@ -602,6 +603,8 @@ static noinline netmem_ref
__page_pool_alloc_netmems_slow(struct page_pool *pool
*/
if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
gfp |= __GFP_NOWARN;
+ if (dma_map)
+ gfp |= pisa_rx_gfp(pool->p.dev);
/* Don't support bulk alloc for high-order pages */
if (unlikely(pp_order))
The network tx side is more complicated because, as discussed
earlier, we need to be sure that the leaf device is one that needs
those special dma-able buffers, and there isn't (or I have not found)
one central place to hook into for tcp, udp, zerocopy,
security things etc., leading to a plethora of small hooks
net/core/skbuff.c | 42 +++-
net/core/sock.c | 62 +++++-
net/ipv4/esp4.c | 4 +-
net/ipv4/ip_output.c | 19 +-
net/ipv4/tcp.c | 23 ++-
net/ipv4/tcp_output.c | 12 +-
net/ipv6/esp6.c | 4 +-
net/ipv6/ip6_output.c | 19 +-
net/tls/tls_device.c | 4 +-
cheers
luigi