Re: [PATCH v2] dma-buf: Split sgl into page-aligned 2G chunks

[David Laight]

On Thu, 2 Jul 2026 00:56:40 -0400
David Hu <xuehaohu@xxxxxxxxxx> wrote:

> On Tue, Jun 30, 2026 at 8:42 AM Jason Gunthorpe <jgg@xxxxxxxx> wrote:
> >
> > On Tue, Jun 23, 2026 at 11:53:50PM +0100, David Laight wrote:
> >  
> > > > If we restrict incoming dmabuf transfers to fit within VFS-centric
> > > > limits (2GB), we impose unnecessary overhead on the RDMA stack, forcing
> > > > it to manage a significantly higher number of memory registrations. By
> > > > cleanly splitting these massive contiguous device buffers into
> > > > page-aligned SGL entries, we directly improve the efficiency of P2P
> > > > transfers and memory registration.  
> > >
> > > But a divide by '4G - PAGE_SIZE' is also non-trivial and (I think affects
> > > a lot of io) when the quotient is always 1.
> > > Splitting into 2G chunks is a lot cheaper.  
> >
> > Doesn't matter this isn't fast path stuff. It is better to use fewer
> > SGL entries, IHMO.
> >  
> > > > Since this change doesn't seem to have a negative impact on standard file
> > > > I/O or break existing VFS constraints, I'm curious why we shouldn't
> > > > support splitting these >4GB P2P transfers? Am I missing something?  
> > >
> > > I was only wondering whether it was needed...
> > > It does bring up the question of why the >4GB transfers even need splitting.
> > > But that is another question.  
> >
> > SGL can only store an unsigned int size, so any large physical range
> > has to be split down.
> >
> > rdma now a days has code to process the sgl and restore back the > 4G
> > sizes since mode RDMA HW can accept that.
> >
> > commit 486055f5e09df959ad4e3aa4ee75b5c91ddeec2e
> > Author: Michael Margolin <mrgolin@xxxxxxxxxx>
> > Date:   Mon Feb 17 14:16:23 2025 +0000
> >
> >     RDMA/core: Fix best page size finding when it can cross SG entries
> >
> > So whatever this produces needs to be compatible with that to undo it.  
> 
> Thank you everyone. It looks like most open issues are sorted out.
> I'll wait for maintainers to weigh in before sending out v3 (which
> will remove the type cast for min() per David L.'s feedback, and
> revert to ALIGN_DOWN(UINT_MAX, PAGE_SIZE) per Jason's feedback).

Does this code get used a lot for 'normal' transfers?
I'm away from my normal systems and can't check.
But if pretty much all of the fragments are small (< 4G) then
it is probably worth adding a check for 'size < limit' before
anything else and optimising that case.

	David

> 
> Hi Jason,
> 
> Thank you for your feedback. I took a closer look at the commit to
> ensure compatibility. This patch is perfectly complementary, and
> actually prevents a failure in an edge case for the latest
> `ib_umem_find_best_pgsz` [1].
> 
> Regards,
> David
> 
> [1] For dma-buf split with `0xFFFFFFFF`, in case of a discontinguity
> in later buffers, we will hit this code path in
> `ib_umem_find_best_pgsz`
> 
> ```
> if (i != 0)
>     mask |= va;
> ```
> (*After `va` had been incremented by `0xFFFFFFFF`, due to `va +=
> sg_dma_len(sg) - pgoff`)
> (*Which will set the lowest bit of `mask` to 1)
> 
> Because `count_trailing_zeros(mask) returns 0`,
> `ib_umem_find_best_pgsz()` will always return 0 in such cases.