Re: [PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Ratheesh Kannoth]

On 2026-09-01 at 09:20:01, Qingfang Deng (qingfang.deng@xxxxxxxxx) wrote:
> Hi,
>
> On 2026/9/1 9:56, Ratheesh Kannoth wrote:
> > qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> > allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> > the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> > regions (including CN10K LMTST areas that span page boundaries), so
> > consumption grows with enabled interfaces and is hard to provision in CMA.
> >
> > Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> > compound pages from the buddy allocator via __get_free_pages(), then map
> > them for device access with dma_map_phys() and dma_unmap_phys() using
> > DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
> > otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
> > retry with GFP_DMA32 when the physical range is outside the device DMA
> > mask, and wire qmem_alloc()/qmem_free() through them instead of
> > dma_alloc_attrs()/dma_free_attrs().
>
> __get_free_pages() can only allocate memory in power-of-two pages. You can
> use alloc_pages_exact() and free_pages_exact() to save memory.
Thank you ! will take it as an enhancement to net-next tree after this patch is merged.

>
> > This works on Octeon because the octeontx2 driver is written for
> > DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> > the driver already uses streaming DMA APIs for packet data while
> > deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> > The same IO coherency lets qmem use a streaming map of buddy-allocated
> > pages instead of a dedicated coherent allocator or CMA reservation. That
> > is valid because the platform is DMA-coherent, not because omitting
> > dma_sync_* magically makes memory coherent.
> >
> > Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> > since the buddy allocator cannot serve them without CMA.
>
> Can you confirm that no existing allocations exceeds MAX_PAGE_ORDER?
Tested this patch on cn10k platforms.

>
> > cc: Geetha sowjanya <gakula@xxxxxxxxxxx>
> > Fixes: 73d33dbc0723 ("octeontx2-af: Use DMA_ATTR_FORCE_CONTIGUOUS attribute in DMA alloc")
> > Signed-off-by: Ratheesh Kannoth <rkannoth@xxxxxxxxxxx>
> >
> > ---
> >
> > v4 -> v5: Fixed compilation issues.
> > 	https://lore.kernel.org/netdev/20260831024210.208447-1-rkannoth@xxxxxxxxxxx/
> >
> > v3 -> v4: Fixed compilation issues.
> > 	https://lore.kernel.org/netdev/apTpKcN_S1xIwRbZ@rkannoth-OptiPlex-7090/
> >
> > v2 -> v3: Addressed sashiko comments
> > 	https://sashiko.dev/#/patchset/20260825045616.3723078-1-rkannoth%40marvell.com
> >
> > v1 -> v2: Rewrote patch as per sashiko comment
> > ---
> >   .../ethernet/marvell/octeontx2/af/common.h    | 93 +++++++++++++++++--
> >   1 file changed, 87 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/common.h b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> > index 779413a383b7..061dd907f7fa 100644
> > --- a/drivers/net/ethernet/marvell/octeontx2/af/common.h
> > +++ b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> > @@ -7,6 +7,11 @@
> >   #ifndef COMMON_H
> >   #define COMMON_H
> > +#include <linux/dma-mapping.h>
> > +#include <linux/dma-map-ops.h>
> > +#include <linux/gfp.h>
> > +#include <linux/mm.h>
> > +
> >   #include "rvu_struct.h"
> >   #define OTX2_ALIGN			128  /* Align to cacheline */
> > @@ -44,6 +49,83 @@ struct qmem {
> >   	u32		qsize;
> >   };
> > +/* Buddy-backed coherent DMA alloc (Option 3): pages from __get_free_pages(),
> > + * DMA-reachable RAM via dma_coherent_ok(), and bus/SMMU mappings via
> > + * dma_map_phys() -> iommu_dma_map_phys() -> iommu_map() on SMMU systems.
> > + */
> > +#define OTX2_DMA_COHERENT_ATTRS	DMA_ATTR_REQUIRE_COHERENT
> > +
> > +static inline bool otx2_dma_phys_in_mask(struct device *dev, phys_addr_t paddr,
> > +					 size_t size)
> > +{
> > +	u64 mask = dma_get_mask(dev);
> > +
> > +	return paddr + size - 1 <= mask;
> > +}
> > +
> > +static inline void *otx2_dma_alloc_coherent(struct device *dev, size_t size,
> > +					    dma_addr_t *dma_handle, gfp_t gfp)
> > +{
> > +	dma_addr_t dma_addr;
> > +	unsigned int order;
> > +	phys_addr_t paddr;
> > +	void *vaddr;
> > +	gfp_t alloc_gfp;
> > +
> > +	if (!dev || !dma_handle || !size)
> > +		return NULL;
> > +
> > +	if (!dev_is_dma_coherent(dev))
> > +		return NULL;
> > +
> > +	size = PAGE_ALIGN(size);
> > +	order = get_order(size);
> > +	if (order > MAX_PAGE_ORDER)
> > +		return NULL;
> > +
> > +	alloc_gfp = (gfp & ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM)) |
> > +		__GFP_ZERO | __GFP_COMP | __GFP_RECLAIM;
> > +
> > +	vaddr = (void *)__get_free_pages(alloc_gfp, order);
> > +	while (vaddr &&
> > +	       !otx2_dma_phys_in_mask(dev, page_to_phys(virt_to_page(vaddr)), size)) {
>
> page_to_phys(virt_to_page(vaddr)) can be simplifed to virt_to_phys(vaddr).
ACK.

>
> > +		free_pages((unsigned long)vaddr, order);
> > +		if (alloc_gfp & GFP_DMA32)
> > +			return NULL;
> > +		alloc_gfp |= GFP_DMA32;
> > +		vaddr = (void *)__get_free_pages(alloc_gfp, order);
> > +	}
> > +	if (!vaddr)
> > +		return NULL;
> > +
> > +	paddr = page_to_phys(virt_to_page(vaddr));
>
> Same here.
ACK.
>
> > +	dma_addr = dma_map_phys(dev, paddr, size, DMA_BIDIRECTIONAL,
> > +				OTX2_DMA_COHERENT_ATTRS);
> > +	if (dma_mapping_error(dev, dma_addr)) {
> > +		free_pages((unsigned long)vaddr, order);
> > +		return NULL;
> > +	}
> > +
> > +	*dma_handle = dma_addr;
> > +	return vaddr;
> > +}
> Kind regards,
> Qingfang