Re: [PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Qingfang Deng]

Hi,

On 2026/9/1 9:56, Ratheesh Kannoth wrote:
> qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> regions (including CN10K LMTST areas that span page boundaries), so
> consumption grows with enabled interfaces and is hard to provision in CMA.
>
> Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> compound pages from the buddy allocator via __get_free_pages(), then map
> them for device access with dma_map_phys() and dma_unmap_phys() using
> DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
> otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
> retry with GFP_DMA32 when the physical range is outside the device DMA
> mask, and wire qmem_alloc()/qmem_free() through them instead of
> dma_alloc_attrs()/dma_free_attrs().

__get_free_pages() can only allocate memory in power-of-two pages. You 
can use alloc_pages_exact() and free_pages_exact() to save memory.

> This works on Octeon because the octeontx2 driver is written for
> DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> the driver already uses streaming DMA APIs for packet data while
> deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> The same IO coherency lets qmem use a streaming map of buddy-allocated
> pages instead of a dedicated coherent allocator or CMA reservation. That
> is valid because the platform is DMA-coherent, not because omitting
> dma_sync_* magically makes memory coherent.
>
> Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> since the buddy allocator cannot serve them without CMA.

Can you confirm that no existing allocations exceeds MAX_PAGE_ORDER?

> cc: Geetha sowjanya <gakula@xxxxxxxxxxx>
> Fixes: 73d33dbc0723 ("octeontx2-af: Use DMA_ATTR_FORCE_CONTIGUOUS attribute in DMA alloc")
> Signed-off-by: Ratheesh Kannoth <rkannoth@xxxxxxxxxxx>
>
> ---
>
> v4 -> v5: Fixed compilation issues.
> 	https://lore.kernel.org/netdev/20260831024210.208447-1-rkannoth@xxxxxxxxxxx/
>
> v3 -> v4: Fixed compilation issues.
> 	https://lore.kernel.org/netdev/apTpKcN_S1xIwRbZ@rkannoth-OptiPlex-7090/
>
> v2 -> v3: Addressed sashiko comments
> 	https://sashiko.dev/#/patchset/20260825045616.3723078-1-rkannoth%40marvell.com
>
> v1 -> v2: Rewrote patch as per sashiko comment
> ---
>   .../ethernet/marvell/octeontx2/af/common.h    | 93 +++++++++++++++++--
>   1 file changed, 87 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/net/ethernet/marvell/octeontx2/af/common.h b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> index 779413a383b7..061dd907f7fa 100644
> --- a/drivers/net/ethernet/marvell/octeontx2/af/common.h
> +++ b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> @@ -7,6 +7,11 @@
>   #ifndef COMMON_H
>   #define COMMON_H
>   
> +#include <linux/dma-mapping.h>
> +#include <linux/dma-map-ops.h>
> +#include <linux/gfp.h>
> +#include <linux/mm.h>
> +
>   #include "rvu_struct.h"
>   
>   #define OTX2_ALIGN			128  /* Align to cacheline */
> @@ -44,6 +49,83 @@ struct qmem {
>   	u32		qsize;
>   };
>   
> +/* Buddy-backed coherent DMA alloc (Option 3): pages from __get_free_pages(),
> + * DMA-reachable RAM via dma_coherent_ok(), and bus/SMMU mappings via
> + * dma_map_phys() -> iommu_dma_map_phys() -> iommu_map() on SMMU systems.
> + */
> +#define OTX2_DMA_COHERENT_ATTRS	DMA_ATTR_REQUIRE_COHERENT
> +
> +static inline bool otx2_dma_phys_in_mask(struct device *dev, phys_addr_t paddr,
> +					 size_t size)
> +{
> +	u64 mask = dma_get_mask(dev);
> +
> +	return paddr + size - 1 <= mask;
> +}
> +
> +static inline void *otx2_dma_alloc_coherent(struct device *dev, size_t size,
> +					    dma_addr_t *dma_handle, gfp_t gfp)
> +{
> +	dma_addr_t dma_addr;
> +	unsigned int order;
> +	phys_addr_t paddr;
> +	void *vaddr;
> +	gfp_t alloc_gfp;
> +
> +	if (!dev || !dma_handle || !size)
> +		return NULL;
> +
> +	if (!dev_is_dma_coherent(dev))
> +		return NULL;
> +
> +	size = PAGE_ALIGN(size);
> +	order = get_order(size);
> +	if (order > MAX_PAGE_ORDER)
> +		return NULL;
> +
> +	alloc_gfp = (gfp & ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM)) |
> +		__GFP_ZERO | __GFP_COMP | __GFP_RECLAIM;
> +
> +	vaddr = (void *)__get_free_pages(alloc_gfp, order);
> +	while (vaddr &&
> +	       !otx2_dma_phys_in_mask(dev, page_to_phys(virt_to_page(vaddr)), size)) {

page_to_phys(virt_to_page(vaddr)) can be simplifed to virt_to_phys(vaddr).

> +		free_pages((unsigned long)vaddr, order);
> +		if (alloc_gfp & GFP_DMA32)
> +			return NULL;
> +		alloc_gfp |= GFP_DMA32;
> +		vaddr = (void *)__get_free_pages(alloc_gfp, order);
> +	}
> +	if (!vaddr)
> +		return NULL;
> +
> +	paddr = page_to_phys(virt_to_page(vaddr));

Same here.

> +	dma_addr = dma_map_phys(dev, paddr, size, DMA_BIDIRECTIONAL,
> +				OTX2_DMA_COHERENT_ATTRS);
> +	if (dma_mapping_error(dev, dma_addr)) {
> +		free_pages((unsigned long)vaddr, order);
> +		return NULL;
> +	}
> +
> +	*dma_handle = dma_addr;
> +	return vaddr;
> +}
Kind regards,
Qingfang