Re: [PATCH v6 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Ratheesh Kannoth]

On 2026-09-02 at 07:40:23, Ratheesh Kannoth (rkannoth@xxxxxxxxxxx) wrote:
> qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> regions (including CN10K LMTST areas that span page boundaries), so
> consumption grows with enabled interfaces and is hard to provision in CMA.
>
> Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> compound pages from the buddy allocator via __get_free_pages(), then map
> them for device access with dma_map_page_attrs(). Add
> otx2_dma_alloc_coherent() and otx2_dma_free_coherent() helpers that
> enforce dev_is_dma_coherent(), retry with GFP_DMA32 when the physical
> range is outside the device DMA mask, and wire qmem_alloc()/qmem_free()
> through them instead of dma_alloc_attrs()/dma_free_attrs().
>
> This works on Octeon because the octeontx2 driver is written for
> DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> the driver already uses streaming DMA APIs for packet data while
> deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> The same IO coherency lets qmem use a streaming map of buddy-allocated
> pages instead of a dedicated coherent allocator or CMA reservation. That
> is valid because the platform is DMA-coherent, not because omitting
> dma_sync_* magically makes memory coherent.
>
> Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> since the buddy allocator cannot serve them without CMA.

My reply to sashiko comments

+----+------------+------------+-----+----------+----------------+---------------+-----------------+
| #  | Location   | Issue      | Sev | Preexist | When it bites  | Suggested fix | My comments     |
+----+------------+------------+-----+----------+----------------+---------------+-----------------+
| 1  | otx2_dma_  | alloc_gfp  | Med | No       | GFP_ATOMIC/    | Preserve      |  All existing callers of this static function
|    | alloc_     | ORs in     |     |          | NOWAIT may     | caller GFP;   |  use GFP_KERNEL. So in sleep in context.
|    | coherent() | __GFP_     |     |          | sleep in       | add RECLAIM   |                 |
|    | GFP        | RECLAIM;   |     |          | atomic ctx     | only when     |                 |
|    | handling   | violates   |     |          | (sched while   | allowed.      |                 |
|    |            | caller GFP |     |          | atomic).       |               |                 |
|    |            | semantics  |     |          | qmem uses      |               |                 |
|    |            |            |     |          | GFP_KERNEL.    |               |                 |
+----+------------+------------+-----+----------+----------------+---------------+-----------------+
| 2  | otx2_dma_  | Coherent   | Hi  | No       | SWIOTLB bounce | Keep coherent |                 |
|    | alloc_     | DMA ->     |     |          | (swiotlb=force | DMA or add    | Commit message clearly explains the reason
|    | coherent() | streaming  |     |          | ARM CCA): CPU  | dma_sync_*;   |                 |
|    | DMA map    | map; no    |     |          | and device use | detect bounce |                 |
|    |            | dma_sync_* |     |          | disjoint mem;  | fallback.     |                 |
|    |            | in queue   |     |          | queue broken.  |               |                 |
+----+------------+------------+-----+----------+----------------+---------------+-----------------+