[QUERY] Flushing cache from userspace using VFIO

[Kishon Vijay Abraham I]

Hi Alex, Cornelia,

I'm trying to see if I can use VFIO (Versatile Framework for userspace I/O
[1]) for communication between two cores within the same SoC. I've tried to put
down a picture like below which tries to communicate between ARM64 (running
Linux) and CORTEX R5 (running firmware). It uses rpmsg/remoteproc for the
control messages and the actual data buffers are directly accessed from the
userspace. The location of the data buffers can be informed to the userspace via
rpmsg_vfio (which has to be built as a rpmsg endpoint).

My question is after the userspace application in ARM64 writes to a buffer in
the SYSTEM MEMORY, can it flush it (through a VFIO IOCTL) before handing the
buffer to the CORTEX R5.

If it's implemented within kernel either we use dma_alloc_coherent() for
allocating coherent memory or streaming DMA APIs like
dma_map_single()/dma_unmap_single() for flushing/invalidate the cache.

Trying to see if that is already supported in VFIO or if not, would it be
acceptable to implement it.

Please let me know your thoughts.

┌───────────────────────────────────────────────────────────────────────────┐
│                                                                           │
│ ┌────────────────────┐                                                    │
│ │                    │                                                    │
│ │ ┌──────────────┐   │                                                    │
│ │ │ userspace    │   │  Data Buffers                                      │
│ │ │ Application  ├───┼──────────────┐                                     │
│ │ │              │   │              │                                     │
│ │ └──────▲──┬────┘   │              │                                     │
│ │        │  │        │              │                                     │
│ │        │  │ user   │              │                 ┌─────────────────┐ │
│ │  ──────┼──┼─────── │              │                 │                 │ │
│ │        │  │ kernel │              │                 │                 │ │
│ │  ┌─────┴──▼────┐   │    ┌─────────┼────────────┐    │                 │ │
│ │  │             │   │    │         │            │    │  Data           │ │
│ │  │  rpmsg_vfio │   │    │  ┌──────▼─────────┐  │    │  Buffers        │ │
│ │  │             │   │    │  │ Reserved Region◄──┼────┼────────┐        │ │
│ │  └─────▲──┬────┘   │    │  │                │  │    │        │        │ │
│ │        │  │        │    │  └────────────────┘  │    │        │        │ │
│ │  ┌─────┴──▼────┐   │    │                      │    │        │        │ │
│ │  │             │   │    │                      │    │ ┌──────┴──────┐ │ │
│ │  │  rpmsg      │   │    │     SYSTEM MEMORY    │    │ │ Application │ │ │
│ │  │             │   │    │       (DDR)          │    │ │   Logic     │ │ │
│ │  └─────▲──┬────┘   │    └──────────────────────┘    │ └───▲────┬────┘ │ │
│ │        │  │        │                                │     │    │      │ │
│ │  ┌─────┴──▼────┐   │Notify Firmware/Control Message │ ┌───┴────▼────┐ │ │
│ │  │             ├───┼────────────────────────────────┼─►             │ │ │
│ │  │  remoteproc │   │Interrupt ARM/Control Message   │ │   Firmware  │ │ │
│ │  │             ◄───┼────────────────────────────────┼─┤             │ │ │
│ │  └─────────────┘   │                                │ └─────────────┘ │ │
│ │      ARM64(Linux)  │                                │   ARM CORTEX R5 │ │
│ └────────────────────┘                                └─────────────────┘ │
│                                                                           │
│                                                                       SoC │
└───────────────────────────────────────────────────────────────────────────┘

Thank You,
Kishon

[1] -> https://youtu.be/WFkdTFTOTpA