Writing to file with O_DIRECT from DMA buffer

[Daniel Estévez]

Hello,

I'm working on a Zynq MPSoC (aarch64) system. I'm implementing a 
userspace application that writes data from a DMA buffer (contiguous in 
physical memory) into a file in an ext4 filesystem on a SSD. I want to 
use O_DIRECT for this, because there is a larger throughput difference 
when benchmarking writes from a buffer located in regular userspace 
memory (250 MB/s without O_DIRECT vs 1200 MB/s with O_DIRECT).

The DMA buffer is declared with a reserved-memory entry in the device 
tree by using a fixed physical address. There is a simple kernel module 
that lets the userspace application mmap() this buffer. The kernel 
module simply does remap_pfn_range() to create the userspace pages 
corresponding to the physical address range declared in the device tree. 
A DMA in the MPSoC FPGA has already written data into the physical 
memory corresponding to this DMA buffer before the application attempts 
to write the data to a file.

When the userspace application attempts to write data from the buffer 
into the file, which has been opened with O_DIRECT, the writes fail with 
EFAULT. Writes succeed if the file is not opened with O_DIRECT, or if 
instead of the DMA buffer, regular memory from the application (obtained 
with malloc) is used to write into the file (in this case even with 
O_DIRECT enabled).

Considering the difference in how the page tables are set up for regular 
application memory (which supports O_DIRECT) versus my DMA buffer (which 
doesn't work with O_DIRECT), it seems to me that I'm missing something. 
I have tried both with the "no-map" attribute in the reserved-memory 
entry that defines the DMA buffer and without it.

As reported by other users, it seems that this also fails if the DMA 
buffer is allocated with dmam_alloc_coherent():

https://support.xilinx.com/s/question/0D54U00005U9xUhSAJ/odirect-access-to-mmaped-dmamalloccoherent-problems?language=en_US

https://www.linuxquestions.org/questions/linux-software-2/direct-io-fails-when-writting-from-cma-allocated-buffer-4175696145/

I have tried to search what requirements the page tables need to have 
for O_DIRECT to work, but didn't found any information.

Best,
Daniel.

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature