[Question] Missing data after DMA read transfer

[Nicolas Morey-Chaisemartin]

Hi everyone,

Short version:
I'm having an issue with direct DMA transfer from a device to host memory.
It seems some of the data is not transferring to the appropriate page.

Some more details:
I'm debugging a home made PCI driver for our board (Kalray), attached to a x86_64 host running centos7 (3.10.0-327.el7.x86_64)

In the current case, a userland application transfers back and forth data through read/write operations on a file.
On the kernel side, it triggers DMA transfers through the PCI to/from our board memory.

We followed what pretty much all docs said about direct I/O to user buffers:

1) get_user_pages() (in the current case, it's at most 16 pages at once)
2) convert to a scatterlist
3) pci_map_sg
4) eventually coalesce sg (Intel IOMMU is enabled, so it's usually possible)
4) A lot of DMA engine handling code, using the dmaengine layer and virt-dma
5) wait for transfer complete, in the mean time, go back to (1) to schedule more work, if any
6) pci_unmap_sg
7) for read (card2host) transfer, set_page_dirty_lock
8) page_cache_release

In 99,9999% it works perfectly.
However, I have one userland application where a few pages are not written by a read (card2host) transfer.
The buffer is memset them to a different value so I can check that nothing has overwritten them.

I know (PCI protocol analyser) that the data left our board for the "right" address (the one set in the sg by pci_map_sg).
I tried reading the data between the pci_unmap_sg and the set_page_dirty, using
        uint32_t *addr = page_address(trans->pages[0]);
        dev_warn(&pdata->pdev->dev, "val = %x\n", *addr);
and it has the expected value.
But if I try to copy_from_user (using the address coming from userland, the one passed to get_user_pages), the data has not been written and I see the memset value.

I manage to build a test case that fails all the time, but never at the same offset within the buffer.
It's always in the middle (never at the start nor end), for a few pages long (varies between runs).


Am I missing something? Could it be possible that I'm not writing to the right page?
If you need more information, feel free to ask


Thanks in advance

Nicolas