On PeerDirect, we have some kind of a middle-ground solution for pinningMay be I am wrong but my understanding is that PeerDirect logic basically
GPU memory. We create a non-ODP MR pointing to VRAM but rely on
user-space and the GPU not to migrate it. If they do, the MR gets
destroyed immediately. This should work on legacy devices without ODP
support, and allows the system to safely terminate a process that
misbehaves. The downside of course is that it cannot transparently
migrate memory but I think for user-space RDMA doing that transparently
requires hardware support for paging, via something like HMM.
About HMM: I do not think that in the current form HMM would fit inI'm hearing most people say ZONE_DEVICE is the way to handle this,Yes, this is definitely something we need. I think Will Davis's patches
which means the missing remaing piece for RDMA is some kind of DMA
core support for p2p address translation..
are a good start.
Another thing I think is that while HMM is good for user-space
applications, for kernel p2p use there is no need for that.
Using ZONE_DEVICE with or without something like DMA-BUF to pin and unpinPotentially there is another issue related to pin/unpin. If memory could
pages for the short duration as you wrote above could work fine for
kernel uses in which we can guarantee they are short.