Re: Enabling peer to peer device transactions for PCIe devices

From: Serguei Sagalovitch
Date: Fri Nov 25 2016 - 12:59:47 EST

Well, I guess there's some consensus building to do. The existing
options are:

* Device DAX: which could work but the problem I see with it is that it
only allows one application to do these transfers. Or there would have
to be some user-space coordination to figure which application gets what
About one application restriction: so it is per memory mapping? I assume that
it should not be problem for one application to do transfer to the several devices
simultaneously? Am I right?

May be we should follow RDMA MR design and register memory for p2p transfer from user

What about the following:

a) Device DAX is created
b) "Normal" (movable, etc.) allocation will be done for PCIe memory and CPU pointer/access will
be requested.
c) p2p_mr_register() will be called and CPU pointer (mmap( on DAX Device)) will be returned.
Accordingly such memory will be marked as "unmovable" by e.g. graphics driver.
d) When p2p is not needed then p2p_mr_unregister() will be called.

What do you think? Will it work?