Re: Enabling peer to peer device transactions for PCIe devices

From: Serguei Sagalovitch
Date: Mon Nov 28 2016 - 16:55:47 EST



On 2016-11-28 04:36 PM, Logan Gunthorpe wrote:
On 28/11/16 12:35 PM, Serguei Sagalovitch wrote:
As soon as PeerDirect mapping is called then GPU must not "move" the
such memory. It is by PeerDirect design. It is similar how it is works
with system memory and RDMA MR: when "get_user_pages" is called then the
memory is pinned.
We haven't touch this in a long time and perhaps it changed, but there
definitely was a call back in the PeerDirect API to allow the GPU to
invalidate the mapping. That's what we don't want.
I assume that you are talking about "invalidate_peer_memory()' callback?
I was told that it is the "last resort" because HCA (and driver) is not
able to handle it in the safe manner so it is basically "abort" everything.