Re: [LSF/MM TOPIC] Discuss least bad options for resolving longterm-GUP usage by RDMA

[Matthew Wilcox]

On Wed, Feb 06, 2019 at 03:28:35PM -0500, Doug Ledford wrote:
> On Wed, 2019-02-06 at 12:20 -0800, Matthew Wilcox wrote:
> > On Wed, Feb 06, 2019 at 03:16:02PM -0500, Doug Ledford wrote:
> > > On Wed, 2019-02-06 at 11:40 -0800, Matthew Wilcox wrote:
> > > > On Wed, Feb 06, 2019 at 07:16:21PM +0000, Christopher Lameter wrote:
> > > > > though? If we only allow this use case then we may not have to worry about
> > > > > long term GUP because DAX mapped files will stay in the physical location
> > > > > regardless.
> > > > 
> > > > ... except for truncate.  And now that I think about it, there was a
> > > > desire to support hot-unplug which also needed revoke.
> > > 
> > > We already support hot unplug of RDMA devices.  But it is extreme.  How
> > > does hot unplug deal with a program running from the device (something
> > > that would have returned ETXTBSY)?
> > 
> > Not hot-unplugging the RDMA device but hot-unplugging an NV-DIMM.
> > 
> > It's straightforward to migrate text pages from one DIMM to another;
> > you remove the PTEs from the CPU's page tables, copy the data over and
> > pagefaults put the new PTEs in place.  We don't have a way to do similar
> > things to an RDMA device, do we?
> 
> We don't have a means of migration except in the narrowly scoped sense
> of queue pair migration as defined by the IBTA and implemented on some
> dual port IB cards.  This narrowly scoped migration even still involves
> notification of the app.
> 
> Since there's no guarantee that any other port can connect to the same
> machine as any port that's going away, it would always be a
> disconnect/reconnect sequence in the app to support this, not an under
> the covers migration.

I don't understand you.  We're not talking about migrating from one IB
card to another, we're talking about changing the addresses that an STag
refers to.