Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

[Jerome Glisse]

On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@xxxxxxxxxx> wrote:
> > On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> >> On 2017/7/20 23:03, Jerome Glisse wrote:
> >> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >> >> On 2017/7/19 10:25, Jerome Glisse wrote:
> >> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:

[...]

> >> > Second device driver are not integrated that closely within mm and the
> >> > scheduler kernel code to allow to efficiently plug in device access
> >> > notification to page (ie to update struct page so that numa worker
> >> > thread can migrate memory base on accurate informations).
> >> >
> >> > Third it can be hard to decide who win between CPU and device access
> >> > when it comes to updating thing like last CPU id.
> >> >
> >> > Fourth there is no such thing like device id ie equivalent of CPU id.
> >> > If we were to add something the CPU id field in flags of struct page
> >> > would not be big enough so this can have repercusion on struct page
> >> > size. This is not an easy sell.
> >> >
> >> > They are other issues i can't think of right now. I think for now it
> >>
> >> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
> >> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
> >> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
> >> demonstrate the whole solution works fine.
> >
> > I am working with NVidia close source driver team to make sure that it works
> > well for them. I am also working on nouveau open source driver for same NVidia
> > hardware thought it will be of less use as what is missing there is a solid
> > open source userspace to leverage this. Nonetheless open source driver are in
> > the work.
> 
> Can you point to the nouveau patches? I still find these HMM patches
> un-reviewable without an upstream consumer.

So i pushed a branch with WIP for nouveau to use HMM:

https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau

Top 16 patches are HMM related (implementic logic inside the driver to use
HMM). The next 16 patches are hardware specific patches and some nouveau
changes needed to allow page fault.

It is enough to have simple malloc test case working:

https://cgit.freedesktop.org/~glisse/compote

There is 2 program here the old one is existing way you use GPU for compute
task while the new one is what HMM allow to achieve ie use malloc memory
directly.


I haven't added yet the device memory support it is in work and i will push
update to this branch and repo for that. Probably next week if no pressing
bug preempt my time.


So there is a lot of ugliness in all this and i don't expect this to be what
end up upstream. Right now there is a large rework of nouveau vm (virtual
memory) code happening to rework completely how we do address space management
within nouveau. This work is prerequisite for a clean implementation for HMM
inside nouveau (it will also lift the 40bits address space limitation that
exist today inside nouveau driver). Once that work land i will work on clean
upstreamable implementation for nouveau to use HMM as well as userspace to
leverage it (this is requirement for upstream GPU driver to have open source
userspace that make use of features). All this is a lot of work and there is
not many people working on this.


They are other initiatives under way related to this that i can not talk about
publicly but if they bare fruit they might help to speedup all this.

Jérôme