Re: Handling NUMA page migration

[Jerome Glisse]

On Tue, Jun 4, 2013 at 11:45 AM, Jerome Glisse <j.glisse@xxxxxxxxx> wrote:
> On Tue, Jun 04, 2013 at 02:14:45PM +0200, Frank Mehnert wrote:
>> On Tuesday 04 June 2013 13:58:07 Robin Holt wrote:
>> > This is probably more appropriate to be directed at the linux-mm
>> > mailing list.
>> >
>> > On Tue, Jun 04, 2013 at 09:22:10AM +0200, Frank Mehnert wrote:
>> > > Hi,
>> > >
>> > > our memory management on Linux hosts conflicts with NUMA page migration.
>> > > I assume this problem existed for a longer time but Linux 3.8 introduced
>> > > automatic NUMA page balancing which makes the problem visible on
>> > > multi-node hosts leading to kernel oopses.
>> > >
>> > > NUMA page migration means that the physical address of a page changes.
>> > > This is fatal if the application assumes that this never happens for
>> > > that page as it was supposed to be pinned.
>> > >
>> > > We have two kind of pinned memory:
>> > >
>> > > A) 1. allocate memory in userland with mmap()
>> > >
>> > >    2. madvise(MADV_DONTFORK)
>> > >    3. pin with get_user_pages().
>> > >    4. flush dcache_page()
>> > >    5. vm_flags |= (VM_DONTCOPY | VM_LOCKED)
>> > >
>> > >       (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND |
>> > >
>> > >        VM_DONTCOPY | VM_LOCKED | 0xff)
>> >
>> > I don't think this type of allocation should be affected.  The
>> > get_user_pages() call should elevate the pages reference count which
>> > should prevent migration from completing.  I would, however, wait for
>> > a more definitive answer.
>>
>> Thanks Robin! Actually case B) is more important for us so I'm waiting
>> for more feedback :)
>>
>> Frank
>>
>> > > B) 1. allocate memory with alloc_pages()
>> > >
>> > >    2. SetPageReserved()
>> > >    3. vm_mmap() to allocate a userspace mapping
>> > >    4. vm_insert_page()
>> > >    5. vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP)
>> > >
>> > >       (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND |
>> > >       0xff)
>> > >
>> > > At least the memory allocated like B) is affected by automatic NUMA page
>> > > migration. I'm not sure about A).
>> > >
>> > > 1. How can I prevent automatic NUMA page migration on this memory?
>> > > 2. Can NUMA page migration also be handled on such kind of memory without
>> > >
>> > >    preventing migration?
>> > >
>> > > Thanks,
>> > >
>> > > Frank
>
> I was looking at migration code lately, and while i am not an expert at all
> in this area. I think there is a bug in the way handle_mm_fault deals, or
> rather not deals, with migration entry.
>
> When huge page is migrated its pmd is replace with a special swp entry pmd,
> which is a non zero pmd but that does not have any of the huge pmd flag set
> so none of the handle_mm_fault path detect it as swap entry. Then believe
> its a valid pmd and try to allocate pte under it which should oops.
>
> Attached patch is what i believe should be done (not even compile tested).
>
> Again i might be missing a subtelty somewhere else and just missed where
> huge migration entry are dealt with.
>
> Cheers,
> Jerome

Never mind i was missing something hugetlb_fault will handle it.

Cheers,
Jerome
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/