Re: Handling NUMA page migration

[Frank Mehnert]

On Tuesday 04 June 2013 16:02:30 Michal Hocko wrote:
> On Tue 04-06-13 14:14:45, Frank Mehnert wrote:
> > On Tuesday 04 June 2013 13:58:07 Robin Holt wrote:
> > > This is probably more appropriate to be directed at the linux-mm
> > > mailing list.
> > > 
> > > On Tue, Jun 04, 2013 at 09:22:10AM +0200, Frank Mehnert wrote:
> > > > Hi,
> > > > 
> > > > our memory management on Linux hosts conflicts with NUMA page
> > > > migration. I assume this problem existed for a longer time but Linux
> > > > 3.8 introduced automatic NUMA page balancing which makes the problem
> > > > visible on multi-node hosts leading to kernel oopses.
> > > > 
> > > > NUMA page migration means that the physical address of a page
> > > > changes. This is fatal if the application assumes that this never
> > > > happens for that page as it was supposed to be pinned.
> > > > 
> > > > We have two kind of pinned memory:
> > > > 
> > > > A) 1. allocate memory in userland with mmap()
> > > > 
> > > >    2. madvise(MADV_DONTFORK)
> > > >    3. pin with get_user_pages().
> > > >    4. flush dcache_page()
> > > >    5. vm_flags |= (VM_DONTCOPY | VM_LOCKED)
> > > >    
> > > >       (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND
> > > >       |
> > > >       
> > > >        VM_DONTCOPY | VM_LOCKED | 0xff)
> > > 
> > > I don't think this type of allocation should be affected.  The
> > > get_user_pages() call should elevate the pages reference count which
> > > should prevent migration from completing.  I would, however, wait for
> > > a more definitive answer.
> > 
> > Thanks Robin! Actually case B) is more important for us so I'm waiting
> > for more feedback :)
> 
> The manual node migration code seems to be OK in case B as well because
> Reserved are skipped (check check_pte_range called from down the
> do_migrate_pages path).
> 
> Maybe auto-numa code is missing this check assuming that it cannot
> encounter reserved pages.
> 
> migrate_misplaced_page relies on numamigrate_isolate_page which relies
> on isolate_lru_page and that one expects a LRU page. Is your Reserved
> page on the LRU list? That would be a bit unexpected.

I will check this.

In the meantime I verified that my testcase does not fail if I pass
'numa_balancing=false' to the kernel, so it's definitely a NUMA balancing
problem.

I also did 'get_page()' on all pages of method B but the testcase so this
didn't help.

Frank
-- 
Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany

Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Geschäftsführer: Jürgen Kunz

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher

Attachment: signature.asc
Description: This is a digitally signed message part.