xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Julien Grall]

(Adding more people in the thread)

Hi,

On 3/12/19 3:59 PM, Julien Grall wrote:
> Hi all,
>
> It looks like all the arm test for linus [1] and next [2] tree
> are now failing. x86 seems to be mostly ok.
>
> The bisector fingered the following commit:
>
> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
> Author: Matthew Wilcox <willy@xxxxxxxxxxxxx>
> Date:   Tue Mar 5 15:46:06 2019 -0800
>
>      mm/memory.c: prevent mapping typed pages to userspace
>      
>      Pages which use page_type must never be mapped to userspace as it would
>      destroy their page type.  Add an explicit check for this instead of
>      assuming that kernel drivers always get this right.
>      
>      Link: http://lkml.kernel.org/r/20190129053830.3749-1-willy@xxxxxxxxxxxxx
>      Signed-off-by: Matthew Wilcox <willy@xxxxxxxxxxxxx>
>      Reviewed-by: Kees Cook <keescook@xxxxxxxxxxxx>
>      Reviewed-by: David Hildenbrand <david@xxxxxxxxxx>
>      Cc: Michael Ellerman <mpe@xxxxxxxxxxxxxx>
>      Cc: Will Deacon <will.deacon@xxxxxxx>
>      Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>      Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
>
> I have tried the latest linus mater as Dom0 and see some issue to get the
> console guest:
>
> 42sh> sudo xl create -c ~/works/guest/guest.cfg
> Parsing config from /home/julien/works/guest/guest.cfg
> xenconsole: Could not read tty from store: Success
>
> I have added a print in the error path added by the commit above to see
> what and where it happens:
>
> [  182.366372] PageAnon 0 PageSlab 0 page_has_type 1
> [  182.371076] WARNING: CPU: 2 PID: 2210 at mm/memory.c:1459 vm_insert_page+0x3e
> 0/0x430
> [  182.378837] Modules linked in:
> [  182.381974] CPU: 2 PID: 2210 Comm: xenstored Not tainted 5.0.0-10742-gea29548
> 1b6e3-dirty #1310
> [  182.390672] Hardware name: ARM Juno development board (r2) (DT)
> [  182.396678] pstate: 40000005 (nZcv daif -PAN -UAO)
> [  182.401553] pc : vm_insert_page+0x3e0/0x430
> [  182.405816] lr : vm_insert_page+0x3e0/0x430
> [  182.410077] sp : ffff000012773bc0
> [  182.413471] x29: ffff000012773bc0 x28: 0000ffff8d3fa000
> [  182.418866] x27: 0000000000000008 x26: 0000000000000001
> [  182.424261] x25: 0000000000000008 x24: 0000ffff8d3fa000
> [  182.429656] x23: 0068000000000f53 x22: ffff8008ab520a00
> [  182.435052] x21: ffff000011644a88 x20: 0000000000000000
> [  182.440454] x19: ffff7e00229b0e80 x18: 0000000000000000
> [  182.445841] x17: 0000000000000000 x16: 0000000000000000
> [  182.451245] x15: 00000000fffffff0 x14: 0000000000000000
> [  182.456631] x13: ffff000012339ff0 x12: 0000000000000006
> [  182.462027] x11: ffff000010ec4980 x10: ffff0000107d01f8
> [  182.467422] x9 : 00000000fffb9fff x8 : ffff8008ab55a0a0
> [  182.472817] x7 : 0000000000000001 x6 : ffff8008bb02a220
> [  182.478212] x5 : ffff8008bb02a220 x4 : 0000000000000000
> [  182.483607] x3 : ffff8008bb032708 x2 : b98ad6a7b7eb2900
> [  182.489002] x1 : 0000000000000000 x0 : 0000000000000025
> [  182.494397] Call trace:
> [  182.496924]  vm_insert_page+0x3e0/0x430
> [  182.500853]  gntdev_mmap+0x188/0x288
> [  182.504495]  mmap_region+0x3dc/0x578
> [  182.508149]  do_mmap+0x2d4/0x478
> [  182.511457]  vm_mmap_pgoff+0xe0/0x108
> [  182.515198]  ksys_mmap_pgoff+0xac/0x308
> [  182.519116]  __arm64_sys_mmap+0x28/0x38
> [  182.523029]  el0_svc_handler+0x88/0x100
> [  182.526943]  el0_svc+0x8/0xc
> [  182.529901] ---[ end trace cf738ac71bfed946 ]---
>
> Does it ring any bell?

It turns out the problem is because the balloon driver will call 
__SetPageOffline() on allocated page. Therefore the page has a type and 
vm_insert_pages will deny the insertion.

My knowledge is quite limited in this area. So I am not sure how we can 
solve the problem.

I would appreciate if someone could provide input of to fix the mapping.

Cheers,

> Cheers,
>
> [1] http://logs.test-lab.xenproject.org/osstest/logs/133695/
> [2] http://logs.test-lab.xenproject.org/osstest/logs/133655/
>
> On 3/11/19 11:52 PM, osstest service owner wrote:
> > branch xen-unstable
> > xenbranch xen-unstable
> > job test-arm64-arm64-xl-xsm
> > testid guest-start
> >
> > Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> > Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> > Tree: xen git://xenbits.xen.org/xen.git
> >
> > *** Found and reproduced problem changeset ***
> >
> >     Bug is in tree:  linux git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> >     Bug introduced:  0ee930e6cafa048c1925893d0ca89918b2814f2c
> >     Bug not present: 2d432cb7091e99881af803cdd67a31969b863005
> >     Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/133723/
> >
> >
> >     commit
>
>
> >     Author: Matthew Wilcox <willy@xxxxxxxxxxxxx>
> >     Date:   Tue Mar 5 15:46:06 2019 -0800
> >     
> >         mm/memory.c: prevent mapping typed pages to userspace
> >         
> >         Pages which use page_type must never be mapped to userspace as it would
> >         destroy their page type.  Add an explicit check for this instead of
> >         assuming that kernel drivers always get this right.
> >         
> >         Link: http://lkml.kernel.org/r/20190129053830.3749-1-willy@xxxxxxxxxxxxx
> >         Signed-off-by: Matthew Wilcox <willy@xxxxxxxxxxxxx>
> >         Reviewed-by: Kees Cook <keescook@xxxxxxxxxxxx>
> >         Reviewed-by: David Hildenbrand <david@xxxxxxxxxx>
> >         Cc: Michael Ellerman <mpe@xxxxxxxxxxxxxx>
> >         Cc: Will Deacon <will.deacon@xxxxxxx>
> >         Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
> >         Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> >
> >
> > For bisection revision-tuple graph see:
> >      http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-linus/test-arm64-arm64-xl-xsm.guest-start.html
> > Revision IDs in each graph node refer, respectively, to the Trees above.
> >
> > ----------------------------------------
> > Running cs-bisection-step --graph-out=/home/logs/results/bisect/linux-linus/test-arm64-arm64-xl-xsm.guest-start --summary-out=tmp/133723.bisection-summary --basis-template=133580 --blessings=real,real-bisect linux-linus test-arm64-arm64-xl-xsm guest-start
> > Searching for failure / basis pass:
> >    133673 fail [host=laxton0] / 133605 [host=laxton1] 133580 ok.
> > Failure / basis pass flights: 133673 / 133580
> > Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> > Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> > Tree: xen git://xenbits.xen.org/xen.git
> > Latest 38e7571c07be01f9f19b355a9306a4e3d5cb0f5b c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> > Basis pass 736706bee3298208343a76096370e4f6a5c55915 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 f393b82fe5ba3ed9cfe2b306ffa53368e55b75af
> > Generating revisions with ./adhoc-revtuple-generator  git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git#736706bee3298208343a76096370e4f6a5c55915-38e7571c07be01f9f19b355a9306a4e3d5cb0f5b git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb9558310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860 git://xenbits.xen.org/qemu-xen.git#de5b678ca4dcdfa83e322491d478d66df56c1986-de5b678ca4dcdfa83e322491d478d66df56c1986 git://xenbits.xen.org/xen.git#f393b82fe5ba3ed9c\
> >    fe2b306ffa53368e55b75af-eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> > Loaded 149180 nodes in revision graph
> > Searching for test results:
> >    133581 blocked irrelevant
> >    133585 blocked irrelevant
> >    133605 [host=laxton1]
> >    133580 pass 736706bee3298208343a76096370e4f6a5c55915 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 f393b82fe5ba3ed9cfe2b306ffa53368e55b75af
> >    133679 fail irrelevant
> >    133689 pass irrelevant
> >    133674 pass 736706bee3298208343a76096370e4f6a5c55915 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 f393b82fe5ba3ed9cfe2b306ffa53368e55b75af
> >    133631 fail irrelevant
> >    133686 fail irrelevant
> >    133678 fail irrelevant
> >    133681 pass irrelevant
> >    133683 pass irrelevant
> >    133685 pass irrelevant
> >    133687 fail irrelevant
> >    133688 fail irrelevant
> >    133673 fail 38e7571c07be01f9f19b355a9306a4e3d5cb0f5b c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133690 fail irrelevant
> >    133720 fail 0ee930e6cafa048c1925893d0ca89918b2814f2c c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133691 fail irrelevant
> >    133692 fail irrelevant
> >    133694 fail 38e7571c07be01f9f19b355a9306a4e3d5cb0f5b c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133721 pass 2d432cb7091e99881af803cdd67a31969b863005 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133723 fail 0ee930e6cafa048c1925893d0ca89918b2814f2c c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133697 fail e5eed351fd5eb73eecc1407cf00309e868379253 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133700 fail f900482da560941f978b0d36660e96f48ea78752 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133701 pass c52e75935f8ded2bd4a75eb08e914bd96802725b c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133702 fail 8bb4e7a2ee26c05a94ae6cb0aec2f82a3523cf35 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133704 pass 9bebefd59084af7c75b66eeee241bf0777f39b88 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133705 pass dc50537bdd1a0804fa2cbc990565ee9a944e66fa c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133709 fail 0ee930e6cafa048c1925893d0ca89918b2814f2c c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133710 pass 677dc9731b54dccaaadbdcea18f8eecc95cee832 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133711 pass afd07389d3f4933c7f7817a92fb5e053d59a3182 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133713 pass 2d432cb7091e99881af803cdd67a31969b863005 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133715 fail 0ee930e6cafa048c1925893d0ca89918b2814f2c c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133718 blocked 2d432cb7091e99881af803cdd67a31969b863005 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> >    133719 pass 2d432cb7091e99881af803cdd67a31969b863005 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> > Searching for interesting versions
> >    Result found: flight 133580 (pass), for basis pass
> >    Result found: flight 133673 (fail), for basis failure
> >    Repro found: flight 133674 (pass), for basis pass
> >    Repro found: flight 133694 (fail), for basis failure
> >    0 revisions at 2d432cb7091e99881af803cdd67a31969b863005 c530a75c1e6a472b0eb9558310b518f0dfcd8860 de5b678ca4dcdfa83e322491d478d66df56c1986 eeb31ee522c7bb8541eb4c037be2c42bfcf0a3c3
> > No revisions left to test, checking graph state.
> >    Result found: flight 133713 (pass), for last pass
> >    Result found: flight 133715 (fail), for first failure
> >    Repro found: flight 133719 (pass), for last pass
> >    Repro found: flight 133720 (fail), for first failure
> >    Repro found: flight 133721 (pass), for last pass
> >    Repro found: flight 133723 (fail), for first failure
> >
> > *** Found and reproduced problem changeset ***
> >
> >     Bug is in tree:  linux git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> >     Bug introduced:  0ee930e6cafa048c1925893d0ca89918b2814f2c
> >     Bug not present: 2d432cb7091e99881af803cdd67a31969b863005
> >     Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/133723/
> >
> >
> >     commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
> >     Author: Matthew Wilcox <willy@xxxxxxxxxxxxx>
> >     Date:   Tue Mar 5 15:46:06 2019 -0800
> >     
> >         mm/memory.c: prevent mapping typed pages to userspace
> >         
> >         Pages which use page_type must never be mapped to userspace as it would
> >         destroy their page type.  Add an explicit check for this instead of
> >         assuming that kernel drivers always get this right.
> >         
> >         Link: http://lkml.kernel.org/r/20190129053830.3749-1-willy@xxxxxxxxxxxxx
> >         Signed-off-by: Matthew Wilcox <willy@xxxxxxxxxxxxx>
> >         Reviewed-by: Kees Cook <keescook@xxxxxxxxxxxx>
> >         Reviewed-by: David Hildenbrand <david@xxxxxxxxxx>
> >         Cc: Michael Ellerman <mpe@xxxxxxxxxxxxxx>
> >         Cc: Will Deacon <will.deacon@xxxxxxx>
> >         Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
> >         Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> >
> > pnmtopng: 189 colors found
> > Revision graph left in /home/logs/results/bisect/linux-linus/test-arm64-arm64-xl-xsm.guest-start.{dot,ps,png,html,svg}.
> > ----------------------------------------
> > 133723: tolerable ALL FAIL
> >
> > flight 133723 linux-linus real-bisect [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/133723/
> >
> > Failures :-/ but no regressions.
> >
> > Tests which did not succeed,
> > including tests which could not be run:
> >    test-arm64-arm64-xl-xsm      12 guest-start             fail baseline untested
> >
> >
> > jobs:
> >    test-arm64-arm64-xl-xsm                                      fail
> >
> >
> > ------------------------------------------------------------
> > sg-report-flight on osstest.test-lab.xenproject.org
> > logs: /home/logs/logs
> > images: /home/logs/images
> >
> > Logs, config files, etc. are available at
> >       http://logs.test-lab.xenproject.org/osstest/logs
> >
> > Explanation of these reports, and of osstest in general, is at
> >       http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
> >       http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
> >
> > Test harness code can be found at
> >       http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxxx
> > https://lists.xenproject.org/mailman/listinfo/xen-devel

--
Julien Grall