Re: VMs freezing when host is running 4.14

[Marc Haber]

4.14.3 is still affected.

I am still bisecting between 4.13 and 4.14, 5 steps to go. Defining a
kernel as "good" if it survived 24 hours on the hosts.

Greetings
Marc


On Wed, Nov 22, 2017 at 04:04:42PM +0100, çéæ wrote:
> From: çéæ <jinpuwang@xxxxxxxxx>
> Subject: Re: VMs freezing when host is running 4.14
> To: Marc Haber <mh+linux-kernel@xxxxxxxxxxxx>
> Cc: LKML <linux-kernel@xxxxxxxxxxxxxxx>, "KVM-ML (kvm@xxxxxxxxxxxxxxx)"
>  <kvm@xxxxxxxxxxxxxxx>
> Date:   Wed, 22 Nov 2017 16:04:42 +0100
> List-ID: <linux-kernel.vger.kernel.org>
> X-Spam-Score: (-) -1.9
> X-Spam-Report: torres.zugschlus.de  Content analysis details:   (-1.9
>  points, 5.0 required)   pts  rule name              description  ----
>  ---------------------- ------------------------------------------- -0.0
>  RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, no
>                              trust
>                              [209.85.215.48 listed in list.dnswl.org]  0.0
>  HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail
>                              domains are different  0.0 FREEMAIL_FROM
>           Sender email is commonly abused enduser mail provider
>                              (jinpuwang[at]gmail.com) -0.0
>  RCVD_IN_MSPIKE_H3      RBL: Good reputation (+3)
>                              [209.85.215.48 listed in wl.mailspike.net]
>  -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
>                              [score: 0.0000] -0.1 DKIM_VALID
>              Message has at least one valid DKIM or DK signature -0.1
>  DKIM_VALID_AU          Message has a valid DKIM or DK signature from
>  author's                             domain  0.1 DKIM_SIGNED
>             Message has a DKIM or DK signature, not necessarily valid  0.1
>  FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and EnvelopeFrom
>                               freemail headers are different -0.0
>  RCVD_IN_MSPIKE_WL      Mailspike good senders
> 
> +cc kvm
> 
> 2017-11-22 10:39 GMT+01:00 Marc Haber <mh+linux-kernel@xxxxxxxxxxxx>:
> > On Tue, Nov 21, 2017 at 05:18:21PM +0100, Marc Haber wrote:
> >> On the affected host, VMs freeze at a rate about two or three per day.
> >> They just stop dead in their tracks, console and serial console become
> >> unresponsive, ping stops, they don't react to virsh shutdown, only to
> >> virsh destroy.
> >
> > I was able to obtain a log of a VM before it became unresponsive. here
> > we go:
> >
> > Nov 22 08:19:01 weave kernel: double fault: 0000 [#1] PREEMPT SMP
> > Nov 22 08:19:01 weave kernel: Modules linked in: crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc sg aesni_intel aes_x86_64 crypto_simd glue_helper cryptd input_leds virtio_balloon virtio_console led_class qemu_fw_cfg ip_tables x_tables autofs4 ext4 mbcache jbd2 fscrypto usbhid sr_mod cdrom virtio_blk virtio_net ata_generic crc32c_intel ehci_pci ehci_hcd usbcore usb_common floppy i2c_piix4 virtio_pci virtio_ring virtio ata_piix i2c_core libata
> > Nov 22 08:19:01 weave kernel: CPU: 1 PID: 8795 Comm: debsecan Not tainted 4.14.1-zgsrv20080 #3
> > Nov 22 08:19:01 weave kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
> > Nov 22 08:19:01 weave kernel: task: ffff88001ef0adc0 task.stack: ffffc900001fc000
> > Nov 22 08:19:01 weave kernel: RIP: 0010:kvm_async_pf_task_wait+0x167/0x200
> > Nov 22 08:19:01 weave kernel: RSP: 0000:ffffc900001ffa10 EFLAGS: 00000202
> > Nov 22 08:19:01 weave kernel: RAX: ffff88001fd11cc0 RBX: ffffc900001ffa30 RCX: 0000000000000002
> > Nov 22 08:19:01 weave kernel: RDX: 0140000000000000 RSI: ffffffff8173514b RDI: ffffffff819bdd80
> > Nov 22 08:19:01 weave kernel: RBP: ffffc900001ffaa0 R08: 0000000000193fc0 R09: ffff880000000000
> > Nov 22 08:19:01 weave kernel: R10: ffffc900001ffac0 R11: 0000000000000000 R12: ffffc900001ffa40
> > Nov 22 08:19:01 weave kernel: R13: 0000000000000be8 R14: ffffffff819bdd80 R15: ffffea0000193f80
> > Nov 22 08:19:01 weave kernel: FS:  00007f97e25dd700(0000) GS:ffff88001fd00000(0000) knlGS:0000000000000000
> > Nov 22 08:19:01 weave kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > Nov 22 08:19:01 weave kernel: CR2: 0000000000483001 CR3: 0000000015df7000 CR4: 00000000000406e0
> > Nov 22 08:19:01 weave kernel: Call Trace:
> > Nov 22 08:19:01 weave kernel: do_async_page_fault+0x6b/0x70
> > Nov 22 08:19:01 weave kernel: ? do_async_page_fault+0x6b/0x70
> > Nov 22 08:19:01 weave kernel: async_page_fault+0x22/0x30
> > Nov 22 08:19:01 weave kernel: RIP: 0010:clear_page_rep+0x7/0x10
> > Nov 22 08:19:01 weave kernel: RSP: 0000:ffffc900001ffb88 EFLAGS: 00010246
> > Nov 22 08:19:01 weave kernel: RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000200
> > Nov 22 08:19:01 weave kernel: RDX: ffff88001ef0adc0 RSI: 0000000000193f80 RDI: ffff8800064fe000
> > Nov 22 08:19:01 weave kernel: RBP: ffffc900001ffc50 R08: 0000000000193fc0 R09: ffff880000000000
> > Nov 22 08:19:01 weave kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000020
> > Nov 22 08:19:01 weave kernel: R13: ffff88001ffd5500 R14: ffffc900001ffce8 R15: ffffea0000193f80
> > Nov 22 08:19:01 weave kernel: ? get_page_from_freelist+0x8c3/0xaf0
> > Nov 22 08:19:01 weave kernel: ? __mem_cgroup_threshold+0x8a/0x130
> > Nov 22 08:19:01 weave kernel: ? free_pcppages_bulk+0x3f6/0x410
> > Nov 22 08:19:01 weave kernel: __alloc_pages_nodemask+0xe4/0xe20
> > Nov 22 08:19:01 weave kernel: ? free_hot_cold_page_list+0x2b/0x50
> > Nov 22 08:19:01 weave kernel: ? release_pages+0x2b7/0x360
> > Nov 22 08:19:01 weave kernel: ? mem_cgroup_commit_charge+0x7a/0x520
> > Nov 22 08:19:01 weave kernel: ? account_entity_enqueue+0x95/0xc0
> > Nov 22 08:19:01 weave kernel: alloc_pages_vma+0x7f/0x1e0
> > Nov 22 08:19:01 weave kernel: __handle_mm_fault+0x9cb/0xf20
> > Nov 22 08:19:01 weave kernel: handle_mm_fault+0xb2/0x1f0
> > Nov 22 08:19:01 weave kernel: __do_page_fault+0x1f2/0x440
> > Nov 22 08:19:01 weave kernel: do_page_fault+0x22/0x30
> > Nov 22 08:19:01 weave kernel: do_async_page_fault+0x4c/0x70
> > Nov 22 08:19:01 weave kernel: async_page_fault+0x22/0x30
> > Nov 22 08:19:01 weave kernel: RIP: 0033:0x56434ef679d8
> > Nov 22 08:19:01 weave kernel: RSP: 002b:00007ffd6b48ad80 EFLAGS: 00010206
> > Nov 22 08:19:01 weave kernel: RAX: 00000000000000eb RBX: 000000000000001d RCX: aaaaaaaaaaaaaaab
> > Nov 22 08:19:01 weave kernel: RDX: 000056434f5eb300 RSI: 000000000000000f RDI: 000056434f3ca6c0
> > Nov 22 08:19:01 weave kernel: RBP: 00000000000000ec R08: 00007f97e2453000 R09: 000056434f5eb3ea
> > Nov 22 08:19:01 weave kernel: R10: 000056434f5eb3eb R11: 000056434f4510a0 R12: 000000000000003a
> > Nov 22 08:19:01 weave kernel: R13: 000056434f3ca500 R14: 000056434f451240 R15: 00007f97e1024750
> > Nov 22 08:19:01 weave kernel: Code: f7 49 89 9d a0 d1 9b 81 48 89 55 98 4c 8d 63 10 e8 4f 02 53 00 eb 20 48 83 7d 98 00 74 3a e8 21 6e 06 00 80 7d c0 00 74 3f fb f4 <fa> 66 66 90 66 66 90 e8 7d 6f 06 00 80 7d c0 00 75 da 48 8d b5
> > Nov 22 08:19:01 weave kernel: RIP: kvm_async_pf_task_wait+0x167/0x200 RSP: ffffc900001ffa10
> > Nov 22 08:19:01 weave kernel: ---[ end trace 4701012ee256be25 ]---
> >
> > Does that help?
> >
> So all guest kernels are 4.14, or also other older kernel?
> 
> > Greetings
> > Marc
> 
> Regards,
> Jack
> 
> >
> > --
> > -----------------------------------------------------------------------------
> > Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
> > Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
> > Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421