Re: Major KVM issues with kernel 4.5 on the host

From: Dr. David Alan Gilbert
Date: Sat Apr 23 2016 - 15:12:21 EST


* Marc Haber (mh+linux-kernel@xxxxxxxxxxxx) wrote:
> On Sat, Apr 23, 2016 at 06:04:29PM +0200, Borislav Petkov wrote:
> > On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> > > Yes, but there are two symptoms. The VM either suffers file system
> > > issues (garbage read from files, or an aborted ext4 journal and
> > > following ro remount) or it stops dead in its tracks.
> >
> > Stops dead? What does that mean exactly? Box is wedged solid and it
> > doesn't react to any key presses?
>
> No ping, no reaction on serial console, no reaction on virtual
> console, no syslog entries.
>
> > Because if so, this could really be a DRAM going bad and a correctable
> > error turning into an uncorrectable. How old is the DRAM in that box?
> > Judging by your CPU, it should be a couple of years...
>
> Uncorrectable errors would still be identified by the ECC hardware,
> and the box wouldn't be perfectly fine with an "old" kernel.

Hmm, your problem does sound like bad hardware, but....
If you've got a nice reliable crash, can you try turning transparent huge pages
off on the host;
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Dave

> > > The box reports about one correctable error per week, so I probably
> > > have a faulty DIMM, but since the issue only surfaces in VMs while the
> > > host system is in perfect working order...
> >
> > So it could be that correctable error turns into an uncorrectable one at
> > some point. But then you should be getting an exception...
>
> Yes, that would be in the logs.
>
> > > And yes, I am pondering to simply replace the box with an Intel CPU.
> >
> > Your CPU is fine, from what I've seen so far.
>
> But we still postulate that the issue does only show on older AMD
> CPUs. Otherwise, I wouldn't be the only one making this experience.
>
> > > I go the way of Debian packages since it is easier to handle the
> > > crypto file systems when the machine is booting up.
> >
> > As long as you're testing the correct bisection kernels...
>
> I am reasonably sure about that, yes.
>
> > > And yes, I think about doing a test reinstall on unencrypted disk to
> > > find out whether encryption plays a role, but I currently need the
> > > machine to urgently to take it out of serice for half a month, and,
> > > again, the host system is in perfect working order, it is just VMs
> > > that barf.
> >
> > Yeah, I can't reproduce it here and I have a very similar box to yours
> > which is otherwise idle, more or less.
> >
> > Another fact which points to potentially DIMM going bad...
>
> Do you want me to memtest for 24 hours?
>
> Greetings
> Marc
>
> --
> -----------------------------------------------------------------------------
> Marc Haber | "I don't trust Computers. They | Mailadresse im Header
> Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
> Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux | Happy \
\ dave @ treblig.org | | In Hex /
\ _________________________|_____ http://www.treblig.org |_______/