Re: perfevents: irq loop stuck!

[Peter Zijlstra]

On Wed, May 14, 2014 at 10:55:40PM -0400, Vince Weaver wrote:
> On Tue, 13 May 2014, Vince Weaver wrote:
> 
> >         pe[32].sample_period=0xc0000000000000bd;
> > 
> > Should it be possible to open an event with a large negative sample_period 
> > like that? 
> 
> so this seems to be a real bug.
> 
> attr->sample_period is a u64 value, but internally it gets cast to
> s64 and is added against itself and so all kinds of unexpected things 
> happen.
> 
> So if you set attr->sample_period to 0xc0000000000000bd in the hopes of 
> sampling the RETIRED_INSTRUCTIONS event every 5 years or so, instead
> what happens is that in
> 	x86_perf_event_set_period()
> the value is cast to a signed 64-bit value, so we are now negative.
> 
> Then "left" is set to period because we are negative.
> 
> Then since left is less than 0, we double the period value.
> This overflows the 64-bit integer and suddenly we are in undefined
> behavior territory and we're lucky the C compiler doesn't decide to
> format the hard drive.
> 
> Anyway we are still less than 0, so then the
> 
> 	if (unlikely(left < 2))
> 		left = 2;
> 
> code kicks in and suddenly our hugely positive sample_period has changed
> to just being "2".  And so we suddenly get a storm of interrupts instead 
> of one every 5 years.
> 
> So, not sure how to fix this without a total re-write, unless we want to 
> cheat and just say sample_period is capped at 62-bits or something.

63 bits should do I think, but yes, we hit a very similar but a few days
ago in the sched_deadline code.

I'm fine with capping it, allowing the full 64bit would mean we need 65
bits (effectively 96 or 128 bit of course) math to make it all work
which would be tedious and give no real gain.

> Also it's unclear why sometimes this can cause a stuck interrupt leading 
> to the "irq loop stuck" message.  I have a reproducible fuzzer test case 
> that will cause this to happen, but can't isolate it down to a simple test 
> case...

So yes.. in theory we should throttle the event after a number of
interrupts to keep the machine alive, and I think that happens because
I've managed to hit it too while trying to reproduce one of the problems
Sasha reported and the machine was largely usuable.

It might trigger the getting stuck msgs because we can reach the 2
events (2 retired instructions isn't that many) in the NMI return path
after we've re-enabled the PMU I suppose.

We'll then not make progress for a while, print the msg, get throttled,
goto 1. This is possible if we're allowed 100+ interrupts per jiffy, so
if you adjust /proc/sys/kernel/perf_event_max_sample_rate to below that
and it doesn't trigger anymore we know the throttle works.

We've also seen hardware seize up on low period counts like that, which
is why there's a minimum of 2, 1 would guarantee an NMI storm/lockup no
matter what.

Attachment: pgp1JYoRCq7vJ.pgp
Description: PGP signature