xen: IPI interrupts not resumed early enough on suspend/resume

From: Ian Campbell
Date: Mon Oct 03 2011 - 11:10:32 EST

Hi Thomas,

Recently I've been chasing an issue where a Xen guest will fail to
resume about 1 time in 100. I eventually managed to bisect this back to
676dc3cf5bc3 "xen: Use IRQF_FORCE_RESUME".

The Xen suspend procedure (drivers/xen/manage.c:do_suspend()) is roughly
(I've omitted some uninteresting parts) as follows:
-> xen_suspend()
HYPERVISOR_suspend() /* Hypercall, returns on resume */
xen_irq_resume() /* Re-establishes evtchn<->irq bindings */

The resume process appears to be coming to a halt at the end of the
stop_machine invocation of xen_suspend(), i.e. after syscore_resume()
but before dpm_resume_noirq().

Looking at the stack traces of all VCPUs when this happens it appears
that they are all idle, which suggests we are missing an event to cause
a reschedule out of the stop_machine thread back into the suspending

One of the effects of 676dc3cf5bc3 was to move the unmasking of the
timer and IPI interrupts from xen_irq_resume() (i.e. within the
stop_machine region) to dpm_resume_noirq() (i.e. outside the
stop_machine region). Since the IPI interrupts includes the reschedule
IPI I rather suspect that is the reason for the problem. I added a hack
to unmask the reched* IPIs at xen_irq_resume() time and so far it seems
to fix things, which backs up my gut feeling.

I can see a few options for how I might go about solving this in a
non-hacky way, which approach do you think would be preferable:

* Add "IRQF_RESUME_EARLY", driven from syscore_resume, and use it
for these interrupts.
* register syscore ops for the Xen event channel subsystem to
unmask the IPIs earlier (would probably look a lot like the code
removed by 676dc3cf5bc3).
* add syscore_ops to Xen smp subsystem to unmask the specific IPIs
(which it binds at start of day) earlier.
* push dpm_(suspend|resume)_noirq down into stop machine region
* use something other than stop_machine to quiesce system and move
to cpu0 for suspend (doesn't seem sensible to reproduce that

Routing IPIs through the regular IRQ path seems a little bit unusual but
it looks like powerpc does something similar in smp_request_message_ipi
and mpic_request_ipis and that code uses the syscore approach. Does
applying that here too seem sane?

Any preference / advice?


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/