[PATCH 1/6] irq_work: Re-raise pending irq_work after system resume

[Adrian Hunter]

irq_work_queue() raises the irq_work self-interrupt only when adding
work to an empty raised_list.  Once raised_list is non-empty, further
queueing assumes the interrupt is already pending and will eventually
process the list.

That assumption can fail across system suspend.  irq_work queued after
arch_suspend_disable_irqs() and before the interrupt controller is
suspended, for example from NMI context, may leave its self-interrupt
pending in hardware.  If that pending interrupt state is lost during
suspend, hibernation restore, or kexec jump, raised_list remains
non-empty after resume but nothing will process it.

Subsequent irq_work added to raised_list will not raise another
interrupt because the list is already non-empty.  Architectures that use
an irq_work interrupt and do not process raised_list from the tick can
therefore leave irq_work pending indefinitely.  A caller waiting in
irq_work_sync() may then spin forever, for example when closing a perf
event whose pending irq_work never executes.

For suspend, hibernation, and kexec jump, only the CPU that remains
online is affected.  The other CPUs are taken offline beforehand, and
smpcfd_dying_cpu() runs irq_work_run() on each CPU before it is removed,
so checking the local raised_list is sufficient.

Add irq_work_resend(), which re-raises the irq_work self-interrupt when
raised_list is non-empty, and invoke it from syscore_resume() after
interrupt controllers have been restored.  Also invoke it from the
syscore_suspend() failure path, after the already-suspended callbacks
have been resumed, because callers do not call syscore_resume() in that
case.  This restores a lost irq_work interrupt on the CPU that remains
online for suspend-to-RAM, hibernation, and kexec jump.

If the original interrupt was not lost, the extra self-interrupt is
harmless.

Example S3 with perf record that always loses a self-IPI:

 # echo deep > /sys/power/mem_sleep
 # cat /sys/power/mem_sleep
 s2idle [deep]
 # perf record -e power:suspend_resume --no-buffering -C 0 -o /tmp/s3.data &
 [2] 2147
 # rtcwake -m mem -s 10
 rtcwake: wakeup from "mem" using /dev/rtc0
 # kill -INT %2
 [ perf record: Woken up 1 times to write data ]
 [ perf record: Captured and wrote 0.892 MB /tmp/s3.data ]

 Now perf record is stuck in irq_work_sync() which spins forever:

 # perf record -C 0 -- sleep 1 | perf report --stdio
 # Overhead  Command  Shared Object      Symbol
 # ........  .......  .................  ................................
 #
     99.98%  perf     [kernel.kallsyms]  [k] irq_work_sync
      0.02%  perf     [kernel.kallsyms]  [k] ktime_get_update_offsets_now

 Eventually resulting in hung tasks:

 INFO: task kworker/u8:1:35 blocked for more than 122 seconds.
       Not tainted 7.3.0 #2
 task:kworker/u8:1    state:D stack:0     pid:35    tgid:35    ppid:2      task_flags:0x4208060 flags:0x00080000
 Workqueue: events_unbound fsnotify_mark_destroy_workfn

Fixes: 76a33061b932 ("irq_work: Force raised irq work to run on irq work interrupt")
Signed-off-by: Adrian Hunter <adrian.hunter@xxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx
---
 drivers/base/syscore.c   | 10 ++++++++++
 include/linux/irq_work.h |  2 ++
 kernel/irq_work.c        | 12 ++++++++++++
 3 files changed, 24 insertions(+)

diff --git a/drivers/base/syscore.c b/drivers/base/syscore.c
index 483adb796654..3f645da46e77 100644
--- a/drivers/base/syscore.c
+++ b/drivers/base/syscore.c
@@ -6,6 +6,7 @@
  */
 
 #include <linux/syscore_ops.h>
+#include <linux/irq_work.h>
 #include <linux/mutex.h>
 #include <linux/module.h>
 #include <linux/suspend.h>
@@ -81,6 +82,8 @@ int syscore_suspend(void)
 		if (syscore->ops->resume)
 			syscore->ops->resume(syscore->data);
 
+	irq_work_resend();
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(syscore_suspend);
@@ -106,6 +109,13 @@ void syscore_resume(void)
 				"Interrupts enabled after %pS\n",
 				syscore->ops->resume);
 		}
+
+	/*
+	 * A pending irq_work self-interrupt may have been lost while suspended.
+	 * Resend it now that interrupt controllers have been restored.
+	 */
+	irq_work_resend();
+
 	trace_suspend_resume(TPS("syscore_resume"), 0, false);
 }
 EXPORT_SYMBOL_GPL(syscore_resume);
diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
index 7184e1273965..0040ea94a4fe 100644
--- a/include/linux/irq_work.h
+++ b/include/linux/irq_work.h
@@ -62,6 +62,7 @@ void irq_work_run(void);
 void irq_work_run_cpu(unsigned int cpu);
 bool irq_work_needs_cpu(void);
 void irq_work_single(void *arg);
+void irq_work_resend(void);
 
 void arch_irq_work_raise(void);
 
@@ -70,6 +71,7 @@ static inline bool irq_work_needs_cpu(void) { return false; }
 static inline void irq_work_run(void) { }
 static inline void irq_work_run_cpu(unsigned int cpu) { }
 static inline void irq_work_single(void *arg) { }
+static inline void irq_work_resend(void) { }
 #endif
 
 #endif /* _LINUX_IRQ_WORK_H */
diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 73eabcbdcd50..5d186ec35770 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -274,6 +274,18 @@ void irq_work_run_cpu(unsigned int cpu)
 	irq_work_run_list(per_cpu_ptr(&lazy_list, cpu));
 }
 
+/*
+ * Raise the self-interrupt again if irq_work is queued, because it may have
+ * been lost across system suspend.
+ */
+void irq_work_resend(void)
+{
+	lockdep_assert_preemption_disabled();
+
+	if (!llist_empty(this_cpu_ptr(&raised_list)))
+		arch_irq_work_raise();
+}
+
 void irq_work_tick(void)
 {
 	struct llist_head *raised = this_cpu_ptr(&raised_list);
-- 
2.53.0