[RESEND PATCH v5 03/12] smp: Refactor remote CPU selection in smp_call_function_any()

[Chuyi Zhou]

Currently, smp_call_function_any() disables preemption across the entire
process of picking a target CPU, enqueueing the IPI, and synchronously
waiting for the remote CPU. Since smp_call_function_single() has already
been optimized to re-enable preemption before the synchronous
csd_lock_wait(), callers of smp_call_function_any() should also benefit
from this optimization to reduce the preemption-disabled critical section.

A naive approach would be to simply remove get_cpu() and put_cpu() from
smp_call_function_any(), leaving the preemption disablement entirely to
smp_call_function_single(). However, doing so opens a dangerous
preemption window between picking the remote CPU (e.g., via
sched_numa_find_nth_cpu()) and dispatching the IPI inside
smp_call_function_single(). If the selected remote CPU is fully offlined
during this window, smp_call_function_single() will fail its
cpu_online() check and return -ENXIO directly to the caller, violating
the guarantee to execute on *any* online CPU in the mask.

To safely enable this optimization, this patch refactors the logic of
smp_call_function_any() and smp_call_function_single(). By moving the
random remote CPU selection into a common __smp_call_function_single(),
and keep the entire selection and IPI dispatch process within a single
preemption-disabled region.

Signed-off-by: Chuyi Zhou <zhouchuyi@xxxxxxxxxxxxx>
---
 kernel/smp.c | 48 ++++++++++++++++++++++++++----------------------
 1 file changed, 26 insertions(+), 22 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 292eefadddbc..9e9dab3b0d51 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -641,17 +641,8 @@ void flush_smp_call_function_queue(void)
 	local_irq_restore(flags);
 }
 
-/**
- * smp_call_function_single - Run a function on a specific CPU
- * @cpu: Specific target CPU for this function.
- * @func: The function to run. This must be fast and non-blocking.
- * @info: An arbitrary pointer to pass to the function.
- * @wait: If true, wait until function has completed on other CPUs.
- *
- * Returns: %0 on success, else a negative status code.
- */
-int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
-			     int wait)
+static int __smp_call_function_single(int cpu, smp_call_func_t func,
+			void *info, const struct cpumask *mask, int wait)
 {
 	call_single_data_t *csd;
 	call_single_data_t csd_stack = {
@@ -668,6 +659,14 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 	 */
 	this_cpu = get_cpu();
 
+	if (mask) {
+		/* Try for same CPU (cheapest) */
+		if (!cpumask_test_cpu(this_cpu, mask))
+			cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(this_cpu));
+		else
+			cpu = this_cpu;
+	}
+
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -712,6 +711,21 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 
 	return err;
 }
+
+/**
+ * smp_call_function_single - Run a function on a specific CPU
+ * @cpu: Specific target CPU for this function.
+ * @func: The function to run. This must be fast and non-blocking.
+ * @info: An arbitrary pointer to pass to the function.
+ * @wait: If true, wait until function has completed on other CPUs.
+ *
+ * Returns: %0 on success, else a negative status code.
+ */
+int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
+			     int wait)
+{
+	return __smp_call_function_single(cpu, func, info, NULL, wait);
+}
 EXPORT_SYMBOL(smp_call_function_single);
 
 /**
@@ -776,17 +790,7 @@ EXPORT_SYMBOL_GPL(smp_call_function_single_async);
 int smp_call_function_any(const struct cpumask *mask,
 			  smp_call_func_t func, void *info, int wait)
 {
-	unsigned int cpu;
-	int ret;
-
-	/* Try for same CPU (cheapest) */
-	cpu = get_cpu();
-	if (!cpumask_test_cpu(cpu, mask))
-		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu));
-
-	ret = smp_call_function_single(cpu, func, info, wait);
-	put_cpu();
-	return ret;
+	return __smp_call_function_single(-1, func, info, mask, wait);
 }
 EXPORT_SYMBOL_GPL(smp_call_function_any);
 
-- 
2.20.1