[PATCH v9 02/14] smp: Enable preemption early in smp_call_function_single()

[Chuyi Zhou]

smp_call_function_single() disables preemption while it validates the
target CPU, prepares the call single data, queues the callback and sends
the IPI.

For the !wait case, preemption protects the per-CPU csd_data from
concurrent modification by another task on the same CPU. For the wait
case, the CSD is stack allocated and no other task can reuse it. CPU
pinning is still required until the callback has been queued and the IPI
has been sent, to ensure that the target CPU cannot be offlined after the
online check but before dispatch.

After generic_exec_single() has queued the callback, the synchronous
csd_lock_wait() invocation at the end of the execution does not require
the caller to remain pinned to the current CPU.

Enable preemption before csd_lock_wait() to shorten the
preemption-disabled section.

Signed-off-by: Chuyi Zhou <zhouchuyi@xxxxxxxxxxxxx>
Tested-by: Paul E. McKenney <paulmck@xxxxxxxxxx>
Reviewed-by: Muchun Song <muchun.song@xxxxxxxxx>
Reviewed-by: Steven Rostedt (Google) <rostedt@xxxxxxxxxxx>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
---
 kernel/smp.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index b58975480e11..292eefadddbc 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -700,11 +700,16 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 
 	err = generic_exec_single(cpu, csd);
 
+	/*
+	 * @csd is stack-allocated when @wait is true. No concurrent access
+	 * except from the IPI completion path, so we can re-enable preemption
+	 * early to reduce latency.
+	 */
+	put_cpu();
+
 	if (wait)
 		csd_lock_wait(csd);
 
-	put_cpu();
-
 	return err;
 }
 EXPORT_SYMBOL(smp_call_function_single);
-- 
2.20.1