[PATCH v8 5/6] workqueue: Imply strict CPU affinity for WQ_RT

[Tvrtko Ursulin]

Imply and set strict CPU affinity for WQ_RT workqueues for even lower
worst case execution latency.

With this, a benchmark which measures the GPU submit latency against a
real-time Vulkan queue when faced with CPU saturation, highlights a
dramatic reduction in maximum observed latencies and further reduction in
the median, and 95- and 98-the percentiles:

    us      7.3.0   WQ_RT   +strict_affinity

   median     30     21           11
    min        2      3            2
    max     1038    941          108
   95pct    2384     49           28
   98pct    2922     63           51

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@xxxxxxxxxx>
Cc: Boris Brezillon <boris.brezillon@xxxxxxxxxxxxx>
Cc: Chia-I Wu <olv@xxxxxxxxxx>
Cc: Liviu Dudau <liviu.dudau@xxxxxxx>
Cc: Matthew Brost <matthew.brost@xxxxxxxxx>
Cc: Steven Price <steven.price@xxxxxxx>
Cc: Tejun Heo <tj@xxxxxxxxxx>
---
 Documentation/core-api/workqueue.rst |  5 +++--
 kernel/workqueue.c                   | 13 ++++++++++---
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/Documentation/core-api/workqueue.rst b/Documentation/core-api/workqueue.rst
index 54697907f2b0..d699c3832b19 100644
--- a/Documentation/core-api/workqueue.rst
+++ b/Documentation/core-api/workqueue.rst
@@ -226,8 +226,9 @@ resources, scheduled and executed.
   implements concurrency management among its workers.
 
 ``WQ_RT``
-  Real-time priority workqueues must be created as unbound. Their worker threads
-  use the FIFO scheduling policy with the lowest applicable priority.
+  Real-time priority workqueues must be created as unbound and will be
+  configured with the strict CPU affinity set. Their worker threads use the FIFO
+  scheduling policy with the lowest applicable priority.
 
   To be used sparingly for use cases such as the real-time GPU rendering
   contexts accessible to privileged clients.
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 637240667cf7..26a26831bee7 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -5723,8 +5723,12 @@ static void
 sanitize_immutable_workqueue_attrs(struct workqueue_struct *wq,
 				   struct workqueue_attrs *attrs)
 {
-	if (wq->flags & WQ_RT)
+	if (wq->flags & WQ_RT) {
 		attrs->prio = RT_PRIORITY;
+		/* Strict CPU affinity for low latency execution. */
+		attrs->affn_scope = WQ_AFFN_CPU;
+		attrs->affn_strict = true;
+	}
 
 	if (wq->flags & __WQ_ORDERED)
 		attrs->ordered = true;
@@ -7847,8 +7851,11 @@ static umode_t wq_sysfs_unbound_group_visible(struct kobject *kobj,
 	if (!(wq->flags & WQ_UNBOUND))
 		return 0;
 
-	/* Do not allow priority changes for RT workers. */
-	if ((wq->flags & WQ_RT) && (attr == &dev_attr_nice.attr))
+	/* Do not allow priority and affinity changes for RT workers. */
+	if ((wq->flags & WQ_RT) &&
+	     (attr == &dev_attr_nice.attr ||
+	      attr == &dev_attr_affinity_scope.attr ||
+	      attr == &dev_attr_affinity_strict.attr))
 		return 0444;
 
 	return attr->mode;
-- 
2.56.0