[PATCH 04/40] nitro_enclaves: Document multi-NUMA CPU pools

[Alexander Graf]

An admin placing enclaves or sandboxes across NUMA nodes needs to know
how the CPU pool, automatic CPU selection and the memory-locality check
interact.

Document those rules in ne_overview.rst, together with the per-node
sysfs directories, and point the cpu_pool ABI file at that section.

Signed-off-by: Alexander Graf <graf@xxxxxxxxxx>
---
 .../sysfs-devices-virtual-misc-nitro_enclaves |  3 ++-
 Documentation/virt/ne_overview.rst            | 27 ++++++++++++++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
index 596054b75e0cf..c549364353186 100644
--- a/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
+++ b/Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves
@@ -39,7 +39,8 @@ Description:
 		avail does not list.
 
 		A read takes the pool lock, so it can wait for a write to
-		ne_cpus to finish.
+		ne_cpus to finish. For how an enclave is given cores from a
+		particular node, see Documentation/virt/ne_overview.rst.
 
 What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/total
 What:		/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/nodeX/used
diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index aecd3704acaf5..223ee7fdec51b 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -48,7 +48,7 @@ be aligned 2 MiB / 1 GiB physically contiguous memory regions (or multiple of
 this size e.g. 8 MiB). The memory can be allocated e.g. by using hugetlbfs from
 user space [2][3][7]. The memory size for an enclave needs to be at least
 64 MiB. The enclave memory and CPUs need to be from the same NUMA node,
-unless the CPU pool spans several of them.
+unless the CPU pool spans several of them; see `Multi-NUMA CPU pools`_.
 
 An enclave runs on dedicated cores. CPU 0 and its CPU siblings need to remain
 available for the primary VM. A CPU pool has to be set for NE purposes by an
@@ -99,3 +99,28 @@ enclave process can exit.
 [6] https://www.kernel.org/doc/html/latest/x86/boot.html
 [7] https://www.kernel.org/doc/html/latest/arm64/hugetlbpage.html
 [8] https://www.kernel.org/doc/html/latest/arm64/booting.html
+
+Multi-NUMA CPU pools
+--------------------
+
+The CPU pool may span multiple NUMA nodes. NE_ADD_VCPU with vcpu_id == 0
+takes the first free core in the pool whatever node it sits on, so an
+enclave built that way can end up holding cores on several nodes.
+NE_ADD_ANY_VCPU names the node instead: the kernel takes a free core on
+that node, and fails rather than crossing to another node once that node
+has none left. User space builds an enclave across nodes by asking for
+each node in turn, and sets numa_node to NE_NUMA_ANY when it does not
+care. Both start from a free thread of a core the enclave already holds,
+and only take a new core once there is none.
+
+The node says where cores come from. An enclave built out of a pool that
+spans nodes has no node of its own, so the rule that its memory be on
+that node does not apply to it and its memory can come from any node.
+
+The pool occupancy is readable under
+/sys/devices/virtual/misc/nitro_enclaves/cpu_pool/, and per node under
+its ``node0``, ``node1``, ... subdirectories, one for each node that was
+online when the device was probed. A node the pool holds nothing on
+reads empty rather than being absent. The driver offlines pool threads,
+so ``nodeN/cpulist`` under /sys/devices/system/node stops listing them
+and answers nothing about the pool.
-- 
2.47.1