[RFC PATCH] tee: optee: allow forcing the reserved shared memory pool

[christian . koever-draxl]

From: Christian Stefan Kövér-Draxl <ckoever@xxxxxxxxxx>

The X98Q's vendor BL32 runs OP-TEE 3.8 and reports sec_caps 0x5, that is
HAVE_RESERVED_SHM | DYNAMIC_SHM, with neither UNREGISTERED_SHM nor
RPC_ARG.

For that combination optee_probe() picks the dynamic pool and sets
OPTEE_SHM_ARG_ALLOC_PRIV, so every message argument is an unregistered
private page passed to secure world by physical address via
OPTEE_SMC_CALL_WITH_ARG. This OP-TEE will not accept that: every call
comes back OPTEE_SMC_RETURN_EBADADDR (4), which optee_open_session()
converts into TEEC_ERROR_COMMUNICATION / TEEC_ORIGIN_COMMS. The result
is that no session can be opened at all - not to the Amlogic firmware
loading PTA the video decoder needs, and not to any built-in PTA either.

Registering memory works fine, and HAVE_RESERVED_SHM is set, so the
existing static pool path is usable. Add optee.no_dyn_shm to select it.
With the reserved pool the device enumeration PTA opens cleanly and the
system PTA answers with its own ACCESS_DENIED, so the ABI is healthy.

The driver arguably should not use unregistered arguments unless secure
world advertised UNREGISTERED_SHM; a parameter keeps the change local
until that is confirmed upstream.

Signed-off-by: Christian Stefan Kövér-Draxl <ckoever@xxxxxxxxxx>
---
Tool use (Documentation/process/generated-content.rst):
This work was done with the help of an AI coding assistant, Claude
(Anthropic), used through Claude Code over many sessions. It did a large
part of the hands-on work: it read the vendor kernel and the firmware
interfaces, probed registers on the board, drafted the code and the
changelogs, and ran the builds, checkpatch, dt_binding_check and the
tests on the hardware. I directed the work, reviewed every change and
the final patches, and take responsibility for them. There is no single
prompt to quote; the sessions were of the form "find out why X fails on
this board, fix it, test it on the box".

Tested on an Amediatech X98Q TV box (Amlogic S905W2, S4 family, 2 GiB),
vendor U-Boot and BL31/OP-TEE, Linux 7.3 plus this work.

 drivers/tee/optee/smc_abi.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
index 51624443359b..1824aead78a6 100644
--- a/drivers/tee/optee/smc_abi.c
+++ b/drivers/tee/optee/smc_abi.c
@@ -34,6 +34,11 @@
 #define CREATE_TRACE_POINTS
 #include "optee_trace.h"
 
+static bool optee_no_dyn_shm;
+module_param_named(no_dyn_shm, optee_no_dyn_shm, bool, 0444);
+MODULE_PARM_DESC(no_dyn_shm,
+		 "Ignore the secure world's dynamic shared memory capability and use the reserved pool instead");
+
 /*
  * This file implement the SMC ABI used when communicating with secure world
  * OP-TEE OS via raw SMCs.
@@ -1770,7 +1775,7 @@ static int optee_probe(struct platform_device *pdev)
 	/*
 	 * Try to use dynamic shared memory if possible
 	 */
-	if (sec_caps & OPTEE_SMC_SEC_CAP_DYNAMIC_SHM) {
+	if ((sec_caps & OPTEE_SMC_SEC_CAP_DYNAMIC_SHM) && !optee_no_dyn_shm) {
 		/*
 		 * If we have OPTEE_SMC_SEC_CAP_RPC_ARG we can ask
 		 * optee_get_msg_arg() to pre-register (by having

base-commit: 7be4bc52fd9e0e55dc9a2e0232b89427d2c4b614
-- 
2.56.0