[RFC PATCH v2 0/3] rust: crypto: library AES-128 / SHA-256 / HMAC + RSA

[Mike Lothian]

This is v2 of the crypto bindings, rebased onto current drm-next --
no functional change from the version prepared but never sent after
v1's review (see "Changes since v1" below for what that review fixed).
It adds a small, reusable kernel::crypto module so in-kernel Rust code
can hash, MAC, encrypt a single AES block, and do RSA public-key
encryption:

  1/3  sha256(), hmac_sha256(), Aes128 (key-prepared-once block cipher)
  2/3  aes_cmac() over the in-tree AES-CMAC library
  3/3  rsa_pubkey_encrypt() over a new lib/crypto RSA primitive

Patch 1 binds the library crypto (lib/crypto) functions directly
(SHA-256 / HMAC-SHA256) and uses one rust_helper_ shim for aes_encrypt()
(its transparent union is unbindable). It runs synchronously in the
calling context with no allocation and is the independently-mergeable,
self-contained contribution.

Patch 2 adds crypto::aes_cmac() over the in-tree AES-CMAC library
(<crypto/aes-cbc-macs.h>) -- the one mode of operation the consumer needs
that lib/crypto already ships -- rather than building CMAC out of bare
AES. The 128-bit key is prepared once and wiped after use.

Patch 3 adds an RSA public-key primitive. Rather than bind crypto_akcipher
(which Eric flagged as a very bad API not to grow), it adds a small
lib/crypto entry point -- lib/crypto/rsa.c, rsa_pubkey_encrypt(), the bare
RSAEP primitive c = m^e mod n [RFC8017 sec 5.1.1] over the MPI library --
and binds that directly: no akcipher tfm, DER key encoding, scatterlists or
async completion. The caller applies its own padding (RSAES-OAEP, ...). It
is gated by a new bool CONFIG_CRYPTO_LIB_RSA (selects MPILIB); because the
Rust wrapper is exported from the built-in kernel crate the symbol must be
in vmlinux, so the option is a bool and the wrapper is
#[cfg(CONFIG_CRYPTO_LIB_RSA)]-gated -- a kernel that does not configure it
gains no dependency. A from-scratch constant-time/allocation-free lib/crypto
RSA (Eric's longer-term direction) is left as future work.

All three were factored out of an in-kernel Rust DisplayLink DL3 dock
driver (which needs SHA/HMAC/AES-CMAC for HDCP 2.2 and RSA for the AKE),
posted alongside as drm/vino ("[RFC PATCH v2 00/6] drm/vino: DisplayLink
DL3 dock driver"); the intent is to land both upstream. The module is
generic. Compile-tested in-tree against current drm-next. Source:
  https://github.com/FireBurn/vino-scripts
  https://github.com/FireBurn/linux/tree/vino
  https://gitlab.freedesktop.org/FireBurn/linux/-/tree/vino

Changes since v1 (Eric Biggers):
 - Drop the bare single-block ECB helper that re-expanded the key per block;
   Aes128 now prepares the key schedule once and reuses it for a keystream.
 - Add aes_cmac() over the in-tree AES-CMAC library (<crypto/aes-cbc-macs.h>)
   instead of building CMAC out of bare AES (now patch 2/3); the vino driver
   drops its hand-rolled CMAC for it.
 - RSA no longer binds crypto_akcipher. v1 exposed an Akcipher transform;
   following Eric's guidance to drop that API, v2 adds a small lib/crypto RSA
   primitive instead (lib/crypto/rsa.c, now patch 3/3) and binds it directly.
   A full constant-time lib/crypto RSA is left as future work.
 - Rebased onto current drm-next; no other functional change.

Mike Lothian (3):
  rust: crypto: add library AES-128 / SHA-256 / HMAC-SHA256 bindings
  rust: crypto: use the in-tree AES-CMAC library
  rust: crypto: add an RSA public-key primitive in lib/crypto

 include/crypto/rsa.h            |  15 +++
 lib/crypto/Kconfig              |   9 ++
 lib/crypto/Makefile             |   3 +
 lib/crypto/rsa.c                | 102 +++++++++++++++++++
 rust/bindings/bindings_helper.h |   4 +
 rust/helpers/crypto.c           |  37 +++++++
 rust/helpers/helpers.c          |   1 +
 rust/kernel/crypto.rs           | 169 ++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs              |   1 +
 9 files changed, 341 insertions(+)
 create mode 100644 include/crypto/rsa.h
 create mode 100644 lib/crypto/rsa.c
 create mode 100644 rust/helpers/crypto.c
 create mode 100644 rust/kernel/crypto.rs

--
2.55.0