[RFC PATCH 0/7] drm/vino: DisplayLink DL3 dock driver (RFC, help wanted)

[Mike Lothian]

Vino is a clean-room, in-kernel Rust DRM driver for DisplayLink DL3 USB
docks (Dell Universal Dock D6000, 17e9:6006), a native replacement for
the out-of-tree EVDI module plus the proprietary DisplayLinkManager
userspace daemon. It is built on the in-tree Rust USB, crypto and DRM/KMS
bindings, which are posted as their own prerequisite series; this series
is the driver that consumes them and touches only drivers/gpu/drm/vino/
plus the two drivers/gpu/drm/{Kconfig,Makefile} wiring lines.

Prerequisite binding series (apply in this order before this one):
  - [RFC PATCH 0/9] rust: usb: synchronous bulk/control transfers + helpers
    https://lore.kernel.org/r/20260617145946.1894-1-mike@xxxxxxxxxxxxxx
  - [RFC PATCH 0/2] rust: crypto: library AES-128 / SHA-256 / HMAC + RSA
    https://lore.kernel.org/r/20260617150143.2152-1-mike@xxxxxxxxxxxxxx
  - [RFC PATCH 0/5] rust: drm: minimal KMS bindings, EDID read, rotation, HDCP defs
    https://lore.kernel.org/r/20260617150232.2210-1-mike@xxxxxxxxxxxxxx

It is posted as an [RFC] in a deliberately INCOMPLETE state to ask for
help with one remaining blocker -- the control-plane engagement "wall"
(see below and patch 3). It was previously a single ~5000-line patch; it
has been split into one self-contained subsystem per patch so it is
reviewable and so others can pick up and fix a single layer in isolation:

  1. skeleton + USB bind + the plaintext connect handshake (proto);
  2. the clean-room HDCP 2.2 AKE/LC/SKE (crypto, rng, hdcp, ake, golden);
  3. the AES-CTR/AES-CMAC ("Dl3Cmac") control-plane seal + arm (cp);
  4. the Vino RawRl mode-2 framebuffer codec (video);
  5. the DRM/KMS device + EP08 framebuffer scanout (drm_sink);
  6. DDC/CI brightness/contrast, DPMS power and DFU device-info;
  7. KUnit self-tests for the protocol and crypto paths.

Each commit builds and the module loads on its own, so the series is
bisectable. With the prerequisite binding series applied it is
git-am-clean and checkpatch.pl --strict reports no errors (the only notes
are the expected "does MAINTAINERS need updating?" on the new files and a
few >100-column verbatim log strings).

Note on tooling: the implementation was AI-assisted (see the Assisted-by:
trailers); the reverse-engineering, the hardware testing on a real D6000,
and responsibility for the code under the DCO are the author's.

What works, all on real hardware (Dell Universal Dock D6000):
 - the plaintext connect handshake over the Rust USB bulk + control API;
 - the clean-room HDCP 2.2 AKE/LC/SKE -- H', L' and V' all verify against
   the dock, establishing the shared session key;
 - the AES-CTR + AES-CMAC control-plane seal, byte-exact against the
   reference daemon's captured wire, plus the plaintext stream-open arm;
 - registration of a real struct drm_device via an atomic KMS pipeline, so
   the dock appears as a mode-settable GEM/dumb DRM card, with a live EP08
   framebuffer-scanout hook on every page-flip.

What does NOT work -- the wall (help wanted):
 After the arm marker the driver sends the first encrypted control-plane
 frame and the dock never acknowledges it (the wsub=0x45 ack count stays
 0), so the CP cipher never engages and no pixels flow. Every
 host-observable channel has been matched to the reference daemon -- the
 bulk wire is byte-identical through the arm and the first encrypted frame,
 the AKE verifies, the seal/MAC/IV are byte-exact, the full EP0 control and
 endpoint sets match, and the arm timing is tighter than the daemon's --
 yet the dock silently drops our encrypted CP while engaging the daemon's.
 The gate appears not to be visible on the host wire. If you have knowledge
 of the DisplayLink DL3 control-plane engagement sequence, or ideas for
 isolating a whole-bus timing/ordering property a per-channel diff cannot
 see, help would be very welcome.

Not-for-merge as-is: the driver carries RE diagnostics behind pr_debug and
a small captured plaintext cap-announce skeleton (golden) that a fully
field-derived builder should eventually replace.

Mike Lothian (7):
  drm/vino: add DisplayLink DL3 dock skeleton and plaintext bring-up
  drm/vino: add the clean-room HDCP 2.2 AKE/LC/SKE
  drm/vino: add the AES-CTR/AES-CMAC control-plane seal and arm
  drm/vino: add the Vino (RawRl mode-2) framebuffer codec
  drm/vino: register a DRM/KMS device and scan out to EP08
  drm/vino: add DDC/CI brightness/contrast, DPMS power and DFU info
  drm/vino: add KUnit self-tests for the protocol and crypto paths

 drivers/gpu/drm/Kconfig          |    2 +
 drivers/gpu/drm/Makefile         |    2 +
 drivers/gpu/drm/vino/Kconfig     |   21 +
 drivers/gpu/drm/vino/Makefile    |    2 +
 drivers/gpu/drm/vino/ake.rs      |  167 +++
 drivers/gpu/drm/vino/cp.rs       |  686 ++++++++++
 drivers/gpu/drm/vino/crypto.rs   |   81 ++
 drivers/gpu/drm/vino/drm_sink.rs | 1466 +++++++++++++++++++++
 drivers/gpu/drm/vino/golden.rs   |   69 +
 drivers/gpu/drm/vino/hdcp.rs     |  167 +++
 drivers/gpu/drm/vino/proto.rs    |   73 ++
 drivers/gpu/drm/vino/rng.rs      |   12 +
 drivers/gpu/drm/vino/video.rs    |  348 +++++
 drivers/gpu/drm/vino/vino.rs     | 2053 ++++++++++++++++++++++++++++++
 14 files changed, 5149 insertions(+)
 create mode 100644 drivers/gpu/drm/vino/Kconfig
 create mode 100644 drivers/gpu/drm/vino/Makefile
 create mode 100644 drivers/gpu/drm/vino/ake.rs
 create mode 100644 drivers/gpu/drm/vino/cp.rs
 create mode 100644 drivers/gpu/drm/vino/crypto.rs
 create mode 100644 drivers/gpu/drm/vino/drm_sink.rs
 create mode 100644 drivers/gpu/drm/vino/golden.rs
 create mode 100644 drivers/gpu/drm/vino/hdcp.rs
 create mode 100644 drivers/gpu/drm/vino/proto.rs
 create mode 100644 drivers/gpu/drm/vino/rng.rs
 create mode 100644 drivers/gpu/drm/vino/video.rs
 create mode 100644 drivers/gpu/drm/vino/vino.rs

-- 
2.54.0