Re: [RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576

[Nicolas Dufresne]

Hi,

Le mercredi 22 juillet 2026 à 19:34 +1200, Jiaxing Hu a écrit :
> This is an RFC for a from-scratch mainline V4L2 driver for the H.264
> hardware video encoder (VEPU510) on the Rockchip RK3576.  It is a
> stateful mem2mem encoder (NV12 in, H.264 Annex-B out) modelled on the
> verisilicon/hantro driver, not on the downstream MPP-service model.
> 
> I'm posting it as an RFC because intra frames work but inter frames do
> not, and I'd like a second pair of eyes on the inter-frame problem
> before this is worth a real submission.

This is really nice to see interest in enabling encoder for this chips set. I'm
adding Detlev in CC here, as he's actively working on RK3588 encoder, which is
quite a similar chip. But there is quite a twist his approach which I'll explain
shortly.

What I believe I've seen walking through your RFC is an encoder based on V4L2
Stateful encoder interface:

https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-encoder.html

While this isn't invalid, specially for encoders, for this type of hardware, the
community agreed direction was to introduce V4L2 Stateless Encoder
specification, using the media request and compound controls to maintain a lower
level interface for this HW. See Paul's proposal for IMX8MP Hantro VC8000E chips
(H.264 only for now):

https://lore.kernel.org/all/20260522101653.2565125-1-paulk@xxxxxxxxxxx/

The downside is that we need a new spec document for this class of driver to
reach upstream (not part of Paul's series). Plus, for each codec, appropriate
encode parameter controls needs to be designed, and they must be proven to work
across multiple hardware (generally at least 2). I still think there is a need
for V4L2 drivers like this. Being high level, they offer a great interface to
protect against HW actions that would be harmful to the Linux operating system.
Specially when you don't have an IOMMU like the IMX8M Plus. But at the same
time, V4L2 memory model often clash with modern graphic stack, making the
integration quite painful.

Here's the twist to Detlev (and myself) approach. In order to make the kernel
module even thinner, and to avoid having to specify new V4L2 interfaces, we
decided to look into making drivers similar to how we do GPU drivers today. This
is possible today thanks to the Vulkan Video standard. So Detlev driver, which
finally got P-Frames support few days ago (he'll explain why this didn't work
initially for him), is split in two part:

- The kernel module, which expose hardware specific interface (he decided to
implement it in Rust)
- The userspace driver, which he's implementing in Mesa project

What I really like of this approach is that the kernel driver is much more
stable, as it expose the HW at a lower level in a codec agnostic way. For the
final application, the interface is the same regardless if its a GPU attached
codec or a standalone codec. Its even the same interface if you are on Windows,
except for the low level zero-copy interop. The rest of the development can
happen in userspace, which is a lot faster to develop and easier to debug.

One thing we notice, is that for performance reason, keeping rate control as
close to the IRQ as possible is important. Its impossible to batch any work if
you have to constantly wait for the previous work to be done in order to compute
the next set of QP values. For that reason, we plan to eventually share these
in-kernel implementation with V4L2. In our imagination, eBPF could also be a
usable option.

I will leave to Detlev to share some early code once he's ready, but considering
you both are working on the same family of encoder, it would certainly be a lot
nicer if both endup with the same type of drivers. Offering two interface for
the same hardware would impose a lot of extra effort that I would rather avoid.

regards,
Nicolas

> 
> What works
> ----------
> 
> Intra-only (I-frame) encoding is confirmed on real hardware (Radxa
> ROCK 4D).  With GOP size 1 the driver produces a valid H.264 stream the
> reference decoder accepts.  This already exercises the full path: V4L2
> m2m, the register programming, the software SPS/PPS prepend, and the
> hardware slice output.
> 
> There is no upstream userspace involved (an encoder needs no request
> API); I drive it with a small ioctl test program that feeds NV12 frames
> and writes the CAPTURE buffers to a .h264 file.
> 
> The open problem: inter (P-frame) frames hang
> ---------------------------------------------
> 
> Every P-frame stalls the encoder's own hardware watchdog (INT_STA bit 8,
> ~20 ms after the kick) and produces essentially no bitstream.  I have
> spent a lot of time narrowing this; it is sharply localized but I cannot
> close it:
> 
>   - The P-frame register writes match a real register-write trace of the
>     vendor stack encoding the same content, byte for byte (I traced the
>     vendor kernel's writes and diffed against this driver's).
> 
>   - The reconstruction the previous frame writes is *valid*: dumping the
>     recon buffer after the I-frame shows correct reconstructed pixels.
> 
>   - If the P-frame reads its own (older, settled) recon slot instead of
>     the immediately-preceding frame's, it completes.  Reading the
>     immediately-preceding frame's *fresh* reconstruction as the
>     reference is what hangs.
> 
>   - It is not FBC: storing the reconstruction uncompressed
>     (enc_pic.rec_fbc_dis = 1) still hangs.
> 
>   - The first frame of a session always works; the first P-frame hangs;
>     after a couple of failures + core resets, later frames sometimes
>     start completing.  It behaves like a warm-up / settling problem on
>     the reference-read path, not a wrong register value.
> 
> So the encoder programs identically to the vendor and reads a valid
> reference, but stalls fetching the previous frame's reconstruction as
> the inter reference on the first inter frame of a session.  My best
> guess is that the vendor does something between consecutive frame
> submissions -- a completion/drain wait, a cache/coherency step, or a
> per-frame re-arm -- that I am missing, but I have not found it.  If
> anyone recognizes this on VEPU5xx, or knows what the reference-read path
> needs between frames, a pointer would be very welcome.
> 
> The shape of this -- the first operation of a power session works, the
> next stalls, and a reset/warm-up sometimes helps -- is the same one I
> ran into bringing up this SoC's NPU (accel/rocket RKNN), where the
> second/chained submit in a session would not fire [1].  I can't claim
> they share a root cause, but if this is a known RK3576-wide submit /
> re-arm quirk rather than an encoder-specific bug, that would be good to
> know.
> 
> [1] https://lore.kernel.org/all/20260718031146.3368811-1-gahing@xxxxxxxxxxxxx/
> 
> Notes for review
> ----------------
> 
>   - The PARAM/SQI register classes are programmed from mpp's constant
>     "default tuning" tables (not derived per-frame), as noted in the
>     code.  They are required: leaving them unwritten stalls even the
>     I-frame.
> 
>   - Rate control is fixed-QP only for now (the bitrate control is
>     advisory).
> 
>   - H.264 baseline/main, single slice, 4:2:0 only.
> 
>   - Tested at 176x144; other resolutions are not yet validated.
> 
> Signed-off-by: Jiaxing Hu <gahing@xxxxxxxxxxxxx>
> 
> Jiaxing Hu (3):
>   dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder
>   media: rockchip: add VEPU510 H.264 encoder driver for RK3576
>   arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes
> 
>  .../bindings/media/rockchip,rk3576-vepu.yaml       |   94 ++
>  arch/arm64/boot/dts/rockchip/rk3576.dtsi           |   50 +
>  drivers/media/platform/rockchip/Kconfig            |    1 +
>  drivers/media/platform/rockchip/Makefile           |    1 +
>  drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 +
>  drivers/media/platform/rockchip/rkvenc/Makefile    |    6 +
>  .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095
> ++++++++++++++++++++
>  .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++
>  drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++
>  drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++
>  10 files changed, 3294 insertions(+)
> --
> 2.43.0
>

Attachment: signature.asc
Description: This is a digitally signed message part