Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

From: Kepplinger-Novakovic Martin

Date: Thu Feb 26 2026 - 06:42:15 EST

Am Donnerstag, dem 26.02.2026 um 08:17 +0100 schrieb Lukas Wunner:
> On Wed, Feb 25, 2026 at 08:47:07AM +0000, Kepplinger-Novakovic Martin wrote:
> > Am Mittwoch, dem 25.02.2026 um 09:13 +0100 schrieb Lukas Wunner:
> > > On Wed, Feb 25, 2026 at 08:02:08AM +0000, Kepplinger-Novakovic Martin wrote:
> > > > ok I can confirm: "git checkout 2f1f34c1bf7b^" indeed is ok and
> > > > 2f1f34c1bf7b is bad.
> > > >
> > > > It's not the same behaviour I described (from v6.18/v6.19. that could be
> > > > a combination of bugs) because on 2f1f34c1bf7b regdb cert verify succeeds,
> > > > only dm-verity fails
> > >
> > > Hm, I assume CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n magically
> > > makes the issue go away?
> >
> > correct. where I see that specific issue (on 2f1f34c1bf7b and v6.7)
> > "caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error.
> > Invalid length or parity, or certain other problems."
> > it then goes away.
> >
> > on v6.18 CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n doesn't seem to help
> > and I see the bugreport's behaviour.
>
> I note that for the RSA verification, since 8552cb04e083 the same buffer
> in memory is used for source and destination of RSA encrypt operation
> invoked by crypto/rsassa-pkcs1.c.
>
> That's fine for the RSA software implementation in crypto/rsa.c but
> I could very well imagine it causes problems with an RSA accelerator,
> particularly because rsa_edesc_alloc() in drivers/crypto/caam/caampkc.c
> now maps the same buffer with DMA_TO_DEVICE and then DMA_FROM_DEVICE.
>
> On v6.19, 8552cb04e083 seems to revert cleanly. Could you try that
> and see if it helps? Be sure to set CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n
> and CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API=y so that we focus on the RSA
> issue for now. We can look at the ahash one afterwards.

hi Lukas, I'm happy you have a look here but that does not seem to be it:

out_buf after caam (out_buf2 is what I print) still holds old data, it now is zeroes (not exactly sure why though) and thus the final check fails one
byte later ("leading zero" is now part of the old data), see one typical case:

[ 2.272135] PKEY: ==>public_key_verify_signature()
[ 2.272165] CAAM rsa init start
[ 2.272180] CAAM rsa init done
[ 2.272191] caam_rsa_pub_key: free old key in ctx
[ 2.272201] caam_rsa_pub_key: write rsa_key->e
[ 2.272210] caam_rsa_pub_key: write rsa_key->n
[ 2.272220] start rsassa_pkcs1_verify
[ 2.272228] slen: 256
[ 2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
[ 2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000 ................
[ 2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000 ................
[ 2.272322] SRC BUF in out_buf1 CRC: 969ee858
[ 2.272335] start caam_rsa_enc
[ 2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000 ..`.@.M.........
[ 2.272377] key:00000010: 00000000 00000000 00000000 00000000 ................
[ 2.272413] edesc:00000000: 00000001 00000001 00000000 00000000 ................
[ 2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c ............l=S.
[ 2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4 ........h/......
[ 2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4 @.`.............
[ 2.272509] CAAM: calling caam_jr_enqueue
[ 2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000 ..`.@.M.........
[ 2.272546] key:00000010: 00000000 00000000 00000000 00000000 ................
[ 2.277444] CAAM: completion callback
[ 2.424765] OUT BUF in out_buf2 CRC: fd0eef11
[ 2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000 ................
[ 2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff ................
[ 2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff ................
[ 2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff ................
[ 2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff ................
[ 2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff ................
[ 2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000 .............010
[ 2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005 ...`.H.e.......
[ 2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30 N.Ua...z..@u0...
[ 2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3 4...............
[ 2.425196] digest (in):00000000: 6155a84e 7aa089cb 7540e613 f28b9a30 N.Ua...z..@u0...
[ 2.425220] digest (in):00000010: 1e98ec34 cecb0e0f 9ee8951a ad8baec3 4...............
[ 2.425239] PKEY: crypto_sig_verify error: -74
[ 2.425262] PKEY: <==public_key_verify_signature() = -74

so long,

martin