Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf
From: Kepplinger-Novakovic Martin
Date: Tue Mar 10 2026 - 04:59:14 EST
Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> On Thu, Feb 26, 2026 at 11:41:56AM +0000, Kepplinger-Novakovic Martin wrote:
> > [ 2.272135] PKEY: ==>public_key_verify_signature()
> > [ 2.272165] CAAM rsa init start
> > [ 2.272180] CAAM rsa init done
> > [ 2.272191] caam_rsa_pub_key: free old key in ctx
> > [ 2.272201] caam_rsa_pub_key: write rsa_key->e
> > [ 2.272210] caam_rsa_pub_key: write rsa_key->n
> > [ 2.272220] start rsassa_pkcs1_verify
> > [ 2.272228] slen: 256
> > [ 2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
> > [ 2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000 ................
> > [ 2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000 ................
> > [ 2.272322] SRC BUF in out_buf1 CRC: 969ee858
> > [ 2.272335] start caam_rsa_enc
> > [ 2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000 ..`.@.M.........
> > [ 2.272377] key:00000010: 00000000 00000000 00000000 00000000 ................
> > [ 2.272413] edesc:00000000: 00000001 00000001 00000000 00000000 ................
> > [ 2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c ............l=S.
> > [ 2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4 ........h/......
> > [ 2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4 @.`.............
> > [ 2.272509] CAAM: calling caam_jr_enqueue
> > [ 2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000 ..`.@.M.........
> > [ 2.272546] key:00000010: 00000000 00000000 00000000 00000000 ................
> > [ 2.277444] CAAM: completion callback
> > [ 2.424765] OUT BUF in out_buf2 CRC: fd0eef11
> > [ 2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000 ................
> > [ 2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff ................
> > [ 2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000 .............010
> > [ 2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005 ...`.H.e.......
> > [ 2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30 N.Ua...z..@u0...
> > [ 2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3 4...............
>
> There's an endianness issue here: 30313000 is the zero byte prescribed
> by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> followed by the first three bytes of hash_prefix_sha256[] in reverse order.
>
> Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> in reverse order. And so on until 20040005, which are the last four
> bytes of the prefix in reverse order.
>
> How are you generating that hexdump? What's the CPU's endianness?
> Is the caam RSA accelerator using a different endianness?
imx6ul is armv7, little endian byte order and the following returns 1 which supports that:
echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6
I always print the hex dump in the following way (here "out_buf" at line
https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247 )
print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);
I don't find anything about imx6ul's CAAM internal endianness in the ref.man.
and can't say much about that.
I simply run mainline, CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n and "crypto: rsassa-pkcs1 - Copy source data for SG list"
reverted.
Again, with this revert, the problem seems to be the same, only that the data that rsassa_pkcs1_verify() is
starting to check here https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266 is still
"old" but now zeroes, not the input-data, thus failing with -EBADMSG instead of -EINVAL.
My feeling is that endianness is not the issue here. I see what you mean, kind of, but let's look at a success-case.
"out_buf" printed here: https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L264
[ 3.863485] out_buf2:00000000: ffff0100 ffffffff ffffffff ffffffff ................
[ 3.863516] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff ................
[ 3.863542] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff ................
[ 3.863567] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff ................
so out_buf[0] is 0x00, out_buf[1] is 0x01, "seeking" forward until !0xff, all succeeds in the lines below.
and sorry for my late response here. I'd be *very* happy to test any ideas!
martin