Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Tue, Mar 10, 2026 at 08:57:36AM +0000, Kepplinger-Novakovic Martin wrote:
> Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> > There's an endianness issue here:  30313000 is the zero byte prescribed
> > by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> > followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> > 
> > Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> > in reverse order.  And so on until 20040005, which are the last four
> > bytes of the prefix in reverse order.
> > 
> > How are you generating that hexdump?  What's the CPU's endianness?
> > Is the caam RSA accelerator using a different endianness?
> 
> imx6ul is armv7, little endian byte order and the following returns 1
> which supports that:
> echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6

Please double-check whether your .config enables CONFIG_CPU_BIG_ENDIAN
or CONFIG_CPU_LITTLE_ENDIAN, just to cover all bases.

> I always print the hex dump in the following way (here "out_buf" at line
> https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247 )
> print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);

Please use 1 instead of 4 as 5th parameter of print_hex_dump().
Using 4 only makes sense if the memory location you want to dump
contains 32-bit values.  That's not the case here as the signature
is a bytestream.

I guess if you use 4, print_hex_dump() dumps the 32-bit values
in big endian order for human readability, but that's confusing
if the memory location actually contains a bytestream.

> Again, with this revert, the problem seems to be the same, only that
> the data that rsassa_pkcs1_verify() is starting to check here
> https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266
> is still "old" but now zeroes, not the input-data, thus failing with
> -EBADMSG instead of -EINVAL.

Actually the "out_buf2" that you've included in this message...

https://lore.kernel.org/all/1a65ac92579fadb4bfc76b32a3a4f1c6df022801.camel@xxxxxxxxxxxxx/

...looks like a valid verified (i.e. encrypted) signature,
the only thing that's weird is the endianness issue and
that there's a bunch of zero bytes at the beginning of
the buffer.

Please re-generate the hexdump of "out_buf" after the call
to crypto_wait_req(), once with a stock kernel and once with
8552cb04e083 reverted, and use 1 as 5th argument to
print_hex_dump().

Thanks,

Lukas