'rcu_preempt detected stalls on CPUs/tasks...' issue of cyclictest on rt-linux

[richard clark]

Hi,
I am running a Ubuntu 20.04.5 LTS on Nvidia Jetson AGX Orin platform
with 12-cores as a guestOS, the kernel version is - 6.1.83-rt28.
Kernel cmdline is:
'root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4
console=ttyTCU0,115200 console=tty0 firmware_class.path=/etc/firmware
fbcon=map:0 net.ifnames=0'

The cyclictest command 'cyclictest -Smp99 -H 3000
--histfile=orin_idle_hyp_4h.hist -D 4h' will hang randomly during the
test, then the minicom console will show below messages:
...

[97619.450889] [CPU11-E] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[97619.450894] [CPU11-E] rcu:   1-...!: (0 ticks this GP)
idle=dc88/0/0x0 softirq=0/0 fqs=2 (false positive?)
[97619.450914] [ CPU1-E] NMI backtrace for cpu 1
[97619.451912] [CPU11-E] rcu: rcu_preempt kthread timer wakeup didn't
happen for 5251 jiffies! g6029253 f0x0 RCU_GP_WAIT_FQS(5)
->state=0x402
[97619.451916] [CPU11-E] rcu:   Possible timer handling issue on cpu=1
timer-softirq=342864
[97619.451918] [CPU11-E] rcu: rcu_preempt kthread starved for 5252
jiffies! g6029253 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
[97619.451921] [CPU11-E] rcu:   Unless rcu_preempt kthread gets
sufficient CPU time, OOM is now expected behavior.
[97619.451923] [CPU11-E] rcu: RCU grace-period kthread stack dump:
[97619.451966] [CPU11-E] rcu: Stack dump where RCU GP kthread last ran:
...
This issue doesn't show if run the Ubuntu 20.04.5 LTS with the same
kernel natively on the Orin board.

Any comments about this or what can I do to triage this issue?