kernel warning percpu ref in obj_cgroup_release (was: Re: linux-next: Tree for Mar 29)

[Christian Borntraeger]

On 29.03.21 11:52, Stephen Rothwell wrote:
> Hi all,
>
> News: there will be no linux-next release on Friday or the following
> Monday.
>
> Changes since 20210326:
>
> The arm64 tree gained a conflict against Linus' tree.
>
> I applied a supplied patch for clang brakage in the kbuild tree.
>
> The net-next tree gained a conflict against the bpf tree.
>
> The drm tree gained a conflict against Linus' tree.
>
> The staging tree gained a conflict against the scmi tree and a semantic
> conflict against the spi tree.
>
> The rust tree gained a conflict against the kbuild tree.
>
> The akpm-current tree lost its its build failure and gained a conflict
> against the gpio-brgl tree.
>
> Non-merge commits (relative to Linus' tree): 7289
>   7213 files changed, 432170 insertions(+), 147471 deletions(-)

This next (328 is fine) triggers several bugs during our KVM CI run:

[ 1506.494716] ------------[ cut here ]------------
[ 1506.494730] percpu ref (obj_cgroup_release) <= 0 (-1) after switching to atomic
[ 1506.494766] WARNING: CPU: 6 PID: 0 at lib/percpu-refcount.c:196 percpu_ref_switch_to_atomic_rcu+0x1ea/0x1f8
[ 1506.494774] Modules linked in: kvm vhost_vsock vmw_vsock_virtio_transport_common vsock vhost vhost_iotlb xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT xt_tcpudp nft_compat nf_nat_tftp nft_objref nf_conntrack_tftp nft_counter bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct dm_service_time nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink zfcp scsi_transport_fc rpcrdma sunrpc dm_multipath rdma_ucm scsi_dh_rdac scsi_dh_emc rdma_cm scsi_dh_alua iw_cm ib_cm mlx5_ib ib_uverbs dm_mod ib_core s390_trng vfio_ccw vfio_mdev mdev vfio_iommu_type1 zcrypt_cex4 vfio eadm_sch sch_fq_codel configfs ip_tables x_tables ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 mlx5_core sha512_s390 sha256_s390 sha1_s390 sha_common nvme nvme_core pkey zcrypt rng_core autofs4 [last unloaded: vfio_ap]
[ 1506.494832] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 5.12.0-20210330.rc4.git0.9d49ed9ca93b.300.fc33.s390x+next #1
[ 1506.494834] Hardware name: IBM 8561 T01 703 (LPAR)
[ 1506.494836] Krnl PSW : 0704c00180000000 00000002d71dd21e (percpu_ref_switch_to_atomic_rcu+0x1ee/0x1f8)
[ 1506.494840]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[ 1506.494842] Krnl GPRS: c0000000fffeffff 00000002f7256818 0000000000000043 00000000fffeffff
[ 1506.494844]            00000000ffffffea 0000038000000001 0000000000000000 000003800000017c
[ 1506.494846]            00000002d7924988 0000000227eb97a0 000003ff5413c7e0 7fffffffffffffff
[ 1506.494848]            0000000080360000 00000002f726b570 00000002d71dd21a 00000380000bba28
[ 1506.494856] Krnl Code: 00000002d71dd20e: e3309fe8ff04        lg      %r3,-24(%r9)
                          00000002d71dd214: c0e5001eb556        brasl   %r14,00000002d75b3cc0
                         #00000002d71dd21a: af000000            mc      0,0
                         >00000002d71dd21e: a7f4ffcc            brc     15,00000002d71dd1b6
                          00000002d71dd222: 0707                bcr     0,%r7
                          00000002d71dd224: 0707                bcr     0,%r7
                          00000002d71dd226: 0707                bcr     0,%r7
                          00000002d71dd228: eb6ff0480024        stmg    %r6,%r15,72(%r15)
[ 1506.494928] Call Trace:
[ 1506.494933]  [<00000002d71dd21e>] percpu_ref_switch_to_atomic_rcu+0x1ee/0x1f8
[ 1506.494940] ([<00000002d71dd21a>] percpu_ref_switch_to_atomic_rcu+0x1ea/0x1f8)
[ 1506.494942]  [<00000002d6b8a6c6>] rcu_do_batch+0x146/0x608
[ 1506.494946]  [<00000002d6b8ec04>] rcu_core+0x124/0x1d0
[ 1506.494948]  [<00000002d75d0222>] __do_softirq+0x13a/0x3c8
[ 1506.494952]  [<00000002d6b05306>] irq_exit+0xce/0xf8
[ 1506.494955]  [<00000002d75c1eb4>] do_ext_irq+0xdc/0x170
[ 1506.494957]  [<00000002d75cdea4>] ext_int_handler+0xc4/0xf4
[ 1506.494959]  [<0000000000000000>] 0x0
[ 1506.494963]  [<00000002d75cd9c2>] default_idle_call+0x42/0x110
[ 1506.494965]  [<00000002d6b411a0>] do_idle+0xd8/0x168
[ 1506.494968]  [<00000002d6b413ee>] cpu_startup_entry+0x36/0x40
[ 1506.494971]  [<00000002d6ac730a>] smp_start_secondary+0x82/0x88
[ 1506.494974] Last Breaking-Event-Address:
[ 1506.494975]  [<00000002d6b71898>] vprintk_emit+0xa8/0x110
[ 1506.494978] Kernel panic - not syncing: panic_on_warn set ...



I will try to bisect this, but if anyone has an idea. CC some candidates.