Re: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker

[Zhang Yi]

On 7/13/2026 6:22 PM, Max Kellermann wrote:
> We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
> that serves as the CacheFiles backing store for a Ceph (fscache)
> client.  The stall is always the journal commit thread spinning on
> journal->j_list_lock:
> 
>   rcu: INFO: rcu_sched self-detected stall on CPU
>   rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
>   CPU: 106 Comm: jbd2/nvme1n1p1-
>   RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
>   Call Trace:
>    jbd2_journal_write_metadata_buffer+0x1c0/0x310
>    jbd2_journal_commit_transaction+0x5e2/0x16e0
>    kjournald2+0xa1/0x220
>    kthread+0xe4/0x1d0

Hi, Max!

Thanks for the report!

I'm a bit curious about which kernel version and CONFIG you
encountered this issue on. IIRC, after commits [1] and [2], the
spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should
always be effective (unless the hardware does not support it, i.e.,
ARCH_NO_PREEMPT). Moreover, the number of buffer_heads in the
checkpoint list of a single transaction should not be large enough
to trigger such soft lockups. So I suspect this issue should be
unlikely to occur on the latest kernel releases.

[1] 7dadeaa6e851 ("sched: Further restrict the preemption modes")
[2] 7c70cb94d29c ("sched: Add Lazy preemption model")

Thanks,
Yi.

> 
> kjournald2 has already passed the "wait for outstanding handles"
> barrier and is in the metadata write-out loop; it is simply unable to
> acquire j_list_lock for >21s.  The lock holder is the jbd2 checkpoint
> shrinker.
> 
> Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
> jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
> walks a transaction's checkpoint list under j_list_lock.  On this
> workload, the lists are large and dominated by busy buffers (dirty /
> under writeback / attached to the running transaction), and
> journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
> proportional to the whole list length, for two reasons:
> 
> 1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
>    the need_resched() check, so the scan does not yield even when a
>    reschedule is pending.
> 
> 2. The scan budget (nr_to_scan) is decremented only for buffers that
>    are actually freed, so busy buffers do not consume it and the loop
>    is not bounded by the shrinker's batch at all.
> 
> Both were introduced by commit b98dba273a0e ("jbd2: remove
> journal_clean_one_cp_list()"), which folded
> journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
> dropped the per-examined-buffer budget that the shrinker previously
> had.
> 
> The result is a self-reinforcing collapse: shrinker instances across
> many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
> journal fills, CacheFiles lookups block in ext4, fscache cookies get
> stuck in LOOKING_UP, and netfs I/O times out.
> 
> Max Kellermann (2):
>   jbd2: check need_resched() when skipping busy checkpoint buffers
>   jbd2: bound shrinker scans by examined checkpoint buffers
> 
>  fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
>