[PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker

[Max Kellermann]

We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
that serves as the CacheFiles backing store for a Ceph (fscache)
client.  The stall is always the journal commit thread spinning on
journal->j_list_lock:

  rcu: INFO: rcu_sched self-detected stall on CPU
  rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
  CPU: 106 Comm: jbd2/nvme1n1p1-
  RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
  Call Trace:
   jbd2_journal_write_metadata_buffer+0x1c0/0x310
   jbd2_journal_commit_transaction+0x5e2/0x16e0
   kjournald2+0xa1/0x220
   kthread+0xe4/0x1d0

kjournald2 has already passed the "wait for outstanding handles"
barrier and is in the metadata write-out loop; it is simply unable to
acquire j_list_lock for >21s.  The lock holder is the jbd2 checkpoint
shrinker.

Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
walks a transaction's checkpoint list under j_list_lock.  On this
workload, the lists are large and dominated by busy buffers (dirty /
under writeback / attached to the running transaction), and
journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
proportional to the whole list length, for two reasons:

1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
   the need_resched() check, so the scan does not yield even when a
   reschedule is pending.

2. The scan budget (nr_to_scan) is decremented only for buffers that
   are actually freed, so busy buffers do not consume it and the loop
   is not bounded by the shrinker's batch at all.

Both were introduced by commit b98dba273a0e ("jbd2: remove
journal_clean_one_cp_list()"), which folded
journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
dropped the per-examined-buffer budget that the shrinker previously
had.

The result is a self-reinforcing collapse: shrinker instances across
many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
journal fills, CacheFiles lookups block in ext4, fscache cookies get
stuck in LOOKING_UP, and netfs I/O times out.

Max Kellermann (2):
  jbd2: check need_resched() when skipping busy checkpoint buffers
  jbd2: bound shrinker scans by examined checkpoint buffers

 fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

-- 
2.47.3