7.2-rc1 regression Folio lock leak in writepage_delalloc()

[Christian Borntraeger]

We have seen random hangs in our daily CI run where qemu/KVM
processes  deadlocks guests with file-backed RAM on btrfs (large data folios)

With the help of claude I think we found the/one problem on an s390
KVM host running 7.2.0-rc3 (KASAN test kernel, but the issue is not
KASAN related). And to be honest here, most of the writeup was created
by claude and I added things where appropriate. Also the patch was
mostly done with the help of claude.

A KVM guest with its RAM backed by a file on btrfs (zstd compression
enabled) locked up together with the host's writeback: two vCPU
threads, an irqfd worker, two flusher workers, khugepaged and a
syncfs caller (dnf) were all stuck in D state for hours.  Analysis
of the crash dump shows a leaked folio lock in btrfs'
writepage_delalloc(); a proposed fix is in the reply mail.

I still need to verify that this patches fixes the deadlock in our
CI but wanted some feedback first.

Dump analysis (shortened)
-------------------------
All blocked tasks funnel into one 64-page (256K) large data folio of
the guest RAM file:

  folio 0x800083fb000, inode 1881035 (the 1.25G s390.ram file)
  flags: PG_locked | PG_waiters | PG_dirty | PG_private | PG_uptodate
         (PG_writeback NOT set)
  btrfs_folio_state: nr_locked == 0, subpage dirty bitmap empty
  still mapped (63/64 PTEs) and on the LRU, no outstanding block I/O

Waiters on that folio lock:
  - 2 vCPU threads + 1 irqfd kworker, all in
    btrfs_page_mkwrite() -> folio_lock, holding mmap_lock (read)
crash> bt 66448
PID: 66448    TASK: 9e934a00          CPU: 9    COMMAND: "CPU 1/KVM"
 #0 [b8b25dbe7d8] __schedule at c0b186d5e78
 #1 [b8b25dbe908] schedule at c0b186d7040
 #2 [b8b25dbe948] io_schedule at c0b186d723c
 #3 [b8b25dbe978] folio_wait_bit_common at c0b167e719c
 #4 [b8b25dbeaf0] btrfs_page_mkwrite at c0b172816fc
 #5 [b8b25dbec98] do_page_mkwrite at c0b168a4ada
 #6 [b8b25dbecf0] do_wp_page at c0b168b2350
 #7 [b8b25dbed70] handle_pte_fault at c0b168bfaf4
 #8 [b8b25dbee58] __handle_mm_fault at c0b168c003e
 #9 [b8b25dbefc0] handle_mm_fault at c0b168c09b6
#10 [b8b25dbf020] __get_user_pages at c0b16899cfc
#11 [b8b25dbf148] get_user_pages_unlocked at c0b1689af1c
#12 [b8b25dbf248] hva_to_pfn at c0a9711e20e [kvm]
#13 [b8b25dbf3f0] __kvm_faultin_pfn at c0a9711ea26 [kvm]
#14 [b8b25dbf4e8] kvm_s390_faultin_gfn at c0a971c092c [kvm]
#15 [b8b25dbf5f8] vcpu_post_run_handle_fault at c0a97148b5e [kvm]
#16 [b8b25dbf6f0] __vcpu_run at c0a9715c1f2 [kvm]
#17 [b8b25dbf808] kvm_arch_vcpu_ioctl_run at c0a9715d3e4 [kvm]
#18 [b8b25dbfbb8] kvm_vcpu_ioctl at c0a97117bd8 [kvm]
#19 [b8b25dbfdd8] __s390x_sys_ioctl at c0b16aa3614
#20 [b8b25dbfe40] __do_syscall at c0b186cdaee
#21 [b8b25dbfe98] system_call at c0b186ebd42
 USER-MODE INTERRUPT FRAME; pt_regs at b8b25dbff38:
 PSW:  0705000180000000 000003ff8a92662c (user space)
 GPRS: 000003ff627faf50 0000000000000036 ffffffffffffffda 000000000000ae80 
       0000000000000000 000003ff627fc8c0 000002aa1f8a7880 000003ff8a8ad310 
       000002aa1e1f3c60 0000000000000000 000000000000ae80 000002aa1f8a2f60 
       000003ff8d3adfa8 000003ff627fc8c0 000003ff627faff0 000003ff627fae88 


  - flusher: extent_write_cache_pages() -> folio_lock
  - delalloc space reclaim worker: same, while holding
    fs_info->delalloc_root_mutex (which in turn blocks
    btrfs_async_reclaim_metadata_space on the mutex)
Behind those: khugepaged in down_write(mmap_lock), and syncfs.

No task in the system owns the folio lock; nothing references the
folio except the six waiters.  The lock was leaked.

Root cause
----------
A folio can carry the folio-level dirty flag with an EMPTY btrfs
subpage dirty bitmap.  btrfs data mappings use filemap_dirty_folio(),
so a generic folio_mark_dirty() sets only the folio flag and xarray
tag - no subpage dirty bits, no delalloc reservation.  On s390 this
happens all the time: the KVM irq adapter path
(arch/s390/kvm/interrupt.c, adapter_indicators_set()) pins the guest
interrupt indicator page with pin_user_pages_remote(FOLL_WRITE),
sets the indicator bit and calls set_page_dirty_lock().  Once a
previously written folio has gone through one complete writeback
cycle (subpage dirty bitmap empty again), the next adapter interrupt
re-dirties it with only the folio flag.

Writeback then does:

  extent_write_cache_pages(): folio_lock(), folio is dirty -> proceed
  extent_writepage() -> writepage_delalloc():
    - btrfs_copy_subpage_dirty_bitmap() -> submit_bitmap is EMPTY
    - the btrfs_folio_set_lock() loop sets nothing (nr_locked stays 0)
    - find_lock_delalloc_range() finds nothing -> goto out
    - out: bitmap_empty(submit_bitmap) is true -> return 1

The "return 1" path means "all dirty ranges were submitted
asynchronously, the async submission owns the folio unlock" - but
nothing was submitted, so extent_writepage() returns and the folio
stays locked forever.  This matches every flag of the dump folio
(locked, dirty, nr_locked == 0, no writeback, still mapped/LRU).

Verifying this in the dump:

- uptodate = 0xffffffffffffffff — all 64 blocks uptodate (consistent with PG_uptodate)
- dirty = 0x0 — the subpage dirty bitmap is EMPTY, exactly as the root cause predicts
- writeback = 0x0 — no writeback in flight (consistent with PG_writeback clear)


Exposure
--------
- Single-block folios are immune: btrfs_copy_subpage_dirty_bitmap()
  unconditionally reports bit 0 for blocks_per_folio == 1.
- Subpage setups (e.g. 64K page size with 4K sectorsize) have been
  exposed since the submission bitmap rework in v6.12
  (bd610c0937aa "btrfs: only unlock the to-be-submitted ranges
  inside a folio").
- 4K page size systems became exposed with large data folio support
  in v7.2-rc1, which routes every large folio through the subpage
  machinery.  That is why we only started seeing this now.

Any GUP-style dirtier can trigger it (KVM adapter interrupts on
s390, vfio, RDMA, io_uring fixed buffers, ...) as long as the target
is a multi-block folio of a btrfs data mapping that was clean at the
time of set_page_dirty_lock().

Reproducer outline: KVM guest on s390 with memory-backend-file on
btrfs + virtio devices using irqfd adapter indicators; hangs within
~25 minutes of guest uptime in our setup.  A targeted reproducer
should also work on x86: mmap a file on btrfs, write it, fsync, let
writeback finish, then pin_user_pages(FOLL_WRITE) +
set_page_dirty_lock() on a page of a large folio and trigger sync.

Proposed fix
------------
Detect the empty-at-entry bitmap right after it has been copied,
before any range lock is set up, clear the stale folio dirty flag
(nothing can ever be written back for it; all dirty flag setters
serialize on the folio lock we hold) and unlock the folio.  Patch
attached below; it survives our compile test and we are preparing a
test run on the affected machine.  Comments welcome - especially on
whether clearing the folio dirty flag is the desired semantic here,
versus e.g. routing such folios through the cow fixup worker to
actually persist GUP-written data.

Thanks

Christian