Re: [PATCH/RFC] btrfs: fix folio lock leak in writepage_delalloc() for folios dirtied behind btrfs' back
From: Christian Borntraeger
Date: Wed Jul 22 2026 - 05:35:30 EST
Am 22.07.26 um 10:59 schrieb Qu Wenruo:
So I now have an userspace O_DIRECT reproducer outside of KVM. (attached) which gave me (on an s390 system, though)
在 2026/7/22 18:05, Christian Borntraeger 写道:
Am 21.07.26 um 23:07 schrieb Qu Wenruo:
First, thank you for taking the time to look into this and trying to understand
things and trying to explain things. this is highly appreciated.
I am still trying to fully understand this myself. A question:
在 2026/7/22 04:41, Christian Borntraeger 写道:
A folio can carry the folio-level dirty flag while its btrfs subpage
dirty bitmap is empty: btrfs data mappings use filemap_dirty_folio(),
so a generic folio_mark_dirty() call sets only the folio flag and the
xarray tag, without setting any subpage dirty bit and without a
delalloc reservation. The typical source is set_page_dirty_lock() on
a GUP pin,
Shouldn't such folio got its ->page_mkwrite() callback get called first?
Isnt that called implicitely at pin time?
I have to admit, I'm not an expert on the MM part, I'm mostly a simple user of the existing MM interfaces.
AFAIK, the last time I brought this thing up, Christoph mentioned that dirtying-folio-without-notifying-fs is a bug, and fs should not and is not able to handle such situation anyway.
And that idea makes a lot of sense to me.
So adding MM list for more help.
[ 189.067731] sysrq: Show Blocked State
[ 189.067872] task:kworker/u1665:5 state:D stack:0 pid:1363 tgid:1363 ppid:2 task_flags:0x4208060 flags:0x00000000
[ 189.067877] Workqueue: writeback wb_workfn (flush-btrfs-1)
[ 189.067884] Call Trace:
[ 189.067886] [<000003a8b2e19284>] __schedule+0x374/0x900
[ 189.067891] [<000003a8b2e1984c>] schedule+0x3c/0xf0
[ 189.067894] [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[ 189.067896] [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[ 189.067902] [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[ 189.067907] [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[ 189.067910] [<000003a8b21ad444>] do_writepages+0xd4/0x190
[ 189.067915] [<000003a8b2324590>] __writeback_single_inode+0x50/0x310
[ 189.067917] [<000003a8b2324b1a>] writeback_sb_inodes+0x2ca/0x690
[ 189.067919] [<000003a8b2324f36>] __writeback_inodes_wb+0x56/0x140
[ 189.067921] [<000003a8b2325498>] wb_writeback+0x368/0x460
[ 189.067923] [<000003a8b23258c6>] wb_do_writeback+0x336/0x3d0
[ 189.067925] [<000003a8b2325d2a>] wb_workfn+0x5a/0x1d0
[ 189.067927] [<000003a8b1f0cee2>] process_one_work+0x1d2/0x480
[ 189.067932] [<000003a8b1f0df30>] worker_thread+0x210/0x420
[ 189.067935] [<000003a8b1f190b8>] kthread+0x148/0x170
[ 189.067939] [<000003a8b1e90358>] __ret_from_fork+0x48/0x220
[ 189.067941] [<000003a8b2e2426a>] ret_from_fork+0xa/0x30
[ 189.067994] task:oob-dirty-repro state:D stack:0 pid:8152 tgid:8151 ppid:7260 task_flags:0x400040 flags:0x00000800
[ 189.067997] Call Trace:
[ 189.067998] [<000003a8b2e19284>] __schedule+0x374/0x900
[ 189.068050] [<000003a8b2e1984c>] schedule+0x3c/0xf0
[ 189.068052] [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[ 189.068054] [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[ 189.068057] [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[ 189.068061] [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[ 189.068063] [<000003a8b21ad444>] do_writepages+0xd4/0x190
[ 189.068066] [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[ 189.068069] [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[ 189.068072] [<000003a8b2637d3c>] btrfs_start_ordered_extent_nowriteback+0x18c/0x1d0
[ 189.068074] [<000003a8b26260ce>] btrfs_page_mkwrite+0x22e/0x8d0
[ 189.068079] [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[ 189.068084] [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[ 189.068086] [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[ 189.068088] [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[ 189.068090] [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[ 189.068094] [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[ 189.068098] [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[ 189.068101] [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[ 189.068106] [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[ 189.068108] [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[ 189.068109] [<000003a8b22d0a70>] vfs_read+0x210/0x370
[ 189.068112] [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[ 189.068114] [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[ 189.068117] [<000003a8b2e24242>] system_call+0x72/0x90
[ 189.068120] task:oob-dirty-repro state:D stack:0 pid:8153 tgid:8151 ppid:7260 task_flags:0x400040 flags:0x00000000
[ 189.068123] Call Trace:
[ 189.068124] [<000003a8b2e19284>] __schedule+0x374/0x900
[ 189.068126] [<000003a8b2e1984c>] schedule+0x3c/0xf0
[ 189.068128] [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[ 189.068130] [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[ 189.068133] [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[ 189.068136] [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[ 189.068138] [<000003a8b21ad444>] do_writepages+0xd4/0x190
[ 189.068141] [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[ 189.068144] [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[ 189.068146] [<000003a8b2637d3c>] btrfs_start_ordered_extent_nowriteback+0x18c/0x1d0
[ 189.068149] [<000003a8b26260ce>] btrfs_page_mkwrite+0x22e/0x8d0
[ 189.068152] [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[ 189.068154] [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[ 189.068156] [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[ 189.068158] [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[ 189.068160] [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[ 189.068163] [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[ 189.068165] [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[ 189.068168] [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[ 189.068170] [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[ 189.068172] [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[ 189.068174] [<000003a8b22d0a70>] vfs_read+0x210/0x370
[ 189.068176] [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[ 189.068178] [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[ 189.068181] [<000003a8b2e24242>] system_call+0x72/0x90
[ 189.068184] task:oob-dirty-repro state:D stack:0 pid:8154 tgid:8151 ppid:7260 task_flags:0x400040 flags:0x00000000
[ 189.068187] Call Trace:
[ 189.068188] [<000003a8b2e19284>] __schedule+0x374/0x900
[ 189.068190] [<000003a8b2e1984c>] schedule+0x3c/0xf0
[ 189.068192] [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[ 189.068193] [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[ 189.068196] [<000003a8b26260fc>] btrfs_page_mkwrite+0x25c/0x8d0
[ 189.068199] [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[ 189.068202] [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[ 189.068203] [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[ 189.068206] [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[ 189.068208] [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[ 189.068210] [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[ 189.068213] [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[ 189.068215] [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[ 189.068218] [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[ 189.068219] [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[ 189.068221] [<000003a8b22d0a70>] vfs_read+0x210/0x370
[ 189.068223] [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[ 189.068225] [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[ 189.068228] [<000003a8b2e24242>] system_call+0x72/0x90
[ 189.068231] task:oob-dirty-repro state:D stack:0 pid:8155 tgid:8151 ppid:7260 task_flags:0x400040 flags:0x00000000
[ 189.068234] Call Trace:
[ 189.068235] [<000003a8b2e19284>] __schedule+0x374/0x900
[ 189.068237] [<000003a8b2e1984c>] schedule+0x3c/0xf0
[ 189.068239] [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[ 189.068241] [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[ 189.068244] [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[ 189.068246] [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[ 189.068249] [<000003a8b21ad444>] do_writepages+0xd4/0x190
[ 189.068252] [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[ 189.068255] [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[ 189.068257] [<000003a8b2637d3c>] btrfs_start_ordered_extent_nowriteback+0x18c/0x1d0
[ 189.068260] [<000003a8b26260ce>] btrfs_page_mkwrite+0x22e/0x8d0
[ 189.068263] [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[ 189.068265] [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[ 189.068267] [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[ 189.068269] [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[ 189.068271] [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[ 189.068274] [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[ 189.068277] [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[ 189.068279] [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[ 189.068281] [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[ 189.068283] [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[ 189.068285] [<000003a8b22d0a70>] vfs_read+0x210/0x370
[ 189.068287] [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[ 189.068289] [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[ 189.068292] [<000003a8b2e24242>] system_call+0x72/0x90
[ 189.068294] task:oob-dirty-repro state:D stack:0 pid:8156 tgid:8151 ppid:7260 task_flags:0x400040 flags:0x00000000
[ 189.068297] Call Trace:
[ 189.068298] [<000003a8b2e19284>] __schedule+0x374/0x900
[ 189.068300] [<000003a8b2e1984c>] schedule+0x3c/0xf0
[ 189.068302] [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[ 189.068304] [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[ 189.068307] [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[ 189.068310] [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[ 189.068313] [<000003a8b21ad444>] do_writepages+0xd4/0x190
[ 189.068315] [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[ 189.068318] [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[ 189.068320] [<000003a8b232bbf2>] sync_file_range+0x122/0x150
[ 189.068323] [<000003a8b232bc84>] ksys_sync_file_range+0x64/0xb0
[ 189.068326] [<000003a8b232bd0a>] __s390x_sys_sync_file_range+0x3a/0x50
[ 189.068328] [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[ 189.068331] [<000003a8b2e24242>] system_call+0x72/0x90// SPDX-License-Identifier: GPL-2.0
/*
* oob-dirty-repro.c - reproduce out-of-band dirty folios on btrfs
* (folio lock leak in writepage_delalloc())
*
* Demonstrates that a "folio-level dirty flag set, fs never notified"
* state can be produced on any architecture with sanctioned, in-tree
* APIs - no s390, no KVM required:
*
* 1. A file on btrfs is written with large buffered writes so the
* page cache backs it with large data folios (v7.2-rc1+).
* 2. The file is mmap'ed MAP_SHARED.
* 3. Thread A loops O_DIRECT reads *into* that mapping. The direct
* I/O read path pins the destination pages (GUP; the initial
* fault runs ->page_mkwrite() and reserves delalloc properly)
* and on bio completion re-dirties them via
* bio_check_pages_dirty() -> bio_release_pages(true)
* -> unpin_user_pages_dirty_lock() -> set_page_dirty_lock().
* 4. Thread B loops sync_file_range(WRITE|WAIT) on the target file.
* Whenever a full clean cycle (clear_page_dirty_for_io(),
* writeback, bits cleared) completes inside thread A's
* submission->completion window, the completion-time
* set_page_dirty_lock() hits a *clean* folio: filemap_dirty_folio()
* sets only the folio flag and the xarray tag - no btrfs subpage
* dirty bit, no delalloc reservation. See the 20-year-old comment
* above bio_set_pages_dirty() in block/bio.c describing exactly
* this ("other code (eg, flusher threads) could clean the pages").
* 5. The next writeback of that folio enters writepage_delalloc()
* with an empty subpage dirty bitmap, takes the
* bitmap_empty() -> return 1 path and leaks the folio lock.
* Every subsequent locker (mkwrite faults, sync, flusher) then
* hangs in D state.
*
* Thread C stores into the mapping (mkwrite traffic, like KVM vCPUs)
* and a watchdog reports when all threads stop making progress.
*
* Usage: gcc -O2 -pthread -Wall -o oob-dirty-repro oob-dirty-repro.c
* ./oob-dirty-repro <dir-on-btrfs> [runtime-seconds]
*
* Expected on 7.2-rc (large data folios enabled): within minutes the
* watchdog reports a stall; the sync/store/io threads are in D state,
* "echo w > /proc/sysrq-trigger" shows them queued on a folio lock
* (folio_wait_bit_common <- btrfs_page_mkwrite / extent_write_cache_pages).
* On kernels without large data folios the leak path is not reachable
* (single-block folios get bit 0 forced in btrfs_copy_subpage_dirty_bitmap());
* the same race then produces the out-of-band state that the (removed)
* COW fixup / "reject out-of-band dirty folios" code used to catch.
*
* WARNING: on affected kernels this wedges writeback of the target
* filesystem (unkillable D-state tasks, syncfs hangs) until reboot.
* Run it on a scratch machine.
*/
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <linux/fs.h>
#include <sys/ioctl.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/statfs.h>
#include <time.h>
#include <unistd.h>
#define BTRFS_SUPER_MAGIC 0x9123683E
#define TARGET_SZ (16UL << 20) /* mmap'ed target file */
#define SRC_SZ (4UL << 20) /* O_DIRECT read source */
#define IO_SZ (2UL << 20) /* per-read size (wide window) */
#define WRITE_CHUNK (1UL << 20) /* buffered init writes (large folios) */
#define STALL_SECS 20
#define NR_IO_THREADS 4
static char *map;
static int target_fd, src_fd;
static atomic_ulong io_iters, sync_iters, store_iters;
static atomic_int stop;
static unsigned long runtime = 600;
static void die(const char *what)
{
perror(what);
exit(1);
}
static void *io_thread(void *arg __attribute__((unused)))
{
/*
* Only the lower half: btrfs verifies read csums in the (pinned)
* destination pages after DMA, so the store thread must not poke
* the same pages, and the NOCOW source avoids csums altogether.
*/
unsigned long nranges = (TARGET_SZ / 2) / IO_SZ;
while (!atomic_load(&stop)) {
off_t off = (random() % nranges) * IO_SZ;
ssize_t r = pread(src_fd, map + off, IO_SZ, 0);
if (r < 0 && errno != EINTR && errno != EIO)
die("O_DIRECT pread into mapping");
atomic_fetch_add(&io_iters, 1);
}
return NULL;
}
static void *sync_thread(void *arg __attribute__((unused)))
{
while (!atomic_load(&stop)) {
if (sync_file_range(target_fd, 0, TARGET_SZ,
SYNC_FILE_RANGE_WAIT_BEFORE |
SYNC_FILE_RANGE_WRITE |
SYNC_FILE_RANGE_WAIT_AFTER) < 0)
die("sync_file_range");
atomic_fetch_add(&sync_iters, 1);
}
return NULL;
}
static void *store_thread(void *arg __attribute__((unused)))
{
struct timespec ts = { 0, 1000000 }; /* 1ms */
while (!atomic_load(&stop)) {
unsigned long off = TARGET_SZ / 2 +
(random() % (TARGET_SZ / 2 / 4096)) * 4096;
map[off] ^= 1; /* mkwrite fault traffic */
atomic_fetch_add(&store_iters, 1);
nanosleep(&ts, NULL);
}
return NULL;
}
int main(int argc, char **argv)
{
char path[4096], *buf;
struct statfs sfs;
pthread_t io[NR_IO_THREADS], sy, st;
unsigned long last_io = 0, last_sync = 0;
unsigned long elapsed = 0, stalled = 0;
setvbuf(stdout, NULL, _IOLBF, 0);
if (argc < 2) {
fprintf(stderr, "usage: %s <dir-on-btrfs> [runtime-secs]\n",
argv[0]);
return 1;
}
if (argc > 2)
runtime = strtoul(argv[2], NULL, 0);
if (statfs(argv[1], &sfs) < 0)
die("statfs");
if ((unsigned int)sfs.f_type != BTRFS_SUPER_MAGIC)
fprintf(stderr, "warning: %s is not on btrfs, "
"the writepage_delalloc() path will not be hit\n",
argv[1]);
/* Target file: large buffered writes -> large data folios. */
snprintf(path, sizeof(path), "%s/oob-target.dat", argv[1]);
target_fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0600);
if (target_fd < 0)
die("open target");
buf = aligned_alloc(4096, WRITE_CHUNK);
if (!buf)
die("aligned_alloc");
memset(buf, 0x5a, WRITE_CHUNK);
for (unsigned long o = 0; o < TARGET_SZ; o += WRITE_CHUNK)
if (pwrite(target_fd, buf, WRITE_CHUNK, o) != WRITE_CHUNK)
die("pwrite target");
if (fsync(target_fd) < 0)
die("fsync target");
/* Source file for the O_DIRECT reads. */
snprintf(path, sizeof(path), "%s/oob-src.dat", argv[1]);
src_fd = open(path, O_RDWR | O_CREAT | O_TRUNC | O_DIRECT, 0600);
if (src_fd < 0)
die("open source (O_DIRECT)");
{ /* NOCOW/nodatasum: keep csum verification out of the picture */
int attr = 0;
ioctl(src_fd, FS_IOC_GETFLAGS, &attr);
attr |= FS_NOCOW_FL;
ioctl(src_fd, FS_IOC_SETFLAGS, &attr);
}
for (unsigned long o = 0; o < SRC_SZ; o += WRITE_CHUNK)
if (pwrite(src_fd, buf, WRITE_CHUNK, o) != WRITE_CHUNK)
die("pwrite source");
if (fsync(src_fd) < 0)
die("fsync source");
free(buf);
map = mmap(NULL, TARGET_SZ, PROT_READ | PROT_WRITE, MAP_SHARED,
target_fd, 0);
if (map == MAP_FAILED)
die("mmap");
printf("running: O_DIRECT reads into MAP_SHARED btrfs mapping vs. "
"sync_file_range, %lus max\n", runtime);
for (int n = 0; n < NR_IO_THREADS; n++)
pthread_create(&io[n], NULL, io_thread, NULL);
pthread_create(&sy, NULL, sync_thread, NULL);
pthread_create(&st, NULL, store_thread, NULL);
for (elapsed = 0; elapsed < runtime; elapsed += 5) {
unsigned long i, s, w;
sleep(5);
i = atomic_load(&io_iters);
s = atomic_load(&sync_iters);
w = atomic_load(&store_iters);
printf(" t=%4lus io=%lu sync=%lu store=%lu\n",
elapsed + 5, i, s, w);
if (i == last_io && s == last_sync) {
stalled += 5;
if (stalled >= STALL_SECS) {
printf("\n*** io+sync THREADS STALLED for %us - "
"folio lock leak reproduced? ***\n"
"Check: grep -l . /proc/%d/task/*/stack "
"| xargs cat\n"
" echo w > /proc/sysrq-trigger; "
"dmesg | tail -100\n"
"Expect: folio_wait_bit_common <- "
"btrfs_page_mkwrite /\n"
" extent_write_cache_pages in "
"D-state threads.\n",
STALL_SECS, getpid());
/* leave the hung process for inspection */
pause();
}
} else {
stalled = 0;
}
last_io = i;
last_sync = s;
}
atomic_store(&stop, 1);
for (int n = 0; n < NR_IO_THREADS; n++)
pthread_join(io[n], NULL);
pthread_join(sy, NULL);
pthread_join(st, NULL);
printf("finished without stall - not reproduced in %lus "
"(try a longer runtime; the race window is\n"
"submission->completion of the O_DIRECT read vs. one full "
"writeback cycle)\n", runtime);
return 0;
}