Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Pedro Falcato]

On Fri, Nov 13, 2026 at 03:50:15PM +0000, Salvatore Dipietro wrote:
> Hi all,
> 
> Thanks Jan, Pedro, Dave, David and Kiryl for the feedback.
> 
> Following Dave's question about how PostgreSQL gets large folios on
> files it writes 8 KiB at a time, we looked deeper and found a
> reproducer that needs only fio and a single disk.
> 
> What happens:
> 
>   1. A buffered sequential read (VACUUM, a backup, a scan) builds large
>      folios through readahead.  They stay large in the page cache until
>      they are evicted.
>   2. The first small write into one of these clean folios charges the
>      whole folio as dirty: a 4 KiB write into a 256 KiB folio charges
>      256 KiB, while writeback only writes the dirty 4 KiB.
>   3. With random writes, most writes land in a different clean folio,
>      so the dirty count reaches the limits with little real data.
>      balance_dirty_pages() then throttles the writers, and they end up
>      paced by how many folios writeback can clean per second rather
>      than by the bytes to write.
> 
> Reproducer: XFS on one EBS gp3 volume (3000 IOPS), m8g.8xlarge
> (32 CPUs, 128 GiB), default read_ahead_kb=128. fio's 1 MiB reads grow
> the readahead window up to the device's max_sectors_kb (256 KiB), so
> the read builds 256 KiB folios.
> 
>   ; create.fio: 16 x 3 GiB files, direct I/O so nothing is cached
>   [global]
>   directory=/mnt/xfs/fio
>   filename_format=f.$jobnum
>   size=3072m
>   numjobs=16
>   group_reporting=1
>   [create]
>   rw=write
>   bs=1m
>   direct=1
>   ioengine=libaio
>   iodepth=16
> 
>   ; load.fio: buffered sequential read; builds the large folios
>   [global]
>   directory=/mnt/xfs/fio
>   filename_format=f.$jobnum
>   size=3072m
>   numjobs=16
>   group_reporting=1
>   [load]
>   rw=read
>   bs=1m
>   direct=0
>   ioengine=psync
> 
>   ; randwrite.fio: random 4 KiB buffered writes, 32 x 312 = ~10k IOPS
>   ; invalidate=0 keeps the cached folios when fio opens the files
>   [randwrite]
>   directory=/mnt/xfs/fio
>   filename=f.0:f.1:f.2:f.3:f.4:f.5:f.6:f.7:f.8:f.9:f.10:f.11:f.12:f.13:f.14:f.15
>   rw=randwrite
>   bs=4k
>   direct=0
>   ioengine=psync
>   invalidate=0
>   numjobs=32
>   rate_iops=312
>   time_based=1
>   runtime=120
>   group_reporting=1
> 
> To reproduce it, run (as root):
> 
>   fio create.fio
>   sync; echo 3 > /proc/sys/vm/drop_caches
>   fio load.fio          # skip this step for the control run
>   fio randwrite.fio
> 
> fio asks for ~10k write IOPS, more than the volume's 3000, so dirty
> data builds up in both runs.  In the control run (no load step) each
> 4 KiB write creates a 4 KiB folio: after 120 s only 3.3G is dirty,
> below the 12.2G background threshold, and nothing is throttled.  After
> the load step the same writes land in 256 KiB folios, and each write
> into a clean folio is charged 256 KiB: the dirty count passes the
> background threshold after 6 s and writers are throttled from 13 s.
> 
> Results on v7.4-rc2, one run per kernel, each on a fresh instance.
> Writer sleep is the total balance_dirty_pages() pause time over the
> 32 writers during the 120 s write phase:
> 
>                          write   p99 / max   writer   max    folio
>                           IOPS     lat ms    sleep s  dirty  size
>   baseline, control      9,984  0.01 / 0.30      0    3.3G    -
>   baseline               4,061    60 / 160     179   23.5G   256K
>   v1 (this series)       9,984  0.01 / 0.67      0    3.6G   256K
>   max order cap 64K      7,071    50 / 280      73   22.3G    64K
>   max order cap 16K      9,984  0.02 / 0.05      0   11.9G    16K
>   split on write         9,984  0.02 / 0.17      0    3.6G   256K
>   evict on write         9,984  0.03 / 0.19      0    3.3G   256K
>   POSIX_FADV_RANDOM      9,984  0.02 / 0.05      0    3.3G     4K
> 
> The control run gave 9,984 IOPS and max latency up to 0.30 ms on every
> kernel.
> 
>   - max order cap: the global cap in mapping_max_folio_order() that Jan
>     suggested trying.  At 64 KiB a 4 KiB write still charges 16x and is
>     throttled.  At 16 KiB this run is not throttled, but dirty reached
>     11.9G, just below the 12.2G background threshold.  The cap that
>     avoids throttling depends on the write size, and it removes large
>     folios for every file, including ones that never see small writes.
>   - split on write: when a buffered write (iomap opts in, not for
>     zeroing or unshare) finds a clean, uptodate, unmapped folio at least
>     4x the write size, folio_split() it down to the write's order before
>     dirtying it.  The rest of the folio stays cached in large folios.
>     On the concerns Jan raised about splitting [1]: split on write only
>     splits clean, uptodate, unmapped folios, so it needs no new
>     filesystem state beyond an opt-in flag in iomap, and if
>     folio_split() fails the write simply proceeds as today; evict on
>     write avoids splitting altogether.
>   - evict on write: same trigger, but mapping_evict_folio() the folio
>     instead of splitting it (~40 lines).  This discards the folio's
>     other cached data; nothing reads it back in this test, so the read
>     cost does not show here.
>   - POSIX_FADV_RANDOM (Dave's question): fio's fadvise_hint=random on
>     the load job.  It avoids the problem without a kernel change, but
>     only where the application opts in: the hint is per open file, so
>     every process that reads a file must issue it, and a reader that
>     doesn't (a backup or copy tool, for example) still builds large
>     folios.  Every application that reads files sequentially and then
>     rewrites them in place would have to adopt it, and keeps the
>     regression until it does.  It also turns readahead off for those
>     files, so their sequential reads lose readahead.

Holy heck, did an LLM write this?

> 
> Split and evict on write keep whole-folio accounting and act at write
> time, so they also cover large folios already in the page cache,
> whoever built them, without each application having to opt in.  Both
> only touch clean, uptodate, unmapped folios at least 4x the write
> size, and only on iomap buffered writes.  Split keeps the rest of the
> folio cached at the cost of the split; evict is simpler but drops it.
> I'd appreciate your opinion on these options, or any other direction
> you think is more appropriate.

I think you'll face strong resistance to any of these options.

-- 
Pedro