Re: [PATCH v7 0/7] ovl: only sync what was written through the overlay

From: Amir Goldstein

Date: Wed Nov 18 2026 - 13:14:33 EST


On Wed, Nov 18, 2026 at 6:03 PM Josef Bacik <josef@xxxxxxxxxxxxxx> wrote:
>
> v1: https://lore.kernel.org/all/20261027-b4-ovl-containerized-sync-v1-0-802e512379a6@xxxxxxxxxxxxxx/
> v2: https://lore.kernel.org/all/20261029-b4-ovl-containerized-sync-v2-0-4e9f08e4a052@xxxxxxxxxxxxxx/
> v3: https://lore.kernel.org/all/20261104-b4-ovl-containerized-sync-v3-0-8daa20144f53@xxxxxxxxxxxxxx/
> v4: https://lore.kernel.org/all/20261106-b4-ovl-containerized-sync-v4-0-6ddde0b7dd8a@xxxxxxxxxxxxxx/
> v5: https://lore.kernel.org/all/20261107-b4-ovl-containerized-sync-v5-0-d34b4ae57fdf@xxxxxxxxxxxxxx/
> v6: https://lore.kernel.org/all/20261110-b4-ovl-containerized-sync-v6-0-43d1be16a4d6@xxxxxxxxxxxxxx/
>
> I've completely redesigned this, because there were so many problems
> that I kept tripping over trying to maintain state between the upper and
> the lower inode. I'm not in love with this design either, but it's the
> simplest. I'm hoping somebody spots something obvious and we can do it a
> different way.

FWIW, I think this is now at least simple enough for me to review ;)

I do expect it may take me some time though.

Thanks,
Amir.



>
> v6->v7:
> - Redone without a flusher. An overlay inode has nothing of its own.
> What is dirty, under writeback or deferred by lazytime is the state of
> the upper inode, and nothing tells overlayfs when that changes. Up to
> v6 the dirty bit of the overlay inode mirrored that state on a bdi of
> the overlay, and most of what the reviews found was the mirror being
> off, or the flusher acting on it. The worst one: with 3000 files open
> for writing and the system over the background threshold the flusher
> of the overlay took 79% of a CPU going over its list. (Sashiko)
> - Now the overlay only keeps a set of the inodes that were modified, an
> xarray, patch 5. Only syncfs(), unmount and a read-only remount walk
> it, and they ask the upper inode of each. Periodic and background
> writeback is all up to the upper file system, like without the series.
> - The xarray has the marks of the page cache and a sync uses them the
> same way. ->sync_fs() without wait tags what is dirty with towrite,
> starts writeback of those and marks them writeback, with wait it waits
> for what is marked writeback. The one difference is that there is no
> completion to clear the writeback mark. Whoever has waited does, and
> only if the upper inode is clean, another sync may have started it
> again.
> - The set holds a reference. An inode leaves it when it's unlinked, or
> when a sync or the shrinker sees the upper inode clean. The shrinker
> part hangs off the ->nr_cached_objects() and ->free_cached_objects()
> hooks. Nobody who walks the set needs a lock against anybody else.
> - Gone: the bdi, ->write_inode(), ->drop_inode(), the lazytime patch,
> "ovl: report failed upper writeback in syncfs()", and the special
> cases for the root and for unhashed inodes. 7 patches, v6 had 8.
> - syncfs() waits for the writeback of its own inodes, whoever started
> it, and for nobody else's.
> - New patch 4: write_inode_for_sync() in fs/fs-writeback.c writes one
> inode the way the flusher does for sync_inodes_sb(), WB_SYNC_ALL with
> for_sync. Up to v6 I pieced that together from sync_inode_metadata()
> without and with wait, so that ext4 wouldn't commit for every inode.
> (Sashiko)
> - Deferred timestamps are handed to the upper file system in a sync, or
> when memory is needed, so lazytime keeps working in between.
> - syncfs() also reports writeback that the upper file system did earlier
> and that failed. The overlay inode remembers where wb_err of the upper
> mapping was when it entered the set, and a sync compares. An fsync()
> in between doesn't take that away.
> - A freeze and a read-only remount of the upper file system are held off
> while an inode is handed over. If one has already begun syncfs() syncs
> the whole upper file system like before. That closes what v6 only
> described in the changelog of patch 7. (Sashiko)
> - New patch 2: an upper file system with SB_I_NO_DATA_INTEGRITY, which
> is fuse, doesn't get relaxed. A sync doesn't wait for the fuse server,
> writing the upper inodes one by one does. (Sashiko)
> - The helpers that set the impure, redirect, opaque and nlink xattrs
> track the inode themselves, and so does ovl_clear_empty(). A link or a
> rename that failed after them left the inode untracked. A setattr that
> fails tracks it too. (Sashiko)
> - Rebased onto 7.4-rc1. v6 was still on the base of the earlier
> versions, and patch 4 conflicted with 4f61ff2ebfb3 ("ovl: use the
> pinned upper dir in ovl_fix_origin()").
>
> v5->v6:
> - Lookup marks a merge directory and its parent when it sets the origin
> and impure xattrs on them. (Sashiko)
> - New patch 8: if somebody else writes the pages of an upper inode and
> that fails, syncfs() on the overlay reports it. It's best effort, the
> limits are in the changelog, and it's on its own so it can be dropped.
> (Sashiko)
> - Left alone: if the deferred timestamps can't be handed over because
> the upper file system is being frozen, the sync of the freeze can miss
> them. They are written after the thaw, it's in the changelog of patch
> 7. (Sashiko)
>
> v4->v5:
> - Added sync_fs_and_bdev() to fs/sync.c for the ->sync_fs() and block
> device part of sync_filesystem(), so ovl_sync_fs() doesn't open code
> it, patch 2. (Christian)
> - ovl_instantiate() marks a new inode and a new hard link dirty.
> (Christian, Sashiko)
> - Copy up marks the parent as well, and the inode also when it only adds
> a link to an indexed upper inode. (Sashiko)
> - Dropped "ovl: make the root inode appear hashed". A fake hash stays in
> the slab object when the inode is freed, and the next overlay inode in
> it looks hashed. The root isn't tracked at all now, ovl_sync_fs()
> writes its upper inode every time. (Sashiko)
> - lazytime got a lot simpler. The overlay inode doesn't get I_DIRTY_TIME
> any more, deferred timestamps are handed to the upper file system when
> ->write_inode() finds them. That takes care of the I/O from dentry
> reclaim, so this doesn't need anything from Jan's deferred inode
> reclaim series. (Sashiko, Christian)
> - ovl_sync_upper_inode() could lose an error.
> - Left alone: fsync=auto still goes by RENAME_WHITEOUT, no flag. What
> that leaves uncovered is under "Things I know about". (Christian)
> - Left alone: if somebody else starts the writeback of an upper inode
> and it fails, syncfs() on the overlay doesn't report it. It never has.
> (Sashiko)
>
> v3->v4:
> - Patch 1 is v2 of Amir's patch. It has ovl_tracks_dirty() and picks the
> mode for tmpfs at mount time, so I dropped my version from patch 2.
> fsync=auto stays auto if it doesn't become relaxed. (Amir)
>
> v2->v3:
> - Patch 1 is Amir's, it adds fsync=relaxed. Only an overlay with that
> syncs just what was written through it, everybody else syncs the whole
> upper file system as before. fsync=auto, the default, picks relaxed if
> the upper file system supports RENAME_WHITEOUT. (Amir)
> - Dropped the fake inodes and the calls in the ovl_do_*() helpers,
> patches 7 to 9 of v2. Upper inodes without an overlay inode are left
> to ->sync_fs() and the block device, like in v1. (Amir)
>
> v1->v2:
> - Dropped the superblock flag and the ext4, xfs and btrfs patches. This
> is all in overlayfs now and works the same on any upper file system.
> (Christoph, Amir)
> - Upper inodes without an overlay inode get a fake one that goes on the
> dirty list, patches 7 to 9.
> - The periodic flusher pass doesn't write the upper inode any more, and
> syncfs() doesn't make ext4 commit per inode.
> - Record write errors in s_wb_err of the overlay.
> - Make the root inode appear hashed so it can be on the dirty list,
> don't mark unhashed inodes dirty, read i_writecount with acquire.
> (Sashiko)
> - Left alone: dirtied_time_when isn't set for an overlay inode that gets
> I_DIRTY_TIME from ->write_inode(). It only matters for the 12 hour
> expiry, where the upper file system writes the timestamps on its own.
> (Sashiko)
> - Rewrote my commit messages. (Christoph)
>
> What it does now:
>
> sync_filesystem() writes every inode on a dirty list, calls ->sync_fs()
> and writes the block device. Only the first step is expensive, because
> of other people's data. With fsync=relaxed overlayfs does that step only
> for the upper inodes of the overlay inodes in its set, ->sync_fs()
> without wait starts them and with wait waits for them, and does the
> other two steps as they are.
>
> Whiteouts and the work directories have no overlay inode, so nobody
> writes those inodes. That's fine if ->sync_fs() or the block device gets
> all metadata to disk, which is what relaxed asks of the upper file
> system.
>
> Things I know about:
>
> - ext4 without a journal supports RENAME_WHITEOUT, so it gets relaxed.
> It's fine there, its metadata is in the buffers of the block device,
> which we write. I tested it like the others.
> - f2fs and ubifs get relaxed as well and I didn't test them. f2fs
> writes its dirty inodes in the checkpoint. ubifs_sync_fs() doesn't,
> but the only thing overlayfs leaves dirty there on an inode it
> doesn't track is the chmod of the work directory to 0 at mount time,
> right after creating it with mode 0. fsync=on is there for anybody
> who needs more.
> - A default mount that gets relaxed shows that in /proc/mounts.
> - A tracked overlay inode, its upper inode and the dentry of that can't
> be reclaimed while the upper inode is dirty, like a dirty inode of
> any file system. After that they stay until a sync or the shrinker
> looks. The shrinker only queues a work item that does the looking.
> The reference it drops can be the last one to an unlinked upper
> inode, and file systems allocate memory under locks that deleting a
> file needs. Deferred timestamps count as dirty, the work item hands
> them to the upper file system.
> - When a sync reports an error it marks it as seen in wb_err of the
> upper mapping, like an fsync() does. A file that is opened on the
> upper inode after that doesn't get it any more.
> - syncfs() looks at every inode in the set, also at those that the
> upper file system has written by now.
> - A file that is open for writing stays in the set, it may be mapped.
> That doesn't cover pages pinned through a mapping. A direct read into
> one that completes after the last munmap() and close() dirties them
> when the inode may have left the set.
> - An atime update puts the inode into the set as well.
> - A read-only remount of the overlay syncs it, so it now fails once
> with the error of writeback that has failed earlier.
> - The error reporting has gaps, in all of them syncfs() returns 0 like
> it does today. With the upper file system frozen or read-only we
> fall back to syncing all of it and don't look at the inodes, so an
> error is reported by the next syncfs() that can. btrfs records an
> error after it has ended writeback, which can be after we looked.
> - Data that an earlier mount of the same upper directory left dirty,
> with fsync=off for example, isn't written by a syncfs() of a later
> mount. Today it is, along with everything else.
> - Every inode of a relaxed overlay gets a slot in the xarray when it's
> allocated, also those that are never modified, so that tracking one
> can't fail. The index is the address of the inode. i_ino isn't unique
> in an overlay and xa_alloc() needs one of the three marks. All
> overlays allocate from one slab cache, so where their inodes are
> mixed a slot can cost a node of the xarray or more. With 64 overlays
> of 2000 files, same debug kernel as below, it's 21 bytes an inode if
> they're walked one after the other, 89 if all at once, 489 if they
> take turns entry by entry. Plus about 6K for every overlay.
> - If the origin of a copied up file can't be decoded, like in a user
> namespace, a later lookup makes a second overlay inode for the same
> upper inode. If the first one is tracked it pins the upper inode
> after that was unlinked, until a sync or the shrinker looks.
>
> Tested like before, see the v1 mail below, now also with ext2 and ext4
> without a journal as the upper file system. New tests for memory
> pressure, a frozen and a read-only upper file system, O_TMPFILE, write
> errors, and syncfs() next to tasks that create files and next to other
> syncfs() calls. Everything that was changed through the overlay is in
> the copy of the block device and the 192M of the neighbours aren't,
> except with ext2, which doesn't get the new mode by default.
>
> To see that the tests can fail I broke the series in 12 places, a line
> or two each, and all of them are caught.
>
> With KASAN and lockdep on, ext4, xfs, btrfs and ext4 without a journal,
> one sample each, without and with the series:
>
> syncfs(), the neighbours' 192M dirty 258-305ms 44-50ms
> unmount, the same 147-273ms 37-43ms
> syncfs(), the 192M under writeback [4] 42-58ms 138-171ms
> syncfs(), 4000 inodes in the set, dirty 52-153ms 71-207ms
> syncfs(), 10000 inodes in the set, clean 55-57ms 144-146ms
>
> So it isn't free. Walking a tree takes 0 to 3% more system time, which
> is what it varies by without the series.
>
> fstests with -overlay, the auto groups of generic and overlay, on ext4
> and on xfs: nothing that passes without the series fails with it, and
> the run time is within 0.3%.
>
> [2] https://lore.kernel.org/r/20261027-b4-ext2-dirsync-v1-1-3dcd012d518d@xxxxxxxxxxxxxx
> [3] https://lore.kernel.org/r/99629e3277656888cbe84bf7490c5e54.josef@xxxxxxxxxxxxxx
> [4] Not btrfs, nothing was in flight there without the series.
>
> --- Original email (v1) ---
>
> Hello,
>
> This picks up Chengguang Xu's "implement containerized syncfs for
> overlayfs", last posted as RFC V6 in November 2021 [1].
>
> syncfs() on an overlay and unmounting one call sync_filesystem() on the
> upper file system, so they write every dirty inode on it no matter who
> dirtied it. We run a lot of containers on one upper file system, and
> stopping one of them writes out the dirty data of all the others. That
> can take a very long time, and s_umount of the upper file system is held
> the whole time, so anything that wants it exclusively queues up and then
> everybody else queues up behind that.
>
> With this an overlay only writes the upper inodes that were changed
> through it, and then has the upper file system commit its metadata.
>
> The V6 discussion ended with a direction but no V7, so that's what I
> tried to follow:
>
> - V6 kept every overlay inode with an upper inode dirty for good. Jan
> and Amir didn't like the cost of walking them. Now an overlay inode
> is marked dirty when the upper inode is changed, mostly from
> ovl_copyattr() as Amir and Miklos suggested.
> - It never looks at the dirty flags of the upper inode, Miklos pointed
> out that they're cleared when writeback starts.
> - ->write_inode() goes through write_inode_now() on the upper inode,
> and keeps the overlay inode dirty while the upper inode is open for
> writing, which covers shared mappings.
> - V6 marked the inodes DONTCACHE, which Jan was worried about. This
> goes back to the v5 patch that keeps dirty overlay inodes cached.
>
> One thing is new. V6 called wait_sb_inodes() on the upper file system to
> catch writeback that was started but not finished, which waits for
> everybody's writeback. Instead ->write_inode() keeps the overlay inode
> dirty while the upper inode has pages tagged dirty or under writeback.
> The tags don't have the problem the inode flags have, the writeback tag
> is set before the dirty tag is cleared. So a clean overlay inode means
> its data is on disk, and syncfs() only waits for its own inodes.
>
> That takes care of the data. Metadata is harder, a lot of what is
> changed on the upper file system has no overlay inode that could be on
> the dirty list: the root of the overlay isn't hashed, and whiteouts,
> index entries, the work directories and their parent have none. File
> systems that log their metadata don't care, ->sync_fs() commits it. ext2
> does, there a whiteout was gone after a crash. I first had overlayfs
> write all of those itself and kept finding more of them, the last one
> was the link count of the shared whiteout when a whiteout is removed.
>
> So patch 1 adds a superblock flag, SB_I_OVL_USE_SYNC_FS, by which a file
> system says that ->sync_fs() commits all of its metadata, and patches 2
> to 4 set it for ext4 with a journal, xfs and btrfs. Without the flag on
> the upper file system nothing changes, no inode is marked dirty and
> syncfs() and unmount call sync_filesystem() like they always did.
>
> Patches 5, 7, 8 and 10 are Chengguang's with changes noted in them,
> patches 6 and 9 are new. I dropped Jan's Reviewed-by tags from V6 since
> the patches changed.
>
> lazytime on the upper file system needs its own handling. The flusher
> leaves the timestamps in memory for up to 12 hours, so the overlay inode
> can't be clean, but it can't stay dirty and pinned for that long either.
> Patch 9 gives it I_DIRTY_TIME like the upper inode has, and forwards
> ->sync_lazytime() to the upper inode.
>
> Things I know about:
>
> - An overlay inode that loses its last reference while it is dirty is
> kept until it has been written and then sits on the LRU, like with
> any other file system. Clean and unlinked ones are still freed right
> away.
> - Every overlay on such an upper file system gets a bdi.
> - I haven't measured syncfs() with a very large number of dirty inodes.
>
> For testing I put several overlays on one upper file system and dirty
> 192M next to the overlay under test, through another overlay and
> directly. Then syncfs() or unmount the one overlay, copy the block
> device as it is at that moment, mount the copy, and check that
> everything that was changed through the overlay is in it and that the
> 192M aren't. That passes with ext4, xfs and btrfs as the upper file
> system. With KASAN and lockdep on, syncfs() goes from 239-284ms to
> 45-48ms and unmount from 126-234ms to 39-44ms. With ext2 and ext4
> without a journal the 192M are written as before.
>
> fstests with -overlay, the auto groups of generic and overlay, gives the
> same results with and without the series on ext4 and on xfs.
>
> [1] https://lore.kernel.org/r/20211122030038.1938875-1-cgxu519@xxxxxxxxxxxx
>
> Thanks,
>
> Josef
>
> ---
> Changes in v7:
> - Link to v6: https://patch.msgid.link/20261110-b4-ovl-containerized-sync-v6-0-43d1be16a4d6@xxxxxxxxxxxxxx
>
> Changes in v6:
> - Link to v5: https://patch.msgid.link/20261107-b4-ovl-containerized-sync-v5-0-d34b4ae57fdf@xxxxxxxxxxxxxx
>
> Changes in v5:
> - Link to v4: https://patch.msgid.link/20261106-b4-ovl-containerized-sync-v4-0-6ddde0b7dd8a@xxxxxxxxxxxxxx
>
> Changes in v4:
> - Link to v3: https://patch.msgid.link/20261104-b4-ovl-containerized-sync-v3-0-8daa20144f53@xxxxxxxxxxxxxx
>
> Changes in v3:
> - Link to v2: https://patch.msgid.link/20261029-b4-ovl-containerized-sync-v2-0-4e9f08e4a052@xxxxxxxxxxxxxx
>
> Changes in v2:
> - Link to v1: https://patch.msgid.link/20261027-b4-ovl-containerized-sync-v1-0-802e512379a6@xxxxxxxxxxxxxx
>
> ---
> Amir Goldstein (1):
> ovl: expand fsync mount options
>
> Chengguang Xu (1):
> ovl: implement containerized syncfs for overlayfs
>
> Josef Bacik (5):
> ovl: no fsync=relaxed without data integrity
> fs: add sync_fs_and_bdev()
> fs: add write_inode_for_sync()
> ovl: add a set of the inodes that a sync has to write
> ovl: track an inode when it is modified
>
> Documentation/filesystems/overlayfs.rst | 45 ++-
> fs/fs-writeback.c | 36 ++
> fs/overlayfs/Kconfig | 21 ++
> fs/overlayfs/Makefile | 2 +-
> fs/overlayfs/copy_up.c | 9 +
> fs/overlayfs/dir.c | 11 +
> fs/overlayfs/file.c | 7 +-
> fs/overlayfs/inode.c | 27 +-
> fs/overlayfs/namei.c | 9 +-
> fs/overlayfs/overlayfs.h | 39 ++-
> fs/overlayfs/ovl_entry.h | 9 +
> fs/overlayfs/params.c | 34 +-
> fs/overlayfs/params.h | 2 +
> fs/overlayfs/readdir.c | 1 +
> fs/overlayfs/super.c | 58 +++-
> fs/overlayfs/sync.c | 591 ++++++++++++++++++++++++++++++++
> fs/overlayfs/util.c | 16 +-
> fs/sync.c | 52 ++-
> include/linux/fs.h | 1 +
> include/linux/pagemap.h | 1 +
> 20 files changed, 920 insertions(+), 51 deletions(-)
> ---
> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
> change-id: 20261026-b4-ovl-containerized-sync-149bed1afe9a
>