[RFC PATCH 0/1] block: long udev open waits during cache teardown
From: Ethan
Date: Tue Nov 17 2026 - 05:51:26 EST
Hi,
This is the first posting of this proposal.
I am requesting feedback on long device-open blocking during final cache
teardown after large buffered writes to a raw block device. In repeated
full mkfs.ntfs runs, formatting succeeds, but a udev process opening the
same device waits for several minutes on disk->open_mutex. The system
reports hung tasks and udev is restarted after watchdog timeouts.
The issue is not simply that formatting or close takes a long time.
The concern is that device lifecycle exclusion is held while large-scale
cache cleanup runs, preventing another device opener from progressing.
The attached patch is an early pre-reclaim prototype for discussing how
to address that exclusion. It is not a current-mainline merge request
or a 6.6.y stable backport request.
Reproduction context and observed impact
----------------------------------------
The recorded system is Phytium ARM64 with about 1 TiB RAM, a 1.92 TB NVMe
device and 512-byte logical sectors. The reproducer is a full, non-quick
mkfs.ntfs run using the original userspace binary and library. Such a
reproducer destroys the target; it was run only on a disposable device.
No userspace cache-drop option or changed write-size option was enabled.
The hung-task reporting threshold remained 120 seconds.
The baseline was 6.6.0.6.6-server-v11-bdevdbg: a vendor 6.6 kernel with
BDEVDBG diagnostics for observing the close path. No preclose cache
reclaim hook was present. This is not an unmodified upstream v6.6 binary.
Three consecutive runs on the same boot produced:
Run Mapping cache at close Final truncate Udev entire open
(4 KiB pages / GiB) (seconds) (seconds)
1 211073040 / 805.180 423.281554 423.281502
2 211110775 / 805.324 447.063839 447.061432
3 211255349 / 805.875 420.785212 420.785106
All three mkfs runs and subsequent NTFS information checks returned 0.
Nevertheless, each run recorded target udev hung-task reports and one
associated udev watchdog restart; the service recovered afterwards.
The three runs contained seven hung-task messages in total. These are
repeated reports of the waiting tasks, not seven independent failures.
No soft lockup, RCU stall, media I/O error or new AER was recorded in
this baseline batch. Other device-event latency was not quantified.
A shortened stack from the first run is:
INFO: task systemd-udevd:95598 blocked for more than 122 seconds.
task:systemd-udevd state:D
__mutex_lock.constprop.0
__mutex_lock_slowpath
mutex_lock
blkdev_get_by_dev.part.0
blkdev_get_by_dev
blkdev_open
do_dentry_open
The same task was subsequently reported blocked for over 245 seconds.
The diagnostic mutex wait was 423281 ms, agreeing with the paired udev
open at millisecond resolution. The entire open includes more than the
mutex wait, so it is not itself an exact mutex measurement. Likewise,
120 seconds is a hung-task reporting threshold, not a timed mutex that
returns an error after 120 seconds.
Where the time is spent
------------------------
The examined vendor block release path is:
mkfs final file release Concurrent udev open
blkdev_release blkdev_open
blkdev_put blkdev_get_by_dev
sync before lock where applicable mutex_lock(open_mutex)
mutex_lock(open_mutex) -> sleeps waiting
blkdev_put_whole
last opener reaches zero
blkdev_flush_mapping
sync_blockdev
kill_bdev
invalidate_bh_lrus
truncate_inode_pages
mutex_unlock(open_mutex) -> can proceed
The last-opener check is significant: the existing final cleanup is
performed when the opener count reaches zero. A new opener is serialized
by the same disk mutex while that cleanup completes. The proposed hook,
in contrast, is per struct file and is not a last-device-opener check.
The diagnostic logs bracket cache teardown and show the target mapping
falling from the page counts above to zero. Paired tracing places nearly
all the final release interval inside truncate_inode_pages. The final
sync was recorded as 0 ms at the diagnostic timer's resolution; this is
not zero cost or proof that every page was always clean. A completed
sync need not evict clean cache pages or remove all associated metadata,
so finishing writes does not eliminate the later cache teardown work.
The cache is a shared block-device mapping, not mkfs process-private RSS.
This supports cache teardown under lifecycle exclusion as the bottleneck
in these runs. It does not establish the separate CPU cost of every BH,
shadow entry or traversal, or prove the same delay on every system.
Proposed direction and scope of this RFC
----------------------------------------
Try vfs_fsync followed by vfs_fadvise(DONTNEED) before fsnotify_close and
the original release, outside open_mutex at the hook. Keep the original
release to clean up remaining pages and concurrent activity. Allow cache
residue; no zero-page invariant or fixed timeout bound is promised.
The objective is to reduce how much cleanup other openers wait behind,
not to eliminate reclaim cost or guarantee a shorter complete close.
The proposed hook applies to block-device files that have a fsync
operation, excluding O_PATH handles. It runs at the final reference to
each such struct file, including read-only and O_DIRECT opens; this is
not a last-device-opener hook. Syncing and evicting a shared mapping at
that point can affect other users. Sync/reclaim may be long and run in
task work, exit or delayed fput context. These are unresolved policy and
placement concerns. Placing the hook before fsnotify_close also defers
notification, which must be distinguished from reduced mutex hold time.
The main questions are whether __fput is an acceptable hook, whether
this should instead remain an explicit userspace action, and whether
a different block close locking design would better address the issue.
If a kernel pre-reclaim policy is appropriate, what eligibility and
accounting could prevent repeated work and interference with other users?
Validation boundaries
----------------------
The implementation is based on the recorded vendor Linux 6.6 source
snapshot. Patch application and comparison with the prepared source
files passed. checkpatch reports 0 errors and one new-file MAINTAINERS
warning. Current mainline has not been used to reproduce or validate
this RFC, and no additional build or runtime test was run while preparing
this posting. The three baseline runs above characterize the problem;
they are not performance or safety results for the proposed patch.
This posting does not claim that all workloads are free of regressions,
or that the change fixes soft lockups or RCU stalls. Buffered-cache and
concurrent-open behavior need dedicated validation.
If the direction is appropriate, the next step is to rebase to the
agreed current tree and validate lifecycle contexts, concurrent buffered
I/O and open/close, error/removal paths, lockdep, CONFIG_BLOCK=n and
complete close and command cost. I would appreciate design feedback
before pursuing a mergeable implementation.
Thanks,
Ethan
Ethan (1):
block: reclaim cache before block file close notification
block/Makefile | 2 ++
block/bdev-preclose.c | 29 +++++++++++++++++++++++++++++
fs/file_table.c | 4 ++++
include/linux/bdev-preclose.h | 14 ++++++++++++++
4 files changed, 49 insertions(+)