Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O
From: Joanne Koong
Date: Mon Jul 06 2026 - 15:27:46 EST
On Mon, Jul 6, 2026 at 4:10 AM changfengnan <changfengnan@xxxxxxxxxxxxx> wrote:
>
> > From: "Christoph Hellwig"<hch@xxxxxxxxxxxxx>
> > Date: Fri, Jul 3, 2026, 20:40
> > Subject: Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O
> > To: "changfengnan"<changfengnan@xxxxxxxxxxxxx>
> > Cc: "Joanne Koong"<joannelkoong@xxxxxxxxx>, "Christoph Hellwig"<hch@xxxxxxxxxxxxx>, "brauner"<brauner@xxxxxxxxxx>, "djwong"<djwong@xxxxxxxxxx>, "ojaswin"<ojaswin@xxxxxxxxxxxxx>, "dgc"<dgc@xxxxxxxxxx>, "linux-xfs"<linux-xfs@xxxxxxxxxxxxxxx>, "linux-fsdevel"<linux-fsdevel@xxxxxxxxxxxxxxx>, "linux-ext4"<linux-ext4@xxxxxxxxxxxxxxx>, "linux-kernel"<linux-kernel@xxxxxxxxxxxxxxx>, <lidiangang@xxxxxxxxxxxxx>, "pankaj.raghav"<pankaj.raghav@xxxxxxxxx>, "Brian Foster"<bfoster@xxxxxxxxxx>
> > On Fri, Jul 03, 2026 at 05:13:12PM +0800, changfengnan wrote:
> > > taken, which introduced some overhead.
> > > I implemented some optimizations based on Christoph’s suggestions and
> > > made some modifications to `iomap_process`;
> > > Now, XFS performance is essentially unchanged, while ext4 still shows a
> > > 1.5% drop; I will continue to investigate the cause.. I’ve attached the patch.
> > > Actually, I had expected that switching to `iomap_next` would improve
> > > performance, since it avoids an indirect call.
> >
> > I don't think it will for you - the direct I/O readers don't have
> > iomap_end methods, so you still have the same number of indirect
> > calls. What could help is to inline the iterator as seen in Joannes'
> > fist demo series. We could even look into doing that only for simple
> > dio in a first step, shifting the burden to use the simple method
> > to the callers (at least for the POC).
>
> I'm not sure if this is what you had in mind, but I made some
> adjustments referencing Joannes' first patch, and using this approach,
> there was no performance degradation.
Thanks, Fengnan. This is awesome that it got the degradation down to zero.
Would you be able to test how this change performs on your benchmarks?:
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index dbe073e181a7..bf0c18cf1017 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,8 @@ static ssize_t ext4_dio_read_iter(struct kiocb
*iocb, struct iov_iter *to)
return generic_file_read_iter(iocb, to);
}
- ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
+ ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL,
+ IOMAP_DIO_NO_IOMAP_END, NULL, 0);
inode_unlock_shared(inode);
file_accessed(iocb->ki_filp);
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index bcd3d4223464..bc3b87e57592 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -968,115 +968,38 @@ static void iomap_dio_simple_end_io(struct bio *bio)
iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
}
-static inline bool
-iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_dio_ops *dops,
- unsigned int dio_flags, size_t done_before)
+static inline void
+iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_next_fn iomap_next,
+ size_t written)
{
- struct inode *inode = file_inode(iocb->ki_filp);
- size_t count = iov_iter_count(iter);
-
- if (dops || done_before)
- return false;
- if (iov_iter_rw(iter) != READ)
- return false;
- if (!count)
- return false;
- /*
- * Simple dio is an optimization for small IO. Filter out large IO
- * early as it's the most common case to fail for typical direct IO
- * workloads.
- */
- if (count > inode->i_sb->s_blocksize)
- return false;
- if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
- IOMAP_DIO_BOUNCE))
- return false;
- if (iocb->ki_pos + count > i_size_read(inode))
- return false;
- if (IS_ENCRYPTED(inode))
- return false;
-
- return true;
+ iomi->iter_start_pos = iomi->pos;
+ iomi->pos += written;
+ iomi->len -= written;
+ iomap_next(iomi, &iomi->iomap, &iomi->srcmap);
}
-/*
- * Fast path for small, block-aligned direct I/Os that map to a single
- * contiguous on-disk extent.
- *
- * iomap_dio_simple_supported() enforces the cheap up-front constraints before
- * entering this path.
- *
- * @dops must be NULL: a non-NULL @dops means the caller wants its
- * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
- * allocated from the filesystem-private @dops->bio_set (whose
front_pad sizes a
- * filesystem-private wrapper around the bio). The fast path instead allocates
- * from the shared iomap_dio_simple_pool, whose front_pad matches struct
- * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
- * back to __iomap_dio_rw() in that case.
- *
- * @done_before must be zero: a non-zero caller-accumulated residual cannot be
- * carried through a single-bio inline completion.
- *
- * @iter must describe a non-empty READ no larger than the inode block size:
- * writes, zero-length I/O, and larger requests need the generic iomap direct
- * I/O path.
- *
- * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
- * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
- * I/O, or bouncing. The range must also stay within i_size and encrypted
- * inodes must use the generic iomap direct I/O path.
- *
- * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
- * decides the request does not fit the fast path. In that case we proceed to
- * the generic __iomap_dio_rw() slow path. Any other errno is a real
result and
- * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
- * caller.
- */
-static ssize_t
-iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
- iomap_next_fn iomap_next, void *private,
- unsigned int dio_flags)
+ssize_t
+__iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+ struct iomap_iter *iomi, iomap_next_fn iomap_next,
+ unsigned int dio_flags)
{
struct inode *inode = file_inode(iocb->ki_filp);
size_t count = iov_iter_count(iter);
bool wait_for_completion = is_sync_kiocb(iocb);
- struct iomap_iter iomi = {
- .inode = inode,
- .pos = iocb->ki_pos,
- .len = count,
- .flags = IOMAP_DIRECT,
- .private = private,
- };
struct iomap_dio_simple *sr;
unsigned int alignment;
struct bio *bio;
ssize_t ret;
- if (iocb->ki_flags & IOCB_NOWAIT)
- iomi.flags |= IOMAP_NOWAIT;
-
- ret = kiocb_write_and_wait(iocb, count);
- if (ret)
- return ret;
-
- inode_dio_begin(inode);
-
- ret = iomap_iter(&iomi, iomap_next);
- if (ret <= 0) {
- inode_dio_end(inode);
- return ret ? ret : -EFAULT;
- }
-
- if (iomi.iomap.type != IOMAP_MAPPED ||
- iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
- (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
+ if (iomi->iomap.type != IOMAP_MAPPED ||
+ iomi->iomap.offset + iomi->iomap.length < iomi->pos + count ||
+ (iomi->iomap.flags & IOMAP_F_INTEGRITY)) {
ret = -ENOTBLK;
goto out_iomap_end;
}
- alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
- if ((iomi.pos | count) & (alignment - 1)) {
+ alignment = iomap_dio_alignment(inode, iomi->iomap.bdev, dio_flags);
+ if ((iomi->pos | count) & (alignment - 1)) {
ret = -EINVAL;
goto out_iomap_end;
}
@@ -1092,14 +1015,14 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
if (user_backed_iter(iter))
dio_flags |= IOMAP_DIO_USER_BACKED;
- bio = bio_alloc_bioset(iomi.iomap.bdev,
+ bio = bio_alloc_bioset(iomi->iomap.bdev,
bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
sr = container_of(bio, struct iomap_dio_simple, bio);
sr->iocb = iocb;
sr->dio_flags = dio_flags;
- bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
+ bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
bio->bi_ioprio = iocb->ki_ioprio;
ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
@@ -1124,8 +1047,8 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
WRITE_ONCE(iocb->private, bio);
}
- iomap_iter_advance(&iomi, count);
- iomap_iter(&iomi, iomap_next);
+ if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+ iomap_dio_simple_finish(iomi, iomap_next, count);
if (!wait_for_completion) {
bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1142,33 +1065,12 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
out_bio_put:
bio_put(bio);
out_iomap_end:
- iomap_iter(&iomi, iomap_next);
+ if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+ iomap_dio_simple_finish(iomi, iomap_next, 0);
inode_dio_end(inode);
return ret;
}
-
-ssize_t
-iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
- unsigned int dio_flags, void *private, size_t done_before)
-{
- struct iomap_dio *dio;
- ssize_t ret;
-
- if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
- done_before)) {
- ret = iomap_dio_simple(iocb, iter, iomap_next,
private, dio_flags);
- if (ret != -ENOTBLK)
- return ret;
- }
-
- dio = __iomap_dio_rw(iocb, iter, iomap_next, dops, dio_flags, private,
- done_before);
- if (IS_ERR_OR_NULL(dio))
- return PTR_ERR_OR_ZERO(dio);
- return iomap_dio_complete(dio);
-}
-EXPORT_SYMBOL_GPL(iomap_dio_rw);
+EXPORT_SYMBOL_GPL(__iomap_dio_simple);
static int __init iomap_dio_init(void)
{
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index 984045af310a..1128f5b526ef 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -89,9 +89,6 @@ int iomap_iter_continue(const struct iomap_iter
*iter, struct iomap *iomap,
bool stale = iomap->flags & IOMAP_F_STALE;
ssize_t advanced = iter->pos - iter->iter_start_pos;
- if (!iomap->length)
- return 1;
-
/*
* Use iter->len to determine whether to continue onto the next mapping.
* Explicitly terminate on error status or if the current iter has not
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47..e43ca419b13a 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -127,7 +127,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
{IOMAP_DIO_FORCE_WAIT, "DIO_FORCE_WAIT" }, \
{IOMAP_DIO_OVERWRITE_ONLY, "DIO_OVERWRITE_ONLY" }, \
{IOMAP_DIO_PARTIAL, "DIO_PARTIAL" }, \
- {IOMAP_DIO_FSBLOCK_ALIGNED, "DIO_FSBLOCK_ALIGNED" }
+ {IOMAP_DIO_FSBLOCK_ALIGNED, "DIO_FSBLOCK_ALIGNED" }, \
+ {IOMAP_DIO_NO_IOMAP_END, "DIO_NO_IOMAP_END" }
DECLARE_EVENT_CLASS(iomap_class,
TP_PROTO(struct inode *inode, struct iomap *iomap),
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index a987ffbf3c02..6a551efff66f 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,7 +251,7 @@ xfs_file_dio_read(
struct iov_iter *to)
{
struct xfs_inode *ip = XFS_I(file_inode(iocb->ki_filp));
- unsigned int dio_flags = 0;
+ unsigned int dio_flags = IOMAP_DIO_NO_IOMAP_END;
const struct iomap_dio_ops *dio_ops = NULL;
ssize_t ret;
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 3b41f123a92d..6213a3b1a0b9 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -10,6 +10,7 @@
#include <linux/mm_types.h>
#include <linux/blkdev.h>
#include <linux/folio_batch.h>
+#include <linux/pagemap.h>
struct address_space;
struct fiemap_extent_info;
@@ -605,15 +606,136 @@ struct iomap_dio_ops {
*/
#define IOMAP_DIO_BOUNCE (1 << 4)
-ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
- unsigned int dio_flags, void *private, size_t done_before);
+/* optimization hint for the simple fast path */
+#define IOMAP_DIO_NO_IOMAP_END (1 << 5)
+
struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before);
ssize_t iomap_dio_complete(struct iomap_dio *dio);
void iomap_dio_bio_end_io(struct bio *bio);
+static inline bool
+iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
+ const struct iomap_dio_ops *dops,
+ unsigned int dio_flags, size_t done_before)
+{
+ struct inode *inode = file_inode(iocb->ki_filp);
+ size_t count = iov_iter_count(iter);
+
+ if (dops || done_before)
+ return false;
+ if (iov_iter_rw(iter) != READ)
+ return false;
+ if (!count)
+ return false;
+ if (count > inode->i_sb->s_blocksize)
+ return false;
+ if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
+ IOMAP_DIO_BOUNCE))
+ return false;
+ if (iocb->ki_pos + count > i_size_read(inode))
+ return false;
+ if (IS_ENCRYPTED(inode))
+ return false;
+ return true;
+}
+
+ssize_t __iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+ struct iomap_iter *iomi, iomap_next_fn iomap_next,
+ unsigned int dio_flags);
+
+/*
+ * Fast path for small, block-aligned direct I/Os that map to a single
+ * contiguous on-disk extent.
+ *
+ * iomap_dio_simple_supported() enforces the cheap up-front constraints before
+ * entering this path.
+ *
+ * @dops must be NULL: a non-NULL @dops means the caller wants its
+ * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
+ * allocated from the filesystem-private @dops->bio_set (whose
front_pad sizes a
+ * filesystem-private wrapper around the bio). The fast path instead allocates
+ * from the shared iomap_dio_simple_pool, whose front_pad matches struct
+ * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
+ * back to __iomap_dio_rw() in that case.
+ *
+ * @done_before must be zero: a non-zero caller-accumulated residual cannot be
+ * carried through a single-bio inline completion.
+ *
+ * @iter must describe a non-empty READ no larger than the inode block size:
+ * writes, zero-length I/O, and larger requests need the generic iomap direct
+ * I/O path.
+ *
+ * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
+ * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
+ * I/O, or bouncing. The range must also stay within i_size and encrypted
+ * inodes must use the generic iomap direct I/O path. IOMAP_DIO_NO_IOMAP_END is
+ * a fast-path optimization the caller can set if there is no work
that needs to
+ * be done after a mapping.
+ *
+ * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
+ * decides the request does not fit the fast path. In that case we proceed to
+ * the generic __iomap_dio_rw() slow path. Any other errno is a real
result and
+ * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
+ * caller.
+ */
+static __always_inline ssize_t
+iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+ iomap_next_fn iomap_next, void *private, unsigned int dio_flags)
+{
+ struct inode *inode = file_inode(iocb->ki_filp);
+ size_t count = iov_iter_count(iter);
+ struct iomap_iter iomi = {
+ .inode = inode,
+ .pos = iocb->ki_pos,
+ .len = count,
+ .flags = IOMAP_DIRECT,
+ .private = private,
+ };
+ ssize_t ret;
+
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ iomi.flags |= IOMAP_NOWAIT;
+
+ ret = kiocb_write_and_wait(iocb, count);
+ if (ret)
+ return ret;
+
+ inode_dio_begin(inode);
+
+ ret = iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
+ if (ret <= 0) {
+ inode_dio_end(inode);
+ /* a zero return means no mapping, which should never happen */
+ return ret ? ret : -EFAULT;
+ }
+
+ return __iomap_dio_simple(iocb, iter, &iomi, iomap_next, dio_flags);
+}
+
+static __always_inline ssize_t
+iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_next_fn
iomap_next,
+ const struct iomap_dio_ops *dops, unsigned int dio_flags,
+ void *private, size_t done_before)
+{
+ struct iomap_dio *dio;
+ ssize_t ret;
+
+ if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
done_before)) {
+ ret = iomap_dio_simple(iocb, iter, iomap_next, private,
+ dio_flags);
+ if (ret != -ENOTBLK)
+ return ret;
+ }
+
+ dio = __iomap_dio_rw(iocb, iter, iomap_next, dops, dio_flags, private,
+ done_before);
+ if (IS_ERR_OR_NULL(dio))
+ return PTR_ERR_OR_ZERO(dio);
+ return iomap_dio_complete(dio);
+}
+
#ifdef CONFIG_SWAP
struct file;
struct swap_info_struct;
@@ -678,21 +800,22 @@ static __always_inline int iomap_process(const
struct iomap_iter *iter,
{
int ret = 0;
- if (iomap->length && end) {
- ssize_t advanced = iter->pos - iter->iter_start_pos;
- loff_t len;
+ if (iomap->length) {
+ if (end) {
+ ssize_t advanced = iter->pos - iter->iter_start_pos;
+ loff_t len;
- len = iomap_length_trim(iter, iter->iter_start_pos,
- iter->len + advanced);
+ len = iomap_length_trim(iter, iter->iter_start_pos,
+ iter->len + advanced);
- ret = end(iter->inode, iter->iter_start_pos, len, advanced,
- iter->flags, iomap);
+ ret = end(iter->inode, iter->iter_start_pos,
len, advanced,
+ iter->flags, iomap);
+ }
+ ret = iomap_iter_continue(iter, iomap, srcmap, ret);
+ if (ret <= 0)
+ return ret;
}
- ret = iomap_iter_continue(iter, iomap, srcmap, ret);
- if (ret <= 0)
- return ret;
-
ret = begin(iter->inode, iter->pos, iter->len, iter->flags, iomap,
srcmap);
I put the changes in a git tree [1] in case that's is more convenient
for you. I think this ends up devirtualizing the iomap_next callback:
before:
$ objdump -dr fs/xfs/xfs.ko | awk '/<xfs_file_dio_read>:/{f=1}
f{print} f&&/^$/{exit}' | grep -E 'iomap_next|iomap_begin|thunk'
a9ac3: R_X86_64_PLT32 __x86_return_thunk-0x4
a9b4b: R_X86_64_32S xfs_read_iomap_next
a9b9c: R_X86_64_PLT32 __x86_return_thunk-0x4
after:
$ objdump -dr fs/xfs/xfs_iomap_next_inlined.ko | awk
'/<xfs_file_dio_read>:/{f=1} f{print} f&&/^$/{exit}' | grep -E
'iomap_next|iomap_begin|thunk'
aa083: R_X86_64_PLT32 __x86_return_thunk-0x4
aa1d4: R_X86_64_PLT32 xfs_read_iomap_next-0x4
aa1ec: R_X86_64_32S xfs_read_iomap_next
aa232: R_X86_64_32S xfs_read_iomap_next
I was unsure how much bloat this adds but I didn't see a noticable
impact when I ran ./scripts/bloat-o-meter:
$ ./scripts/bloat-o-meter fs/xfs/xfs.ko fs/xfs/xfs_iomap_next_inlined.ko
add/remove: 0/0 grow/shrink: 11/1 up/down: 589/-4 (585)
Function old new delta
xfs_file_dio_read 309 685 +376
xfs_file_dio_write_aligned 355 402 +47
xfs_file_dio_write_unaligned 545 591 +46
xfs_file_dio_write_atomic 457 502 +45
xfs_dax_write_iomap_next 267 291 +24
xfs_atomic_write_cow_iomap_next 1151 1175 +24
xfs_zoned_direct_write_iomap_next 217 224 +7
xfs_direct_write_iomap_next 77 84 +7
xfs_buffered_write_iomap_next 373 379 +6
xfs_xattr_iomap_next 360 365 +5
xfs_seek_iomap_next 665 667 +2
xfs_read_iomap_next 681 677 -4
Total: Before=1977941, After=1978526, chg +0.03%
where the baseline xfs.ko was built at the commit in [2] and the
xfs_iomap_next_inlined.ko was built at the commit in [3]. If these
changes are acceptable to you, then I'll put this into v3 of the
iomap_next series.
Thanks,
Joanne
[1] https://github.com/joannekoong/linux/commits/iomap_simple_dio_iomap_next/
[2] https://github.com/joannekoong/linux/commit/1e82c8988ee56e9e7202d5ee09c7f7ea7ac7a832
[3] https://github.com/joannekoong/linux/commit/86ecc9de53bb1b8b1287950fbb745a5a8e49f736