[RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322

[孙魁 (Kui Sun)]

Dear Kernel Maintainers,

During our upgrade from Linux kernel 5.15 to Linux kernel 6.18, we observed a significant performance regression in random I/O workloads―with a maximum degradation of 27.7%.
This issue is particularly pronounced in single-threaded, small-block I/O scenarios。

To illustrate the impact, we conducted benchmark tests using AnTuTu on Unisoc T615 devices.
The results are summarized below:

Table  1：Random Read/Write Speed Scores
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            20906   20508   21362   20925.33
T615    6.18            20164   21107   21077   20782.67

Table  2：Multi-threaded Mixed Random Read/Write Scores
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            4670    4701    4457    4609.33
T615    6.18            4311    4697    4471    4493.00

Table   3：Mixed Random Read/Write Speed Scores（Single-threaded）
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            18604   18314   17732   18216.67
T615    6.18            13372   13081   13081   13178.00（↓27.66%）

Notably, only the single-threaded test (Table 3) shows severe degradation, while multi-threaded tests exhibit minimal change (<3%).
This strongly suggests the regression is tied to increased per-request scheduling or interrupt overhead in low-concurrency, small-block (e.g., 4KB) I/O paths.

Root Cause Identification

Through investigation, we identified that upstream commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, merged in April 2025, is likely responsible for this performance drop.
After locally reverting this commit on kernel 6.18, performance fully recovered:

Table 4：Mixed Random Read/Write Speed Scores（After Revert）
Device  Kernel Version                  Test1   Test2   Test3   Average
T615    5.15                                    18604   18314   17732   18216.67
T615    6.18                            13372   13081   13081   13178.00（↓27.66%）
T615    6.18（reverted 3c7ac40)  18314   18604   18604   18507.33

Technical Analysis

We believe the change introduced additional interrupt or scheduling latency.
In multi-threaded workloads, a single interrupt can process multiple 4KB requests (e.g., 8 requests), amortizing the scheduling cost to M/8 per request (where M is the total overhead).
In contrast, single-threaded I/O handles only one request per interrupt, incurring the full cost M per operation.
Consequently, single-threaded small I/O is highly sensitive to such latency increases, explaining the disproportionate impact observed in Table 3.

Request and Recommendations

Given the tangible impact on mobile user experience, we kindly request the community to:
1.      Consider reverting commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, or
2.      Re-evaluate the proposed change in light of its effect on low-concurrency I/O paths, as discussed here:
https://lore.kernel.org/lkml/88d31a258feb36425ad73d0323077972f85f8341.camel@xxxxxxxxxx/
3,      Could we add a flag to allow our UFS driver to choose between using an interrupt or an interrupt thread? [1]

Furthermore, we recommend that future evaluations of similar changes include:
*       Small-block (e.g., 4KB or 8KB) random read/write benchmarks, and
*       Single-threaded workloads,
as these are critical for mobile and embedded systems.

We appreciate your attention and are happy to provide additional data or assist in validating potential fixes.

[1]: https://lore.kernel.org/linux-scsi/20260710065948.467514-1-kui.sun@xxxxxxxxxx/T/#u