Re: [RFC PATCH 0/3] Neural Storage Driver - learning page cache prefetcher

[Jan Kara]

Hi Ayhan,

I'm sorry for a bit delayed reply.

On Thu 20-08-26 08:26:31, ayhan aydın wrote:
> Thanks for the review and for the questions — they pushed us to go back
> and measure this properly. Since the last exchange we ran a full
> window-size matrix with complete latency and overhead numbers, and it
> answers directly what both of you asked. Let me start with the data.
> 
> These results come from our current in-development tree. We wanted to
> share them with you before pushing anything, so the tree referenced in
> the RFC (upstream-prep-v1, 2026-08-17) is still the earlier snapshot.
> Since that snapshot we made three changes that led to these numbers:
> 
>   1. Jump handling — when a prediction lands outside the kernel's
>      readahead window we now always issue WILLNEED|NOREUSE explicitly,
>      instead of leaving the strategy branch to decide.
>   2. Kernel-window awareness — when the kernel's own readahead window
>      is already >= 1MB the module detects this and completely silences
>      itself (confirmed by its own counters), so it never competes with
>      or disturbs a kernel window that is already sufficient.

Err, ok. I guess I'm somewhat missing the intended purpose of your changes
then. Is your idea that if read_ahead_kb (i.e., maximum allowed readahead)
is set too low, your algorithm will fix it up?

How I was understanding your intention is that your algorithm aims to
provide a better guess for appropriate readahead given the workload *within
the bounds set by read_ahead_kb*. Because read_ahead_kb is there for
sysadmin to allow restricting readahead (e.g. to avoid excessive memory
consumption from readahead).

>   3. Full instrumentation — per-run p50/p95/p99/p999 latency, per-pass
>      usr/sys CPU overhead, and disk-I/O accounting.
> 
> We'll push the updated tree to GitHub once this discussion settles.
> 
> Full metrics matrix — methodology: interleaved ON/OFF on the same
> machine, drop_caches before every run, 3 runs per configuration, 6.2 GB
> SQLite database and an 8 GB sequential file. Disk I/O is byte-identical
> between ON and OFF in all 96 runs — NSD never issues extra I/O.
> 
> seq64 — 8 GB sequential file, 64k requests, n=131K/run:
> 
>   kb    | OFF avg  ON avg   diff  | p50 OFF->ON  | p99 OFF->ON   |
> overhead (usr+sys)
>   128   | 22.02s   17.11s   -22%  | 50->23 us     | 812->805 us   | 5.94->4.33s
>   256   | 17.08s   17.22s   +1%   | 20->15 us     | 824->1129 us  | 4.07->4.13s
>   512   | 16.81s   16.96s   +1%   | 16->14 us     | 1092->1903 us | 4.24->3.60s
>   1024  | 17.25s   17.03s   -1%   | 15->16 us     | 1903->1883 us | 3.67->3.74s

OK, so for read_ahead_kb >= 256k the difference disappears with your disk.

> SQLite Q2 full-table scan — 6.2 GB:
> 
>   kb    | OFF avg  ON avg   diff  | overhead (usr+sys)
>   128   | 24.51s   20.48s   -16%  | +0.95s
>   256   | 20.05s   16.60s   -17%  | +1.26s
>   512   | 15.31s   12.18s   -20%  | +1.06s
>   1024  | 12.22s   12.09s   -1%   | +0.26s

Interesting that here the difference disappears only with read_ahead_kb =
1024k. Isn't it because based on 2. above your algorithm disabled itself?
Also how come when your algorithm is enabled, results were still improving
as you were increasing read_ahead_kb?

> random4k — 100k random 4k requests:
> 
>   kb    | OFF avg  ON avg   diff  | p99
>   128   | 25.0s    25.5s    +2%   | ~0.98ms
>   256   | 26.9s    26.0s    -3%   | ~0.99ms
>   512   | 25.7s    25.5s    -1%   | ~0.98ms
>   1024  | 25.4s    24.7s    -3%   | ~0.99ms
> 
> random_repeat — 100k offsets x 3 passes, n=300K:
> 
>   kb    | OFF avg  ON avg   diff  | p99
>   128   | 26.8s    25.1s    -6%   | 0.39ms
>   256   | 25.1s    24.0s    -4%   | 0.38ms
>   512   | 24.5s    24.4s    0%    | 0.37ms
>   1024  | 25.0s    24.7s    -1%   | 0.39ms

I'd say no matterial difference here with read_ahead_kb >= 256k

> Honza, you asked whether the difference goes away at 1M — and yes, it
> does, for both workloads. The 128k default is the dominant factor; at
> 1M the kernel alone already reaches this SSD's bandwidth ceiling, and
> NSD recognises that and silences itself entirely at >= 1M windows.
> 
> But the 512k row is the one I'd really like your view on. The kernel
> with a 512k window alone (15.31s) trails the kernel with 1M (12.22s)
> by about 25%, yet NSD with a 512k window reaches 12.18s — the same
> throughput at half the window. Our block-layer traces show OFF@512k
> keeps issuing ~512k requests, so the device is not the limit. It looks
> like the kernel's window growth is capped at ra_pages, while the
> fadvise-driven expansion NSD triggers is not — which, incidentally, is
> why distros end up hand-tuning this value. Is that cap deliberate, for
> latency or cache-pollution control, or is there room for the growth
> policy to self-adapt?

I'm not sure I correctly understand the question so please rephrase it if I
answer something else but it is deliberate, that explicit readahead (e.g.
through fadvise(2) but also if you just do large read(2)) is not bound by
read_ahead_kb.

> Willy, on the architecture: hooking only vfs_read and skipping the
> fault path was a deliberate choice to bound the first iteration, but
> you're right that it's a gap — filemap_fault bypasses vfs_read
> entirely, so we're currently blind to mmap'd workloads. Hooking it
> with the same fprobe is the next step. On replacing the PG_readahead
> folio marking: the matrix above convinced us this is the right
> direction. The kernel's growth policy, not the device, is the real
> bottleneck at default settings, so we're moving toward a design that
> manages ra_state directly and measures its own waste — per-file
> tracking of consumed versus expired prefetches with adaptive
> aggressiveness, which is really just replacing the implicit
> assumptions behind PG_readahead marking with explicit measurement.
> Backward access patterns are already handled by the stride predictor
> (negative deltas, forward/backward counted separately); a dedicated
> backward-pattern benchmark will be added to the suite.

Just one note here: I wouldn't focus too much on default settings because
most people tune them to different values with udev rules. What is
interesting is whether your algorithm beats current readahead algorithm
with proper read_ahead_kb for the device (i.e., usually 1m these days).

> Two things we're still working on, for honesty: at 256-512k the
> sequential p99 regresses (~0.8ms -> ~1.1ms and ~1.1ms -> ~1.9ms) as
> the prefetch regions push the device queue — the usage-tracking work
> above targets exactly that. And the random_repeat gains (4-6% at
> 128-256k) suggest the predictor does help beyond what window size can
> fix; p95 there improves from 296 to 243 us.
> 
> Questions we'd value your input on:
>   1. Is fadvise-driven expansion beyond ra_pages acceptable as a
>      mechanism, or should the growth policy itself change?

Not sure what Matthew thinks but my opinion is that automated readahead
(regardless of algorithm used for deciding it) simply should be limited to
read_ahead_kb. That is the long-term Linux behavior and that is the knob
sysadmin has to constrain readahead algorithm if he wishes and we should
not break it.

>   2. What would a minimal integrated design look like — replacing
>      PG_readahead marking with direct ra_state control?

I'll defer this to Matthew since I'm actually not sure how he imagined this
to work.

>   3. If we iterate: fault path first, or usage-tracking first?

I don't have a strong preference here :)

								Honza
-- 
Jan Kara <jack@xxxxxxxx>
SUSE Labs, CR