Re: [PATCH 0/1] sched: Restore PREEMPT_NONE as default

[Andres Freund]

Hi,

On 2026-04-08 20:08:15 +0000, Salvatore Dipietro wrote:
> On 2026-04-04 17:42 UTC, Andres Freund wrote:
> > Salvatore, could you repeat that benchmark in some variations?
> > 1) Use huge pages
>
> Enabling Transparent Huge page on the system and on the Postgres configuration,
> the regression disappears and both system has a throughput in the 185k tps range.
> Looking at /proc/vmstat, I can see high volume of minor fault page rate that
> indicate the memory pressure when HP are not enabled.
>
> | Instance      | Arch | Baseline   | Preempt None  | Ratio |
> |               |      |            |               |       |
> | m8g.24xlarge  | ARM  | 186,664.56 | 189,934.34    | 1.01x |

Phew, that's a relief.


> On 2026-04-05  1:40 UTC, Andres Freund wrote:
> > Now, this machine is smaller and a different arch, so who knows.
>
> To compare the results, I have run the same reproducer with huge page off on
> different instances architecture and size. I can see that, in most of the cases,
> regression is present. In particular for Graviton, increasing the instance
> size, increases the regression as well since it creates more contention on
> the resources.
>
> | Instance     | Arch | Baseline  | Preempt None | Ratio |
> |              |      |           |              |       |
> | m8g.2xlarge  | ARM  | 23,438.98 | 21,378.73    | 0.91x |
> | m8g.4xlarge  | ARM  | 40,843.86 | 42,496.78    | 1.04x |
> | m8g.8xlarge  | ARM  | 49,096.64 | 85,796.66    | 1.75x |
> |              |      |           |              |       |
> | m7i.2xlarge  | x86  | 16,615.54 | 23,381.16    | 1.40x |
> | m7i.4xlarge  | x86  | 28,759.26 | 32,758.62    | 1.14x |
> | m7i.8xlarge  | x86  | 73,456.28 | 83,419.36    | 1.14x |
> | m7i.24xlarge | x86  | 63,489.67 | 67,314.40    | 1.06x |

Were these all run with the same client counts?  I'm a bit baffled that on arm
the regression is most pronounced on the large end, while it seems smaller on
the large end on x86.



> On 2026-04-05  1:40 UTC, Andres Freund wrote:
> > Could you run something like the following while the benchmark is running:
> >  SELECT backend_type, wait_event_type, wait_event, state, count(*) FROM pg_stat_activity where wait_event_type NOT IN ('Activity') GROUP BY backend_type, wait_event_type, wait_event, state order by count(*) desc \watch 1
> > and show what you see at the time your profile shows the bad contention?
>
> On baseline, I constantly see SpinDelay as first record with significantly higher count
> than the other wait event types while, with the patch, WALWrite is constantly
> the first record.
>
> Baseline:
>
>   backend_type  | wait_event_type |      wait_event      |        state        | count
> ----------------+-----------------+----------------------+---------------------+-------
>  client backend | Timeout         | SpinDelay            | active              |   838
>  client backend | LWLock          | WALWrite             | idle in transaction |    10
>  client backend | Client          | ClientRead           | idle in transaction |     4
>  client backend | LWLock          | WALWrite             | active              |     3
>  client backend | Timeout         | SpinDelay            | idle                |     2
>  client backend | Client          | ClientRead           | idle                |     1
>  client backend | Client          | ClientRead           | active              |     1
>  client backend | IO              | WalSync              | idle in transaction |     1
>  checkpointer   | Timeout         | CheckpointWriteDelay |                     |     1
> (9 rows)

Makes sense.  Since this not low-level-postgres-knowledge group, it's probably
worth pointing out that this wait event will only be reported while in
exponential backoff for a spinlock, not while actively spinning on the lock
(it'd be too expensive to set it when just delayed on a spinlock for a handful
of cycles).

So what this likely signifies is that the lock holder got scheduled out while
encountering a minor fault tlb miss, the spinlocks entered exponential backoff
after spinning for a bit, and things don't really progress until the lock
holder is scheduled again.

The fact it doesn't happen with huge pages seems to pretty reliably indicate
that it is related to the page faults happening with the spinlock held.


> With patch (PREEMPT_NONE):
>
>
>   backend_type  | wait_event_type |      wait_event      |        state        | count
> ----------------+-----------------+----------------------+---------------------+-------
>  client backend | LWLock          | WALWrite             | active              |   922
>  client backend | IPC             | ProcarrayGroupUpdate | active              |    26
>  client backend | Client          | ClientRead           | active              |    24
>  client backend | IO              | DataFileRead         | active              |    11
>  client backend | LWLock          | WALWrite             | idle                |     5
>  client backend | Timeout         | SpinDelay            | active              |     4
>  client backend | IO              | DataFileWrite        | active              |     3
>  client backend | IO              | WalSync              | active              |     2
>  client backend | LWLock          | WALWrite             | idle in transaction |     1
>  walwriter      | LWLock          | WALWrite             |                     |     1
>  checkpointer   | IO              | DataFileSync         |                     |     1
>  client backend | IO              | DataFileRead         | idle                |     1
> (12 rows)

FWIW, everything being stalled on WALWrite indicates not directly lock
contention but waiting for WAL flushes to complete (yes, we ought to make that
difference easier to understand).  Which is what you'd expect in this workload
with remote storage.


> On 2026-04-07 11:19 UTC, Mark Rutland wrote:
> > Salvatore, was there a specific reason to test with PG_HUGE_PAGES=off
> > rather than PG_HUGE_PAGES=try?
>
> We test with various configurations to ensure customers don't encounter
> regressions regardless of their setup choices, even if some configurations
> aren't optimal for maximum performance.

There's not optimal and there's a 100GB cold buffer pool with 10x more active
connections than CPU cores, with 4k pages, and only ever running during the
warm up period, never achieving steady state (where the spinlocked path never
is used) :)

(Note that in real OLTP systems with a similar number of busy clients you'll
typically have far fewer actually active connections due to network latency to
the client and reaction time in the client - a locally run pgbench will show
this drastically more than real scenarios.)


If you were to find the same regression with a smaller buffer pool and a not
quite so crazily overloaded machine, I'd be much more concerned, because at a
few GB of buffer pool it's a lot more common and more reasonable to not have
huge pages configured - but you're also going to have a hard time staying in
the warm up period for long enough to matter...


Greetings,

Andres Freund