Re: [PATCH v2 bpf-next 0/3] bpf: Import external pages into an arena
From: Vineeth Pillai
Date: Fri Nov 20 2026 - 08:45:43 EST
Hi Andrea,
Thanks for the v2 and for sharing the scx_pvhost/scx_pvguest demo
branch. Thought of discussing the possibility of using the pvsched
transport with the bpf arena framework implemented in this series.
On Tue, Nov 17, 2026 at 5:30 AM Andrea Righi <arighi@xxxxxxxxxx> wrote:
>
> As discussed in the BPF track and the sched_ext microconference at Linux
> Plumbers 2026 [1][2], shared BPF arenas can provide a generic channel for
> bidirectional zero-copy communication between BPF programs running on a host
> and its guests, for example to exchange scheduling hints between host and
> guest sched_ext schedulers.
>
> v1 [3] exposed a pinned arena as a sized bpffs file and gave the guest its
> share of it as a NUMA node. Following the review, this version drops that
> approach and teaches the arena to map pages that it didn't allocate, based on
> Alexei's prototype [4].
>
> Background
> ==========
>
> An arena can only map pages it allocated itself, so its memory can be shared
> only with the BPF programs and the user space that map the arena.
>
> Sharing it with a guest through a NUMA node, as in v1, has two fundamental
> problems: the node is ordinary guest RAM, so the guest arena's pages go back to
> the guest's buddy allocator when the guest scheduler exits, while the host can
> still write to them; and memory block granularity makes it wasteful (16M of
> guest memory for one shared page) and limits it to a few hundred guests per
> host scheduler.
>
> Design
> ======
>
> The arena gains a single primitive, arena_map_ext_pages(), which maps pages
> supplied from outside at a free arena address. Each call creates one import,
> kept in an interval tree with its pages, which belong to their supplier, not to
> the arena:
>
> - bpf_arena_free_pages() over any of them releases the whole import: the
> pages are unmapped and unpinned, and the range goes back to the arena, so
> the arena doesn't grow as guests come and go,
> - they are released when the arena is destroyed,
> - they are not counted in the arena memory usage,
> - the arena mapping raises SIGBUS on them (SIGSEGV with
> BPF_F_SEGV_ON_FAULT): user space accesses them through their supplier.
>
> Two sleepable kfuncs for syscall programs supply the pages, one for each side
> of a host/guest shared arena:
>
> int bpf_arena_import_memfd(void *map, void __arena *addr,
> u32 page_cnt, int fd, u64 off);
>
> Host side: pins the pages of a memfd, or of any file on tmpfs or hugetlbfs,
> with memfd_pin_folios() like udmabuf does. The host hands the same file to the
> VMM as a memory backend, with no change to the VMM.
>
> int bpf_arena_import_dax(void *map, void __arena *addr,
> u32 page_cnt, u64 src);
>
> Guest side: the VMM exposes the file as a virtio-pmem or NVDIMM device, which
> the guest turns into a device-dax device (/dev/daxX.Y). The kfunc pins the
> device-dax pages mapped at an address of the calling task, and holds a
> reference on their dev_pagemap until it releases them.
>
> Both sides then see the same pages as arena memory, at an offset known to both
> of them, so they can agree on a layout in-band, with plain BPF loads and
> stores, and keep pointers to it in the rest of their arena state.
>
> Use case: paravirt scheduling
> =============================
>
> The git tree below has an additional [DEMO] commit, not meant for upstream,
> that provides a real, tested use case of this functionality: a pair of
> sched_ext schedulers, scx_pvhost on the host and scx_pvguest in every guest,
> which share a few pages per VM through their arenas. The host publishes which
> vCPUs it has preempted, its idle CPUs and every VM's fair share of the host
> CPUs; the guests publish their demand and the state of their vCPUs. With that,
> the guests stop queueing work behind preempted vCPUs and fold onto their share
> of the host CPUs only while that pays off.
>
As Josh mentioned on v1, we have been developing pvsched [5] -- a
lightweight guest/VMM/host paravirt transport (with QEMU and Cloud
Hypervisor support) based on the "guest allocates ordinary RAM, passes
GPAs via a small PCI BAR + synchronous doorbell, host pins via
/dev/pvsched" model. We recently added external-policy support so the
shared pages after a small negotiation header are completely opaque to
the kernel framework.
Looking at Patch 1, arena_map_ext_pages() (along with import_memfd for
host/userspace sharing) already takes an arbitrary struct page ** array
and maps it contiguously into arena VA space. That means for VMs we can
layer your shared-arena model directly on top of the pvsched transport
(on both host and guest) and get the best of both approaches:
1. No extra memory backend or devdax/nvdimm stack:
Instead of a separate 16 MiB pmem file per VM (plus guest ZONE_DEVICE
memmap overhead, libnvdimm/dax_pmem config, and sysfs rebinding) for
sharing a few pages at a hardcoded 2 MiB offset, the guest allocates
ordinary 4 KiB RAM pages (including from the right guest NUMA nodes)
and registers their GPAs in BAR1. Both sides still map that page
list as a flat, contiguous virtual slice in their BPF arenas via
arena_map_ext_pages(), and Patch 2 (import_dax, along with its
dev_pagemap unbind blocking and mem_hotplug_lock ordering) wouldn't
be needed for VMs.
2. Clean lifetime and synchronous revocation:
In the guest, pvsched_guest owns the allocated RAM pages across
guest scheduler reloads, so scx_pvguest exiting never returns shared
pages to the guest buddy allocator while attached; on driver unbind
or kexec, a synchronous DOORBELL=0 detaches on the host first. On
the host, /dev/pvsched detach callbacks know the exact instant a VM
detaches or exits and can synchronously unmap the imported arena
PTEs (relying on ex_handler_bpf zeroing loads/dropping stores),
avoiding stale writes or blocked device unbinds.
3. Up-front negotiation, authoritative vCPU identity, and doorbell:
BAR0 + the page header give an explicit handshake for policy name,
version, and layout before anything goes live, and /dev/pvsched
binds pages directly to VMM runner TIDs and KVM vcpu_ids (avoiding
inotify + /proc/*/maps + thread comm scraping across QEMU, CHV,
Firecracker, crosvm, etc.). BAR0 also provides a kick doorbell and
optional KVM boundary hooks when a policy wants event-driven wakeups
in addition to polling.
4. Consumer-agnostic on both ends:
Decoupling the transport from the consumer lets scx_pvhost <->
scx_pvguest use shared BPF arenas as in your demo, while also
allowing mixed setups -- e.g., keeping stock EEVDF in the guest
(publishing via thin in-kernel hooks without replacing guest task
scheduling, which you noted in scx-pvsched.md's TODO) consumed by a
host sched_ext scheduler or kernel module.
5. Live migration and Confidential VMs:
With memfd + arena->kern_vm, host BPF writes go through init_mm and
aren't tracked by user PTE / KVM dirty logging during pre-copy
migration, and x-ignore-shared conflicts with VMs whose main RAM
uses shared memfd (e.g. for vhost-user). With guest-RAM pages +
pvsched, pages migrate as normal guest RAM: source VMM detaches at
stop-and-copy (unmapping the arena import and marking pages dirty
before the final sync), and destination VMM replays BAR1 into
destination /dev/pvsched before vCPU resume (immediately disabling
via header status if the destination runs no host scheduler, with
zero blackout). Guest-allocated pages also map naturally to CoCo VMs
via set_memory_decrypted() before GPA registration.
On the pvsched side, supporting scx_pvhost/scx_pvguest cleanly would
just mean:
- making the pvsched PCI transport + /dev/pvsched arch-independent so
arm64 works alongside x86_64,
- allowing an ordered multi-page VM slice in BAR1 (not just 1 page per
vCPU), and
- exposing a thin kernel helper around arena_map_ext_pages() /
arena_unmap_import() in kernel/bpf/arena.c so host and guest kfuncs
can import and synchronously revoke those pages.
I am working on a PoC using the paravirt transport from pvsched for the
bpf arena framework in this series. Will share it soon.
We are also thinking of extracting the transport side of pvsched into a
standalone pvshm framework, with pvsched as a consumer of pvshm. I am
envisioning pvshm as the transport framework for guest/host BPF arena
imports as well, while pvsched remains a standalone scheduling
framework supporting kernel-module or BPF-based policies.
Please let me know your thoughts about this.
Thanks,
Vineeth
[5] https://github.com/pvsched/linux
> On an NVIDIA Vera host (arm64), with 8-vCPU VMs on QEMU or Firecracker, at 2x
> and 6x overcommit, compared to stock VMs, the paravirt pair is 2.2-3x as fast on
> hackbench, 1.3-1.9x on a kernel build and serves 1.6-2x the requests of schbench
> and nginx; tools/sched_ext/scx-pvsched.md in the [DEMO] commit describes the
> design, the results and how to run it.
>
> Known limitations
> =================
>
> - Imports are released whole: freeing part of an import releases all of it.
> - Imported device-dax pages keep the device in use: unbinding it blocks until
> the arena releases them. Revoking them would need a notification from the
> dax core and a way to tell the BPF programs that the memory is gone.
> - BPF programs have no machine check recovery: on a real NVDIMM, the supplier
> has to keep bad pages out of the arena.
> - There is no doorbell: the sched_ext use case polls the shared pages at
> scheduling events.
> - Confidential VMs would need the guest to keep the shared pages shared with
> the host, which the device-dax import doesn't handle yet.
>
> References
> ==========
>
> [1] https://lpc.events/event/20/contributions/2429/
> [2] https://lpc.events/event/20/contributions/2482/
> [3] https://lore.kernel.org/r/20261010224902.1232178-1-arighi@xxxxxxxxxx/
> [4] https://lore.kernel.org/r/DM6HVNSP3USV.AJSWS289Z3I9@xxxxxxxxx/
>
> Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-pvsched
>
> Changes in v2:
> - Drop BPF_F_ARENA_NO_FREE, BPF_F_ARENA_EXPORT and the sized bpffs file, and
> map pages that the arena didn't allocate instead (Alexei Starovoitov)
> - Import the host's pages from a memfd with memfd_pin_folios(), the VMM uses
> the same file as a memory backend (Alexei Starovoitov)
> - Replace the guest NUMA node with a device-dax import, so the guest's pages
> never go back to its buddy allocator (Alexei Starovoitov, Josh Don)
> - Release an import with bpf_arena_free_pages(), instead of keeping the arena
> at its high watermark (Alexei Starovoitov, Josh Don)
> - Map the shared pages at an offset known to both sides, instead of
> discovering it out of band (Josh Don)
> - Track imports in an interval tree instead of a second range tree and a
> list, so that a release finds its import without allocating
> - Link to v1: https://lore.kernel.org/r/20261010224902.1232178-1-arighi@xxxxxxxxxx/
>
> Andrea Righi (3):
> bpf: Allow importing memfd pages into an arena
> bpf: Allow importing device-dax pages into an arena
> selftests/bpf: Test importing pages into an arena
>
> kernel/bpf/arena.c | 461 ++++++++++++++++++++-
> .../selftests/bpf/prog_tests/arena_import.c | 451 ++++++++++++++++++++
> tools/testing/selftests/bpf/progs/arena_import.c | 93 +++++
> 3 files changed, 994 insertions(+), 11 deletions(-)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_import.c
> create mode 100644 tools/testing/selftests/bpf/progs/arena_import.c