Re: [PATCH bpf-next 0/5] bpf: Expose pinned arenas as sized bpffs files

[Josh Don]

Hi Andrea,

Thanks for sending this out, I'd really like to converge on a shared
memory model that will work for both the goals of phantom tracker and
for pvsched :) Been thinking about the tradeoffs of this vs the
current structure of shared memory that we have in pvsched v3. I think
this definitely brings some nice properties, but I have some
questions/suggestions inline below.

On Sat, Oct 10, 2026 at 3:49 PM Andrea Righi <arighi@xxxxxxxxxx> wrote:
>
> As discussed in the BPF track and the sched_ext microconference at Linux
> Plumbers 2026 [1][2], shared BPF arenas can provide a generic channel
> for bidirectional zero-copy communication between BPF programs running
> on a host and its guests. One potential use is to exchange scheduling
> hints between host and guest sched_ext schedulers through shared memory.
>
> The goal is to let both schedulers access the same underlying pages as
> BPF arena memory. Exchanging hints then requires no copies between
> separate host and guest buffers. With the arena file interface proposed
> here, the setup uses existing QEMU/KVM memory backends and guest NUMA
> interfaces, without further kernel or hypervisor changes to implement
> the communication channel.

Seems this relies on QEMU support for supplying offset+size for a
virtual NUMA node; looking quickly this would seem to need additional
work to support on other VMMs like CHV (and especially crosvm). Not a
showstopper but just a comment that I think the PCI device model is
easier to integrate in other VMMs.

> A host scheduler's userspace component can create and pin an arena map,
> then use an arena allocator to partition it into per-guest chunks.
> Userspace passes each chunk to QEMU as a shared file-backed memory
> backend, using a bounded offset and length in the sized bpffs file.
> QEMU exposes only that chunk as a NUMA node inside the corresponding
> guest; the rest of the host arena is not mapped into guest RAM.

So this is the host allocates, guest uses model. Contrasted with
pvsched v3's guest allocate, host pins.

Guest still needs some way of communicating back to the host which
pages correspond to which data. Ie. a mapping of VCPU -> address
within this arena memory that the host can then use to lookup per-VCPU
info written by the guest. This comes for free with the guest
allocation model in pvsched. I see you allude to this requirement
below.

Another thing missing here is a doorbell, so this approach doesn't
obviate the need for at least one guest BAR. If we have that, it's not
a stretch to use an additional BAR for communicating the mappings (as
pvsched does in v3).

> The guest scheduler can create its own arena and allocate pages from
> that NUMA node. Those guest physical pages are backed by the host arena,
> so host and guest BPF programs can access the same memory through their
> respective arenas.

Two nice things here that we don't currently have with pvsched. One is
that the host BPF programs can directly read/write the arena memory
without needing an additional special kfunc to grab the right pointer
(for pvsched specific BPF hooks in the host we intend to pass the
pointer in the BPF context, but for sched_ext programs we intended to
add a kfunc to e.g. lookup the pointer by VCPU).  Second, the memory
here is arena memory, which is quite nice. Currently pvsched just
interprets it as a struct with a large data blob. Ideally we'd like to
convert the pvsched model to use an arena.

Some additional drawbacks of the guest arena allocation you describe here:
- The extra NUMA node can presumably be used for unrelated
allocations; there's no way to isolate it for just the purposes of our
arena? Guest kernel and other guest userspace could allocate on that
node.
- Memory allocations targeting that NUMA node (for the arena) could
fail to find free space and fall back to another NUMA node; that at
least seems solvable by forcing allocations to only come from the
targetted node though
- This removes flexiblity of the guest to allocate memory how it
likes. For example, with pvsched the guest can allocate the per-VCPU
pages from the guest virtual nodes where those VCPUs reside.
- I don't think this works for confidential VMs? You'd end up needing
the guest to share the pages back with the host, which is essentially
what the guest-allocates model already does

I would definitely like to have the memory come to BPF via an arena
for the flexibility of use, but I'm still a bit hung up on the host
allocation part. I had been thinking about how to make the arena work
using guest allocations, but nothing super concrete. Essentially the
guest would create an arena and allocate pages, and stash the GPAs in
the pvsched BAR. The host would need to pull those pages into its
arena and deterministically allow access to them.

> After identifying the corresponding arena offsets,
> both sides exchange data through direct BPF loads and stores. Arena
> pointers are not relocated between mappings; a shared protocol can use
> validated slice-relative offsets instead.
>
> QEMU is one potential consumer. The interface is generic: it exposes a
> pinned BPF arena as a sized bpffs file that ordinary userspace consumers
> can open and mmap, including consumers that require a file size when
> setting up shared memory.
>
> The series separates page retention from file export:
>
> * BPF_F_ARENA_NO_FREE makes BPF page-free operations no-ops, retaining
>   allocated pages until the map is destroyed. This flag alone preserves
>   ordinary arena pin behavior.

Does this imply that the arena remains allocated at its highest
watermark, regardless if we have VMs leave the system (and thus the
host attempt to free the arena pages that corresponded to the departed
VMs)?

Smaller note: we probably want to at least make the free operation
zero out the page so that no data is leftover when we reuse that page
(ie. for a new VM).

> * BPF_F_ARENA_EXPORT exposes the pin as a sized file and requires
>   BPF_F_ARENA_NO_FREE, so BPF programs cannot remove backing pages while
>   external consumers may still use them.
>
> The map FD retains the canonical BPF address range. Exported mappings
> can select different virtual addresses and map disjoint slices. Faults
> resolve by file offset, and page zaps cover only the overlapping range
> in each exported VMA. Consumers establish the canonical range through
> map_extra or a full-capacity map FD mapping before mapping the pin.
> For exported arenas, the canonical range matches the fixed file size;
> ordinary and retention-only arenas still allow shorter mappings.
>
> Pin permissions distinguish readers from writers. O_RDONLY consumers
> can create read-only shared mappings, but cannot request writable
> mappings or acquire write permission through mprotect(). Private
> mappings and file size changes are rejected. Open files and mappings
> retain the arena after unpinning until the last reference is released.
> The capacity is initialized before inode publication and can be queried
> with fstat() or lseek(fd, 0, SEEK_END).
>
> Exported mappings retain the arena's lifecycle restrictions: they are
> not inherited across fork and cannot be split, expanded, or relocated.
> The documentation describes these limits and native tests exercise them.
>
> Access through open() and mmap() intentionally uses filesystem
> permissions and the applicable file, mmap, and BPF map LSM checks.
> olicies specific to the bpf() syscall, including seccomp filters on that
> syscall, do not govern this file interface.
>
> [1] https://lpc.events/event/20/contributions/2429/
> [2] https://lpc.events/event/20/contributions/2482/
>
> Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git bpf-pinned-arena

Best,
Josh