[PATCH 00/13] Introduce NVIDIA vGPU manager and VFIO variant driver

[Zhi Wang]

Add the nova-core support needed to create and manage NVIDIA vGPU
instances, together with the NVIDIA vGPU VFIO variant driver that consumes
the lifecycle interface.

The control path introduced by this series is:

 +-----------------------+         +-----------------------+
 | Linux guest           |         | Windows guest         |
 | NVIDIA guest driver   |         | NVIDIA guest driver   |
 +-----------+-----------+         +-----------+-----------+
             |                                 |
 +-----------v-----------+         +-----------v-----------+
 | QEMU (VFIO user)      |         | QEMU (VFIO user)      |
 +-----------+-----------+         +-----------+-----------+
             +----------------+----------------+
                              |
                      +-------v-------+
                      |   VFIO core   |
                      +-------+-------+
                              |
        +---------------------v---------------------+
        | NVIDIA vGPU VFIO variant driver           |
        +---------------------+---------------------+
                              |
              +---------------+----------------------+
              | binds PCI VF                         | PF lifecycle API
              |                                      | open / close / reset
              v                                      v
     +--------+--------+               +-------------+-------------+
     |    PCI VF(s)    |               | nova-core (PF driver)     |
     |  (virtual GPU)  |               |       vGPU Manager        |
     +--------+--------+               |     VRAM / channels ...   |
              |                        +-------------+-------------+
              |                                      |
+-------------v--------------------------------------v----------------+
|                      NVIDIA physical GPU                            |
|            VF resources                 PF / GSP / VRAM             |
+---------------------------------------------------------------------+

On the nova-core side, the PF driver owns the GPU firmware-facing part
of vGPU management. It already detects vGPU mode before GSP boot and
supplies the SR-IOV topology to the GPU System Processor firmware
(GSP-RM). This series extends it to:

  - Retain the firmware-reported VMMU segment size and FIFO engine
    ordering after GSP_INIT, together with nova-core's channel capacity.
  - Add VramBlock and VramRegion ownership plus Bar1Map for bounded CPU
    mappings of VRAM-backed control structures.
  - Query GSP-RM through its command queue for the vGPU type already
    assigned to a VF and its properties.
  - Build a layout-validated, profile-wide pool of paired framebuffer and
    management-heap slots, reserve contiguous channel ranges, and keep
    live allocations in a per-PF instance registry.
  - Use the typed r000 firmware bindings for control and response layouts,
    message IDs, and firmware-defined region sizes.
  - Handle GMC transactions over the GSP queues, match GMC replies by
    command and sequence, dispatch interleaved RM RPCs from the shared
    GSP-to-CPU message queue, and consume stale GMC messages.
  - Implement the GMCAPI bootload, shutdown, and cleanup operations with
    typed channel and resource maps for each plugin instance.
  - Establish the BAR1-backed PluginRpc channel, negotiate its protocol,
    and send the VM configuration and plugin BME-state update.
  - Use per-instance CeUtils channels to scrub guest framebuffer memory
    on allocation, reset, and shutdown. If ownership or completion is
    uncertain, keep the affected VRAM and channels out of the allocators.
  - Expose the three per-instance plugin log buffers through debugfs with
    the header needed by nvlog_decoder.
  - Keep resource locking, teardown, and rollback inside nova-core.
  - Select the firmware-defined 48-VM WPR2 heap when a device advertises
    more than 32 VFs.

On the VFIO side, the NVIDIA vGPU VFIO variant driver:

  - Binds only to an NVIDIA VF explicitly selected through
    driver_override.
  - Derives the guest function ID (GFID) from the SR-IOV VF index and
    registers a vfio-pci-core device.
  - Presents the vGPU PCI device and subsystem IDs returned by nova-core,
    and limits the reported framebuffer BAR size.
  - Delegates the remaining PCI and VFIO operations to vfio-pci-core and
    uses the standard VFIO physical-device helpers for IOMMUFD.

The ownership boundary between the two drivers is visible at these
lifecycle points:

  - Open: after vfio_pci_core_enable(), the variant driver passes the PF
    returned by pci_physfn(vf), the GFID, the VF's PCI
    domain/bus/device/function (DBDF), and the calling process TGID to
    nvidia_vgpu_open(). nova-core validates the PF and GFID, creates and
    activates the instance, and returns the guest-visible PCI IDs and
    BAR1 size. On failure, the variant driver disables the vfio-pci-core
    device; on success, it calls
    vfio_pci_core_finish_enable().
  - Reset: the variant driver first calls nvidia_vgpu_reset(). nova-core
    sends the plugin reset RPC and scrubs the guest framebuffer. Only
    after that succeeds is VFIO_DEVICE_RESET passed to vfio-pci-core for
    the PCI function reset.
  - Close: the variant driver invokes the void nvidia_vgpu_close()
    interface to request plugin shutdown, framebuffer scrubbing, and
    resource release from nova-core, then closes the vfio-pci-core
    device.

The cross-module interface is therefore limited to three GPL-only
symbols in NOVA_CORE_VGPU: nvidia_vgpu_open(), nvidia_vgpu_close(), and
nvidia_vgpu_reset().

This is a ground-up rework of the earlier vGPU RFC [1]. In particular,
the vGPU manager has been moved from the VFIO driver into the nova-core
implementation, and the VFIO side no longer carries copies of RM firmware
headers. The VFIO driver attaches to the VF and uses its PF's nova-core
state for management, as discussed in that thread.

The patches are organized as follows:

  1-4   Initialize VgpuManager and add the memory and firmware building
        blocks.
  5-8   Allocate instances and implement GMC, plugin boot, and RPC.
  9-10  Scrub guest framebuffer memory and expose plugin diagnostics.
 11-12  Export the lifecycle API and add its VFIO consumer.
 13     Select the larger WPR2 heap for 48-VF devices.

The series is based on the nova-core r000 GSP, memory-management,
SR-IOV, Rust bitmap/id-pool, and PCI abstraction work. The exact base
commit is recorded below, and the complete prerequisite stack is
available in [2].

[1] https://lore.kernel.org/kvm/20250903221111.3866249-1-zhiw@xxxxxxxxxx/
[2] https://github.com/zhiwang-nvidia/nova-core/tree/zhi/nova-vgpu-wip-nova-gsp-20260902

Alok Kumar (1):
  gpu: nova-core: vgpu: add VRAM slot allocator

Zhi Wang (12):
  gpu: nova-core: vgpu: add post-GSP-boot vGPU initialization
  gpu: nova-core: mm: add VramBlock and Bar1Map
  gpu: nova-core: vgpu: add r000 plugin bindings
  gpu: nova-core: vgpu: add instance create/destroy
  gpu: nova-core: gsp: add GMC transaction helpers
  gpu: nova-core: vgpu: add vGPU bootload
  gpu: nova-core: vgpu: implement PluginRpc channel and config params
  gpu: nova-core: vgpu: scrub guest framebuffer memory with CeUtils
  gpu: nova-core: vgpu: export plugin log buffers via debugfs
  gpu: nova-core: vgpu: export lifecycle operations to VFIO
  vfio/nvidia-vgpu: add the NVIDIA vGPU VFIO variant driver
  gpu: nova-core: reserve the 48-VM WPR2 heap

 drivers/gpu/nova-core/driver.rs               |  42 +-
 drivers/gpu/nova-core/fb.rs                   |   2 +-
 drivers/gpu/nova-core/gpu.rs                  | 184 +++--
 drivers/gpu/nova-core/gpu/channel.rs          |   3 +
 drivers/gpu/nova-core/gsp.rs                  |  29 +-
 drivers/gpu/nova-core/gsp/boot.rs             |  13 +-
 drivers/gpu/nova-core/gsp/cmdq.rs             | 356 +++++++--
 drivers/gpu/nova-core/gsp/commands.rs         |  78 +-
 drivers/gpu/nova-core/gsp/fw.rs               |  63 +-
 drivers/gpu/nova-core/gsp/fw/commands.rs      | 185 ++++-
 .../gpu/nova-core/gsp/fw/r000_00/bindings.rs  | 175 +++++
 drivers/gpu/nova-core/gsp/hal/gh100.rs        |   2 +-
 drivers/gpu/nova-core/gsp/hal/tu102.rs        |   2 +-
 drivers/gpu/nova-core/mm.rs                   |   6 +-
 drivers/gpu/nova-core/mm/bar_user.rs          | 183 ++++-
 drivers/gpu/nova-core/mm/vram.rs              | 187 +++++
 drivers/gpu/nova-core/nova_core_exports.c     |   5 +
 drivers/gpu/nova-core/vgpu.rs                 |  91 ---
 drivers/gpu/nova-core/vgpu/bootload.rs        | 162 ++++
 drivers/gpu/nova-core/vgpu/consts.rs          |  33 +
 drivers/gpu/nova-core/vgpu/fw.rs              | 490 ++++++++++++
 drivers/gpu/nova-core/vgpu/fw/commands.rs     |  28 +
 drivers/gpu/nova-core/vgpu/instance.rs        | 705 ++++++++++++++++++
 drivers/gpu/nova-core/vgpu/log.rs             | 167 +++++
 drivers/gpu/nova-core/vgpu/mod.rs             | 170 +++++
 drivers/gpu/nova-core/vgpu/plugin_rpc.rs      | 274 +++++++
 drivers/gpu/nova-core/vgpu/scrubber.rs        | 474 ++++++++++++
 drivers/gpu/nova-core/vgpu/vfio.rs            | 282 +++++++
 drivers/gpu/nova-core/vgpu/vram.rs            | 140 ++++
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/Makefile                     |   2 +
 drivers/vfio/pci/nvidia-vgpu/Kconfig          |  16 +
 drivers/vfio/pci/nvidia-vgpu/Makefile         |   2 +
 drivers/vfio/pci/nvidia-vgpu/main.c           | 253 +++++++
 include/drm/nvidia_vgpu.h                     |  28 +
 rust/bindings/bindings_helper.h               |   1 +
 36 files changed, 4536 insertions(+), 299 deletions(-)
 create mode 100644 drivers/gpu/nova-core/mm/vram.rs
 delete mode 100644 drivers/gpu/nova-core/vgpu.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/bootload.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/consts.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/fw.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/fw/commands.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/instance.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/log.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/mod.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/plugin_rpc.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/scrubber.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/vfio.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/vram.rs
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvidia-vgpu/main.c
 create mode 100644 include/drm/nvidia_vgpu.h


base-commit: a1dc9fdea9ab15e5df1bd6af1f6de2e40079f02d
-- 
2.53.0