[PATCH v3 00/31] Introduce NVIDIA vGPU manager

[Zhi Wang]

Add the nova-core support needed to create and manage NVIDIA vGPU
instances on the current whole-GPU SR-IOV path.

The overall control path is shown below. This v3 posting contains the
nova-core manager and PF-side lifecycle APIs. Typed PF-service
publication and the VFIO variant driver follow separately.

 +-----------------------+         +-----------------------+
 | Linux guest           |         | Windows guest         |
 | NVIDIA guest driver   |         | NVIDIA guest driver   |
 +-----------+-----------+         +-----------+-----------+
             |                                 |
 +-----------v-----------+         +-----------v-----------+
 | QEMU (VFIO user)      |         | QEMU (VFIO user)      |
 +-----------+-----------+         +-----------+-----------+
             +----------------+----------------+
                              |
                      +-------v-------+
                      |   VFIO core   |
                      +-------+-------+
                              |
        +---------------------v---------------------+
        | NVIDIA vGPU VFIO variant driver           |
        +---------------------+---------------------+
                              |
              +---------------+----------------------+
              | binds PCI VF                         | PF lifecycle API
              |                                      | open / close / reset
              v                                      v
     +--------+--------+               +-------------+-------------+
     |    PCI VF(s)    |               | nova-core (PF driver)     |
     |  (virtual GPU)  |               |       vGPU Manager        |
     +--------+--------+               |     VRAM / channels ...   |
              |                        +-------------+-------------+
              |                                      |
+-------------v--------------------------------------v----------------+
|                      NVIDIA physical GPU                            |
|            VF resources                 PF / GSP / VRAM             |
+---------------------------------------------------------------------+

The manager queries the type assigned to a VF by GSP-RM, reserves its
channel and VRAM resources, boots and configures the GSP plugin, and
handles shutdown. It also scrubs guest VRAM before reuse and exposes
per-instance plugin logs through debugfs.

Host resources remain owned until firmware teardown and BAR unmapping
complete. Failed creation rolls back through the instance owner; if
firmware ownership is uncertain or teardown fails, the manager retains
the affected resources until device removal.

This continues the rework of the earlier vGPU RFC [6]: firmware-facing
management belongs to nova-core, while the VFIO driver binds to the VF
and uses the PF's services. The companion VFIO driver does not need its
own copies of the RM firmware headers.

The patches are organized as follows:

  1-7   Boot state, runtime parameters and the 48-VM WPR2 heap.
  8-11  VRAM allocation, region views and BAR mappings.
 12-13  Synchronous GMC transactions and completion events.
 14-18  Plugin bindings, property queries and instance allocation.
 19-27  Plugin boot, shutdown and RPC.
 28-30  CeUtils scrubbing and plugin logs.
 31     SR-IOV PF lifecycle APIs.

Changes since v2:
* Rebase onto the newer GIN interrupt/GSP event and r000 firmware stack,
  and drop changes already supplied by the prerequisites.
* Add RAII for BAR mappings and VRAM slots, creation rollback, and pinned
  manager teardown while its firmware and memory dependencies are alive.
* Add the PF-side lifecycle APIs after the plugin-log support.

Previous versions:
v2: [1]
v1: https://lore.kernel.org/nova-gpu/20260905081116.106613-1-zhiw@xxxxxxxxxx/

Dependencies:
This series is based on drm-rust-next at 896ed0833627, which includes the
Nova memory-management work, plus the GIN/GSP event [2], NVKV [3], r000
GSP [4], Rust bitmap/ID-pool [5] and PCI prerequisite stack. 

The complete integration tree, including the prerequisites and the
companion VFIO driver, is available at:

  https://github.com/zhiwang-nvidia/nova-core/tree/zhi/nova-vgpu-20260928

[1] https://lore.kernel.org/nova-gpu/20260914074907.1399629-1-zhiw@xxxxxxxxxx/
[2] https://lore.kernel.org/nova-gpu/20260912044400.677097-1-jhubbard@xxxxxxxxxx/
[3] https://lore.kernel.org/nova-gpu/20260827-b4-nvkv-v2-0-0de9d5c8658c@xxxxxxxxxx/
[4] https://lore.kernel.org/nova-gpu/20260918010719.1176945-1-jhubbard@xxxxxxxxxx/
[5] https://lore.kernel.org/nova-gpu/20260827-chid-v8-0-bc74c77d0214@xxxxxxxxxx/
[6] https://lore.kernel.org/kvm/20250903221111.3866249-1-zhiw@xxxxxxxxxx/

Alok Kumar (2):
  gpu: nova-core: gsp: decode the FIFO engine table
  gpu: nova-core: vgpu: add VRAM slot allocator

Zhi Wang (29):
  gpu: nova-core: gsp: pass boot context through setup helpers
  gpu: nova-core: gsp: decouple boot context from VgpuManager
  gpu: nova-core: vgpu: detect boot state independently
  gpu: nova-core: gpu: add a channel ID pool for vGPU
  gpu: nova-core: vgpu: initialize runtime parameters after GSP boot
  gpu: nova-core: vgpu: reserve the 48-VM WPR2 heap
  gpu: nova-core: mm: borrow BarUser for temporary BAR1 access
  gpu: nova-core: mm: add VramBlock
  gpu: nova-core: mm: add VramRegion
  gpu: nova-core: mm: add BarMapping
  gpu: nova-core: gsp: add synchronous GMC transactions
  gpu: nova-core: gsp: wait for GMC completion events
  gpu: nova-core: vgpu: add r000 plugin bindings
  gpu: nova-core: gsp: factor out NVKV payload conversion
  gpu: nova-core: vgpu: query VF assignments and properties
  gpu: nova-core: vgpu: add instance create/destroy
  gpu: nova-core: vgpu: encode vGPU boot requests
  gpu: nova-core: vgpu: add GSP plugin communication buffers
  gpu: nova-core: vgpu: add instance boot
  gpu: nova-core: vgpu: add instance shutdown
  gpu: nova-core: vgpu: initialize GSP plugin RPC buffers
  gpu: nova-core: vgpu: add GSP plugin RPC transactions
  gpu: nova-core: vgpu: negotiate the GSP plugin RPC version
  gpu: nova-core: vgpu: send GSP plugin configuration parameters
  gpu: nova-core: vgpu: update the GSP plugin BME state
  gpu: nova-core: vgpu: add CeUtils commands
  gpu: nova-core: vgpu: scrub guest VRAM with CeUtils
  gpu: nova-core: vgpu: export plugin log buffers via debugfs
  gpu: nova-core: vgpu: introduce SR-IOV PF APIs

 Documentation/gpu/nova/core/interrupts.rst    |  19 +-
 drivers/gpu/nova-core/fb.rs                   |  28 +-
 drivers/gpu/nova-core/gpu.rs                  | 133 +++-
 drivers/gpu/nova-core/gpu/channel.rs          |   4 +
 drivers/gpu/nova-core/gsp.rs                  |  30 +-
 drivers/gpu/nova-core/gsp/boot.rs             |   2 +-
 drivers/gpu/nova-core/gsp/cmdq.rs             | 225 +++++-
 drivers/gpu/nova-core/gsp/commands.rs         |  50 +-
 drivers/gpu/nova-core/gsp/fw.rs               |  22 +-
 drivers/gpu/nova-core/gsp/fw/commands.rs      |  81 ++-
 .../gpu/nova-core/gsp/fw/r000_00/bindings.rs  | 175 +++++
 drivers/gpu/nova-core/gsp/hal/gh100.rs        |   2 +-
 drivers/gpu/nova-core/gsp/hal/tu102.rs        |   2 +-
 drivers/gpu/nova-core/gsp/nvkv.rs             |  39 ++
 drivers/gpu/nova-core/gsp/nvkv/encode.rs      |  12 +
 drivers/gpu/nova-core/mm.rs                   |   6 +-
 drivers/gpu/nova-core/mm/bar_user.rs          | 183 ++++-
 drivers/gpu/nova-core/mm/vram.rs              | 131 ++++
 drivers/gpu/nova-core/nova_core.rs            |   6 +
 drivers/gpu/nova-core/regs.rs                 |  31 +-
 drivers/gpu/nova-core/vgpu.rs                 | 111 ++-
 drivers/gpu/nova-core/vgpu/commands.rs        | 320 +++++++++
 drivers/gpu/nova-core/vgpu/fw.rs              |  70 ++
 drivers/gpu/nova-core/vgpu/fw/commands.rs     | 471 +++++++++++++
 drivers/gpu/nova-core/vgpu/gsp_plugin_comm.rs | 496 ++++++++++++++
 drivers/gpu/nova-core/vgpu/gsp_plugin_rpc.rs  | 170 +++++
 drivers/gpu/nova-core/vgpu/instance.rs        | 642 ++++++++++++++++++
 drivers/gpu/nova-core/vgpu/log.rs             | 171 +++++
 drivers/gpu/nova-core/vgpu/scrubber.rs        | 173 +++++
 drivers/gpu/nova-core/vgpu/vgpu_api.rs        | 158 +++++
 drivers/gpu/nova-core/vgpu/vram.rs            | 179 +++++
 31 files changed, 3983 insertions(+), 159 deletions(-)
 create mode 100644 drivers/gpu/nova-core/mm/vram.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/commands.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/fw.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/fw/commands.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/gsp_plugin_comm.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/gsp_plugin_rpc.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/instance.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/log.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/scrubber.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/vgpu_api.rs
 create mode 100644 drivers/gpu/nova-core/vgpu/vram.rs


base-commit: b840061c3893d74b9827f05caffeaf08e707aed5