[PATCH 0/4] Introduce nvgrace-egm driver for Extended GPU Memory

[Ankit Agrawal]

Background
==========
NVIDIA Grace-based systems (Grace Hopper, Grace Blackwell Superchip) have
a special bios-configurable mode called EGM (Extended GPU Memory) that
allows a carveout to be created from the system memory. This carveout is a
contiguous region of special range of system DRAM reserved by the UEFI
boot firmware.

This range does not appear in the EFI memory map and is therefore invisible
to the host kernel. i.e. there are no struct pages for it and it cannot
be managed by the normal page allocator. This range is described in ACPI
as vendor DSD properties on the GPU PCI device node (see ACPI Design
below). It is intended to be used by a VMM to back an entire virtual
machine's physical memory. The VMM maps the range into its address space
via mmap() on a char device and passes it to KVM as the backing store for
guest RAM similar to memfd or hugetlbfs. KVM builds Stage-2 page tables
mapping the guest's GPA range linearly to the EGM SPAs i.e. a guest offset
X maps to SPA (egm_base + X).

The range is special because the GPU on these systems has direct high-speed
access to it that bypasses the host system IOMMU. Normal PCI DMA is
gated by the IOMMU and requires an explicit IOMMU mapping. The GPU's path
to this memory goes via NVLink/NVSwitch directly to the memory controller
and does not involve the host IOMMU. Since KVM maps guest GPA to EGM SPA
linearly, the GPU can reach guest memory at those same SPAs without
needing any additional host IOMMU mapping. Normal system memory that the
kernel sees is protected by the IOMMU in the usual way and this range is
deliberately kept separate.

To make this work a driver is needed to parse the ACPI, capture the
physical memory range, expose it as a char device and allow it to be
mmap'd for KVM. A VMM like qemu can open this char dev and then use it
much like a hugetlbfs or memfd to back the VM. Due to the security
difference the memory has to be strictly separated from the rest of the
hypervisor memory. Exposing the range as a DMABUF for iommufd is planned
as future work and is not part of this series.

This hardware feature allows the system memory to be access across nodes
on a multi-node system such as Grace Blackwell NVL72. It can also provide
throughput improvements because the GPU can access VM system memory via
its high-bandwidth fabric rather than going through the IOMMU-gated PCIe
path. Moreover skipping IOMMU translations could also improve performance
for any use cases targeted towards high performance.

Due to the security difference, the EGM range is strictly isolated from
the rest of the hypervisor memory (which is covered through IOMMU).

A driver is needed to:
  - Parse ACPI to locate the physical range (base SPA, size, proximity
    domain)
  - Expose the range as a char device (/dev/egmX, X = NUMA proximity
    domain) that a VMM such as QEMU can open and mmap to back guest RAM
  - Zero the region on first open to isolate successive VM uses
  - Track and expose retired ECC pages to the VMM via ioctl
  - Register the PFN range with the memory_failure infrastructure for
    runtime uncorrectable error handling


ACPI Design
===========
The EGM region properties are exposed as NVIDIA vendor DSD properties on
the GPU's PCI ACPI device node:

  nvidia,egm-pxm                     NUMA proximity domain of the region
  nvidia,egm-base-pa                 Physical base address (SPA) of the
                                     region
  nvidia,egm-size                    Length of the region in bytes
  nvidia,egm-retired-pages-data-base Physical address of the firmware-
                                     provided retired ECC pages table

This ACPI construct helps with describing and discovering this physically
present memory range that is intentionally absent from the EFI memory map
and is tied to a specific device for its access properties.

On a multi-socket system there is one EGM region per socket. Multiple GPUs
on the same socket advertise identical region properties (same proximity
domain, base address and size). The driver deduplicates by proximity
domain: the first GPU probed for a given domain creates the char device
and subsequent GPUs on the same socket are linked to it via sysfs.


Implementation
==============
Patch 1 introduces the driver skeleton: Kconfig/Makefile/MAINTAINERS, the
egm device class, the char device minor number range (MAX_EGM_NODES=4
matching the largest 4-socket Grace configuration), ACPI property walking
(nvidia,egm-pxm / nvidia,egm-base-pa / nvidia,egm-size), per-region state
tracking with PXM-based deduplication, char device creation (/dev/egmX),
GPU-to-EGM sysfs symlinks and the egm_size sysfs attribute.

Patch 2 implements the file operations: memory scrubbing on the first
open() using memremap()/memset() in 1 GiB chunks with cond_resched() to
avoid RCU stalls and mmap() via remap_pfn_range().

Patch 3 adds retired ECC page handling: reads the firmware-provided
retired-pages table from the ACPI DSD property
nvidia,egm-retired-pages-data-base at module init, stores the page offsets
in a per-device xarray and exposes them to userspace via the
EGM_RETIRED_PAGES_LIST ioctl (UAPI header include/uapi/linux/egm.h).
Each firmware entry covers a 64 KiB retired region. The driver inserts one
xarray entry per kernel page so a lookup at the running page size hits the
right retired range on both 4K and 64K kernels.

Patch 4 registers the EGM PFN range with the memory_failure infrastructure
on the first open() and unregisters it on the last close(). The
pfn_to_vma_pgoff callback validates that a reported PFN falls within the
EGM region, converts it to an in-region file offset, and inserts it into
the retired-pages xarray so runtime uncorrectable errors are surfaced
alongside ACPI-reported retired pages via the same ioctl.


Enablement
==========
EGM mode is enabled through a flag in the platform firmware (SBIOS). The
size of the host-visible partition is separately configurable; all
remaining system DRAM is reserved as the EGM region invisible to the
hypervisor.


RFC Posting
===========
Link: https://lore.kernel.org/all/20260223155514.152435-1-ankita@xxxxxxxxxx/


Verification
============
Tested on a 2-socket (4 GPU) Grace Blackwell platform by booting a VM with
QEMU [1] and kernel branch [2] and confirming EGM capability is visible
inside the guest and basic cuda workloads. Note that this has dependency
on the generic dmabuf importer/exporter work in progress [3]. Until then, a
bypass PFNMAP workaround is used [4] (also part of the kernel branch [2].


Link: https://github.com/NVIDIA/QEMU/tree/nvidia_stable-10.1 [1]
Link: https://github.com/ankita-nv/linux/tree/v7.2-egm-01072026 [2]
Link: https://lore.kernel.org/all/0-v1-b5cab63049c0+191af-dmabuf_map_type_jgg@xxxxxxxxxx/ [3]
Link: https://github.com/ankita-nv/linux/commit/f2a915d00916c5ef5e25f7bc2ad5c10efd36fb4b [4]


Suggested-by: Alex Williamson <alex@xxxxxxxxxxx>
Suggested-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
Suggested-by: Aniket Agashe <aniketa@xxxxxxxxxx>
Suggested-by: Vikram Sethi <vsethi@xxxxxxxxxx>
Suggested-by: Matthew R. Ochs <mochs@xxxxxxxxxx>

Ankit Agrawal (4):
  platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM
    regions
  platform/nvidia: Implement mmap and memory scrubbing for EGM chardev
  platform/nvidia: Handle retired ECC pages and expose via ioctl
  platform/nvidia: Register EGM PFNMAP range with memory_failure

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 MAINTAINERS                                   |   6 +
 drivers/platform/Kconfig                      |   1 +
 drivers/platform/Makefile                     |   1 +
 drivers/platform/nvidia/Kconfig               |  10 +
 drivers/platform/nvidia/Makefile              |   3 +
 drivers/platform/nvidia/egm.c                 | 874 ++++++++++++++++++
 include/uapi/linux/egm.h                      |  28 +
 8 files changed, 924 insertions(+)
 create mode 100644 drivers/platform/nvidia/Kconfig
 create mode 100644 drivers/platform/nvidia/Makefile
 create mode 100644 drivers/platform/nvidia/egm.c
 create mode 100644 include/uapi/linux/egm.h

-- 
2.34.1