Re: [PATCH drm-next 00/14] [RFC] DRM GPUVA Manager & Nouveau VM_BIND UAPI

From: Danilo Krummrich
Date: Wed Jan 18 2023 - 11:51:57 EST



On 1/18/23 17:30, Alex Deucher wrote:
On Wed, Jan 18, 2023 at 11:19 AM Danilo Krummrich <dakr@xxxxxxxxxx> wrote:

On 1/18/23 16:37, Christian König wrote:
Am 18.01.23 um 16:34 schrieb Danilo Krummrich:
Hi Christian,

On 1/18/23 09:53, Christian König wrote:
Am 18.01.23 um 07:12 schrieb Danilo Krummrich:
This patch series provides a new UAPI for the Nouveau driver in
order to
support Vulkan features, such as sparse bindings and sparse residency.

Furthermore, with the DRM GPUVA manager it provides a new DRM core
feature to
keep track of GPU virtual address (VA) mappings in a more generic way.

The DRM GPUVA manager is indented to help drivers implement
userspace-manageable
GPU VA spaces in reference to the Vulkan API. In order to achieve
this goal it
serves the following purposes in this context.

1) Provide a dedicated range allocator to track GPU VA
allocations and
mappings, making use of the drm_mm range allocator.

This means that the ranges are allocated by the kernel? If yes that's
a really really bad idea.

No, it's just for keeping track of the ranges userspace has allocated.

Ok, that makes more sense.

So basically you have an IOCTL which asks kernel for a free range? Or
what exactly is the drm_mm used for here?

Not even that, userspace provides both the base address and the range,
the kernel really just keeps track of things. Though, writing a UAPI on
top of the GPUVA manager asking for a free range instead would be
possible by just adding the corresponding wrapper functions to get a
free hole.

Currently, and that's what I think I read out of your question, the main
benefit of using drm_mm over simply stuffing the entries into a list or
something boils down to easier collision detection and iterating
sub-ranges of the whole VA space.

Why not just do this in userspace? We have a range manager in
libdrm_amdgpu that you could lift out into libdrm or some other
helper.

The kernel still needs to keep track of the mappings within the various VA spaces, e.g. it silently needs to unmap mappings that are backed by BOs that get evicted and remap them once they're validated (or swapped back in).


Alex




Regards,
Christian.


- Danilo


Regards,
Christian.


2) Generically connect GPU VA mappings to their backing
buffers, in
particular DRM GEM objects.

3) Provide a common implementation to perform more complex mapping
operations on the GPU VA space. In particular splitting and
merging
of GPU VA mappings, e.g. for intersecting mapping requests
or partial
unmap requests.

The new VM_BIND Nouveau UAPI build on top of the DRM GPUVA manager,
itself
providing the following new interfaces.

1) Initialize a GPU VA space via the new
DRM_IOCTL_NOUVEAU_VM_INIT ioctl
for UMDs to specify the portion of VA space managed by the
kernel and
userspace, respectively.

2) Allocate and free a VA space region as well as bind and
unbind memory
to the GPUs VA space via the new DRM_IOCTL_NOUVEAU_VM_BIND
ioctl.

3) Execute push buffers with the new DRM_IOCTL_NOUVEAU_EXEC ioctl.

Both, DRM_IOCTL_NOUVEAU_VM_BIND and DRM_IOCTL_NOUVEAU_EXEC, make use
of the DRM
scheduler to queue jobs and support asynchronous processing with DRM
syncobjs
as synchronization mechanism.

By default DRM_IOCTL_NOUVEAU_VM_BIND does synchronous processing,
DRM_IOCTL_NOUVEAU_EXEC supports asynchronous processing only.

The new VM_BIND UAPI for Nouveau makes also use of drm_exec
(execution context
for GEM buffers) by Christian König. Since the patch implementing
drm_exec was
not yet merged into drm-next it is part of this series, as well as a
small fix
for this patch, which was found while testing this series.

This patch series is also available at [1].

There is a Mesa NVK merge request by Dave Airlie [2] implementing the
corresponding userspace parts for this series.

The Vulkan CTS test suite passes the sparse binding and sparse
residency test
cases for the new UAPI together with Dave's Mesa work.

There are also some test cases in the igt-gpu-tools project [3] for
the new UAPI
and hence the DRM GPU VA manager. However, most of them are testing
the DRM GPU
VA manager's logic through Nouveau's new UAPI and should be
considered just as
helper for implementation.

However, I absolutely intend to change those test cases to proper
kunit test
cases for the DRM GPUVA manager, once and if we agree on it's
usefulness and
design.

[1]
https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next /
https://gitlab.freedesktop.org/nouvelles/kernel/-/merge_requests/1
[2] https://gitlab.freedesktop.org/nouveau/mesa/-/merge_requests/150/
[3]
https://gitlab.freedesktop.org/dakr/igt-gpu-tools/-/tree/wip_nouveau_vm_bind

I also want to give credit to Dave Airlie, who contributed a lot of
ideas to
this patch series.

Christian König (1):
drm: execution context for GEM buffers

Danilo Krummrich (13):
drm/exec: fix memory leak in drm_exec_prepare_obj()
drm: manager to keep track of GPUs VA mappings
drm: debugfs: provide infrastructure to dump a DRM GPU VA space
drm/nouveau: new VM_BIND uapi interfaces
drm/nouveau: get vmm via nouveau_cli_vmm()
drm/nouveau: bo: initialize GEM GPU VA interface
drm/nouveau: move usercopy helpers to nouveau_drv.h
drm/nouveau: fence: fail to emit when fence context is killed
drm/nouveau: chan: provide nouveau_channel_kill()
drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm
drm/nouveau: implement uvmm for user mode bindings
drm/nouveau: implement new VM_BIND UAPI
drm/nouveau: debugfs: implement DRM GPU VA debugfs

Documentation/gpu/driver-uapi.rst | 11 +
Documentation/gpu/drm-mm.rst | 43 +
drivers/gpu/drm/Kconfig | 6 +
drivers/gpu/drm/Makefile | 3 +
drivers/gpu/drm/amd/amdgpu/Kconfig | 1 +
drivers/gpu/drm/drm_debugfs.c | 56 +
drivers/gpu/drm/drm_exec.c | 294 ++++
drivers/gpu/drm/drm_gem.c | 3 +
drivers/gpu/drm/drm_gpuva_mgr.c | 1323
+++++++++++++++++
drivers/gpu/drm/nouveau/Kbuild | 3 +
drivers/gpu/drm/nouveau/Kconfig | 2 +
drivers/gpu/drm/nouveau/include/nvif/if000c.h | 23 +-
drivers/gpu/drm/nouveau/include/nvif/vmm.h | 17 +-
.../gpu/drm/nouveau/include/nvkm/subdev/mmu.h | 10 +
drivers/gpu/drm/nouveau/nouveau_abi16.c | 23 +
drivers/gpu/drm/nouveau/nouveau_abi16.h | 1 +
drivers/gpu/drm/nouveau/nouveau_bo.c | 152 +-
drivers/gpu/drm/nouveau/nouveau_bo.h | 2 +-
drivers/gpu/drm/nouveau/nouveau_chan.c | 16 +-
drivers/gpu/drm/nouveau/nouveau_chan.h | 1 +
drivers/gpu/drm/nouveau/nouveau_debugfs.c | 24 +
drivers/gpu/drm/nouveau/nouveau_drm.c | 25 +-
drivers/gpu/drm/nouveau/nouveau_drv.h | 92 +-
drivers/gpu/drm/nouveau/nouveau_exec.c | 310 ++++
drivers/gpu/drm/nouveau/nouveau_exec.h | 55 +
drivers/gpu/drm/nouveau/nouveau_fence.c | 7 +
drivers/gpu/drm/nouveau/nouveau_fence.h | 2 +-
drivers/gpu/drm/nouveau/nouveau_gem.c | 83 +-
drivers/gpu/drm/nouveau/nouveau_mem.h | 5 +
drivers/gpu/drm/nouveau/nouveau_prime.c | 2 +-
drivers/gpu/drm/nouveau/nouveau_sched.c | 780 ++++++++++
drivers/gpu/drm/nouveau/nouveau_sched.h | 98 ++
drivers/gpu/drm/nouveau/nouveau_svm.c | 2 +-
drivers/gpu/drm/nouveau/nouveau_uvmm.c | 575 +++++++
drivers/gpu/drm/nouveau/nouveau_uvmm.h | 68 +
drivers/gpu/drm/nouveau/nouveau_vmm.c | 4 +-
drivers/gpu/drm/nouveau/nvif/vmm.c | 73 +-
.../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c | 168 ++-
.../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.h | 1 +
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c | 32 +-
drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h | 3 +
include/drm/drm_debugfs.h | 25 +
include/drm/drm_drv.h | 6 +
include/drm/drm_exec.h | 144 ++
include/drm/drm_gem.h | 75 +
include/drm/drm_gpuva_mgr.h | 527 +++++++
include/uapi/drm/nouveau_drm.h | 216 +++
47 files changed, 5266 insertions(+), 126 deletions(-)
create mode 100644 drivers/gpu/drm/drm_exec.c
create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c
create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.c
create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.h
create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.c
create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.h
create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.c
create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.h
create mode 100644 include/drm/drm_exec.h
create mode 100644 include/drm/drm_gpuva_mgr.h


base-commit: 0b45ac1170ea6416bc1d36798414c04870cd356d