Re: [PATCH RESEND v2 00/18] DCD: Add support for Dynamic Capacity Devices (DCD)

From: Ira Weiny
Date: Fri Aug 16 2024 - 10:18:45 EST


Please ignore this RESEND. It is a very old version and sent by mistake.

Ira


Ira Weiny wrote:
> A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
> device that implements dynamic capacity. Dynamic capacity feature
> allows memory capacity to change dynamically, without the need for
> resetting the device.
>
> Even though this is marked v2 by b4, this is effectively a whole new
> series for DCD support. Quite a bit of the core support was completed
> by Navneet in [4]. However, the architecture through the CXL region,
> DAX region, and DAX Device layers is completely different. Particular
> attention was paid to:
>
> 1) managing skip resources in the hardware device
> 2) ensuring the host OS only sent a release memory mailbox
> response when all DAX devices are done using an extent
> 3) allowing dax devices to span extents
> 4) allowing dax devices to use parts of extents
>
> I could say all of the review comments from v1 are addressed but frankly
> the series has changed so much that I can't guarantee anything.
>
> The series continues to be based on the type-2 work posted from Dan.[2]
> However, my branch with that work is a bit dated. Therefore I have
> posted this series on github here.[5]
>
> Testing was sped up with cxl-test and ndctl dcd support. A preview of
> that work is on github.[6] In addition Fan Ni's Qemu DCD series was
> used part of the time.[3]
>
> The major parts of this series are:
>
> - Get the dynamic capacity (DC) region information from cxl device
> - Configure device DC regions reported by hardware
> - Enhance CXL and DAX regions for DC
> a. maintain separation between the hardware extents and the CXL
> region extents to provide for the addition of interleaving in
> the future.
> - Get and maintain the hardware extent lists for each device via an
> initial extent list and DC event records
> a. Add capacity Events
> b. Add capacity response
> b. Release capacity events
> d. Release capacity response
> - Notify region layers of extent changes
> - Allow for DAX devices to be created on extents which are surfaced
> - Maintain references on extents which are in use
> a. Send Release capacity Response only when DAX devices are not
> using memory
> - Allow DAX region extent labels to change to allow for flexibility in
> DAX device creation in the future (further enhancements are required
> to ndctl for this)
> - Trace Dynamic Capacity events
> - Add cxl-test infrastructure to allow for faster unit testing
>
> To: Dan Williams <dan.j.williams@xxxxxxxxx>
> Cc: Navneet Singh <navneet.singh@xxxxxxxxx>
> Cc: Fan Ni <fan.ni@xxxxxxxxxxx>
> Cc: Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx>
> Cc: Davidlohr Bueso <dave@xxxxxxxxxxxx>
> Cc: Dave Jiang <dave.jiang@xxxxxxxxx>
> Cc: Alison Schofield <alison.schofield@xxxxxxxxx>
> Cc: Vishal Verma <vishal.l.verma@xxxxxxxxx>
> Cc: Ira Weiny <ira.weiny@xxxxxxxxx>
> Cc: linux-cxl@xxxxxxxxxxxxxxx
> Cc: linux-kernel@xxxxxxxxxxxxxxx
>
> [1] https://lore.kernel.org/all/64326437c1496_934b2949f@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.notmuch/
> [2] https://lore.kernel.org/all/168592149709.1948938.8663425987110396027.stgit@xxxxxxxxxxxxxxxxxxxxxxxxx/
> [3] https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/
> [4] https://lore.kernel.org/r/20230604-dcd-type2-upstream-v1-0-71b6341bae54@xxxxxxxxx
> [5] https://github.com/weiny2/linux-kernel/commits/dcd-v2-2023-08-28
> [6] https://github.com/weiny2/ndctl/tree/dcd-region2
>
> ---
> Changes in v2:
> - iweiny: Complete rework of the entire series
> - Link to v1: https://lore.kernel.org/r/20230604-dcd-type2-upstream-v1-0-71b6341bae54@xxxxxxxxx
>
> ---
> Ira Weiny (15):
> cxl/hdm: Debug, use decoder name function
> cxl/mbox: Flag support for Dynamic Capacity Devices (DCD)
> cxl/region: Add Dynamic Capacity decoder and region modes
> cxl/port: Add Dynamic Capacity mode support to endpoint decoders
> cxl/port: Add Dynamic Capacity size support to endpoint decoders
> cxl/region: Add Dynamic Capacity CXL region support
> cxl/mem: Read extents on memory device discovery
> cxl/mem: Handle DCD add and release capacity events.
> cxl/region: Expose DC extents on region driver load
> cxl/region: Notify regions of DC changes
> dax/bus: Factor out dev dax resize logic
> dax/region: Support DAX device creation on dynamic DAX regions
> tools/testing/cxl: Make event logs dynamic
> tools/testing/cxl: Add DC Regions to mock mem data
> tools/testing/cxl: Add Dynamic Capacity events
>
> Navneet Singh (3):
> cxl/mem: Read Dynamic capacity configuration from the device
> cxl/mem: Expose device dynamic capacity configuration
> cxl/mem: Trace Dynamic capacity Event Record
>
> Documentation/ABI/testing/sysfs-bus-cxl | 56 ++-
> drivers/cxl/core/core.h | 1 +
> drivers/cxl/core/hdm.c | 215 ++++++++-
> drivers/cxl/core/mbox.c | 646 +++++++++++++++++++++++++-
> drivers/cxl/core/memdev.c | 77 ++++
> drivers/cxl/core/port.c | 19 +
> drivers/cxl/core/region.c | 418 +++++++++++++++--
> drivers/cxl/core/trace.h | 65 +++
> drivers/cxl/cxl.h | 99 +++-
> drivers/cxl/cxlmem.h | 138 +++++-
> drivers/cxl/mem.c | 50 ++
> drivers/cxl/pci.c | 8 +
> drivers/dax/Makefile | 1 +
> drivers/dax/bus.c | 263 ++++++++---
> drivers/dax/bus.h | 1 +
> drivers/dax/cxl.c | 213 ++++++++-
> drivers/dax/dax-private.h | 61 +++
> drivers/dax/extent.c | 133 ++++++
> tools/testing/cxl/test/mem.c | 782 +++++++++++++++++++++++++++-----
> 19 files changed, 3005 insertions(+), 241 deletions(-)
> ---
> base-commit: c76cce37fb6f3796e8e146677ba98d3cca30a488
> change-id: 20230604-dcd-type2-upstream-0cd15f6216fd
>
> Best regards,
> --
> Ira Weiny <ira.weiny@xxxxxxxxx>
>