Re: [PATCH RESEND v2 00/18] DCD: Add support for Dynamic Capacity Devices (DCD)

[Ira Weiny]

Please ignore this RESEND.  It is a very old version and sent by mistake.

Ira


Ira Weiny wrote:
> A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
> device that implements dynamic capacity.  Dynamic capacity feature
> allows memory capacity to change dynamically, without the need for
> resetting the device.
> 
> Even though this is marked v2 by b4, this is effectively a whole new
> series for DCD support.  Quite a bit of the core support was completed
> by Navneet in [4].  However, the architecture through the CXL region,
> DAX region, and DAX Device layers is completely different.  Particular
> attention was paid to:
> 
> 	1) managing skip resources in the hardware device
> 	2) ensuring the host OS only sent a release memory mailbox
> 	   response when all DAX devices are done using an extent
> 	3) allowing dax devices to span extents
> 	4) allowing dax devices to use parts of extents
> 
> I could say all of the review comments from v1 are addressed but frankly
> the series has changed so much that I can't guarantee anything.
> 
> The series continues to be based on the type-2 work posted from Dan.[2]
> However, my branch with that work is a bit dated.  Therefore I have
> posted this series on github here.[5]
> 
> Testing was sped up with cxl-test and ndctl dcd support.  A preview of
> that work is on github.[6]  In addition Fan Ni's Qemu DCD series was
> used part of the time.[3]
> 
> The major parts of this series are:
> 
> - Get the dynamic capacity (DC) region information from cxl device
> - Configure device DC regions reported by hardware
> - Enhance CXL and DAX regions for DC
> 	a. maintain separation between the hardware extents and the CXL
> 	   region extents to provide for the addition of interleaving in
> 	   the future.
> - Get and maintain the hardware extent lists for each device via an
>   initial extent list and DC event records
>         a. Add capacity Events
> 	b. Add capacity response
> 	b. Release capacity events
> 	d. Release capacity response
> - Notify region layers of extent changes
> - Allow for DAX devices to be created on extents which are surfaced
> - Maintain references on extents which are in use
> 	a. Send Release capacity Response only when DAX devices are not
> 	   using memory
> - Allow DAX region extent labels to change to allow for flexibility in
>   DAX device creation in the future (further enhancements are required
>   to ndctl for this)
> - Trace Dynamic Capacity events
> - Add cxl-test infrastructure to allow for faster unit testing
> 
> To: Dan Williams <dan.j.williams@xxxxxxxxx>
> Cc: Navneet Singh <navneet.singh@xxxxxxxxx>
> Cc: Fan Ni <fan.ni@xxxxxxxxxxx>
> Cc: Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx>
> Cc: Davidlohr Bueso <dave@xxxxxxxxxxxx>
> Cc: Dave Jiang <dave.jiang@xxxxxxxxx>
> Cc: Alison Schofield <alison.schofield@xxxxxxxxx>
> Cc: Vishal Verma <vishal.l.verma@xxxxxxxxx>
> Cc: Ira Weiny <ira.weiny@xxxxxxxxx>
> Cc: linux-cxl@xxxxxxxxxxxxxxx
> Cc: linux-kernel@xxxxxxxxxxxxxxx
> 
> [1] https://lore.kernel.org/all/64326437c1496_934b2949f@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.notmuch/
> [2] https://lore.kernel.org/all/168592149709.1948938.8663425987110396027.stgit@xxxxxxxxxxxxxxxxxxxxxxxxx/
> [3] https://lore.kernel.org/all/6483946e8152f_f1132294a2@iweiny-mobl.notmuch/
> [4] https://lore.kernel.org/r/20230604-dcd-type2-upstream-v1-0-71b6341bae54@xxxxxxxxx
> [5] https://github.com/weiny2/linux-kernel/commits/dcd-v2-2023-08-28
> [6] https://github.com/weiny2/ndctl/tree/dcd-region2
> 
> ---
> Changes in v2:
> - iweiny: Complete rework of the entire series
> - Link to v1: https://lore.kernel.org/r/20230604-dcd-type2-upstream-v1-0-71b6341bae54@xxxxxxxxx
> 
> ---
> Ira Weiny (15):
>       cxl/hdm: Debug, use decoder name function
>       cxl/mbox: Flag support for Dynamic Capacity Devices (DCD)
>       cxl/region: Add Dynamic Capacity decoder and region modes
>       cxl/port: Add Dynamic Capacity mode support to endpoint decoders
>       cxl/port: Add Dynamic Capacity size support to endpoint decoders
>       cxl/region: Add Dynamic Capacity CXL region support
>       cxl/mem: Read extents on memory device discovery
>       cxl/mem: Handle DCD add and release capacity events.
>       cxl/region: Expose DC extents on region driver load
>       cxl/region: Notify regions of DC changes
>       dax/bus: Factor out dev dax resize logic
>       dax/region: Support DAX device creation on dynamic DAX regions
>       tools/testing/cxl: Make event logs dynamic
>       tools/testing/cxl: Add DC Regions to mock mem data
>       tools/testing/cxl: Add Dynamic Capacity events
> 
> Navneet Singh (3):
>       cxl/mem: Read Dynamic capacity configuration from the device
>       cxl/mem: Expose device dynamic capacity configuration
>       cxl/mem: Trace Dynamic capacity Event Record
> 
>  Documentation/ABI/testing/sysfs-bus-cxl |  56 ++-
>  drivers/cxl/core/core.h                 |   1 +
>  drivers/cxl/core/hdm.c                  | 215 ++++++++-
>  drivers/cxl/core/mbox.c                 | 646 +++++++++++++++++++++++++-
>  drivers/cxl/core/memdev.c               |  77 ++++
>  drivers/cxl/core/port.c                 |  19 +
>  drivers/cxl/core/region.c               | 418 +++++++++++++++--
>  drivers/cxl/core/trace.h                |  65 +++
>  drivers/cxl/cxl.h                       |  99 +++-
>  drivers/cxl/cxlmem.h                    | 138 +++++-
>  drivers/cxl/mem.c                       |  50 ++
>  drivers/cxl/pci.c                       |   8 +
>  drivers/dax/Makefile                    |   1 +
>  drivers/dax/bus.c                       | 263 ++++++++---
>  drivers/dax/bus.h                       |   1 +
>  drivers/dax/cxl.c                       | 213 ++++++++-
>  drivers/dax/dax-private.h               |  61 +++
>  drivers/dax/extent.c                    | 133 ++++++
>  tools/testing/cxl/test/mem.c            | 782 +++++++++++++++++++++++++++-----
>  19 files changed, 3005 insertions(+), 241 deletions(-)
> ---
> base-commit: c76cce37fb6f3796e8e146677ba98d3cca30a488
> change-id: 20230604-dcd-type2-upstream-0cd15f6216fd
> 
> Best regards,
> -- 
> Ira Weiny <ira.weiny@xxxxxxxxx>
>