Re: [PATCH v6 23/27] dax/region: Create resources on sparse DAX regions

[Jonathan Cameron]

On Tue, 05 Nov 2024 12:38:45 -0600
ira.weiny@xxxxxxxxx wrote:

> From: Navneet Singh <navneet.singh@xxxxxxxxx>
> 
> DAX regions which map dynamic capacity partitions require that memory be
> allowed to come and go.  Recall sparse regions were created for this
> purpose.  Now that extents can be realized within DAX regions the DAX
> region driver can start tracking sub-resource information.
> 
> The tight relationship between DAX region operations and extent
> operations require memory changes to be controlled synchronously with
> the user of the region.  Synchronize through the dax_region_rwsem and by
> having the region driver drive both the region device as well as the
> extent sub-devices.
> 
> Recall requests to remove extents can happen at any time and that a host
> is not obligated to release the memory until it is not being used.  If
> an extent is not used allow a release response.
> 
> When extents are eligible for release.  No mappings exist but data may
> reside in caches not yet written to the device.  Call
> cxl_region_invalidate_memregion() to write back data to the device prior
> to signaling the release complete.
> 
> Speculative writes after a release may dirty the cache such that a read
> from a newly surfaced extent may not come from the device.  Call
> cxl_region_invalidate_memregion() prior to bringing a new extent online
> to ensure the cache is marked invalid.
> 
> While these invalidate calls are inefficient they are the best we can do
> to ensure cache consistency without back invalidate.  Furthermore this
> should occur infrequently with sufficiently large extents and work loads
> to not be too bad of an impact.
> 
> The DAX layer has no need for the details of the CXL memory extent
> devices.  Expose extents to the DAX layer as device children of the DAX
> region device.  A single callback from the driver aids the DAX layer to
> determine if the child device is an extent.  The DAX layer also
> registers a devres function to automatically clean up when the device is
> removed from the region.
> 
> There is a race between extents being surfaced and the dax_cxl driver
> being loaded.  The driver must therefore scan for any existing extents
> while still under the device lock.
> 
> Respond to extent notifications.  Manage the DAX region resource tree
> based on the extents lifetime.  Return the status of remove
> notifications to lower layers such that it can manage the hardware
> appropriately.
> 
> Signed-off-by: Navneet Singh <navneet.singh@xxxxxxxxx>
> Co-developed-by: Ira Weiny <ira.weiny@xxxxxxxxx>
> Signed-off-by: Ira Weiny <ira.weiny@xxxxxxxxx>
Same again.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx>