Re: [PATCH v13 17/18] cxl/memfeature: Add CXL memory device PPR control feature

[Jonathan Cameron]

On Wed, 9 Oct 2024 13:41:18 +0100
<shiju.jose@xxxxxxxxxx> wrote:

> From: Shiju Jose <shiju.jose@xxxxxxxxxx>
> 
> Post Package Repair (PPR) maintenance operations may be supported by CXL
> devices that implement CXL.mem protocol. A PPR maintenance operation
> requests the CXL device to perform a repair operation on its media.
> For example, a CXL device with DRAM components that support PPR features
> may implement PPR Maintenance operations. DRAM components may support two
> types of PPR: Hard PPR (hPPR), for a permanent row repair, and Soft PPR
> (sPPR), for a temporary row repair. sPPR is much faster than hPPR, but the
> repair is lost with a power cycle.
> 
> During the execution of a PPR Maintenance operation, a CXL memory device:
> - May or may not retain data
> - May or may not be able to process CXL.mem requests correctly, including
> the ones that target the DPA involved in the repair.
> These CXL Memory Device capabilities are specified by Restriction Flags
> in the sPPR Feature and hPPR Feature.
> 
> sPPR maintenance operation may be executed at runtime, if data is retained
> and CXL.mem requests are correctly processed. For CXL devices with DRAM
> components, hPPR maintenance operation may be executed only at boot because
> data would not be retained.
> When a CXL device identifies a failure on a memory component, the device
> may inform the host about the need for a PPR maintenance operation by using
> an Event Record, where the Maintenance Needed flag is set. The Event Record
> specifies the DPA that should be repaired. A CXL device may not keep track
> of the requests that have already been sent and the information on which
> DPA should be repaired may be lost upon power cycle.
> The userspace tool requests for maintenance operation if the number of
> corrected error reported on a CXL.mem media exceeds error threshold.
> 
> CXL spec 3.1 section 8.2.9.7.1.2 describes the device's sPPR (soft PPR)
> maintenance operation and section 8.2.9.7.1.3 describes the device's
> hPPR (hard PPR) maintenance operation feature.
> 
> CXL spec 3.1 section 8.2.9.7.2.1 describes the sPPR feature discovery and
> configuration.
> 
> CXL spec 3.1 section 8.2.9.7.2.2 describes the hPPR feature discovery and
> configuration.
> 
> Add support for controlling CXL memory device PPR feature.
> Register with EDAC driver, which gets the memory repair attr descriptors
> from the EDAC memory repair driver and exposes sysfs repair control
> attributes for PRR to the userspace. For example CXL PPR control for the
> CXL mem0 device is exposed in /sys/bus/edac/devices/cxl_mem0/mem_repairX/
> 
> Tested with QEMU patch for CXL PPR feature.
> https://lore.kernel.org/all/20240730045722.71482-1-dave@xxxxxxxxxxxx/
> 
> Signed-off-by: Shiju Jose <shiju.jose@xxxxxxxxxx>
Trivial comments inline.  This description should call out that initial
support is sPPR only, though hPPR is very easy to add.

Jonathan

> ---
>  drivers/cxl/core/memfeature.c | 335 +++++++++++++++++++++++++++++++++-
>  1 file changed, 329 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/cxl/core/memfeature.c b/drivers/cxl/core/memfeature.c
> index 567406566c77..a0c9a6bd73c0 100644
> --- a/drivers/cxl/core/memfeature.c
> +++ b/drivers/cxl/core/memfeature.c
> @@ -18,8 +18,9 @@
>  #include <linux/limits.h>
>  #include <cxl.h>
>  #include <linux/edac.h>
> +#include "core.h"
>  
> -#define CXL_DEV_NUM_RAS_FEATURES	2
> +#define CXL_DEV_NUM_RAS_FEATURES	3
>  #define CXL_DEV_HOUR_IN_SECS	3600
>  
>  #define CXL_SCRUB_NAME_LEN	128
> @@ -723,6 +724,294 @@ static const struct edac_ecs_ops cxl_ecs_ops = {
>  	.set_threshold = cxl_ecs_set_threshold,
>  };
>  
> +/* CXL memory soft PPR & hard PPR control definitions */
Add some specification references for the various structures
etc.

> +static const uuid_t cxl_sppr_uuid =
> +	UUID_INIT(0x892ba475, 0xfad8, 0x474e, 0x9d, 0x3e, 0x69, 0x2c, 0x91,     \
> +		  0x75, 0x68, 0xbb);
> +
> +static const uuid_t cxl_hppr_uuid =
> +	UUID_INIT(0x80ea4521, 0x786f, 0x4127, 0xaf, 0xb1, 0xec, 0x74, 0x59,     \
> +		  0xfb, 0x0e, 0x24);
> +

> +#define CXL_MEMDEV_PPR_DEVICE_INITIATED_MASK BIT(0)
> +#define CXL_MEMDEV_PPR_FLAG_DPA_SUPPORT_MASK BIT(0)
> +#define CXL_MEMDEV_PPR_FLAG_NIBBLE_SUPPORT_MASK BIT(1)
> +#define CXL_MEMDEV_PPR_FLAG_MEM_SPARING_EV_REC_SUPPORT_MASK BIT(2)
> +
> +#define CXL_MEMDEV_PPR_RESTRICTION_FLAG_MEDIA_ACCESSIBLE_MASK BIT(0)
> +#define CXL_MEMDEV_PPR_RESTRICTION_FLAG_DATA_RETAINED_MASK BIT(2)
> +
> +#define CXL_MEMDEV_PPR_SPARING_EV_REC_EN_MASK BIT(0)
> +
> +struct cxl_memdev_ppr_rd_attrs {
> +	u8 max_op_latency;
> +	__le16 op_cap;
> +	__le16 op_mode;
> +	u8 op_class;
> +	u8 op_subclass;
> +	u8 rsvd[9];

Down to here is the common header. Maybe break that out as a separate
structure as we will get more maintenance features.
Also makes the spec reference simpler as some of the flags are
in the generic part (the device initiated one)


> +	u8 ppr_flags;
> +	__le16 restriction_flags;
> +	u8 ppr_op_mode;
> +}  __packed;
> +


> +
> +static int cxl_do_query_ppr(struct device *dev, void *drv_data)
> +{
> +	struct cxl_ppr_context *cxl_ppr_ctx = drv_data;
> +
> +	if (!cxl_ppr_ctx->dpa)
> +		return -EINVAL;
> +
> +	return cxl_mem_ppr_set_attrs(dev, drv_data, CXL_PPR_PARAM_DO_QUERY);
> +}
> +
> +static int cxl_do_ppr(struct device *dev, void *drv_data)
> +{
> +	struct cxl_ppr_context *cxl_ppr_ctx = drv_data;
> +	int ret;
> +
> +	if (!cxl_ppr_ctx->dpa)
> +		return -EINVAL;

blank line here (as in do_query above)

> +	ret = cxl_mem_ppr_set_attrs(dev, drv_data, CXL_PPR_PARAM_DO_PPR);
> +
> +	return ret;
return cxl_mem_ppr_set_attrs()

> +}