Re: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node

[Jarkko Sakkinen]

On Fri, 2021-10-29 at 16:53 +0200, Greg Kroah-Hartman wrote:
> On Fri, Oct 29, 2021 at 03:18:57PM +0300, Jarkko Sakkinen wrote:
> > The amount of SGX memory on the system is determined by the BIOS and it
> > varies wildly between systems.  It can be from dozens of MB's on desktops
> > or VM's, up to many GB's on servers.  Just like for regular memory, it is
> > sometimes useful to know the amount of usable SGX memory in the system.
> > 
> > Add an attribute for the amount of SGX memory in bytes to each NUMA
> > node. The path is /sys/devices/system/node/node[0-9]*/sgx/size.
> > Calculate these values by summing up EPC section sizes for each node
> > during the driver initalization.
> > 
> > Cc: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
> > Signed-off-by: Jarkko Sakkinen <jarkko@xxxxxxxxxx>
> > ---
> > v9:
> > * Fix racy initialization of sysfs attributes:
> >   https://lore.kernel.org/linux-sgx/YXOsx8SvFJV5R7lU@xxxxxxxxx/
> > 
> > v8:
> > * Fix a bug in sgx_numa_init(): node->dev should be only set after
> >   sysfe_create_group().  Otherwise, sysfs_remove_group() will issue a
> >   warning in sgx_numa_exit(), when sgx_create_group() is unsuccessful,
> >   because the group does not exist.
> > 
> > v7:
> > * Shorten memory_size to size. The prefix makes the name only longer
> >   but does not clarify things more than "size" would.
> > * Use device_attribute instead of kobj_attribute.
> > * Use named attribute group instead of creating raw kobject just for
> >   the "sgx" subdirectory.
> > 
> > v6:
> > * Initialize node->size to zero in sgx_setup_epc_section(), when the
> >   node is first accessed.
> > 
> > v5
> > * A new patch based on the discussion on
> >   https://lore.kernel.org/linux-sgx/3a7cab4115b4f902f3509ad8652e616b91703e1d.camel@xxxxxxxxxx/T/#t
> > ---
> >  Documentation/ABI/stable/sysfs-devices-node |  7 +++++
> >  arch/x86/include/asm/sgx.h                  |  2 ++
> >  arch/x86/kernel/cpu/sgx/main.c              | 31 +++++++++++++++++++++
> >  arch/x86/kernel/cpu/sgx/sgx.h               |  2 ++
> >  drivers/base/node.c                         | 16 ++++++++++-
> >  5 files changed, 57 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node
> > index 484fc04bcc25..12dc2149e8e0 100644
> > --- a/Documentation/ABI/stable/sysfs-devices-node
> > +++ b/Documentation/ABI/stable/sysfs-devices-node
> > @@ -176,3 +176,10 @@ Contact:   Keith Busch <keith.busch@xxxxxxxxx>
> >  Description:
> >                 The cache write policy: 0 for write-back, 1 for write-through,
> >                 other or unknown.
> > +
> > +What:          /sys/devices/system/node/nodeX/sgx/size
> > +Date:          October 2021
> > +Contact:       Jarkko Sakkinen <jarkko@xxxxxxxxxx>
> > +Description:
> > +               Total available physical SGX memory, also known as Enclave Page
> > +               Cache (EPC), in bytes.
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > index 05f3e21f01a7..43d1ccae6c62 100644
> > --- a/arch/x86/include/asm/sgx.h
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -365,6 +365,8 @@ struct sgx_sigstruct {
> >   * comment!
> >   */
> >  
> > +extern const struct attribute_group sgx_node_group;
> > +
> >  #ifdef CONFIG_X86_SGX_KVM
> >  int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
> >                      int *trapnr);
> > diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
> > index a6e313f1a82d..86d6d309ff29 100644
> > --- a/arch/x86/kernel/cpu/sgx/main.c
> > +++ b/arch/x86/kernel/cpu/sgx/main.c
> > @@ -714,9 +714,12 @@ static bool __init sgx_page_cache_init(void)
> >                         spin_lock_init(&sgx_numa_nodes[nid].lock);
> >                         INIT_LIST_HEAD(&sgx_numa_nodes[nid].free_page_list);
> >                         node_set(nid, sgx_numa_mask);
> > +                       sgx_numa_nodes[nid].size = 0;
> >                 }
> >  
> >                 sgx_epc_sections[i].node =  &sgx_numa_nodes[nid];
> > +               sgx_numa_nodes[nid].dev = &node_devices[nid]->dev;
> > +               sgx_numa_nodes[nid].size += size;
> >  
> >                 sgx_nr_epc_sections++;
> >         }
> > @@ -790,6 +793,34 @@ int sgx_set_attribute(unsigned long *allowed_attributes,
> >  }
> >  EXPORT_SYMBOL_GPL(sgx_set_attribute);
> >  
> > +#ifdef CONFIG_NUMA
> > +static ssize_t size_show(struct device *dev, struct device_attribute *attr, char *buf)
> > +{
> > +       unsigned long size = 0;
> > +       int nid;
> > +
> > +       for (nid = 0; nid < num_possible_nodes(); nid++) {
> > +               if (dev == sgx_numa_nodes[nid].dev) {
> > +                       size = sgx_numa_nodes[nid].size;
> > +                       break;
> > +               }
> > +       }
> > +
> > +       return sysfs_emit(buf, "%lu\n", size);
> > +}
> > +DEVICE_ATTR_RO(size);
> 
> static?

The named ("sgx") attribute group is exported:

extern const struct attribute_group sgx_node_group;

and added as an additional group of the node device:

static const struct attribute_group node_dev_group = {
	.attrs = node_dev_attrs,
};

static const struct attribute_group *node_dev_groups[] = {
	&node_dev_group,
#ifdef CONFIG_X86_SGX
	&sgx_node_group,
#endif /* CONFIG_X86_SGX */
	NULL,
};
 
An alternative would to define a helper, e.g.

ssize_t sgx_get_node_size(struct device *dev);

And use this in drivers/base/node.c to implement the attribute. I'm
fine either way...

/Jarkko