Re: [PATCH v5 2/7] x86/sgx: Add infrastructure to identify SGX EPC pages

[Luck, Tony]

On Tue, Sep 21, 2021 at 03:32:14PM -0700, Dave Hansen wrote:
> On 9/21/21 1:50 PM, Luck, Tony wrote:
> >> Did we ever figure out how much space storing really big ranges in the
> >> xarray consumes?
> > No. Willy said the existing xarray code would be less than optimal with
> > this usage, but that things would be much better when he applied some
> > maple tree updates to the internals of xarray.
> > 
> > If there is some easy way to measure the memory backing an xarray I'm
> > happy to get the data. Or if someone else can synthesize it ... the two
> > ranges on my system that are added to the xarray are:
> > 
> > $ dmesg | grep -i sgx
> > [    8.496844] sgx: EPC section 0x8000c00000-0x807f7fffff
> > [    8.505118] sgx: EPC section 0x10000c00000-0x1007fffffff
> > 
> > I.e. two ranges of a bit under 2GB each.
> > 
> > But I don't think the overhead can be too hideous:
> > 
> > $ grep MemFree /proc/meminfo
> > MemFree:        1048682016 kB
> > 
> > I still have ~ 1TB free. Which is much greater that the 640 KB which should
> > be "enough for anybody" :-).
> 
> There is a kmem_cache_create() for the xarray nodes.  So, you should be
> able to see the difference in /proc/meminfo's "Slab" field.  Maybe boot
> with init=/bin/sh to reduce the noise and look at meminfo both with and
> without SGX your patch applied, or just with the xarray bits commented out.
> 
> I don't quite know how the data structures are munged, but xas_alloc()
> makes it look like 'struct xa_node' is allocated from
> radix_tree_node_cachep.  If that's the case, you should also be able to
> see this in even more detail in:
> 
> # grep radix /proc/slabinfo
> radix_tree_node   432305 482412    584   28    4 : tunables    0    0
>  0 : slabdata  17229  17229      0
> 
> again, on a system with and without your new code enabled.


Booting with init=/bin/sh and running that grep command right away at
the prompt:

With the xa_store_range() call commented out of my kernel:

radix_tree_node     9800   9968    584   56    8 : tunables    0    0    0 : slabdata    178    178      0


With xa_store_range() enabled:

radix_tree_node     9950  10136    584   56    8 : tunables    0    0    0 : slabdata    181    181      0



The head of the file says these are the field names:

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>

So I think this means that I have (9950 - 9800) * 584 = 87600 more bytes
allocated. Maybe that's a lot? But percentage-wise is seems in the
noise. E.g. We allocate one "struct sgx_epc_page" for each SGX page.
On my system I have 4GB of SGX EPC, so around 32 MB of these structures.

-Tony