Re: [PATCH 00/26] DCD: Add support for Dynamic Capacity Devices (DCD)

[Jonathan Cameron]

On Sun, 5 May 2024 21:24:14 -0700
Ira Weiny <ira.weiny@xxxxxxxxx> wrote:

> Jonathan Cameron wrote:
> > On Wed, 1 May 2024 16:49:24 -0700
> > Ira Weiny <ira.weiny@xxxxxxxxx> wrote:
> >   
> > > Jonathan Cameron wrote:  
> > > >     
> > > > > 
> > > > > Fan Ni's latest v5 of Qemu DCD was used for testing.[2]    
> > > > Hi Ira, Navneet.    
> > > > > 
> > > > > Remaining work:
> > > > > 
> > > > > 	1) Integrate the QoS work from Dave Jiang
> > > > > 	2) Interleave support    
> > > > 
> > > > 
> > > > More flag.  This one I think is potentially important and don't
> > > > see any handling in here.    
> > > 
> > > Nope I admit I missed the spec requirement.
> > >   
> > > > 
> > > > Whilst an FM could in theory be careful to avoid sending a
> > > > sparse set of extents, if the device is managing the memory range
> > > > (which is possible all it supports) and the FM issues an Initiate Dynamic
> > > > Capacity Add with Free (again may be all device supports) then we
> > > > can't stop the device issuing a bunch of sparse extents.
> > > > 
> > > > Now it won't be broken as such without this, but every time we
> > > > accept the first extent that will implicitly reject the rest.
> > > > That will look very ugly to an FM which has to poke potentially many
> > > > times to successfully allocate memory to a host.    
> > > 
> > > This helps me to see see why the more bit is useful.
> > >   
> > > > 
> > > > I also don't think it will be that hard to support, but maybe I'm
> > > > missing something?     
> > > 
> > > Just a bunch of code and refactoring busy work.  ;-)  It's not rocket
> > > science but does fundamentally change the arch again.
> > >   
> > > > 
> > > > My first thought is it's just a loop in cxl_handle_dcd_add_extent()
> > > > over a list of extents passed in then slightly more complex response
> > > > generation.    
> > > 
> > > Not exactly 'just a loop'.  No matter how I work this out there is the
> > > possibility that some extents get surfaced and then the kernel tries to
> > > remove them because it should not have.  
> > 
> > Lets consider why it might need to back out.
> > 1) Device sends an invalid set of extents - so maybe one in a later message
> >    overlaps with an already allocated extent.   Device bug, handling can
> >    be extremely inelegant - up to crashing the kernel.  Worst that happens
> >    due to race is probably a poison storm / machine check fun?  Not our
> >    responsibility to deal with something that broken (in my view!) Best effort
> >    only.
> > 
> > 2) Host can't handle the extent for some reason and didn't know that until
> >    later - can just reject the ones it can't handle.   
> 
> 3) Something in the host fails like ENOMEM on a later extent surface which
>    requires the host to back out of all of them.
> 
> 3 should be rare and I'm working toward it.  But it is possible this will
> happen.
> 
> If you have a 'prepare' notify it should avoid most of these because the
> extents will be mostly formed.  But there are some error paths on the actual
> surface code path.

True. If these are really small allocations then elegant handling feels like
a nice to have rather than a requirement.

> 
> >   
> > > 
> > > To be most safe the cxl core is going to have to make 2 round trips to the
> > > cxl region layer for each extent.  The first determines if the extent is
> > > valid and creates the extent as much as possible.  The second actually
> > > surfaces the extents.  However, if the surface fails then you might not
> > > get the extents back.  So now we are in an invalid state.  :-/  WARN and
> > > continue I guess?!??!  
> > 
> > Yes. Orchestrator can decide how to handle - probably reboot server in as
> > gentle a fashion as possible.
> >   
> 
> Ok
> 
> >   
> > > 
> > > I think the safest way to handle this is add a new kernel notify event
> > > called 'extent create' which stops short of surfacing the extent.  [I'm
> > > not 100% sure how this is going to affect interleave.]
> > > 
> > > I think the safest logic for add is something like:
> > > 
> > > 	cxl_handle_dcd_add_event()
> > > 		add_extent(squirl_list, extent);
> > > 
> > > 		if (more bit) /* wait for more */
> > > 			return;
> > > 
> > > 		/* Create extents to hedge the bets against failure */
> > > 		for_each(squirl_list)
> > > 			if (notify 'extent create' != ok)
> > > 				send_response(fail);
> > > 				return;
> > > 		
> > > 		for_each(squirl_list)
> > > 			if (notify 'surface' != ok)
> > > 				/*
> > > 				 * If the more bit was set, some extents
> > > 				 * have been surfaced and now need to be
> > > 				 * removed...
> > > 				 *
> > > 				 * Try to remove them and hope...
> > > 				 */  
> > 
> > If we failed to surface them all another option is just tell the device
> > that.   Responds with the extents that successfully surfaced and reject
> > all others (or all after the one that failed?)  So for the lower layers
> > send the device a response that says "thanks but I only took these ones"
> > and for the upper layers pretend "I was only offered these ones"
> >   
> 
> But doesn't that basically break the more bit?  I'm willing to do that as it is
> easier for the host.

Don't think so.  We can always accept part of the offered extents in same
way we can accept part of a single offered extent if we like.
The more flag just means we only get to do that communication of what
we accepted once. So we have to reply with what we want and don't set
more flag in last message - thus indicating we don't want the rest.
(making sure we also tidy up the log for the ones we rejected)

> 
> > > 				WARN_ON('surface extents failed');
> > > 				for_each(squirl_list)
> > > 					notify 'remove without response'
> > > 				send_response(fail);
> > > 				return;
> > > 
> > > 		send_response(squirl_list, accept);
> > > 
> > > The logic for remove is not changed AFAICS because the device must allow
> > > for memory to be released at any time so the host is free to release each
> > > of the extents individually despite the 'more' bit???  
> > 
> > Yes, but only after it accepted them - which needs to be done in one go.
> > So you can't just send releases before that (the device will return an
> > error and keep them in the pending list I think...)  
> 
> :-(  OK so this more bit is really more...  no pun intended.  Because this
> breaks the entire model I have if I have to treat these as a huge atomic unit.
> 
> Let me think on that a bit more.  Obviously it is just tagging an iterating the
> extents to find those associated with a more bit on accept.  But it will take
> some time to code up.

The ability to give up at any point (though you need to read and clear the extents
that are left) should get around a lot of the complexity but sure it's
not a trivial thing to support.

I'd flip a 'something went wrong flag' on the the first failure, carry on the
walk not surfacing anything else, but clearing the logs etc, then finally reply
with what succeeded before that 'went wrong' flag was set.

> 
> >   
> > >   
> > > > 
> > > > I don't want this to block getting initial DCD support in but it
> > > > will be a bit ugly if we quickly support the more flag and then end
> > > > up with just one kernel that an FM has to be careful with...    
> > > 
> > > I'm not sure which is worse.  Given your use case above it seems like the
> > > more bit may be more important for 'dumb' devices which want to add
> > > extents in blocks before responding to the FM.  Thus complicating the FM.
> > > 
> > > It seems 'smarter' devices which could figure this out (not requiring the
> > > more bit) are the ones which will be developed later.  So it seems the use
> > > case time line is the opposite of what we need right now.  
> > 
> > Once we hit shareable capacity (which the smarter devices will use) then
> > this become the dominant approach to non contiguous allocations because
> > you can't add extents with a given tag in multiple goes.  
> 
> Why not?  Sharing is going to require some synchronization with the
> orchestrator and can't the user app just report it did not get all it's memory
> and wait for more?  With the same tag?

Hmm. I was sure the spec said sharing did not allow addition of capacity after
first creation, but now can't find it.  If you did do it though, fun
occurs when you then pass it on to the second device because you have
to do that via tag alone.

I believe this is about simplification on the device side because
offers of extents to other hosts are done by tag. If you allow extra ones
to turn up there are race conditions to potentially deal with.

7.6.7.6.5 Initiate Dynamic Capacity add.

"Enable shared Access" Enable access to extents previously added to another
host in a DC region that reports the "sharable" flag, as designated by the
specific tag value.

Note it is up to the device to offer the same capacity to all hosts for
which this is issued.  There is no extent list or length provided.


> 
> > 
> > So I'd expect the more flag to be more common not less over time.  
> > > 
> > > For that reason I'm inclined to try and get this in.
> > >   
> > 
> > Great - but I'd not worry too much about bad effects if you get invalid
> > lists from the device.  If the only option is shout and panic, then fine
> > though I'd imagine we can do slightly better than that, so maybe warn
> > extensively and don't let the region be used.  
> 
> It is not just about invalid lists.  It is that setting up the extent devices
> may fail and waiting for the devices to be set up means that they are user
> visible.  So that is the chicken and the egg...
> 
> This is unlikely and perhaps the partials should just be surfaced and accept
> whatever works.  Then let it all tear down later if it does not all go.
> 
> But I was trying to honor the accept 'all or nothing' as that is what has been
> stated as the requirement of the more bit.

That's not quite true - for shared it is all or nothing (after first host anyway) but
for other capacity it is 'accept X and reject Y in one go'.  You don't need to
take it all but you only get one go to say what you did accept.

> 
> But it seems that it does not __have__ to be atomic.  Or at least the partials
> can be cleaned up and all tried again.

With care you can accept up to a point, then give those back if you like - or
carry on and use them.

Jonathan

> 
> Ira
> 
> > 
> > Jonathan
> >   
> > > Ira
> > >   
> >   
> 
> 
>