Re: [PATCH RFC] vfio/mdev: delay uevent after initialization complete

[Cornelia Huck]

[cc:ing Greg for his opinion on this; retaining quoting for context]

On Tue, 13 Feb 2018 17:15:02 -0700
Alex Williamson <alex.williamson@xxxxxxxxxx> wrote:

> On Tue, 13 Feb 2018 14:09:01 +0100
> Cornelia Huck <cohuck@xxxxxxxxxx> wrote:
> 
> > On Mon, 12 Feb 2018 14:20:57 -0700
> > Alex Williamson <alex.williamson@xxxxxxxxxx> wrote:
> >   
> > > On Fri,  9 Feb 2018 11:27:16 +0100
> > > Cornelia Huck <cohuck@xxxxxxxxxx> wrote:
> > >     
> > > > The registration code first registers the mdev device, and then
> > > > proceeds to populate sysfs. An userspace application that listens
> > > > for the ADD uevent is therefore likely to look for sysfs entries
> > > > that have not yet been created.
> > > > 
> > > > The canonical way to fix this is to use attribute groups that are
> > > > registered by the driver core before it sends the ADD uevent; I
> > > > unfortunately did not find a way to make this work in this case,
> > > > though.
> > > > 
> > > > An alternative approach is to suppress uevents before we register
> > > > with the core and generate the ADD uevent ourselves after the
> > > > sysfs infrastructure is in place.
> > > > 
> > > > Signed-off-by: Cornelia Huck <cohuck@xxxxxxxxxx>
> > > > ---
> > > > 
> > > > This feels like a band-aid, but I can't figure out how to handle creating
> > > > attribute groups when there's a callback in the parent involved.
> > > > 
> > > > This should address the issue with libvirt's processing of mdevs raised in
> > > > https://www.redhat.com/archives/libvir-list/2018-February/msg00023.html
> > > > - although libvirt will still need to deal with older kernels, of course.
> > > > 
> > > > Best to consider this an untested patch :)      
> > > 
> > > I agree, this feels like a band-aide.  If every device in the kernel
> > > needs to suppress udev events until until some key component is added,
> > > that suggests that either udev is broken in general or not being used
> > > as intended.      
> > 
> > I think udev is working exactly as designed - it's more a problem of
> > when the kernel is sending what kind of notification to userspace, and
> > the particular issue here is how the code sending the event (driver
> > core) and the code assembling part of the user interface (mdev)
> > interact.
> >   
> > > Zongyong submitted a different proposal to fix this
> > > here[1].  That proposal seems a bit more sound and has precedence
> > > elsewhere in the kernel.  What do you think of that approach?  We
> > > don't need both afaict.  Thanks,
> > > 
> > > Alex
> > > 
> > > [1]https://patchwork.kernel.org/patch/10196197/    
> > 
> > Zongyong's patch is sending an additional CHANGE uevent, and I agree
> > that doing both does not make sense. However, I think the CHANGE uevent
> > is not quite suitable in this case, and delaying the ADD uevent is
> > better.
> > 
> > [Warning, the following may be a bit rambling.]
> > 
> > The Linux driver model works under the assumption that any device is
> > represented as an in-kernel object that exposes information and knobs
> > through sysfs. As long as the device exists, userspace can poke at the
> > sysfs entries and retrieve information or configure things.
> > 
> > The idea of the 'ADD' uevent is basically to let userspace know that
> > there is now a new device with its related sysfs entries, and it may
> > look at it and configure it. IOW, if I (as a userspace application) get
> > the ADD uevent, I expect to be able to look at the device's sysfs
> > entries and find all the files/directories that are usually there,
> > without having to wait.
> > 
> > This expectation is broken if a device is first registered with the
> > driver core (generating the ADD uevent) and the driver adds sysfs
> > attributes later. To fix this, the driver core added a way to specify
> > default attribute groups for the device, which are registered by the
> > driver core itself before it generates the ADD uevent. Unfortunately, I
> > did not see a way to do this here (which does not mean there isn't).
> > The alternative was to prevent the driver core from sending the ADD
> > uevent and do it from the mdev code when it was ready.
> > 
> > The 'CHANGE' uevent, on the other hand, tells userspace that something
> > has changed for the device (that already existed). I (as a userspace
> > application) would expect to see it if, for example, the information
> > exposed via sysfs has changed, or maybe even if new, optional, entries
> > have appeared and I might want to rescan. With Zongyong's patch,
> > userspace gets the CHANGE uevent for something that was already
> > expected to be there, and is now _really_ there. It does give userspace
> > an indication that it can now work with the device (which certainly
> > improves things), but I would prefer to get rid of the too-early uevent
> > completely so that userspace does not get notified at all before the
> > device is completely present in sysfs.
> > 
> > So, in short, my patch does 'don't tell userspace until we're really
> > done', and Zongyong's patch does 'tell userspace again when we're
> > really done'.  
> 
> This all sounds reasonable, but don't we have this synchronization
> problem _everywhere_?  I apologize for referencing this bug because it's
> not public (https://bugzilla.redhat.com/show_bug.cgi?id=1376907) but
> the gist of it is that soft-unplugging PCI devices using the remove
> entry in sysfs and re-adding with rescan sysfs entry results in libvirt
> seeing the ADD uevent before the PCI config attribute is created and it
> balks on the device. 

Yikes. So it seems that _any_ PCI device is incomplete when the ADD
uevent is sent? Or does this just apply to an unfortunately large
number of drivers?

> So at the PCI core we have this same issue and
> developers are saying that there's no guarantee that sysfs entries
> won't be added and removed at any time in the lifecycle of the device
> and it's not the kernel's responsibility to provide that
> synchronization. 

I think this is mixing up two things:
- sysfs entries that are there by default. These not being in place
  when the ADD uevent is sent sounds broken.
- Optional sysfs entries that might change. There may be a case for
  those to be added/removed later on (probably paired with a CHANGE
  uevent.)

> So what are we to do?  If this issue exists in a far
> more predominant device subsystem than mdev, do we need to reset
> developer expectations about what sort of synchronization, if any, the
> kernel is intending to provide?  Should we agree and document some best
> practices around ordering of (or suppressing of) uevents so that we can
> consistently cleanup subsystems, or define that such ordering is
> officially not guaranteed?  Thanks,
> 
> Alex

I'd like to clarify expectations here as well. My understanding has
been what I outlined above, and what I think allows userspace to rely
on asynchronous notifications without having to resort to polling,
waiting, etc.