Re: [PATCH RFC 04/15] drivers/base: Add support for a new IMS irq domain

[Dey, Megha]

Hi Jason,

On 4/23/2020 1:11 PM, Jason Gunthorpe wrote:
> On Tue, Apr 21, 2020 at 04:34:11PM -0700, Dave Jiang wrote:
> > diff --git a/drivers/base/ims-msi.c b/drivers/base/ims-msi.c
> > new file mode 100644
> > index 000000000000..738f6d153155
> > +++ b/drivers/base/ims-msi.c
> > @@ -0,0 +1,100 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * Support for Device Specific IMS interrupts.
> > + *
> > + * Copyright  2019 Intel Corporation.
> > + *
> > + * Author: Megha Dey <megha.dey@xxxxxxxxx>
> > + */
> > +
> > +#include <linux/dmar.h>
> > +#include <linux/irq.h>
> > +#include <linux/mdev.h>
> > +#include <linux/pci.h>
> > +
> > +/*
> > + * Determine if a dev is mdev or not. Return NULL if not mdev device.
> > + * Return mdev's parent dev if success.
> > + */
> > +static inline struct device *mdev_to_parent(struct device *dev)
> > +{
> > +	struct device *ret = NULL;
> > +	struct device *(*fn)(struct device *dev);
> > +	struct bus_type *bus = symbol_get(mdev_bus_type);
> > +
> > +	if (bus && dev->bus == bus) {
> > +		fn = symbol_get(mdev_dev_to_parent_dev);
> > +		ret = fn(dev);
> > +		symbol_put(mdev_dev_to_parent_dev);
> > +		symbol_put(mdev_bus_type);
>
> No, things like this are not OK in the drivers/base
>
> Whatever this is doing needs to be properly architected in some
> generic way.

Basically what I am trying to do here is to determine if the device is 
an mdev device or not. mdev devices have no IRQ domain associated to it 
and use their parent dev's IRQ domain to allocate interrupts.

The issue is that
1. all the vfio-mdev code today can be compiled as a module
2. None of the mdev macros/functions are being used outside of the 
drivers/vfio/mdev code path (where they are defined).

Hence, these definitions are not visible outside of drivers/vfio/mdev 
when compiled as a module and thus I have used symbol_get/put.

I will try asking the mdev folks if they would have a better solution 
for this or some of this code can be mde more generic.

> > +static int dev_ims_prepare(struct irq_domain *domain, struct device *dev,
> > +			   int nvec, msi_alloc_info_t *arg)
> > +{
> > +	if (dev_is_mdev(dev))
> > +		dev = mdev_to_parent(dev);
>
> Like maybe the caller shouldn't be passing in a mdev in the first
> place, or some generic driver layer scheme is needed to go from a
> child device (eg a mdev or one of these new virtual bus things) to the
> struct device that owns the IRQ interface.

In our current use case, IMS interrupts are only used by guest (mdev's), 
although they can be used by host as well. So the 'dev' passed by the 
caller of platform_msi_domain_alloc_irqs_group() is effectively an mdev.

I am not sure about how we could have a generic code to convert the 
'child' mdev device to struct device. Do you have any suggestions on how 
we could do this?

> > +	init_irq_alloc_info(arg, NULL);
> > +	arg->dev = dev;
> > +	arg->type = X86_IRQ_ALLOC_TYPE_IMS;
>
> Also very bewildering to see X86_* in drivers/base

Well, this needs to go for sure. I will replace it with something more 
generic.
> > +struct irq_domain *arch_create_ims_irq_domain(struct irq_domain *parent,
> > +					      const char *name)
> > +{
> > +	struct fwnode_handle *fn;
> > +	struct irq_domain *domain;
> > +
> > +	fn = irq_domain_alloc_named_fwnode(name);
> > +	if (!fn)
> > +		return NULL;
> > +
> > +	domain = msi_create_irq_domain(fn, &ims_ir_domain_info, parent);
> > +	if (!domain)
> > +		return NULL;
> > +
> > +	irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
> > +	irq_domain_free_fwnode(fn);
> > +
> > +	return domain;
> > +}
>
> I'm still not really clear why all this is called IMS.. This looks
> like the normal boilerplate to setup an IRQ domain? What is actually
> 'ims' in here?

It is just a way to create a new domain specifically for IMS interrupts. 
Although, since there is a platform_msi_create_irq_domain already, which 
does something similar, I will use the same for IMS as well.

Also, since there is quite a stir over the name 'IMS' do you have any 
suggestion for a more generic name for this?

> > diff --git a/drivers/vfio/mdev/mdev_private.h b/drivers/vfio/mdev/mdev_private.h
> > index 7d922950caaf..c21f1305a76b 100644
> > +++ b/drivers/vfio/mdev/mdev_private.h
> > @@ -36,7 +36,6 @@ struct mdev_device {
> >   };
> >   
> >   #define to_mdev_device(dev)	container_of(dev, struct mdev_device, dev)
> > -#define dev_is_mdev(d)		((d)->bus == &mdev_bus_type)
> >   
> >   struct mdev_type {
> >   	struct kobject kobj;
> > diff --git a/include/linux/mdev.h b/include/linux/mdev.h
> > index 0ce30ca78db0..fa2344e239ef 100644
> > +++ b/include/linux/mdev.h
> > @@ -144,5 +144,8 @@ void mdev_unregister_driver(struct mdev_driver *drv);
> >   struct device *mdev_parent_dev(struct mdev_device *mdev);
> >   struct device *mdev_dev(struct mdev_device *mdev);
> >   struct mdev_device *mdev_from_dev(struct device *dev);
> > +struct device *mdev_dev_to_parent_dev(struct device *dev);
> > +
> > +#define dev_is_mdev(dev) ((dev)->bus == symbol_get(mdev_bus_type))
>
> NAK on the symbol_get

As I mentioned earlier, given the way the current mdev code is 
structured, the only way to use dev_is_mdev or other macros/functions in 
the mdev subsystem outside of drivers/vfio/mdev is to use 
symbol_get/put. Obviously, seems like this is not a correct thing to do, 
I will have to find another way.

> Jason