Re: [RFC] PCI: Unassigned Expansion ROM BARs

[Alex Williamson]

On Fri, 2015-09-25 at 09:35 -0500, Bjorn Helgaas wrote:
> On Thu, Sep 24, 2015 at 09:35:20PM -0700, Yinghai Lu wrote:
> > On Thu, Sep 24, 2015 at 12:01 PM, Yinghai Lu <yinghai@xxxxxxxxxx> wrote:
> > 
> > > Or do we want to keep a white list to say which device should have
> > > ROM bar as mush have, and other is optional to have ?
> > 
> > Subject: [RFC PATCH] PCI: Add pci_dev_need_rom_bar()
> > 
> > Only set that for
> > 1. if BIOS/firmware already set ROM bar.
> > 2. via quirks for some devices.
> > 
> > We assign those needed ROM bar as required
> > and other ROM bars as optional resources.
> 
> I'd rather not have a whitelist if we can avoid it.  We'd be
> continually adding new devices to it, and it makes the system harder
> to understand because there's no consistent rule about how ROMs are
> handled.
> 
> Alex mentioned the idea of ripping the ROM, and I'd like to explore
> that idea a little more.  What if we could temporarily assign space
> for the ROM during enumeration, read the contents, cache the contents
> somewhere, then deallocate the actual BAR space?  We could hang onto
> the cache and give it to anybody who later needs the ROM.
> 
> I know there are probably issues here, but I don't know what they are,
> so I'd like to at least have a conversation about it.

Ok, so for background in the case I mentioned, we can often use the
pci-sysfs rom interface to get a copy of the device ROM, which we then
pass to QEMU and we avoid any access to the physical ROM from the VM.
We can obviously get the ROM from other sources too, but that's not
really relevant here.

If we want to extend this idea into the kernel, creating a buffer that
holds the ROM contents that we access rather than mapping and enabling
the ROM to provide access, we first need to get access to the ROM at
least once.  The simplification here is that we can do this on boot and
we can re-use the space allocated to the standard BARs since we don't
need space for both the ROM and the standard BARs at the same time.  I
think that in the vast majority of cases, we're going to find that the
ROM BAR is smaller than the largest standard MMIO BAR of the device or
at least smaller than the minimum bridge aperture.  This suggests that
we could simply record the BAR values, reprogram them to zero, then
overlap the ROM BAR to the orignal addresses, enable, rip the ROM, then
restore the configuration without needing to adjust any bridge
apertures.

That sounds pretty good, so long as we can consider the ROM to be
perfectly static.  I don't know if anything relies on an in-place update
or if there are ROMs that are less read-only than others.  Maybe it's
safe to assume that or at least safe to assume that if the BIOS didn't
leave room for them, then we can consider them static.  It might be
interesting to strace some of the userspace firmware update programs for
add-in cards to see if they re-read the ROM to determine if it has
changed.

We already sort of do this for VGA ROMs.  When a driver tries to map the
boot VGA ROM they actually map the shadow copy at 0xC0000 (iirc) rather
than the one on the device.  This actually sort of sucks because this
particular shadow copy certainly is not read-only and in all the glory
that is VGA, the shadow sometimes gets modified by the execution of the
VGA ROM itself and we no longer have access to the pristine device
version of it (bad for device assignment of primary graphics).

Now, if we want to make our own shadow copies for all ROMs, it seems
pretty clear that we first need to get access to the ROM, which means we
need to figure out if the BIOS mapped it.  If the ROM BAR is outside of
the bridge aperture (catching both 0 and 0xFFF..000) or overlaps a
standard ROM BAR, we can consider it unprogrammed.  In that case we need
to try to do the trick above with unmapping standard BARs to get the
shadow copy.  Otherwise we should be able to get the shadow copy in
place (maybe a question of which we prefer to use in this case).  I
would be willing to risk that if the BIOS didn't leave room for the ROM
and we can't map it into the space used by other BARs or it doesn't fit
the bridge aperture, we can spit out a printk warning and skip it.  I
expect a very low failure rate.

What dependencies would we have on the BIOS programming of the ROM BAR
if we took such an approach?  It seems like we should always be able to
detect invalid contents in the ROM BAR and it doesn't matter if they
think we should have access or not, we own the device and can give
ourselves access.  Thanks,

Alex

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/