Re: [PATCH] x86/gpu: add JSL stolen memory support

[Bjorn Helgaas]

On Fri, Nov 06, 2020 at 10:39:16AM +0100, Daniel Vetter wrote:
> On Thu, Nov 5, 2020 at 3:17 PM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> > On Thu, Nov 05, 2020 at 11:46:06AM +0200, Joonas Lahtinen wrote:
> > > Quoting Bjorn Helgaas (2020-11-04 19:35:56)
> > > > [+cc Jani, Joonas, Rodrigo, David, Daniel]
> > > >
> > > > On Wed, Nov 04, 2020 at 05:35:06PM +0530, Tejas Upadhyay wrote:
> > > > > JSL re-uses the same stolen memory as ICL and EHL.
> > > > >
> > > > > Cc: Lucas De Marchi <lucas.demarchi@xxxxxxxxx>
> > > > > Cc: Matt Roper <matthew.d.roper@xxxxxxxxx>
> > > > > Signed-off-by: Tejas Upadhyay <tejaskumarx.surendrakumar.upadhyay@xxxxxxxxx>
> > > >
> > > > I don't plan to do anything with this since previous similar patches
> > > > have gone through some other tree, so this is just kibitzing.
> > > >
> > > > But the fact that we have this long list of Intel devices [1] that
> > > > constantly needs updates [2] is a hint that something is wrong.
> > >
> > > We add an entry for every new integrated graphics platform. Once the
> > > platform is added, there have not been changes lately.
> > >
> > > > IIUC the general idea is that we need to discover Intel gfx memory by
> > > > looking at device-dependent config space and add it to the E820 map.
> > > > Apparently the quirks discover this via PCI config registers like
> > > > I830_ESMRAMC, I845_ESMRAMC, etc, and tell the driver about it via the
> > > > global "intel_graphics_stolen_res"?
> > >
> > > We discover what is called the graphics data stolen memory. It is regular
> > > system memory range that is not CPU accessible. It is accessible by the
> > > integrated graphics only.
> > >
> > > See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/kernel/early-quirks.c?h=v5.10-rc2&id=814c5f1f52a4beb3710317022acd6ad34fc0b6b9
> > >
> > > > That's not the way this should work.  There should some generic, non
> > > > device-dependent PCI or ACPI method to discover the memory used, or at
> > > > least some way to do it in the driver instead of early arch code.
> > >
> > > It's used by the early BIOS/UEFI code to set up initial framebuffer.
> > > Even if i915 driver is never loaded, the memory ranges still need to
> > > be fixed. They source of the problem is that the OEM BIOS which are
> > > not under our control get the programming wrong.
> > >
> > > We used to detect the memory region size again at i915 initialization
> > > but wanted to eliminate the code duplication and resulting subtle bugs
> > > that caused. Conclusion back then was that storing the struct resource
> > > in memory is the best trade-off.
> > >
> > > > How is this *supposed* to work?  Is there something we can do in E820
> > > > or other resource management that would make this easier?
> > >
> > > The code was added around Haswell (HSW) device generation to mitigate
> > > bugs in BIOS. It is traditionally hard to get all OEMs to fix their
> > > BIOS when things work for Windows. It's only later years when some
> > > laptop models are intended to be sold with Linux.
> > >
> > > The alternative would be to get all the OEM to fix their BIOS for Linux,
> > > but that is not very realistic given past experiences. So it seems
> > > a better choice to to add new line per platform generation to make
> > > sure the users can boot to Linux.
> >
> > How does Windows do this?  Do they have to add similar code for each
> > new platform?
> 
> Windows is chicken and doesn't move any mmio bar around on its own.
> Except if the bios explicitly told it somehow (e.g. for the 64bit bar
> stuff amd recently announced for windows, that linux supports since
> years by moving the bar). So except if you want to preemptively
> disable the pci code that does this anytime there's an intel gpu, this
> is what we have to do.

I think Windows *does* move BARs (they use the more generic
terminology of "rebalancing PNP resources") in some cases [3,4].  Of
course, I'm pretty sure Windows will only assign PCI resources inside
the windows advertised in the host bridge _CRS.

Linux *used* to ignore that host bridge _CRS and could set BARs to
addresses that appeared available but were in fact used by the
platform somehow.  But Linux has been paying attention to host bridge
_CRS for a long time now, so it should also only assign resources
inside those windows.

So I'm missing what the real problem is.  Joonas mentioned BIOS bugs
above, but I don't know what that means.

Here's what I can figure out so far, tell me if I'm in the weeds:

  - The intel_graphics_stolen() early quirk:
      1) Initializes global struct resource intel_graphics_stolen_res
      2) Adds the region to E820 map as E820_TYPE_RESERVED

  - i915_driver_hw_probe() uses intel_graphics_stolen_res, but this
    happens after the PCI core is initialized, so early quirks
    wouldn't be required for this use.

  - So I guess the E820 map update is what requires the early quirk?
    But I don't know exactly what depends on this update.

I haven't found the connection to the early BIOS/UEFI code and the
initial framebuffer yet.

> And given that this 64bit mmio bar support in windows still requires
> an explicit bios upgrade for the explicit opt in I don't think this
> will change anytime soon.
> 
> We have a similar ugly problem with kvm, since you can't use these
> ranges freely (they're very special in hw), and the kvm maintainers
> are equally annoyed that they have to keep supporting RRMR to block
> that range, just because of intel integrated graphics. Apparently
> windows is again totally fine with this.
> -Daniel
> 
> 
> >
> > > > > ---
> > > > >  arch/x86/kernel/early-quirks.c | 1 +
> > > > >  1 file changed, 1 insertion(+)
> > > > >
> > > > > diff --git a/arch/x86/kernel/early-quirks.c b/arch/x86/kernel/early-quirks.c
> > > > > index a4b5af03dcc1..534cc3f78c6b 100644
> > > > > --- a/arch/x86/kernel/early-quirks.c
> > > > > +++ b/arch/x86/kernel/early-quirks.c
> > > > > @@ -549,6 +549,7 @@ static const struct pci_device_id intel_early_ids[] __initconst = {
> > > > >       INTEL_CNL_IDS(&gen9_early_ops),
> > > > >       INTEL_ICL_11_IDS(&gen11_early_ops),
> > > > >       INTEL_EHL_IDS(&gen11_early_ops),
> > > > > +     INTEL_JSL_IDS(&gen11_early_ops),
> > > > >       INTEL_TGL_12_IDS(&gen11_early_ops),
> > > > >       INTEL_RKL_IDS(&gen11_early_ops),
> > > > >  };
> > > >
> > > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/early-quirks.c?h=v5.10-rc2#n518
> > > >
> > > > [2]
> > > >   May 2020 efbee021ad02 ("x86/gpu: add RKL stolen memory support")
> > > >   Jul 2019 6b2436aeb945 ("x86/gpu: add TGL stolen memory support")
> > > >   Mar 2019 d53fef0be4a5 ("x86/gpu: add ElkhartLake to gen11 early quirks")
> > > >   May 2018 db0c8d8b031d ("x86/gpu: reserve ICL's graphics stolen memory")
> > > >   Dec 2017 33aa69ed8aac ("x86/gpu: add CFL to early quirks")
> > > >   Jul 2017 2e1e9d48939e ("x86/gpu: CNL uses the same GMS values as SKL")
> > > >   Jan 2017 bc384c77e3bb ("x86/gpu: GLK uses the same GMS values as SKL")
> > > >   Oct 2015 00ce5c8a66fb ("drm/i915/kbl: Kabylake uses the same GMS values as Skylake")
> > > >   Mar 2015 31d4dcf705c3 ("drm/i915/bxt: Broxton uses the same GMS values as Skylake")
> > > >   ...

[3] https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/stopping-a-device-to-rebalance-resources
[4] http://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-174cf47d23cd/cpa070_wh06.ppt ("PCI Express In Depth For Windows Vista and Beyond")