Re: [PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[Anirudh Srinivasan]

Hello Bjorn,

On Wed, Aug 26, 2026 at 2:55 PM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
>
> [+cc Ilpo]
>
> On Mon, Aug 24, 2026 at 11:57:07AM -0500, Anirudh Srinivasan wrote:
> > Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
> > software talking to the card. This BAR cannot be assigned on hosts whose
> > PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
> > failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
> > because bridge windows are sized as the sum of all enabled child BARs.
> > This makes the card completely unusable on such systems.
> >
> >     spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
> >     pci_bus 0000:00: root bus resource [bus 00-ff]
> >     pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
> >     pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
> >     pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]
> >
> >     <snip>
> >
> >     pci 0000:01:00.0: [1e52:b140] type 00 class 0x120000 PCIe Endpoint
> >     pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x1fffffff 64bit pref]
> >     pci 0000:01:00.0: BAR 2 [mem 0x1120000000-0x11200fffff 64bit pref]
> >     pci 0000:01:00.0: BAR 4 [mem 0xfffffff800000000-0xffffffffffffffff 64bit pref]
> >
> >     <snip>
> >
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
> >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
> >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
> >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: releasing
> >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: releasing
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
> >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
> >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> >
> > Add a header fixup that walks the root-bus apertures and drops BAR 4
> > when it is larger than every root-bus aperture. Resource assignment
> > skips resources with zero flags (pdev_resource_assignable()), so the
> > bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
> > Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
> > tt-bh-linux are able to run on the card.
> >
> > Signed-off-by: Anirudh Srinivasan <asrinivasan@xxxxxxxxxxxxxxxxxxx>
> > ---
> >  drivers/pci/quirks.c | 32 ++++++++++++++++++++++++++++++++
> >  1 file changed, 32 insertions(+)
> >
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index b09f27f7846fc..df54e379c116b 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -6415,3 +6415,35 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
> >  #endif
> > +
> > +/*
> > + * Drop unused 32 GiB BAR 4 on Tenstorrent Blackhole cards so the card is
> > + * usable on systems with smaller host bridge apertures.
> > + */
> > +static void quirk_tenstorrent_blackhole_bar4(struct pci_dev *pdev)
> > +{
> > +     struct resource *res = &pdev->resource[4];
> > +     struct pci_bus *bus = pdev->bus;
> > +     struct resource *win;
> > +
> > +     if (!(res->flags & IORESOURCE_MEM))
> > +             return;
> > +
> > +     /* Find the root bus; its resource list holds the host apertures */
> > +     while (bus->parent)
> > +             bus = bus->parent;
> > +
> > +     pci_bus_for_each_resource(bus, win) {
> > +             if (!win || !(win->flags & IORESOURCE_MEM))
> > +                     continue;
> > +             if (resource_size(res) <= resource_size(win))
> > +                     return;
> > +     }
> > +
> > +     pci_info(pdev, "BAR 4 %pR larger than every host bridge aperture, dropping it\n",
> > +              res);
> > +     res->start = 0;
> > +     res->end = 0;
> > +     res->flags = 0;
>
> This seems problematic because the hardware BAR still exists even if
> we zero out res->flags.
>
> This is a problem because pci_enable_device() will successfully enable
> a device and turn on PCI_COMMAND_MEMORY even though the hardware BAR
> may contain junk.  And if a driver ioremaps this BAR, it is
> ioremapping physical address 0, which doesn't end well.
>
> I know this from bitter experience :)
>
> If you have some device-specific way to actually disable the BAR in
> the hardware, that could work.

lspci for this device doesn't show this BAR region. Are you saying
that BAR4 still exists (but points to 0) ? Our driver doesn't seem to
have any issues with this, but some other driver may not play well
with this.

$ lspci -d 1e52: -vvv
0000:01:00.0 Processing accelerators: Tenstorrent Inc Blackhole
        Subsystem: Tenstorrent Inc p150a
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 27
        Region 0: Memory at 1800000000 (64-bit, prefetchable) [size=512M]
        Region 2: Memory at 1820000000 (64-bit, prefetchable) [size=1M]
        Capabilities: <access denied>
        Kernel driver in use: tenstorrent

>
> > +}
> > +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TENSTORRENT, 0xb140, quirk_tenstorrent_blackhole_bar4);
> >
> > --
> > 2.43.0
> >