Re: [PATCH] PCI: Warn about MEM resource size being too big

[Alan Mikhak]

On Wed, Apr 1, 2020 at 2:01 PM Alan Mikhak <alan.mikhak@xxxxxxxxxx> wrote:
>
> On Wed, Apr 1, 2020 at 1:29 PM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> >
> > On Tue, Mar 31, 2020 at 01:36:04PM -0700, Alan Mikhak wrote:
> > > On Tue, Mar 31, 2020 at 1:12 PM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> > > > On Mon, Mar 30, 2020 at 05:19:47PM -0700, Alan Mikhak wrote:
> > > > > Output a warning for MEM resource size with
> > > > > non-zero upper 32-bits.
> > > > >
> > > > > ATU programming functions limit the size of
> > > > > the translated region to 4GB by using a u32 size
> > > > > parameter. Function dw_pcie_prog_outbound_atu()
> > > > > does not program the upper 32-bit ATU limit
> > > > > register. This may result in undefined behavior
> > > > > for resource sizes with non-zero upper 32-bits.
> > > > >
> > > > > For example, a 128GB address space starting at
> > > > > physical CPU address of 0x2000000000 with size of
> > > > > 0x2000000000 needs the following values programmed
> > > > > into the lower and upper 32-bit limit registers:
> > > > >  0x3fffffff in the upper 32-bit limit register
> > > > >  0xffffffff in the lower 32-bit limit register
> > > > >
> > > > > Currently, only the lower 32-bit limit register is
> > > > > programmed with a value of 0xffffffff but the upper
> > > > > 32-bit limit register is not being programmed.
> > > > > As a result, the upper 32-bit limit register remains
> > > > > at its default value after reset of 0x0. This would
> > > > > be a problem for a 128GB PCIe space because in
> > > > > effect its size gets reduced to 4GB.
> > > > >
> > > > > ATU programming functions can be changed to
> > > > > specify a u64 size parameter for the translated
> > > > > region. Along with this change, the internal
> > > > > calculation of the limit address, the address of
> > > > > the last byte in the translated region, needs to
> > > > > change such that both the lower 32-bit and upper
> > > > > 32-bit limit registers can be programmed correctly.
> > > > >
> > > > > Changing the ATU programming functions is high
> > > > > impact. Without change, this issue can go
> > > > > unnoticed. A warning may prompt the user to
> > > > > look into possible issues.
> > > >
> > > > So this is basically a warning, and we could actually *fix* the
> > > > problem with more effort?  I vote for the fix.
> > >
> > > The fix would impact all PCIe drivers that depend on dwc.
> >
> > Is that another way of saying "the fix would *fix* all the drivers
> > that depend on dwc"?
>
> Thanks Bjorn for your comments.
>
> Not at all. I'm not suggesting that. I'm just stating the dilemma.
>
> One option is, as you may be alluding, the *fix* would include
> modification to all drivers that depend on dwc to at least not break
> the build.. Whoever embarks on such a *fix* would have to take
> that on before submitting the patch.
>
> Another option is to produce an alternate ATU programming
> API for the Linux PCI sub-system to support u64 size. That
> way individual driver owners can choose if and when to migrate
> their drivers to the new API on their own timeline. Such an
> alternative API can also be generic to support not only
> Designware PCIe controllers but others.
>
> >
> > > I would have no way of validating such a fix without
> > > breaking it for everyone let alone the bandwidth it needs.
> > > All drivers that depend on dwc seem to be currently happy
> > > with the u32 size limit. I suggest we add the warning but
> > > keep this issue in mind for a solution that allows existing
> > > PCe drivers to phase into the fix on their own individual
> > > schedules, if they need to.
> >
> > Obviously it would *nice* to test all the drivers that depend on dwc,
> > but if you're fixing a problem, you verify the fix on your system, and
> > the relevant people review it, I don't think exhaustive testing is a
> > hard requirement, and I certainly wouldn't expect you to do it.
>
> That is a relief for whoever commits to take this on.
>
> >
> > If we want to live with a 32-bit limit, I think we should change the
> > relevant interfaces to use u32 so there's not a way to venture into
> > this region of undefined behavior.  I don't think "warning + undefined
> > behavior" is a very maintainable situation.
>
> I cannot live with the 32-bit limit. I need a 64-bit solution. I had
> to implement a solution that suits my needs. I have worked
> out some of the issue. It is generic enough for my use with PCIe
> controllers from more than one vendor. But, it requires pulling a
> lot of code from Designware layer into a separate framework
> which I believe can become common for Linux PCI subsystem.
> If it gets in, others who need 64-bit can migrate over to it without
> being migrated involuntarily.
>

The "undefined" behavior part of the problem can be fixed
separately in the function dw_pcie_prog_outbound_atu_unroll()
by modifying its internal calculation of the limit address and
programming the ATU upper limit address register.

With the above fix in dw_pcie_prog_outbound_atu_unroll(),
the total problem reduces to the size issued being reduced
to 4G max.

> >
> > > > > This limitation also means that multiple ATUs
> > > > > would need to be used to map larger regions.
> > > > >
> > > > > Signed-off-by: Alan Mikhak <alan.mikhak@xxxxxxxxxx>
> > > > > ---
> > > > >  drivers/pci/controller/dwc/pcie-designware-host.c | 6 +++++-
> > > > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > > > index 395feb8ca051..37a8c71ef89a 100644
> > > > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > > > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > > > @@ -325,6 +325,7 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > > > >       struct pci_bus *child;
> > > > >       struct pci_host_bridge *bridge;
> > > > >       struct resource *cfg_res;
> > > > > +     resource_size_t mem_size;
> > > > >       u32 hdr_type;
> > > > >       int ret;
> > > > >
> > > > > @@ -362,7 +363,10 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > > > >               case IORESOURCE_MEM:
> > > > >                       pp->mem = win->res;
> > > > >                       pp->mem->name = "MEM";
> > > > > -                     pp->mem_size = resource_size(pp->mem);
> > > > > +                     mem_size = resource_size(pp->mem);
> > > > > +                     if (upper_32_bits(mem_size))
> > > > > +                             dev_warn(dev, "MEM resource size too big\n");
> > > > > +                     pp->mem_size = mem_size;
> > > > >                       pp->mem_bus_addr = pp->mem->start - win->offset;
> > > > >                       break;
> > > > >               case 0:
> > > > > --
> > > > > 2.7.4
> > > > >