Re: [PATCH v2 1/1] PCI: dwc: Fix index 0 incorrectly being interpreted as a free ATU slot

[Frank Li]

On Tue, Apr 02, 2024 at 11:23:49AM +0200, Niklas Cassel wrote:
> On Tue, Apr 02, 2024 at 10:42:28AM +0530, Manivannan Sadhasivam wrote:
> > On Fri, Mar 22, 2024 at 07:19:01AM +0100, Niklas Cassel wrote:
> > > On Fri, Mar 22, 2024 at 10:56:23AM +0530, Manivannan Sadhasivam wrote:
> > > > On Thu, Mar 21, 2024 at 01:07:32PM -0500, Bjorn Helgaas wrote:
> > > > > On Thu, Mar 21, 2024 at 10:43:45PM +0530, Manivannan Sadhasivam wrote:
> > > > > > On Mon, Mar 04, 2024 at 05:46:16PM -0500, Frank Li wrote:
> > > > > > > dw_pcie_ep_inbound_atu()
> > > > > > > {
> > > > > > > 	...
> > > > > > > 	if (!ep->bar_to_atu[bar])
> > > > > > > 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
> > > > > > > 	else
> > > > > > > 		free_win = ep->bar_to_atu[bar];
> > > > > > > 	...
> > > > > > > }
> > > > > > > 
> > > > > > > The atu index 0 is valid case for atu number. The find_first_zero_bit()
> > > > > > > will return 6 when second time call into this function if atu is 0. Suppose
> > > > > > > it should use branch 'free_win = ep->bar_to_atu[bar]'.
> > > > > > > 
> > > > > > > Change 'bar_to_atu' to free_win + 1. Initialize bar_to_atu as 0 to indicate
> > > > > > > it have not allocate atu to the bar.
> > > > > > 
> > > > > > I'd rewrite the commit message as below:
> > > > > > 
> > > > > > "The mapping between PCI BAR and iATU inbound window are maintained in the
> > > > > > dw_pcie_ep::bar_to_atu[] array. While allocating a new inbound iATU map for a
> > > > > > BAR, dw_pcie_ep_inbound_atu() API will first check for the availability of the
> > > > > > existing mapping in the array and if it is not found (i.e., value in the array
> > > > > > indexed by the BAR is found to be 0), then it will allocate a new map value
> > > > > > using find_first_zero_bit().
> > > > > > 
> > > > > > The issue here is, the existing logic failed to consider the fact that the map
> > > > > > value '0' is a valid value for BAR0. Because, find_first_zero_bit() will return
> > > > > > '0' as the map value for BAR0 (note that it returns the first zero bit
> > > > > > position).
> > > > > > 
> > > > > > Due to this, when PERST# assert + deassert happens on the PERST# supported
> > > > > > platforms, the inbound window allocation restarts from BAR0 and the existing
> > > > > > logic to find the BAR mapping will return '6' for BAR0 instead of '0' due to the
> > > > > > fact that it considers '0' as an invalid map value.
> > > > > > 
> > > > > > So fix this issue by always incrementing the map value before assigning to
> > > > > > bar_to_atu[] array and then decrementing it while fetching. This will make sure
> > > > > > that the map value '0' always represents the invalid mapping."
> > > > > 
> > > > > This translates C code to English in great detail, but still doesn't
> > > > > tell me what's broken from a user's point of view, how urgent the fix
> > > > > is, or how it should be handled.
> > > > > 
> > > > > DMA doesn't work because ATU setup is wrong?  Driver MMIO access to
> > > > > the device doesn't work?  OS crashes?  How?  Incorrectly routed access
> > > > > causes UR response?  Happens on every boot?  Only after a reboot or
> > > > > controller reset?  What platforms are affected?  "PERST# supported
> > > > > platforms" is not actionable without a lot of research or pre-existing
> > > > > knowledge.  Should this be backported to -stable?
> > > > > 
> > > > 
> > > > Severity is less for the bug fixed by this patch. We have 8 inbound iATU windows
> > > > on almost all of the platforms and after PERST# assert + deassert, BAR0 uses map
> > > > '6' instead of '0'.
> > > > 
> > > > This has no user visibility since the mapping will go fine and we have only 6
> > > > BARs. So I'd not mark this as as critical fix that needs special attention.
> > > 
> > > So we will have 6 mappings configured, but only 5 BARs, so two mappings for
> > > BAR0. The iATU looks at them in order, so index 0 will override index 6.
> > > 
> > > We are lucky that the endpoint subsystem does not clean up allocations properly
> > > right now (you have an outstanding series which fixes this).
> > > 
> > > If the endpoint subsystem did clean up resources properly, we would DMA to the
> > > area that was previously allocated for BAR0, instead of the new area for BAR0.
> > > 
> > 
> > How would DMA happen to the previously allocated area? When the BARs are cleared
> > properly and then allocated again, BAR0 will get the map of 0 again and then the
> > iATU will map window 0 with BAR0. Only if we don't clear the BARs properly (as
> > like now), then it will result in BAR0 having 2 identical mappings and even with
> > that there won't be any issue since both map 0 and 6 are valid.
> > 
> > Am I missing anything?
> 
> Like Bjorn summarize it:
> "We dodge the bullet as long as the mappings for BAR 0 are identical,
> which doesn't feel like much comfort."
> 
> Yes, right now we don't have a cleanup of either the backing memory for
> the BAR, or the iATUs mapping the PCI address to backing memory.
> (We allocate the backing memory for the BARs in .bind(), and free it in
> unbind().)
> 
> So the superfluous iATU6 mapping will be the same as the iATU0 mapping.
> 
> After your series, we will still allocate and free the backing memory
> in .bind()/.unbind(), but we will set/clear the iATU mapping in the
> .init()/.deinit() EPF callbacks.
> 
> 
> > 
> > > Perhaps just include this fix in your series that actually cleans up the BARs?
> > > 
> > 
> > Yeah, makes sense. Once we agree on a finalized commit message in this thread,
> > I'll include this patch in my series.
> 
> I think that we have spent too much time on this patch already.
> 
> My suggestion is that you simply apply it to pci/endpoint branch directly and
> fixup the commit message (like Bjorn usually does with [bhelgaas: commit log])
> after Frank's Sign-off.

I combined mani and your suggested message to 

https://lore.kernel.org/imx/20240326193540.3610570-1-Frank.Li@xxxxxxx/

I am okay to fine tune it by yourself. 

Frank

> 
> 
> Kind regards,
> Niklas