Re: [PATCH v9 3/7] PCI: Add parent_bus_offset to resource_entry

[Frank Li]

On Wed, Feb 26, 2025 at 06:23:26PM -0600, Bjorn Helgaas wrote:
> On Tue, Jan 28, 2025 at 05:07:36PM -0500, Frank Li wrote:
> > Introduce `parent_bus_offset` in `resource_entry` and a new API,
> > `pci_add_resource_parent_bus_offset()`, to provide necessary information
> > for PCI controllers with address translation units.
> >
> > Typical PCI data flow involves:
> >   CPU (CPU address) -> Bus Fabric (Intermediate address) ->
> >   PCI Controller (PCI bus address) -> PCI Bus.
> >
> > While most bus fabrics preserve address consistency, some modify addresses
> > to intermediate values. The `parent_bus_offset` enables PCI controllers to
> > translate these intermediate addresses correctly to PCI bus addresses.
> >
> > Pave the road to remove hardcoded cpu_addr_fixup() and similar patterns in
> > PCI controller drivers.
> > ...
>
> > +++ b/drivers/pci/of.c
> > @@ -402,7 +402,17 @@ static int devm_of_pci_get_host_bridge_resources(struct device *dev,
> >  			res->flags &= ~IORESOURCE_MEM_64;
> >  		}
> >
> > -		pci_add_resource_offset(resources, res,	res->start - range.pci_addr);
> > +		/*
> > +		 * IORESOURCE_IO res->start is io space start address.
> > +		 * IORESOURCE_MEM res->start is cpu start address, which is the
> > +		 * same as range.cpu_addr.
> > +		 *
> > +		 * Use (range.cpu_addr - range.parent_bus_addr) to align both
> > +		 * IO and MEM's parent_bus_offset always offset to cpu address.
> > +		 */
> > +
> > +		pci_add_resource_parent_bus_offset(resources, res, res->start - range.pci_addr,
> > +						   range.cpu_addr - range.parent_bus_addr);
>
> I don't know exactly where it needs to go, but I think we can call
> .cpu_addr_fixup() once at startup on the base of the region.  This
> will tell us the offset that applies to the entire region, i.e.,
> parent_bus_offset.
>
> Then we can remove all the .cpu_addr_fixup() calls in
> cdns_pcie_host_init_address_translation(),
> cdns_pcie_set_outbound_region(), and dw_pcie_prog_outbound_atu().
>
> Until we can get rid of all the .cpu_addr_fixup() implementations,
> We'll still have that single call at startup (I guess once for cadence
> and another for designware), but it should simplify the current
> callers quite a bit.

I don't think it can simple code. cdns_pcie_set_outbound_region() and
dw_pcie_prog_outbound_atu() are called by EP functions, which have not use
"resource" to manage outbound windows. And there are IO/CFG/MEM space at
host side, It will involve more .cpu_addr_fixup() calls to setup these
range.

Frank
>
> >  	}
> >
> >  	/* Check for dma-ranges property */
> > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > index 47b31ad724fa5..0d7e67b47be47 100644
> > --- a/include/linux/pci.h
> > +++ b/include/linux/pci.h
> > @@ -1510,6 +1510,8 @@ static inline void pci_release_config_region(struct pci_dev *pdev,
> >  void pci_add_resource(struct list_head *resources, struct resource *res);
> >  void pci_add_resource_offset(struct list_head *resources, struct resource *res,
> >  			     resource_size_t offset);
> > +void pci_add_resource_parent_bus_offset(struct list_head *resources, struct resource *res,
> > +					resource_size_t offset, resource_size_t parent_bus_offset);
> >  void pci_free_resource_list(struct list_head *resources);
> >  void pci_bus_add_resource(struct pci_bus *bus, struct resource *res);
> >  struct resource *pci_bus_resource_n(const struct pci_bus *bus, int n);
> > diff --git a/include/linux/resource_ext.h b/include/linux/resource_ext.h
> > index ff0339df56afc..b6ec6cc318203 100644
> > --- a/include/linux/resource_ext.h
> > +++ b/include/linux/resource_ext.h
> > @@ -24,6 +24,7 @@ struct resource_entry {
> >  	struct list_head	node;
> >  	struct resource		*res;	/* In master (CPU) address space */
> >  	resource_size_t		offset;	/* Translation offset for bridge */
> > +	resource_size_t		parent_bus_offset; /* Parent bus address offset for bridge */
> >  	struct resource		__res;	/* Default storage for res */
> >  };
> >
> >
> > --
> > 2.34.1
> >