Re: [PATCH v3 2/2] PCI: qcom: Avoid DBI and ATU register space mirror to BAR/MMIO region
From: Prudhvi Yarlagadda
Date: Mon Jul 29 2024 - 19:58:29 EST
Hi Bjorn,
On 7/29/2024 3:51 PM, Bjorn Helgaas wrote:
> On Thu, Jul 25, 2024 at 04:03:56PM -0700, Prudhvi Yarlagadda wrote:
>> Hi Bjorn,
>>
>> Thanks for the review comments.
>>
>> On 7/24/2024 11:43 AM, Bjorn Helgaas wrote:
>>> On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote:
>>>> PARF hardware block which is a wrapper on top of DWC PCIe controller
>>>> mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE
>>>> register to get the size of the memory block to be mirrored and uses
>>>> PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base
>>>> address of DBI and ATU space inside the memory block that is being
>>>> mirrored.
>>>>
>>>> When a memory region which is located above the SLV_ADDR_SPACE_SIZE
>>>> boundary is used for BAR region then there could be an overlap of DBI and
>>>> ATU address space that is getting mirrored and the BAR region. This
>>>> results in DBI and ATU address space contents getting updated when a PCIe
>>>> function driver tries updating the BAR/MMIO memory region. Reference
>>>> memory map of the PCIe memory region with DBI and ATU address space
>>>> overlapping BAR region is as below.
>>>>
>>>> |---------------|
>>>> | |
>>>> | |
>>>> ------- --------|---------------|
>>>> | | |---------------|
>>>> | | | DBI |
>>>> | | |---------------|---->DBI_BASE_ADDR
>>>> | | | |
>>>> | | | |
>>>> | PCIe | |---->2*SLV_ADDR_SPACE_SIZE
>>>> | BAR/MMIO|---------------|
>>>> | Region | ATU |
>>>> | | |---------------|---->ATU_BASE_ADDR
>>>> | | | |
>>>> PCIe | |---------------|
>>>> Memory | | DBI |
>>>> Region | |---------------|---->DBI_BASE_ADDR
>>>> | | | |
>>>> | --------| |
>>>> | | |---->SLV_ADDR_SPACE_SIZE
>>>> | |---------------|
>>>> | | ATU |
>>>> | |---------------|---->ATU_BASE_ADDR
>>>> | | |
>>>> | |---------------|
>>>> | | DBI |
>>>> | |---------------|---->DBI_BASE_ADDR
>>>> | | |
>>>> | | |
>>>> ----------------|---------------|
>>>> | |
>>>> | |
>>>> | |
>>>> |---------------|
>>>>
>>>> Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not
>>>> used for BAR region which is why the above mentioned issue is not
>>>> encountered. This issue is discovered as part of internal testing when we
>>>> tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence
>>>> we are trying to fix this.
>>>>
>>>> As PARF hardware block mirrors DBI and ATU register space after every
>>>> PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write
>>>> U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid
>>>> mirroring DBI and ATU to BAR/MMIO region. Write the physical base address
>>>> of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and
>>>> PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU
>>>> blocks are at expected memory locations.
>>>>
>>>> The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2
>>>> and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF
>>>> Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and
>>>> PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3.
>>>> PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF
>>>> Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init()
>>>> functions of the respective PARF versions to program applicable
>>>> PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR
>>>> register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro.
>>>>
>>>> Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@xxxxxxxxxxx>
>>>> Reviewed-by: Mayank Rana <quic_mrana@xxxxxxxxxxx>
>>>> ---
>>>> drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++-------
>>>> 1 file changed, 45 insertions(+), 17 deletions(-)
>>>>
>>>> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
>>>> index 0180edf3310e..6976efb8e2f0 100644
>>>> --- a/drivers/pci/controller/dwc/pcie-qcom.c
>>>> +++ b/drivers/pci/controller/dwc/pcie-qcom.c
>>>> @@ -45,6 +45,7 @@
>>>> #define PARF_PHY_REFCLK 0x4c
>>>> #define PARF_CONFIG_BITS 0x50
>>>> #define PARF_DBI_BASE_ADDR 0x168
>>>> +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C
>>>> #define PARF_MHI_CLOCK_RESET_CTRL 0x174
>>>> #define PARF_AXI_MSTR_WR_ADDR_HALT 0x178
>>>> #define PARF_AXI_MSTR_WR_ADDR_HALT_V2 0x1a8
>>>> @@ -52,8 +53,13 @@
>>>> #define PARF_LTSSM 0x1b0
>>>> #define PARF_SID_OFFSET 0x234
>>>> #define PARF_BDF_TRANSLATE_CFG 0x24c
>>>> -#define PARF_SLV_ADDR_SPACE_SIZE 0x358
>>>> +#define PARF_DBI_BASE_ADDR_V2 0x350
>>>> +#define PARF_DBI_BASE_ADDR_V2_HI 0x354
>>>> +#define PARF_SLV_ADDR_SPACE_SIZE_V2 0x358
>>>> +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI 0x35C
>>>> #define PARF_NO_SNOOP_OVERIDE 0x3d4
>>>> +#define PARF_ATU_BASE_ADDR 0x634
>>>> +#define PARF_ATU_BASE_ADDR_HI 0x638
>>>> #define PARF_DEVICE_TYPE 0x1000
>>>> #define PARF_BDF_TO_SID_TABLE_N 0x2000
>>>> #define PARF_BDF_TO_SID_CFG 0x2c00
>>>> @@ -107,9 +113,6 @@
>>>> /* PARF_CONFIG_BITS register fields */
>>>> #define PHY_RX0_EQ(x) FIELD_PREP(GENMASK(26, 24), x)
>>>>
>>>> -/* PARF_SLV_ADDR_SPACE_SIZE register value */
>>>> -#define SLV_ADDR_SPACE_SZ 0x10000000
>>>> -
>>>> /* PARF_MHI_CLOCK_RESET_CTRL register fields */
>>>> #define AHB_CLK_EN BIT(0)
>>>> #define MSTR_AXI_CLK_EN BIT(1)
>>>> @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci)
>>>> dw_pcie_dbi_ro_wr_dis(pci);
>>>> }
>>>>
>>>> +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie)
>>>> +{
>>>> + struct dw_pcie *pci = pcie->pci;
>>>> +
>>>> + if (pci->dbi_phys_addr) {
>>>> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf +
>>>> + PARF_DBI_BASE_ADDR);
>>>> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE);
>>>> + }
>>>> +}
>>>> +
>>>> +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie)
>>>> +{
>>>> + struct dw_pcie *pci = pcie->pci;
>>>> +
>>>> + if (pci->dbi_phys_addr) {
>>>> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf +
>>>> + PARF_DBI_BASE_ADDR_V2);
>>>> + writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf +
>>>> + PARF_DBI_BASE_ADDR_V2_HI);
>>>> +
>>>> + if (pci->atu_phys_addr) {
>>>> + writel(lower_32_bits(pci->atu_phys_addr), pcie->parf +
>>>> + PARF_ATU_BASE_ADDR);
>>>> + writel(upper_32_bits(pci->atu_phys_addr), pcie->parf +
>>>> + PARF_ATU_BASE_ADDR_HI);
>>>> + }
>>>> +
>>>> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2);
>>>> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI);
>>>> + }
>>>> +}
>>>
>>> These functions write CPU physical addresses into registers. I don't
>>> know where these registers live. If they are on the PCI side of the
>>> world, they most likely should contain PCI bus addresses, not CPU
>>> physical addresses.
>>>
>>> In some systems, PCI bus addresses are the same as CPU physical
>>> addresses, but on many systems they are not the same, so it's better
>>> if we don't make implicit assumptions that they are the same.
>>
>> On Qualcomm platforms, CPU physical address and PCI bus addresses
>> for DBI and ATU registers are same. PARF registers live outside the
>> PCI address space in the system memory.
>>
>> There is a mapping logic in the QCOM PARF wrapper which detects any
>> incoming read/write transactions from the NOC towards PCIe
>> controller and checks its addresses against PARF_DBI_BASE_ADDR and
>> PARF_ATU_BASE_ADDR registers so that these transactions can be
>> routed to DBI and ATU registers inside the PCIe controller.
>>
>> So, these PARF registers needs to be programmed with base CPU
>> physical addresses of DBI and ATU as the incoming DBI and ATU
>> transactions from the NOC contain CPU physical adresses.
>
> Can you add a comment to this effect that these registers are
> effectively in the CPU address domain, not the PCI bus domain?
> Otherwise the next person who reviews this will have the same
> question, and somebody may even try to "fix" this by converting it to
> a PCI bus address.
>
> Bjorn
ACK. I will add a comment in the next patch.
Thanks,
Prudhvi