Re: [PATCH v6 0/8] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback

[Koichiro Den]

On Thu, Feb 12, 2026 at 12:57:52AM +0900, Koichiro Den wrote:
> On Tue, Feb 10, 2026 at 05:30:10PM +0100, Niklas Cassel wrote:
> > On Tue, Feb 10, 2026 at 11:07:16PM +0900, Koichiro Den wrote:
> > > On Tue, Feb 10, 2026 at 01:24:29PM +0100, Niklas Cassel wrote:
> > > > On Mon, Feb 09, 2026 at 09:53:08PM +0900, Koichiro Den wrote:
> > > > > Tested on
> > > > > =========
> > > > > 
> > > > > I tested the embedded (DMA) doorbell fallback path (via pci-epf-test) on
> > > > > R-Car Spider boards:
> > > > > 
> > > > >   $ ./pci_endpoint_test -t DOORBELL_TEST
> > > > >   TAP version 13
> > > > >   1..1
> > > > >   # Starting 1 tests from 1 test cases.
> > > > >   #  RUN           pcie_ep_doorbell.DOORBELL_TEST ...
> > > > >   #            OK  pcie_ep_doorbell.DOORBELL_TEST
> > > > >   ok 1 pcie_ep_doorbell.DOORBELL_TEST
> > > > >   # PASSED: 1 / 1 tests passed.
> > > > >   # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
> > > > > 
> > > > > with the following message observed on the EP side:
> > > > > 
> > > > >   [   80.464653] pci_epf_test pci_epf_test.0: Using embedded (DMA) doorbell fallback
> > > > > 
> > > > > (Note: for the test to pass on R-Car Spider, one of the following was required:
> > > > >  - echo 1048576 > functions/pci_epf_test/func1/pci_epf_test.0/bar2_size
> > > > >  - apply https://lore.kernel.org/all/20251023072217.901888-1-den@xxxxxxxxxxxxx/)
> > > > 
> > > > I applied this series on top of branch pci/controller/dwc
> > > > on Rock 5B (pcie-dw-rockchip.c).
> > > > 
> > > > On EP side:
> > > > [   39.218533] pci_epf_test pci_epf_test.0: Can't find MSI domain for EPC
> > > > [   39.219125] pci_epf_test pci_epf_test.0: Using embedded (DMA) doorbell fallback
> > > > 
> > > > On RC side:
> > > > #  RUN           pcie_ep_doorbell.DOORBELL_TEST ...
> > > > [   40.297892] pci-endpoint-test 0000:01:00.0: Failed to trigger doorbell in endpoint
> > > > # pci_endpoint_test.c:279:DOORBELL_TEST:Expected 0 (0) == ret (-22)
> > > > # pci_endpoint_test.c:279:DOORBELL_TEST:Test failed for Doorbell
> > > > 
> > > > # DOORBELL_TEST: Test failed
> > > > #          FAIL  pcie_ep_doorbell.DOORBELL_TEST
> > > > not ok 23 pcie_ep_doorbell.DOORBELL_TEST
> > > > 
> > > > Any suggestions?
> > > > 
> > > > (All BARs in pcie-dw-rockchip.c is marked as BAR_RESIZABLE.)
> > > 
> > > Thank you for testing.
> > > 
> > > If the failure was observed in a scenario other than a plain
> > > `./pci_endpoint_test -t DOORBELL_TEST`, could you please try again with [1]
> > > applied as well?
> > > 
> > > [1] https://lore.kernel.org/linux-pci/20260202145407.503348-1-den@xxxxxxxxxxxxx/
> > 
> > I applied that series, but I got the same problem.
> > 
> > I added debug, and the EP side does use the correct address for the eDMA:
> > [   26.279457] msg_addr: 0xa403800a0
> > [   26.279898] phys_addr: 0xa40300000 offset: 0x800a0
> > 
> > 
> > If I write to the msg_addr directly on the EP using devmem, I do see the print
> > that I added in the IRQ handler:
> > # devmem 0xa403800a0 32 0
> > [  155.861989] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  158.809160] dw_edma_interrupt_emulated:696
> > [  158.809543] pci_epf_test_doorbell_primary:729
> > # [  158.809986] pci_epf_test_doorbell_handler:703
> > # devmem 0xa403800a0 32 0
> > [  161.241326] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  163.466054] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  167.378662] dw_edma_interrupt_emulated:696
> > [  167.379045] pci_epf_test_doorbell_primary:729
> > # [  167.379512] pci_epf_test_doorbell_handler:703
> > # devmem 0xa403800a0 32 0
> > [  168.880179] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  170.492176] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  171.729154] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  173.481271] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  174.985787] dw_edma_interrupt_emulated:696
> > # devmem 0xa403800a0 32 0
> > [  176.517131] dw_edma_interrupt_emulated:696
> > [  176.517511] pci_epf_test_doorbell_primary:729
> > # [  176.517963] pci_epf_test_doorbell_handler:703
> > 
> > But not on every write....
> > 
> > I'm not sure, but could this perhaps be because we are missing this patch:
> > https://lore.kernel.org/dmaengine/20260105075904.1254012-1-den@xxxxxxxxxxxxx/
> 
> Thank you for the detailed debugging.
> 
> I don't have a Rock 5B to reproduce locally, but your log indicates that
> the emulated interrupt is not always delivered on the same eDMA IRQ line.
> On RK3588 (rk3588-extra.dtsi) there are multiple eDMA IRQs (dma0-4), while
> pci-epf-test requests only epf->db_msg[0].virq (IRQ 53 in your
> /proc/interrupts). For R-Car S4 Spider, chip->nr_irqs == 1 and I wasn't
> able to verify whether my earlier concern here:
> https://lore.kernel.org/linux-pci/p4ommmpcjegvb4lafzecf67tzmdodtuqexeoifcn5eh7xqyp2y@ss76d3ubbsw7/
> 
>   > The proposed dmaengine_{register,unregister}_selfirq() APIs are
>   > device-wide (i.e. not per channel), so I'm not sure which "channel" you
>   > refer to here.  Also, when chip->nr_irqs > 1 on EP, dw-edma distributes
>   > channels across multiple IRQ vectors, and it's unclear (at least to me)
>   > which IRQ vector the emulated interrupt ("fake irq") is expected to be
>   > delivered on.
> 
> actually holds true in practice. Your report makes it clear that the
> emulated interrupt can indeed be delivered on different IRQ vectors.
> 
> One hypothesis is that: we currently program msg.data = 0 for the
> "embedded" doorbell, and we write to DMA_READ_INT_STATUS_OFF. The register
> field (RD_DONE_INT_STATUS) is defined per-channel, and the register
> supports interrupt emulation by writes, so it might be that writing BIT(n)
> selects the channel/irq line, while writing 0 does not consistently map to
> a specific one.
> 
> Could you try a quick experiment on the Rock 5B EP side?
> 
>   devmem 0xa403800a0 32 1
>   devmem 0xa403800a0 32 2
>   devmem 0xa403800a0 32 4
>   devmem 0xa403800a0 32 8
> 
> and see if the interrupt consistently lands on a specific one of IRQ53-56
> for each value?
> 
> If that is the case, we can make msg.data non-zero value instead of always 0.
> 
> If that is not the case, [...]

Thanks to the experiment by Niklas, it turns out that this is not the case.

> [...] then we may need to consider two less ideal
> options:
>   - switch back to the ~v3 approach, where we run the registered hook
>     exactly at the time when the emulated interrupt is deasserted. (ref.
>     https://lore.kernel.org/linux-pci/20260204145440.950609-6-den@xxxxxxxxxxxxx/)
>   - or, require users to request_irq() for all irq vectors associated with
>     all channels. However, this would not be very attractive from a design
>     perspective.

So now revisiting this, in hindsight, this part (the two options) was
premature. I'm considering taking a different approach, like:

    - alloc a single virtual IRQ dedicated to the emulated interrupt,
      associate it with dummy_irq_chip + handle_simple_irq, and have the
      parent handlers clear the emulated interrupt (i.e. write 0 to
      INT_CLEAR) before invoking generic_handle_irq() from parent handlers
      unconditionally (*). Or, create a dedicated irq_chip and use
      handle_level_irq(), where the emulated interrupt is cleared in the
      irq_ack path. The virtual IRQ is taught via
      pci_epc_get_aux_resources().

      (*): Because the Done/Abort bits are not set by the write into the
	   INT_STATUS, and because a DMA transfer completion may set those
	   bits between the write and the IRQ handling, as I understand it,
	   we still need to call generic_handle_irq() unconditionally.

I'm planning to rework the relevant code for v7.

Kind regards,
Koichiro

> 
> > 
> > # dmesg | grep eDMA
> > [    1.243339] rockchip-dw-pcie a40000000.pcie-ep: eDMA: unroll T, 2 wr, 2 rd
> > 
> > # cat /proc/interrupts | grep edma
> >  53:          8          0          0          0          0          0          0          0    GICv3 303 Level     dw-edma-core:a40000000.pcie-ep, pci-ep-test-doorbell
> >  54:          7          0          0          0          0          0          0          0    GICv3 304 Level     dw-edma-core:a40000000.pcie-ep
> >  55:         15          0          0          0          0          0          0          0    GICv3 301 Level     dw-edma-core:a40000000.pcie-ep
> >  56:          7          0          0          0          0          0          0          0    GICv3 302 Level     dw-edma-core:a40000000.pcie-ep
> > 
> > 
> > 
> > Anyway, I was still curious why this did never worked when writing from the
> > host side, even when running the test case many, many times.
> > AFAICT, the inbound translation looks correct.
> > 
> > RK3588 (pcie-dw-rockchip.c) exposes the DMA registers in BAR4 by default.
> > If I hack pci-epf-test on top of your patch to unconditionally return BAR4 with
> > offset 0xa0, it works. So my best guess is that the fixed inbound translation
> > in BAR4 (to the eDMA registers) somehow messes with the inbound translation if
> > another BAR tries to use an inbound translation to the eDMA registers as well.
> 
> Thanks a lot for letting me know that. I see two possible ways forward:
> 
>   (a) extend PCI_EPC_AUX_DMA_CTRL_MMIO to optionally describe that the DMA
>       MMIO window is already mapped to a fixed BAR and should be reused, so
>       EPFs avoid creating a second mapping to the same target. I guess it
>       could be treated as a quirk for "rockchip,rk3588-pcie-ep".
> 
>   (b) alternatively, clear the default BAR4 mapping on RK3588 at least
>       temporarily when using the pci-epf-msi doorbell fallback.
> 
> Koichiro
> 
> > 
> > 
> > Kind regards,
> > Niklas
> > 
>