Re: [PATCH v2] riscv: dts: spacemit: k1: give each PCIe port its own DMA window
From: Frédéric Mahé
Date: Tue Nov 17 2026 - 04:51:56 EST
A follow-up question on the hole that remains for each port.
On the BPI-F3 the remaining pcie1 hole (CPU 0x1_1000_0000-0x1_1fff_ffff)
still costs a lot: anonymous pages often come from the low end of
ZONE_NORMAL, and when fio's buffers land there, 4k random reads lose
about half their IOPS to SWIOTLB bouncing.
The port only uses a small part of its 256 MB window here:
pci 0001:00:00.0: BAR 0 [mem 0x90000000-0x900fffff]: assigned
pci 0001:00:00.0: BAR 1 [mem 0x90100000-0x901fffff]: assigned
pci 0001:00:00.0: bridge window [mem 0x90200000-0x902fffff]: assigned
So I tried a test DT that excludes only bus 0x9000_0000-0x903f_ffff
from pcie1's dma-ranges (the root port BARs and the bridge window, plus
margin), on top of your v2:
dma-ranges = <0x0 0x00000000 0x0 0x00000000 0x0 0x80000000>,
<0x0 0x80000000 0x1 0x00000000 0x0 0x10000000>,
<0x0 0x90400000 0x1 0x10400000 0x0 0x0fc00000>,
<0x0 0xa0000000 0x1 0x20000000 0x3 0x60000000>;
Results (v7.4-rc1, 16 GB, NVMe on pcie1):
- 15529 pages in CPU 0x1_1040_0000-0x1_1fff_ffff: 4 KiB O_DIRECT write
and read per page, no bounce, every page read back correctly
- fio crc32c verify (16 GiB direct, 8 GiB buffered) OK, no AER
- 4k random read, IOPS (1 job QD32 / 4 jobs QD32 / 8 jobs QD16):
v2 27.5k / 86k / 137k, test DT 47.5k / 185k / 193k
(same as booting with mem=2G, where nothing bounces)
I did not try DMA into the bridge window itself, as that would target
the drive's own BAR.
So it looks like the root port only claims its programmed BARs and
bridge window, not the whole 256 MB aperture. Troy, is that how the K1
root port works? If so, would a smaller aperture for pcie1 (the M.2
slot on most boards, e.g. 16 MB), with a matching smaller hole, be
acceptable? The other ports could keep large windows.
Thanks,
Frédéric