[PATCH v5 13/14] PCI: Fix composite resource sizing

[Ilpo Järvinen]

The bridge window sizing algorithm aims to pack child resources back to
back. With composite child resources whose sizes are not a power of two,
back-to-back placement may not be possible. The non-aligning remainder
placement is limited because it must be adjacent to the rest of the
composite resource.

Add bridge window gap size calculation.

When there are only two resources and at least one of them is
composite, the gap end address aligns to meet the smaller of the two
alignment requirements. If there is only one composite resource, the
smaller alignment may be that of the composite resource.

Other combinations fall back to filling each resource's align sized
blocks fully. Sometimes, smaller size may be possible but due to how
sizing and assignment are made in different phases, it is not always
possible to predict where each resource is assigned. Thus, the sizing
has to play it safe.

Fixes: 9036bd0efcb6 ("PCI: Align head space better")
Reported-by: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
Reported-by: Nikolas Joshua Britton <nbritton@xxxxxxxxx>
Link: https://lore.kernel.org/linux-pci/20260903063124.9316-1-nbritton@xxxxxxxxx/
Reported-by: Maciej Grochowski <Maciej.Grochowski@xxxxxxxx>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@xxxxxxxxxxxxxxx>
---

I removed Bjorn's Tested-by tag as this patch has been through a major
rework.

There is one known failure mode (pointed out by AI review) with >= 3
resources, each with a different alignment requirement following a
certain pattern and the smallest align one must also be a composite
resource. Additional requirement is that the window must have an
odd-multiple start address.

While that corner case could be addressed by always forcing larger gaps
(at the cost of everyone paying the extra gap "tax"), I see no reason
to overengineer this for corner cases that might not occur in real
systems. If issues are reported, this can be revisited and perhaps more
complex fallback logic added (to avoid incurring the gap "tax"
unilaterally).

Adding the gap logic fixes known regressions seen in real systems, so
it is an important improvement even as is.
---
 drivers/pci/setup-bus.c | 57 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 55 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index b6be6cd16006..86c9557bc2fb 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1267,6 +1267,9 @@ static resource_size_t __pbus_size_mem(struct pci_bus *bus,
 	struct pci_dev *dev;
 	resource_size_t winsize = 0, win_align;
 	resource_size_t aligns[PCI_RES_ALIGN_RANGE] = {};
+	resource_size_t partial_blocks = 0, max_partial = 0;
+	unsigned int nonaligning = 0, rescount = 0;
+	resource_size_t remainders = 0, gap = 0;
 	int max_order = 0;
 
 	win_align = pci_min_window_alignment(bus, b_res->flags);
@@ -1277,7 +1280,7 @@ static resource_size_t __pbus_size_mem(struct pci_bus *bus,
 
 		pci_dev_for_each_resource(dev, r, i) {
 			const char *r_name = pci_resource_name(dev, i);
-			resource_size_t r_size, align;
+			resource_size_t r_size, align, remainder;
 
 			if (!pdev_resources_assignable(dev) ||
 			    !pdev_resource_should_fit(r))
@@ -1320,13 +1323,63 @@ static resource_size_t __pbus_size_mem(struct pci_bus *bus,
 				aligns[order] += align;
 			if (order > max_order)
 				max_order = order;
+
+			rescount++;
+
+			if (r_size <= align)
+				continue;
+			remainder = r_size - ALIGN_DOWN(r_size, align);
+			if (!remainder)
+				continue;
+
+			nonaligning++;
+			remainders += remainder;
+			partial_blocks += align;
+			max_partial = max(max_partial, align);
 		}
 	}
 
+	if (nonaligning && rescount == 2) {
+		resource_size_t second_align = max_partial;
+
+		/*
+		 * With two resources having differing alignment
+		 * requirements, the gap size can be optimized by aligning it
+		 * only up to the smaller of the alignment requirements.
+		 *
+		 * This also allows two composite remainders to share the
+		 * same block when the resources have the same alignment
+		 * requirement which is common with multi-GPU setups where
+		 * each bridge window contains a small BAR and a large BAR.
+		 * The small BARs can share the same align block.
+		 *
+		 * As this sizing phase cannot influence resource assignment
+		 * order and placement, this is only guaranteed to work with
+		 * two non-aligning resources.
+		 */
+		if (nonaligning == 2) {
+			second_align = partial_blocks - max_partial;
+		} else {
+			for (unsigned int i = 0; i < max_order; i++) {
+				if (!aligns[i])
+					continue;
+
+				second_align = min(second_align,
+						   1ULL << (i + __ffs(SZ_1M)));
+				break;
+			}
+		}
+
+		gap = ALIGN(remainders, second_align) - remainders;
+	} else if (nonaligning >= 2 ||
+		   (nonaligning && rescount > nonaligning)) {
+		gap = partial_blocks - remainders;
+	}
+
 	*head_align = calculate_head_align(aligns, max_order);
 	*head_align = max(*head_align, win_align);
 
-	return ALIGN(max(winsize, min_size), win_align);
+	return ALIGN(max(winsize + gap, min_size), win_align);
 }
 
 /**
-- 
2.47.3