[PATCH v2 03/11] iommu/arm-smmu-v3: Add arm_smmu_drain_queue_for_iopf() helper

[Nicolin Chen]

A poll-until-empty form does not converge on the shared EVTQ or PRIQ since
other masters keep advancing PROD with unrelated traffic. A fixed snapshot
bounds the wait even under sustained unrelated load, since the target does
not move with subsequent enqueues.

Add a helper that drains an SMMU EVTQ or PRIQ up to a PROD snapshot taken
on entry, waiting until CONS reaches the snapshot. Every entry already in
the queue is consumed by the threaded IRQ handler before this returns. A
subsequent change uses it from the IOPF attach-release path.

SMMUv3 guarantees that after a CMD_SYNC following STE/CD invalidation and
any CMD_ATC_INV, no further entries tied to the pre-SYNC configuration will
appear in either queue. A caller that has completed that sequence captures
its entire in-flight cohort in the snapshot.

Read CONS from MMIO each iteration so the comparison does not rely on the
cached llq->cons, which the threaded handler owns. Disable WFE since SMMU
does not signal SEV on EVTQ/PRIQ PROD advance. Call cond_resched() inside
the loop so this caller does not starve the threaded IRQ handler it waits
on when both share a CPU on a non-preemptible kernel.

This is a prerequisite to apply bug fix calling iopf_queue_flush_dev().

Fixes: cfea71aea921 ("iommu/arm-smmu-v3: Put iopf enablement in the domain attach path")
Cc: stable@xxxxxxxxxxxxxxx # v6.16
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Nicolin Chen <nicolinc@xxxxxxxxxx>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 27 +++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index cf41b3cf5985f..4794a15f351c4 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -894,6 +894,33 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
 					   cmds->num, true);
 }
 
+/* Drain an SMMU EVTQ or PRIQ to a PROD snapshot taken on entry */
+static int arm_smmu_drain_queue_for_iopf(struct arm_smmu_device *smmu,
+					 struct arm_smmu_queue *q)
+{
+	struct arm_smmu_queue_poll qp;
+	u32 prod, cons;
+	int ret = 0;
+
+	/* Snapshot PROD; entries [old_cons, prod) are the cohort to drain */
+	prod = readl_relaxed(q->prod_reg);
+	queue_poll_init(smmu, &qp);
+	qp.wfe = false; /* No SEV on EVTQ/PRIQ PROD advance */
+	/* Read MMIO each iteration; llq->cons is the threaded handler's */
+	do {
+		cons = readl_relaxed(q->cons_reg);
+		if (__queue_empty(&q->llq, cons, prod) ||
+		    __queue_consumed(&q->llq, cons, prod))
+			return 0;
+		cond_resched();
+	} while (!(ret = queue_poll(&qp)));
+
+	dev_warn_ratelimited(smmu->dev,
+			     "queue drain timed out at prod=0x%x cons=0x%x\n",
+			     prod, cons);
+	return ret;
+}
+
 static void arm_smmu_page_response(struct device *dev, struct iopf_fault *unused,
 				   struct iommu_page_response *resp)
 {
-- 
2.43.0