RE: [PATCH 1/1] Documentation: hyperv: Update docs for PCI pass-thru device support

[Michael Kelley]

From: Naman Jain <namjain@xxxxxxxxxxxxxxxxxxx> Sent: Tuesday, November 3, 2026 12:43 AM
> 
> On 11/1/2026 7:52 AM, Michael Kelley wrote:
> > Add a section to the documentation describing the IRQ domain and IRQ chip
> > hierarchy that is created for virtual PCI devices, including the
> > differences on x86 and arm64, and the reasons for those differences.
> >
> > In the context of the IRQ domain and chip hierarchies, describe how the
> > CPU selected for effective affinity is spread across CPUs in the affinity
> > masks. Again, account for the differences in x86 and arm64.
> >
> > Finally, rework existing text on how MSI messages are composed. Provide
> > an expanded section on the overall approach that Hyper-V takes to MSI
> > messages. The section provides more context on the handling of MSI messages
> > in the Hyper-V vPCI driver, and why it is structured the way it is.
> >
> > Signed-off-by: Michael Kelley <mhklinux@xxxxxxxxxxx>
> > ---
> 
> <snip>
> 
> > +MSI Messages
> > +------------
> > +In a typical guest on KVM, the VMM traps guest writes to a vPCI
> > +device's MSI/MSI-X registers/tables. The guest writes a virutal
> > +message (with a guest APIC address and a guest vector), and VMM
> > +emulation translates the message to program the physical device and
> > +physical device interrupt remapping (IR) tables on the guest's
> > +behalf. The guest never sees or handles the host-level MSI message.
> > +
> > +Hyper-V takes a different approach in which MSI/MSI-X register or
> > +table writes are not trapped. The guest writes the device's MSI/MSI-X
> > +registers/tables directly with a host-valid MSI message. The physical
> > +IOMMU and its remapping table are still owned by the host, so the
> > +guest must ask the host for the proper message.
> > +
> 
> Could we distinguish MSI-X table writes from MSI configuration-space
> writes here?
> 
> pci_write_msg_msi() programs the MSI address/data through
> pci_write_config_*(), which goes through the Hyper-V config-space
> access mechanism. As described in the existing "PCI Device Setup"
> and "PCI config space access in a CoCo VM" sections, those accesses
> trap to Hyper-V in normal VMs, or use explicit hypercalls in CoCo VMs.
> 
> The distinction seems to be that Hyper-V requires the guest to obtain
> a host-valid MSI message explicitly over VMBus, rather than relying
> on interception of guest MSI programming to establish the interrupt
> mapping. That is separate from whether the programming accesses
> themselves are trapped. We should probably make that distinction instead
> of saying that both MSI and MSI-X writes are untrapped?

Yes, you are right. The current text is imprecise and conflates the MSI
and MSI-X cases. Both cases require the guest to request the MSI
message from the Hyper-V host, but only for MSI-X is the address/data
written to an address that is not trapped. My assumption is that Hyper-V
takes this approach for MSI so that the same mechanism is used for
both MSI and MSI-X, even though in the MSI case the address/data is
written to a PCI config space address that is trapped (or uses a hypercall
in the CoCo VM case).

I'll reword this section to make the distinction clear.

FWIW, the code comment for hv_pci_restore_msi_state() conflates
MSI and MSI-X in the same way. I'll see about a patch to tweak that.

> 
> > +The "asking" is done by the .irq_compose_msi_msg function for the
> > +"Hyper-V PCIe MSI" chip, which is hv_compose_msi_msg(). The
> > +implementation requires sending a VMBus message to the Hyper-V host
> > +and awaiting an interrupt indicating receipt of a reply message.
> > +Since .irq_compose_msi_msg can be called with IRQ spin locks held, it
> > +doesn't work to do the normal sleep until awakened by the interrupt.
> > +Instead hv_compose_msi_msg() must send the VMBus message, and
> > +poll for the completion message. As further complexity, the vPCI
> > +device could be ejected/rescinded while the polling is in progress,
> > +so this scenario must be detected as well. See comments in the code
> > +regarding this very tricky area. A synchronous hypercall to get the
> > +MSI message would be a much nicer interface, but the code makes do
> > +with what's available.
> > +
> > +Once the host-valid MSI message is obtained, the guest must treat it
> > +as a black-box that the guest does not interpret or modify. The only
> > +valid guest action is to write it to the MSI/MSI-X registers/tables
> > +using the .irq_chip_write_msi_msg function on the HV-PCI-MSIX-<PCIaddr>
> > +chip. That function is pci_msi_domain_write_msg(), which calls
> > +__pci_write_msi_msg().
> > +
> > +If the affinity of an IRQ changes, the steps must be
> > +repeated. A new host-valid MSI message is requested via the
> > +.irq_compose_msi_msg function and that message is written
> > +to the MSI/MSI-X registers using the .irq_chip_write_msi_msg
> > +function.
> 
> I don't think ordinary affinity changes require a new MSI message,
> including with the companion arm64 spreading patch applied.
> 
> hv_pcie_init_dev_msi_info() installs irq_chip_set_affinity_parent
> instead of the generic compose-and-write affinity handler.
> 
> On arm64, the new hv_pci_arm64_set_affinity() selects a CPU and passes
> a single-CPU mask to the parent. gic_set_affinity() then updates the
> SPI's GICD_IROUTER and effective affinity, without recomposing or
> rewriting the device's MSI message.
> 
> Similarly, on x86, the new CPU/vector is communicated through the
> Hyper-V unmask/retarget path using the existing MSI descriptor.
> 
> Could this paragraph describe updating the routing of the existing
> message, rather than obtaining and programming a new one?

Indeed, I agree. The .irq_set_affinity function is originally set
to msi_domain_set_affinity() by msi_lib_init_dev_msi_info().
msi_domain_set_affinity() is the compose-and-write affinity
handler you referred to.

But hv_pcie_init_dev_msi_info() overwrites msi_domain_set_affinity()
with irq_chip_set_affinity_parent(). Evidently the mechanisms you
describe above for changing the target guest CPU are sufficient.
Just to be sure, I ran a test on an arm64 VM in Azure with a Mellanox
VF, and indeed there are no calls to hv_compose_msi_msg() when
manually changing the affinity of one of the Mellanox IRQs. I wish
I knew more about what exactly Hyper-V is doing under-the-hood
so could better interpret the Linux-side code!

I'll update this portion of the text as well.

Thanks for the careful review. Let me know if there are other
aspects of this topic that you've figured out the hard way that
should be documented. Hopefully getting this written down in
one place will help someone new coming along in the future.

Michael