Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper

[Ankit Agrawal]

>> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated
>> > as an IO Port BAR, it occurs to me that there's no config space
>> > emulation of BAR2 (or BAR3) here.  Doesn't this mean that QEMU registers
>> > the BAR as 32-bit, non-prefetchable?  ie. VFIOBAR.type & .mem64 are
>> > wrong?
>>
>> Maybe I didn't understand the question, but the PCI config space read/write
>> would still be handled by vfio_pci_core_read/write() which returns the
>> appropriate flags. I have checked that the device BARs are 64b and
>> prefetchable in the VM.
>
> vfio_pci_core_read/write() accesses the physical device, which doesn't
> implement BAR2.  Why would an unimplemented BAR2 on the physical device
> report 64-bit, prefetchable?
>
> QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
> vfio_bar_prepare() and passes this type to pci_register_bar() in
> vfio_bar_register().  Without an implementation of a config space read
> op in the variant driver and with no physical implementation of BAR2 on
> the device, I don't see how we get correct values in these fields.

I think I see the cause of confusion. There are real PCIe compliant BARs
present on the device, just that it isn't being used once the C2C
interconnect is active. The BARs are 64b prefetchable. Here it the lspci
snippet of the device on the host.
# lspci -v -s 9:1:0.0
0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
        Subsystem: NVIDIA Corporation Device 16eb
        Physical Slot: 0-5
        Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, IOMMU group 19
        Memory at 661002000000 (64-bit, prefetchable) [size=16M]
        Memory at 662000000000 (64-bit, prefetchable) [size=128G]
        Memory at 661000000000 (64-bit, prefetchable) [size=32M]

I suppose this answers the BAR sizing question as well?