Re: [PATCH 0/3] KVM: x86: KVM_MEM_PCI_HOLE memory

[Michael S. Tsirkin]

On Tue, Jul 28, 2020 at 04:37:38PM +0200, Vitaly Kuznetsov wrote:
> This is a continuation of "[PATCH RFC 0/5] KVM: x86: KVM_MEM_ALLONES
> memory" work: 
> https://lore.kernel.org/kvm/20200514180540.52407-1-vkuznets@xxxxxxxxxx/
> and pairs with Julia's "x86/PCI: Use MMCONFIG by default for KVM guests":
> https://lore.kernel.org/linux-pci/20200722001513.298315-1-jusual@xxxxxxxxxx/
> 
> PCIe config space can (depending on the configuration) be quite big but
> usually is sparsely populated. Guest may scan it by accessing individual
> device's page which, when device is missing, is supposed to have 'pci
> hole' semantics: reads return '0xff' and writes get discarded.
> 
> When testing Linux kernel boot with QEMU q35 VM and direct kernel boot
> I observed 8193 accesses to PCI hole memory. When such exit is handled
> in KVM without exiting to userspace, it takes roughly 0.000001 sec.
> Handling the same exit in userspace is six times slower (0.000006 sec) so
> the overal; difference is 0.04 sec. This may be significant for 'microvm'
> ideas.
> 
> Note, the same speed can already be achieved by using KVM_MEM_READONLY
> but doing this would require allocating real memory for all missing
> devices and e.g. 8192 pages gives us 32mb. This will have to be allocated
> for each guest separately and for 'microvm' use-cases this is likely
> a no-go.
> 
> Introduce special KVM_MEM_PCI_HOLE memory: userspace doesn't need to
> back it with real memory, all reads from it are handled inside KVM and
> return '0xff'. Writes still go to userspace but these should be extremely
> rare.
> 
> The original 'KVM_MEM_ALLONES' idea had additional optimizations: KVM
> was mapping all 'PCI hole' pages to a single read-only page stuffed with
> 0xff. This is omitted in this submission as the benefits are unclear:
> KVM will have to allocate SPTEs (either on demand or aggressively) and
> this also consumes time/memory.

Curious about this: if we do it aggressively on the 1st fault,
how long does it take to allocate 256 huge page SPTEs?
And the amount of memory seems pretty small then, right?

> We can always take a look at possible
> optimizations later.
> 
> Vitaly Kuznetsov (3):
>   KVM: x86: move kvm_vcpu_gfn_to_memslot() out of try_async_pf()
>   KVM: x86: introduce KVM_MEM_PCI_HOLE memory
>   KVM: selftests: add KVM_MEM_PCI_HOLE test
> 
>  Documentation/virt/kvm/api.rst                |  19 ++-
>  arch/x86/include/uapi/asm/kvm.h               |   1 +
>  arch/x86/kvm/mmu/mmu.c                        |  19 +--
>  arch/x86/kvm/mmu/paging_tmpl.h                |  10 +-
>  arch/x86/kvm/x86.c                            |  10 +-
>  include/linux/kvm_host.h                      |   7 +-
>  include/uapi/linux/kvm.h                      |   3 +-
>  tools/testing/selftests/kvm/Makefile          |   1 +
>  .../testing/selftests/kvm/include/kvm_util.h  |   1 +
>  tools/testing/selftests/kvm/lib/kvm_util.c    |  81 +++++++------
>  .../kvm/x86_64/memory_slot_pci_hole.c         | 112 ++++++++++++++++++
>  virt/kvm/kvm_main.c                           |  39 ++++--
>  12 files changed, 243 insertions(+), 60 deletions(-)
>  create mode 100644 tools/testing/selftests/kvm/x86_64/memory_slot_pci_hole.c
> 
> -- 
> 2.25.4