[PATCH v13] Linux Xen PVH support (v13)

From: Konrad Rzeszutek Wilk
Date: Fri Jan 03 2014 - 15:34:59 EST

The patches, also available at

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/pvh.v13

implements the neccessary functionality to boot a PV guest in PVH mode.

This blog has a great description of what PVH is:

These patches are based on v3.13-rc6. If I had failed to
address your review I am terrible sorry - it was an oversight.
Please poke at the patch again.

Changes since v13: [http://mid.gmane.org/1388550945-25499-1-git-send-email-konrad.wilk@xxxxxxxxxx]
- Rework per David and Stefano review.
- Fix regression with Xen 4.1.
- Use native_cpuid instead of xen_cpuid.

v12: [http://mid.gmane.org/1387313503-31362-1-git-send-email-konrad.wilk@xxxxxxxxxx]
- Rework per Stefano's review.
- Split some patches up for easier review.
- Bugs fixed.

v11 as compared to v10: [https://lkml.org/lkml/2013/12/12/625]:
- Split patches in a more logical sense, squash some
- Dropped Acked-by's from folks
- Fleshed out descriptions

Regression wise - there are no bugs with Xen 4.[1,2,3,4].

That is if you compile/boot it with
CONFIG_XEN_PVH=y or "# CONFIG_XEN_PVH is not set" - in both cases as
either dom0 or domU there are no bugs. Also launched it as 32/64 bit
dom0 with 32/64 domU as PV or PVHVM, and along with SLES11, SLES12,
F15->F19 (32/64), OL5, OL6, RHEL5 (32/64) FreeBSD HVM, NetBSD PV without issues.

With Xen 4.1, there was a regression, (see
and it this patchset has the fix for it.

PARAVIRT OPS / x86_init /apic /smp ops

The paravirt ops that are in usage are:

pv_mmu_ops.flush_tlb_others = xen_flush_tlb_others;

These are still used:

pv_info = xen_info;
pv_init_ops = xen_init_ops;
pv_apic_ops = xen_apic_ops;
pv_time_ops = xen_time_ops;

And the x86_init,apic, and smp_ops ops are still in force.

This is just the first step so there might be some other ones
that are needed that I failed to enumerate.

The pv_cpu_ops is not used. From pv_mmu_ops only one is used.


The only things needed to make this work as PVH are:

0) Get the latest version of Xen and compile/install it.
See http://wiki.xen.org/wiki/Compiling_Xen_From_Source for details

1) Clone above mentioned tree

See http://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs#Configuring_the_Kernel
for details. The steps are:

cd $HOME
git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git linux
cd linux
git checkout origin/stable/pvh.v11

2) Compile with CONFIG_XEN_PVH=y

a) From scratch:

make defconfig
make menuconfig
Processor type and features ---> Linux guest support --->
Paravirtualization layer for spinlocks
Xen guest support (which will now show you:)
Support for running as a PVH guest (NEW)

in case you like to edit .config, it is:


You will also have to enable the block, network drivers, console, etc
which are in different submenus.

b). Based on your current distro.

cp /boot/config-`uname -r` $HOME/linux/.config
make menuconfig
Processor type and features ---> Linux guest support --->
Support for running as a PVH guest (NEW)

3) Launch it with 'pvh=1' in your guest config (for example):

extra="console=hvc0 debug kgdboc=hvc0 nokgdbroundup initcall_debug debug"
vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
vfb = [ 'vnc=1, vnclisten=,vncunused=1']

using 'xl'. Xend 'xm' does not have PVH support.

It will bootup as a normal PV guest, but 'xen-detect' will report it as an HVM

Items that have not been tested extensively or at all:
- Migration (xl save && xl restore for example).

- 32-bit guests (won't even present you with a CONFIG_XEN_PVH option)

- PCI passthrough

- Running it in dom0 mode (as the patches for that are not yet in Xen upstream).
If you want to try that, you can merge/pull Mukesh's branch:

cd $HOME/xen
git pull git://oss.oracle.com/git/mrathor/xen.git dom0pvh-v6

.. and use this bootup parameter ("dom0pvh=1"). Remember to recompile
and install the new version of Xen. This patchset
does not contain the patches neccessary to setup guests - but I can
create one easily enough.

- Memory ballooning
- Multiple VBDs, NICs, etc.

Things that are broken:
- CPUID filtering. There are no filtering done at all which means that
certain cpuid flags are exposed to the guest. The x2apic will cause
a crash if the NMI handler is invoked. The APERF will cause inferior
scheduling decisions.

If you encounter errors, please email with the following (pls note that the
guest config has 'on_reboot="preserve", on_crash="preserve" - which you should
have in your guest config to contain the memory of the guest):

a) xl dmesg
b) xl list
c) xenctx -s $HOME/linux/System.map -f -a -C <domain id>
[xenctx is sometimes found in /usr/lib/xen/bin/xenctx ]
d) the console output from the guest
e) Anything else you can think off.

Stash away your vmlinux file (it is too big to send via email) - as I might
need it later on.

That is it!

Thank you!

arch/arm/include/asm/xen/page.h | 1 +
arch/arm/xen/enlighten.c | 9 +-
arch/x86/include/asm/xen/page.h | 8 +-
arch/x86/xen/Kconfig | 5 ++
arch/x86/xen/enlighten.c | 100 +++++++++++++++++-----
arch/x86/xen/grant-table.c | 62 ++++++++++++++
arch/x86/xen/irq.c | 5 +-
arch/x86/xen/mmu.c | 166 +++++++++++++++++++++----------------
arch/x86/xen/p2m.c | 15 +++-
arch/x86/xen/setup.c | 40 +++++++--
arch/x86/xen/smp.c | 49 +++++++----
arch/x86/xen/xen-head.S | 25 +++++-
arch/x86/xen/xen-ops.h | 1 +
drivers/xen/events.c | 14 ++--
drivers/xen/gntdev.c | 2 +-
drivers/xen/grant-table.c | 87 ++++++++++++++-----
drivers/xen/platform-pci.c | 10 ++-
drivers/xen/xenbus/xenbus_client.c | 3 +-
include/xen/grant_table.h | 9 +-
include/xen/interface/elfnote.h | 13 +++
include/xen/xen.h | 14 ++++
21 files changed, 483 insertions(+), 155 deletions(-)

Konrad Rzeszutek Wilk (7):
xen/pvh: Don't setup P2M tree.
xen/mmu/p2m: Refactor the xen_pagetable_init code (v2).
xen/mmu: Cleanup xen_pagetable_p2m_copy a bit.
xen/grants: Remove gnttab_max_grant_frames dependency on gnttab_init.
xen/grant-table: Refactor gnttab_init
xen/grant: Implement an grant frame array struct (v2).
xen/pvh: Piggyback on PVHVM for grant driver (v4)

Mukesh Rathor (12):
xen/p2m: Check for auto-xlat when doing mfn_to_local_pfn.
xen/pvh/x86: Define what an PVH guest is (v3).
xen/pvh: Early bootup changes in PV code (v4).
xen/pvh: MMU changes for PVH (v2)
xen/pvh/mmu: Use PV TLB instead of native.
xen/pvh: Setup up shared_info.
xen/pvh: Load GDT/GS in early PV bootup code for BSP.
xen/pvh: Secondary VCPU bringup (non-bootup CPUs)
xen/pvh: Update E820 to work with PVH (v2)
xen/pvh: Piggyback on PVHVM for event channels (v2)
xen/pvh: Piggyback on PVHVM XenBus.
xen/pvh: Support ParaVirtualized Hardware extensions (v3).

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/