RE: [PATCH hyperv-next v5 03/11] Drivers: hv: Enable VTL mode for arm64

[Michael Kelley]

From: Arnd Bergmann <arnd@xxxxxxxx> Sent: Monday, March 10, 2025 2:21 PM
> 
> On Mon, Mar 10, 2025, at 22:01, Michael Kelley wrote:
> > From: Arnd Bergmann <arnd@xxxxxxxx> Sent: Saturday, March 8, 2025 1:05 PM
> >> >  config HYPERV_VTL_MODE
> >> >  	bool "Enable Linux to boot in VTL context"
> >> > -	depends on X86_64 && HYPERV
> >> > +	depends on (X86_64 || ARM64)
> >> >  	depends on SMP
> >> > +	select OF_EARLY_FLATTREE
> >> > +	select OF
> >> >  	default n
> >> >  	help
> >>
> >> Having the dependency below the top-level Kconfig entry feels a little
> >> counterintuitive. You could flip that back as it was before by doing
> >>
> >>       select HYPERV_VTL_MODE if !ACPI
> >>       depends on ACPI || SMP
> >>
> >> in the HYPERV option, leaving the dependency on HYPERV in
> >> HYPERV_VTL_MODE.
> >
> > I would argue that we don't ever want to implicitly select
> > HYPERV_VTL_MODE because of some other config setting or
> > lack thereof.  VTL mode is enough of a special case that it should
> > only be explicitly selected. If someone omits ACPI, then HYPERV
> > should not be selectable unless HYPERV_VTL_MODE is explicitly
> > selected.
> >
> > The last line of the comment for HYPERV_VTL_MODE says
> > "A kernel built with this option must run at VTL2, and will not run
> > as a normal guest."  In other words, don't choose this unless you
> > 100% know that VTL2 is what you want.
> 
> It sounds like the latter is the real problem: enabling a feature
> should never prevent something else from working. Can you describe
> what VTL context is and why it requires an exception to a rather
> fundamental rule here? If you build a kernel that runs on every
> single piece of arm64 hardware and every hypervisor, why can't
> you add HYPERV_VTL_MODE to that as an option?
> 

VTL = Virtual Trust Level, and VSM = Virtual Secure Mode, are Hyper-V's
terminology for offering multiple execution environments with
hierarchical trust in the context of a single VM. A normal guest
operating system runs at VTL 0, and there are no other VTLs in use.
But in some environments, additional software may run as a paravisor
layer between the normal guest OS and the hypervisor. This software
runs at some other VTL > 0, and has a higher privilege level within
the VM than software running at VTL 0 (which is the lowest privilege).
VTL 2 is used today in the Azure cloud with CoCo VMs to run a
paravisor, and there may be other uses in the future. See [1] if you
want more details on VSM and VTLs. Also [2] for the CoCo VM use
case.

Ideally, a Linux kernel image could detect at runtime what VTL it is
running at, and "do the right thing". Unfortunately, on x86 Linux this
has proved difficult (or perhaps impossible) because the amount of
boot-time setup required to ask the question about the current VTL
is significant. The idiosyncrasies and historical baggage of x86 requires
that Linux do some x86-specific initialization steps for VTL > 0
before the question can be asked. Hence the introduction of
CONFIG_HYPERV_VTL_MODE, and the behavior that when it is
selected, the kernel image won't run normally in VTL 0.

I'll go out on a limb and say that I suspect on arm64 a runtime
determination based on querying the VTL *could* be made (though
I'm not the person writing the code). But taking advantage of that
on arm64 produces an undesirable dichotomy with x86.

Roman may have further thoughts on the topic, but that's
what I know about how we got here.

Michael

[1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm
[2] https://techcommunity.microsoft.com/blog/windowsosplatform/openhcl-the-new-open-source-paravisor/4273172