Re: [PATCH v14 004/113] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module

[Huang, Kai]

On Sun, 2023-05-28 at 21:18 -0700, Yamahata, Isaku wrote:
> +static void __init vmx_tdx_on(void *info)
> +{
> +       atomic_t *err = info;
> +       int r;
> +
> +       r = vmx_hardware_enable();
> +       if (!r)
> +	       r = tdx_cpu_enable();
> +       if (r)
> +	       atomic_set(err, r);
> +}
> +
> +static void __init vmx_off(void *unused)
> +{
> +       vmx_hardware_disable();
> +}
> +
> +int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
> +{
> +	atomic_t err = ATOMIC_INIT(0);
> +	int r = 0;
> +
> +	if (!enable_ept) {
> +		pr_warn("Cannot enable TDX with EPT disabled\n");
> +		return -EINVAL;
> +	}
> +
> +	/* tdx_enable() in tdx_module_setup() requires cpus lock. */
> +	cpus_read_lock();
> +	on_each_cpu(vmx_tdx_on, &err, true);	/* TDX requires vmxon. */
> +	r = atomic_read(&err);
> +	if (!r)
> +		r = tdx_module_setup();
> +	on_each_cpu(vmx_off, NULL, true);
> +	cpus_read_unlock();
> +
> +	return r;
> +}

As we discussed in v13, this code doesn't track which CPUs have run
vmx_hardware_enable() successfully.  Thus if ...

	on_each_cpu(vmx_tdx_on, &err, true);	/* TDX requires vmxon. */

... fails on some cpu due to whatever reason, in ...
	
	on_each_cpu(vmx_off, NULL, true);

... vmx_hardware_disable() will fail to do VMXOFF for those cpus that haven't
done VMXON successfully yet, resulting in BUG_ON(!kvm_rebooting) being triggered
in kvm_spurious_fault().

We need a per-cpu flag to track whether cpu has done VMXON successfully.