Re: [PATCH v9 07/18] x86/virt/tdx: Do TDX module per-cpu initialization

[Huang, Kai]

On Mon, 2023-02-13 at 14:28 -0800, Dave Hansen wrote:
> On 2/13/23 13:13, Huang, Kai wrote:
> > Perhaps I didn't explain clearly in the comment.  Below is the updated one:
> > 
> >                 /*
> >                  * The previous call of __tdx_enable() may only have
> >                  * initialized part of present cpus during module
> >                  * initialization, and new cpus may have become online
> >                  * since then w/o doing per-cpu initialization.
> >                  *
> >                  * For example, a new CPU can become online when KVM is
> >                  * unloaded, in which case tdx_cpu_enable() is not called since
> >                  * KVM's CPU online callback has been removed.
> >                  *
> >                  * To make sure all online cpus are TDX-runnable, always
> >                  * do per-cpu initialization for all online cpus here
> >                  * even the module has been initialized.
> >                  */
> 
> This is voodoo.
> 
> I want a TDX-specific hotplug CPU handler.  Period.  Please make that
> happen.  
> 

Yes 100% agreed.

> Put that code in this patch.  That handler should:
> 
> 	1. Run after the KVM handler (if present)
> 	2. See if VMX is on
> 	3. If VMX is on:
> 	 3a. Run smp_func_module_lp_init(), else
> 	 3b. Mark the CPU as needing smp_func_module_lp_init()
> 
> Then, in the 'case TDX_MODULE_INITIALIZED:', you call a function to
> iterate over the cpumask that was generated in 3b.
> 
> That makes the handoff *EXPLICIT*.  You know exactly which CPUs need
> what done to them.  A CPU hotplug either explicitly involves doing the
> work to make TDX work on the CPU, or explicitly defers the work to a
> specific later time in a specific later piece of code.

In 3b. we don't need to "explicitly mark the  CPU as needing
smp_func_module_lp_init()".  We already have __cpu_tdx_mask to track whether
LP.INIT has been done on one cpu and we can use that to determine:

	Any online cpu which isn't set in __cpu_tdx_mask needs to do LP.INIT in
	tdx_enable().

And the function module_lp_init_online_cpus() already handles that, and it can
be called directly in tdx_enable() path (as shown in this patch).

I'll do above as you suggested, but just use __cpu_tdx_mask as explained above.

( My main concern is "Run after the KVM handler" seems a little bit hacky to me.
Logically, it's more reasonable to have the TDX callback _before_ KVM's but not
_after_.  If any user (KVM) has done tdx_enable() successfully, the TDX code
should give the user a "TDX-runnable" cpu before user (KVM)'s own callback is
involved. Anyway as mentioned above, I'll do above as you suggested.)