Re: [PATCH v5 12/22] x86/virt/tdx: Reset software states during TDX module shutdown

[Huang, Kai]

> @@ -1179,6 +1179,7 @@ EXPORT_SYMBOL_FOR_KVM(tdx_enable);
>  int tdx_module_shutdown(void)
>  {
>  	struct tdx_module_args args = {};
> +	int ret, cpu;
>  
>  	/*
>  	 * Shut down the TDX module and prepare handoff data for the next
> @@ -1188,7 +1189,22 @@ int tdx_module_shutdown(void)
>  	 * modules as new modules likely have higher handoff version.
>  	 */
>  	args.rcx = tdx_sysinfo.handoff.module_hv;
> -	return seamcall_prerr(TDH_SYS_SHUTDOWN, &args);
> +	ret = seamcall_prerr(TDH_SYS_SHUTDOWN, &args);
> +	if (ret)
> +		return ret;
> +
> +	tdx_module_status = TDX_MODULE_UNINITIALIZED;
> +	sysinit_done = false;
> +	sysinit_ret = 0;
> +
> +	/*
> +	 * By reaching here CPUHP is disabled and all present CPUs
> +	 * are online. It's safe to just loop all online CPUs and
> +	 * reset the per-cpu flag.
> +	 */
> +	for_each_online_cpu(cpu)
> +		per_cpu(tdx_lp_initialized, cpu) = false;

Since you have removed the requirement that P-SEAMLDR.INSTALL must be done
on all CPUs, and removed the relevant patch, the "all present CPUs are
online" part isn't correct anymore.

And using for_each_online_cpu() isn't enough since this doesn't reset the
tdx_lp_initialized for offline CPUs.

One way is to just use for_each_possible_cpu() here so tdx_lp_initialized
for all CPUs are reset.  Since the "CPUHP is disabled" part is still correct
AFAICT (since stop_machine() disables CPUHP internally during the
operation), resetting tdx_lp_initialized for offline CPUs won't race with
CPUHP.

And assuming this series will be applied after Sean's VMXON series, we will
have a TDX-specific CPUHP callback tdx_online_cpu() in TDX x86 core to do
tdx_cpu_enable(), which will then enable TDX again on the new-online CPU.

Btw, w/o Sean's VMXON series, currently only KVM provides the TDX-specific
CPUHP callback.  So it seems if module update is done when KVM is not
loaded, there will be no TDX-specific CPUHP callback to re-enable TDX for
the new-online CPU.  This means any SEAMCALL on that CPU will fail before
KVM module is loaded again (which will then re-register the TDX-specific
CPUHP and run tdx_cpu_enable() for all online CPUs). 

But I don't think we should consider this case.