Re: [External] Re: [PATCH v9 0/8] Parallel CPU bringup for x86_64

[Oleksandr Natalenko]

On 21.02.2023 21:04, Usama Arif wrote:
> On 21/02/2023 19:10, David Woodhouse wrote:
> > On Tue, 2023-02-21 at 13:14 +0100, Oleksandr Natalenko wrote:
> > > With this in place:
> > >
> > > ```
> > >          early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(0);
> > >          initial_gs = per_cpu_offset(0);
> > >          smpboot_control = 0;
> > > ```
> > >
> > > the resume does not work.
> >
> > Yeah, I think it's always running on CPU0 after the other CPUs are
> > taken down anyway.
> >
> > We definitely *do* need to clear smpboot_control because we absolutely
> > want it using the temp_stack we explicitly put into initial_stack, not
> > finding its own idle thread.
> >
> > The problem was that it was never being restored to STARTUP_SECONDARY
> > in the parallel modes, because that's a one-time setup in
> > native_smp_prepare_cpus(). So we can just restore it in
> > arch_thaw_secondary_cpus_begin() too, by working this into patch 6 of
> > the series.
> >
> > (Usama, I think my tree is fairly out of date now so I'll let you do
> > that? Thanks!).
>
> Sounds good! Will send out the next revision with below diff, 
> checkpatch
> fixes and rebased to 6.2 (testing it now). The below fix looks good! 
> Oleksandr, would you mind testing out suspend/resume with the below 
> diff on your AMD machine just to make sure it fixes the issue before I 
> send out the next revision with it. Thanks!

Right, so I applied the whole series + this fix, and the suspend/resume 
works. Thanks!

Reported-by: Oleksandr Natalenko <oleksandr@xxxxxxxxxxxxxx>
Tested-by: Oleksandr Natalenko <oleksandr@xxxxxxxxxxxxxx>

There's another open question pending though: why would this series 
cause booting up one CPU only on an older AMD CPU. But I'd expect 
Piotr's fellow to chime in occasionally.

> > diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> > index 50621793671d..3db77a257ae2 100644
> > --- a/arch/x86/kernel/smpboot.c
> > +++ b/arch/x86/kernel/smpboot.c
> > @@ -1571,6 +1571,17 @@ void __init native_smp_prepare_cpus(unsigned 
> > int max_cpus)
> >     void arch_thaw_secondary_cpus_begin(void)
> >   {
> > +	/*
> > +	 * On suspend, smpboot_control will have been zeroed to allow the
> > +	 * boot CPU to use explicitly passed values including a temporary
> > +	 * stack. Since native_smp_prepare_cpus() won't be called again,
> > +	 * restore the appropriate value for the parallel startup modes.
> > +	 */
> > +	if (do_parallel_bringup) {
> > +		smpboot_control = STARTUP_SECONDARY |
> > +			(x2apic_mode ? STARTUP_APICID_CPUID_0B : STARTUP_APICID_CPUID_01);
> > +	}
> > +
> >   	set_cache_aps_delayed_init(true);
> >   }

--
  Oleksandr Natalenko (post-factum)