Re: [PATCH] powerpc/pseries/cpuhp: respect current SMT when adding new CPU

[Nathan Lynch]

Michal Suchánek <msuchanek@xxxxxxx> writes:
> On Mon, Feb 13, 2023 at 08:46:50AM -0600, Nathan Lynch wrote:
>> Laurent Dufour <ldufour@xxxxxxxxxxxxx> writes:
>> > When a new CPU is added, the kernel is activating all its threads. This
>> > leads to weird, but functional, result when adding CPU on a SMT 4 system
>> > for instance.
>> >
>> > Here the newly added CPU 1 has 8 threads while the other one has 4 threads
>> > active (system has been booted with the 'smt-enabled=4' kernel option):
>> >
>> > ltcden3-lp12:~ # ppc64_cpu --info
>> > Core   0:    0*    1*    2*    3*    4     5     6     7
>> > Core   1:    8*    9*   10*   11*   12*   13*   14*   15*
>> >
>> > There is no SMT value in the kernel. It is possible to run unbalanced LPAR
>> > with 2 threads for a CPU, 4 for another one, and 5 on the latest.
>> >
>> > To work around this possibility, and assuming that the LPAR run with the
>> > same number of threads for each CPU, which is the common case,
>> 
>> I am skeptical at best of baking that assumption into this code. Mixed
>> SMT modes within a partition doesn't strike me as an unreasonable
>> possibility for some use cases. And if that's wrong, then we should just
>> add a global smt value instead of using heuristics.
>> 
>> > the number
>> > of active threads of the CPU doing the hot-plug operation is computed. Only
>> > that number of threads will be activated for the newly added CPU.
>> >
>> > This way on a LPAR running in SMT=4, newly added CPU will be running 4
>> > threads, which is what a end user would expect.
>> 
>> I could see why most users would prefer this new behavior. But surely
>> some users have come to expect the existing behavior, which has been in
>> place for years, and developed workarounds that might be broken by this
>> change?
>> 
>> I would suggest that to handle this well, we need to give user space
>> more ability to tell the kernel what actions to take on added cores, on
>> an opt-in basis.
>> 
>> This could take the form of extending the DLPAR sysfs command set:
>> 
>> Option 1 - Add a flag that tells the kernel not to online any threads at
>> all; user space will online the desired threads later.
>> 
>> Option 2 - Add an option that tells the kernel which SMT mode to apply.
>
> powerpc-utils grew some drmgr hooks recently so maybe the policy can be
> moved to userspace?

I'm not sure whether the hook mechanism would come into play, but yes, I
am suggesting that user space be given the option of overriding the
kernel's current behavior.