Re: [PATCH printk v1 03/13] printk: use percpu flag instead of cpu_online()

[John Ogness]

I have taken some time to investigate the percpu implementation so that
I could provide clear answers here.

On 2022-02-15, Petr Mladek <pmladek@xxxxxxxx> wrote:
> I am not 100% sure. But it seems that static per-CPU variables might
> actually be used since the boot.

You are correct, but until per-cpu areas are setup, they all point to
CPU0. Normally that is not a problem since usually code is using
this_cpu_ptr() (which will always be the CPU0 in early boot), rather
than specifying foreign CPUs with per_cpu_ptr().

> Most likely, only dynamically allocated per-cpu variables have to wait
> until the per-cpu areas are initialized.

It is also important to wait if data will be stored that is no longer
valid after per-cpu areas are setup. setup_per_cpu_areas() copies the
static CPU0 per-cpu value to all the newly setup per-cpu areas.

This is actually the cause for the mystery [0] of failing irq_work when
printk_deferred was added with commit
15341b1dd409749fa5625e4b632013b6ba81609b ("char/random: silence a
lockdep splat with printk()". The problem was that the irq work was
being queued and _claimed_ before per-cpu areas were setup. But the
actual work functions are not called there. So when the irq_work was
copied to all the new per-cpu areas and the actual work was carried out,
the work item that was cleared was the old static variable and not the
new per-cpu area variable. This means all the irq_work flags were left
in the claimed state (PENDING|BUSY) and could no longer be claimed.

By avoiding queueing irq_work before setup_per_cpu_areas(), we correctly
avoided this problem. (I considered sending a patch so that
irq_work_claim() will fail if a global @percpu_complete is not yet
set. But for now, our set_percpu_data_ready() solution is at least good
enough for the printk subsystem.)

> We should probably revisit the code and remove the fallback to
> normal static variables.

Definitely. Now it is clear that @printk_count and @printk_count_nmi do
not need early variants.

John

[0] https://lore.kernel.org/lkml/aa0732c6-5c4e-8a8b-a1c1-75ebe3dca05b@xxxxxxxxxxxxxxxxxxxxxx/