Re: per-cpu operation madness vs validation

[Christoph Lameter]

On Tue, 26 Jul 2011, Peter Zijlstra wrote:

> Hi all,
>
> so recently our per-cpu ops have exploded.. who can say (without
> looking) what the difference is between percpu_read() and
> this_cpu_read() ?
>
> Now Thomas recently did a fresh -rt and ran into the fun problem of
> trying to reconstruct the requirements on a lot of the per-cpu grub
> we've grown over the past few releases.
>
> Does it require preempt-disable, bh disable, irq-disable, is it perhaps
> covered by a lock, what!? I'm sure Thomas can point you to a few gems if
> you're interested in specifics.

The this_cpu stuff attempts to make it more consistent

__this_cpu_xx stuff is to be run in preempt disable contexts or in context
where interrupts were disabled.

this_cpu_xx stuff is for context where preemption is possible.

irqsafe_cpu_xx stuff is for contexts where access is also possible from
an interrupt context.

Those all fold to the same operation on x86. The x86 cpu operations with
segment prefix are interrupt and preempt safe. The context issues come
into play in fallback scenarios where other architectures do not have
instructions that can perform the same things as x86.

> Now the reason is of course that -rt changes some things, even when
> using migrate_disable() it might be multiple processes might try and
> access the per-cpu variable, needing serialization.

On x86 these operations are rt safe by the pure fact that all these
operations are the same interrupt safe instruction. The difficulties come in for other
architectures. In those cases preemption or interrupts need to be
disabled.

> Even for mainline such validation is useful, suppose you intended the
> variable to only be accessed by task context, and thus disabling
> preemption is plenty to fully serialize things. However unbeknown to the
> original author someone adds usage from IRQ context, and voila things
> start breaking.

Right.

> The point of course is, how are we going to go about doing this, I'm
> sure adding a proper conditional to each and every per-cpu op is going
> to be a herculean task, and most of it utterly boring.

Basically we need to track the contexts in which references to a certain
per cpu variable are established. Then constraints can be enforced. For
example:

1. A variable used in an interupt context with __this_cpu ops requires
irqsafe_cpu_xxx when used in context where interrupts are enabled.

2. A variable used in a non-preemptible context with __this_cpu ops
requires this_cpu ops in a preemptible context.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/