Re: [RFC PATCH] introduce sys_membarrier(): process-wide memorybarrier

[Steven Rostedt]

On Sat, 2010-01-09 at 20:44 -0500, Mathieu Desnoyers wrote:

> > So what if we have a linear decrease in performance with the number of
> > threads on the write side?
> 
> Hrm, looking at arch/x86/include/asm/mmu_context.h
> 
> switch_mm(), which is basically called each time the scheduler needs to
> change the current task, does a
> 
> cpumask_clear_cpu(cpu, mm_cpumask(prev));
> 
> and
> 
> cpumask_set_cpu(cpu, mm_cpumask(next));
> 
> which precise goal is to stop the flush ipis for the previous mm. The
> 100$ question is : why do we have to confirm that the thread is indeed
> on the runqueue (taking locks and everything) when we could simply just
> bluntly use the mm_cpumask for our own IPIs ?


I was just looking at that code, and was thinking the same thing ;-)

> cpumask_clear_cpu and cpumask_set_cpu translate into clear_bit/set_bit.
> cpumask_next does a find_next_bit on the cpumask.
> 
> clear_bit/set_bit are atomic and not reordered on x86. PowerPC also uses
> ll/sc loops in bitops.h, so I think it should be pretty safe to assume
> that mm_cpumask is, by design, made to be used as cpumask to send a
> broadcast IPI to all CPUs which run threads belonging to a given
> process.
> 
> So, how about just using mm_cpumask(current) for the broadcast ? Then we
> don't even need to allocate our own cpumask neither.
> 
> Or am I missing something ? I just sounds too simple.

I think we can use it. If for some reason it does not satisfy what you
need then I also think the TLB flushing is also broken.

IIRC, (Paul help me out on this), what Paul said earlier, we are trying
to protect against this scenario:

(from Paul's email:)


> 
>         CPU 1                           CPU 2
>      -----------                    -------------
> 
>         <user space>                    <kernel space, switching to task>
> 
>                                         ->curr updated
> 
>                                         <long code path, maybe mb?>
> 
>                                         <user space>
> 
>                                         rcu_read_lock(); [load only]
> 
>                                         obj = list->next
> 
>         list_del(obj)
> 
>         sys_membarrier();
>         < kernel space >
> 
>         if (task_rq(task)->curr != task)
>         < but load to obj reordered before store to ->curr >
> 
>         < user space >
> 
>         < misses that CPU 2 is in rcu section >


If the TLB flush misses that CPU 2 has a threaded task, and does not
flush CPU 2s TLB, it can also risk the same type of crash.

> 
>         [CPU 2's ->curr update now visible]
> 
>         [CPU 2's rcu_read_lock() store now visible]
> 
>         free(obj);
> 
>                                         use_object(obj); <=== crash!
> 

Think about it. If you change a process mmap, say you updated a mmap of
a file by flushing out one page and replacing it with another. If the
above missed sending to CPU 2, then CPU 2 may still be accessing the old
page of the file, and not the new one.

I think this may be the safe bet.

-- Steve


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/