Re: Prevent inconsistent CPU state after sequence of dlclose/dlopen

[Mathieu Desnoyers]

On 2025-01-10 12:24, Adhemerval Zanella Netto wrote:
> On 10/01/25 14:15, Mathieu Desnoyers wrote:
> > On 2025-01-10 12:10, Florian Weimer wrote:
> > > * Mathieu Desnoyers:
> > >
> > > > On 2025-01-10 11:54, Peter Zijlstra wrote:
> > > > > On Fri, Jan 10, 2025 at 10:55:36AM -0500, Mathieu Desnoyers wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I was discussing with Mark Rutland recently, and he pointed out that a
> > > > > > sequence of dlclose/dlopen mapping new code at the same addresses in
> > > > > > multithreaded environments is an issue on ARM, and possibly on Intel/AMD
> > > > > > with the newer TLB broadcast maintenance.
> > > > > What is the exact race? Should not munmap() invalidate the TLBs
> > > > > before
> > > > > it allows overlapping mmap() to complete?
> > > >
> > > > The race Mark mentioned (on ARM) is AFAIU the following scenario:
> > > >
> > > > CPU 0                     CPU 1
> > > >
> > > > - dlopen()
> > > >     - mmap PROT_EXEC @addr
> > > >                             - fetch insn @addr, CPU state expects unchanged insn.
> > > >                             - execute unrelated code
> > > > - dlclose(addr)
> > > >     - munmap @addr
> > > > - dlopen()
> > > >     - mmap PROT_EXEC @addr
> > > >                             - fetch new insn @addr. Incoherent CPU state.
> > >
> > > Unmapping an object while code is executing in it is undefined.
> >
> > That's not the scenario though. In this scenario, CPU 1 executes
> > _unrelated code_ while we unmap @addr.
>
> But in this scenario you still a concurrent dlclose while you have a running
> thread executing code from that module, right? Or am I still missing something
> here?

No.

> Or, are you saying that even after dlopen returns (assuming the scenario where
> it maps the code in a previous used mapping), the CPU is in an inconsistent
> state unless MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE is issued?

Yes, this is it.

And the issue is not specific to dlopen/dlclose. We can have a similar
issue if we have a sequence of:

dlopen  @addr
dlclose
mmap PROT_EXEC|PROT_WRITE  @addr
  - JIT writes some code and jumps to it.

So it appears to be something we may want to fix at the kernel level.

Thanks,

Mathieu


--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com