Re: [PATCH v6 3/3] rust: sync: Add SpinLockIrq

From: Boqun Feng
Date: Thu Oct 03 2024 - 08:52:57 EST


On Wed, Oct 02, 2024 at 10:53:32PM +0200, Thomas Gleixner wrote:
> On Mon, Sep 16 2024 at 17:28, Lyude Paul wrote:
> > A variant of SpinLock that is expected to be used in noirq contexts, and
> > thus requires that the user provide an kernel::irq::IrqDisabled to prove
> > they are in such a context upon lock acquisition. This is the rust
> > equivalent of spin_lock_irqsave()/spin_lock_irqrestore().
>
> This fundamentally does not work with CONFIG_PREEMPT_RT. See:
>
> https://www.kernel.org/doc/html/latest/locking/locktypes.html
>
> for further information. TLDR:
>
> On RT enabled kernels spin/rw_lock are substituted by sleeping locks. So
> you _cannot_ disable interrupts before taking the lock on RT enabled
> kernels. That will result in a 'might_sleep()' splat.
>

One thing I was missing when I suggested Lyude with the current API is
that local_irq_save() disables interrupts even on RT. I was under the
impression that local_irq_save() will only disable preemption per:

https://lwn.net/Articles/146861/

but seems it's not the case right now: we move the RT vs non-RT games
and hardware interrupt disabling vs preemption/migration disabling to
local_lock_*() I guess?

> There is a reason why the kernel has two distinct spinlock types:
>
> raw_spinlock_t and spinlock_t
>
> raw_spinlock_t is a real spinning lock independent of CONFIG_PREEMPT_RT,
> spinlock_t is mapped to raw_spinlock_t on CONFIG_PREEMPT_RT=n and to a
> rtmutex based implementation for CONFIG_PREEMPT_RT=y.
>
> As a consequence spin_lock_irq() and spin_lock_irqsave() will _NOT_
> disable interrupts on a CONFIG_PREEMPT_RT=y kernel.
>
> The proposed rust abstraction is not suitable for that.
>
> At this phase of rust integration there is no need to wrap
> raw_spinlock_t, so you have two options to solve that:
>
> 1) Map Rust's SpinLockIrq() to spin_lock_irqsave() and
> spin_unlock_irqrestore() which does the right thing
>
> 2) Play all the PREEMPT_RT games in the local irq disable abstraction
>
> #1 is the right thing to do because no driver should rely on actually
> disabling interrupts on the CPU. If there is a driver which does that,
> then it's not compatible with RT and should use a local lock instead.
>
> local locks aside of being RT compatible have the benefit that they give
> scope to the protected region/data, while a plain local_irq_disable()
> does not.
>
> Don't even think about exposing this 'with_irq_disabled' interface
> unless you are trying to move actual core code like the scheduler or low
> level interrupt handling to rust.
>
> Create explicit interrupt safe interfaces which map to the underlying
> locking primitives instead.
>

Then we should have a SpinLockIrq<T> type, and a function:

fn with_locked<U>(&self, cb: impl FnOnce(&mut T) -> U) -> U {
<spin_lock_irqsave()>
let ret = cb(...);
<spin_lock_irqrestore()>
ret
}

FYI, the reason that we cannot have a SpinLockIrq::lock() return a guard
that holds the irq state is because:

https://github.com/Rust-for-Linux/linux/issues/998

namely:

// interrupts are enabled here.
let a = spin_lock_irq1.lock();
let b = spin_lock_irq2.lock();

drop(a); // releasing spin_lock_irq1 and restore the irq state.
// `b` exists with interrupts enabled, which breaks the
// invariants of b.

(technically we can, but that requires a rework of how nested
irq_save()s are handled, that's another can of worms)

Regards,
Boqun

> Thanks,
>
> tglx
>
>