Re: [GIT pull] locking/urgent for v5.15-rc2

[Linus Torvalds]

On Sun, Sep 19, 2021 at 11:28 AM Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:
>
>
> -       atomic_add(READER_BIAS - bias, &rwb->readers);
> +       /*
> +        * _release() is needed in case that reader is in fast path, pairing
> +        * with atomic_try_cmpxchg() in rwbase_read_trylock(), provides RELEASE
> +        */
> +       (void)atomic_add_return_release(READER_BIAS - bias, &rwb->readers);

Ugh. This really needs fixing.

atomic_add() is already much more than release-ordered on x86, and
atomic_add_return_release() is much more expensive on some uarchs.

I think it should be easy to add a atomic_add_release() function, and
it might be as simple as the attached patch, allowing architectures to
add their own arch_atomic_add_release() as needed.

I've pulled this, but please don't do things like the above hack.

            Linus
 arch/x86/include/asm/atomic.h               | 2 ++
 include/linux/atomic/atomic-arch-fallback.h | 4 ++++
 include/linux/atomic/atomic-instrumented.h  | 7 +++++++
 3 files changed, 13 insertions(+)

diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 5e754e895767..6c5814177c73 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -55,6 +55,8 @@ static __always_inline void arch_atomic_add(int i, atomic_t *v)
 		     : "ir" (i) : "memory");
 }
 
+#define arch_atomic_add_release arch_atomic_add
+
 /**
  * arch_atomic_sub - subtract integer from atomic variable
  * @i: integer value to subtract
diff --git a/include/linux/atomic/atomic-arch-fallback.h b/include/linux/atomic/atomic-arch-fallback.h
index a3dba31df01e..ae437d961bd1 100644
--- a/include/linux/atomic/atomic-arch-fallback.h
+++ b/include/linux/atomic/atomic-arch-fallback.h
@@ -165,6 +165,10 @@ arch_atomic_set_release(atomic_t *v, int i)
 #define arch_atomic_set_release arch_atomic_set_release
 #endif
 
+#ifndef arch_atomic_add_release
+#define arch_atomic_add_release (void)arch_atomic_add_return_release
+#endif
+
 #ifndef arch_atomic_add_return_relaxed
 #define arch_atomic_add_return_acquire arch_atomic_add_return
 #define arch_atomic_add_return_release arch_atomic_add_return
diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h
index a0f654370da3..485b89804b9d 100644
--- a/include/linux/atomic/atomic-instrumented.h
+++ b/include/linux/atomic/atomic-instrumented.h
@@ -56,6 +56,13 @@ atomic_add(int i, atomic_t *v)
 	arch_atomic_add(i, v);
 }
 
+static __always_inline void
+atomic_add_release(int i, atomic_t *v)
+{
+	instrument_atomic_read_write(v, sizeof(*v));
+	arch_atomic_add_release(i, v);
+}
+
 static __always_inline int
 atomic_add_return(int i, atomic_t *v)
 {