[RFC PATCH v4 0/2] binder: split alloc->mutex to improve performance

[Bo Zhang]

Hi,

This is v4 of the binder alloc lock optimization. Thanks to the Sashiko
automated review for the feedback on v3, and to Alice Ryhl for the
earlier reviews.

The series splits the binder allocator lock into two, with distinct
roles:

  - alloc->lock (spinlock): owns the non-sleeping metadata - pages[],
    the LRU list, the rb-trees and free_async_space. This is the hot
    path hit on every binder transaction.

  - install_mutex: only serializes the sleeping PTE operations
    (vm_insert_page vs the shrinker's zap_vma_range) for a given alloc.

pages[] and the LRU are always updated together under alloc->lock, so
the buffer allocation path always observes a consistent state. The
shrinker takes install_mutex with mutex_trylock() (skipping on failure),
which avoids a self-deadlock when install-side reclaim re-enters the
shrinker, and keeps the shrinker out of any blocking cycle with
mmap_lock.

Performance (binderThroughputTest, Qualcomm SM8850, 2 workers, 10 runs)
under concurrent drop_caches:

                    mutex (baseline)    spinlock + install_mutex
  throughput:    27k-59k iter/s       84k-89k iter/s
  average:       0.031-0.068ms        0.021-0.022ms
  P99:           0.088-0.148ms        0.046-0.056ms

Changes since v3:
- Fix an AA self-deadlock: the shrinker now uses mutex_trylock() on
  install_mutex, so install-side vm_insert_page() recursing into direct
  reclaim and re-entering the shrinker on the same thread no longer
  deadlocks (Sashiko).
- Fix a use-after-free: pages[index]=NULL is done under alloc->lock
  (together with the LRU isolate) instead of under install_mutex, so
  binder_lru_freelist_del() can no longer read a pointer the shrinker is
  about to free (Sashiko).
- Fix an RT-task livelock: since pages[] and the LRU are now consistent
  under alloc->lock, list_lru_del() never fails and the -EAGAIN retry
  path is removed entirely (Sashiko).
- The mutex_trylock() also removes the ABBA concern with mmap_lock, so
  the install side's mmap_lock fallback returns to a plain blocking
  acquire (no more -EAGAIN/retry).

Changes since v2:
- Fixed the ABBA/-EBUSY/next-buffer issues raised on v2 (superseded by
  the simpler v4 locking above).

Changes since v1:
- Dropped the spinlock-only approach that raced install against shrinker
  zap; added install_mutex to serialize them (Alice).

v3: https://lore.kernel.org/all/20260904110448.23086-1-zhangbo0325@xxxxxxxxx/
v2: https://lore.kernel.org/all/20260831123545.3655557-1-zhangbo56@xxxxxxxxxx/
v1: https://lore.kernel.org/all/20260805152752.1924434-1-zhangbo56@xxxxxxxxxx/

Bo Zhang (2):
  binder: switch alloc->mutex to spinlock for buffer metadata
  binder: add install_mutex to serialize page install and shrinker zap

 drivers/android/binder_alloc.c | 95 +++++++++++++++++++---------------
 drivers/android/binder_alloc.h | 11 ++--
 2 files changed, 61 insertions(+), 45 deletions(-)

-- 
2.34.1