Re: [PATCH v6 3/3] mm: implement page refcount locking via dedicated bit

[Kiryl Shutsemau]

On Sat, Sep 12, 2026 at 10:50:10PM +0300, Ilya Gladyshev wrote:
> The current page refcount implementation uses a single counter value
> (zero) as dead. So, to prevent incrementing a dead refcount in
> folio_try_get(), it fundamentally requires a CAS loop.
> 
> This CAS loop can act as a serialization point and can become a
> significant bottleneck during high-frequency file read operations
> [1][2].
> 
> This patch reallocates the refcount value range:
> 
> (1) refcount < 0 means dead refcount (uninit / frozen)
> (2) refcount = 0 allowed only as a temporary state (see below)
> (3) refcount > 0 is a regular reference count
> 
> In other words, refcount is now split into "dead bit" and a 31-bit
> counter.
> 
> Refcount decrement now works as follows:
> 1. Counter decrement
> 2. If it is now zero, try to put it deep inside the dead zone
>    (CAS to INT_MIN). Or you can view it as "set up frozen bit and reset
>    counter".
> 3. This CAS can fail only if someone grabbed a reference in-between --
>    that's okay, this page is their problem now.
> 
> The size of the dead zone allows performing an optimistic increment
> inside page_ref_add_unless_frozen(), replacing the previous read + CAS
> loop with a single RMW operation. This reduces cache line bouncing and
> improves scalability, especially in NUMA scenarios.
> 
> [1]: https://lore.kernel.org/all/20251017141536.577466-1-kirill@xxxxxxxxxxxxx/
> [2]: https://lore.kernel.org/all/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@xxxxxxxxxxxxxx/
> 
> Reviewed-by: Artem Kuzin <artem.kuzin@xxxxxxxxxx>
> Co-developed-by: Ivan Gorbunov <ivgorbunov@xxxxxx>
> Signed-off-by: Ivan Gorbunov <ivgorbunov@xxxxxx>
> Signed-off-by: Ilya Gladyshev <ilya.gladyshev@xxxxxxxxx>
> Acked-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxx>
> ---
>  include/linux/page-flags.h | 13 +++++++++++++
>  include/linux/page_ref.h   | 30 +++++++++++++++++++++++++-----
>  2 files changed, 38 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 7a863572adce..b19721e0e7ca 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -196,6 +196,19 @@ enum pageflags {
>  
>  #define PAGEFLAGS_MASK		((1UL << NR_PAGEFLAGS) - 1)
>  
> +/* Most significant bit in page refcount */
> +#define PAGEREF_FROZEN_BIT BIT(31)

I am not sure about the naming here. I would expect _BIT to be 31.
Maybe we should name it _MASK?

#define PAGEREF_FROZEN_BIT	31
#define PAGEREF_FROZEN_MASK	BIT(PAGEREF_FROZEN_BIT)

> +
> +/* Page reference counter can be in 3 logical states,
> + * which are described below with their value representation
> + *        state              |         value
> + * (1)  safe with  owners    |   1...INT_MAX
> + * (2)  safe with no owners  |         0
> + * (3)  frozen               |  INT_MIN....-1
> + *
> + * State (2) can only temporarily occur inside dec_and_test.
> + */
> +

Do we what to enforce your comment about (2)? VM_WARN() in
page_ref_dec(), page_ref_sub() and _return() variants?

BTW, I wounder if all these VM_WARNs makes DEBUG_VM=y kernel too slow?
Do we need a separate config option to debug refcounts?

>  #ifndef __GENERATING_BOUNDS_H
>  
>  /*
> diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h
> index 82ff3a99297a..cc7a9d7db504 100644
> --- a/include/linux/page_ref.h
> +++ b/include/linux/page_ref.h
> @@ -64,7 +64,7 @@ static inline void __page_ref_unfreeze(struct page *page, int v)
>  
>  static inline bool __page_count_is_frozen(int count)
>  {
> -	return count == 0;
> +	return count & PAGEREF_FROZEN_BIT;
>  }
>  
>  static inline bool page_is_frozen(const struct page *page)
> @@ -79,7 +79,12 @@ static inline bool folio_is_frozen(const struct folio *folio)
>  
>  static inline int page_ref_count(const struct page *page)
>  {
> -	return atomic_read(&page->_refcount);
> +	int val = atomic_read(&page->_refcount);
> +
> +	if (unlikely(val & PAGEREF_FROZEN_BIT))
> +		return 0;
> +
> +	return val;

I was worried about the branch in the hot path and wanted to propose a
branchless hack -- val & ~(val >> 31) -- but compilers seem to be good
enough to generate CMOV. The hack produces one more instruction.

>  }
>  
>  /**
> @@ -150,7 +155,7 @@ static inline void init_page_count(struct page *page)
>  
>  static inline void set_page_count_frozen(struct page *page)
>  {
> -	set_page_count(page, 0);
> +	set_page_count(page, PAGEREF_FROZEN_BIT);

Are you sure we covered the cases that do frozen -> 1 transition with
page_ref_inc()?

Patch 2 switches __init_zone_device_page() to set_page_count_frozen().
Who unfreezes them?

Looking dax_fault_iter(), it seems to be done by folio_ref_inc() there
for FS_DAX case. Switching to PAGEREF_FROZEN_BIT would break it, no?

It seems some more ground work needed.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov