[PATCH v10 00/33] Memory folios

From: Matthew Wilcox (Oracle)
Date: Tue May 11 2021 - 17:48:16 EST


Managing memory in 4KiB pages is a serious overhead. Many benchmarks
benefit from a larger "page size". As an example, an earlier iteration
of this idea which used compound pages (and wasn't particularly tuned)
got a 7% performance boost when compiling the kernel.

Using compound pages or THPs exposes a weakness of our type system.
Functions are often unprepared for compound pages to be passed to them,
and may only act on PAGE_SIZE chunks. Even functions which are aware of
compound pages may expect a head page, and do the wrong thing if passed
a tail page.

We also waste a lot of instructions ensuring that we're not looking at
a tail page. Almost every call to PageFoo() contains one or more hidden
calls to compound_head(). This also happens for get_page(), put_page()
and many more functions. There does not appear to be a way to tell gcc
that it can cache the result of compound_head(), nor is there a way to
tell it that compound_head() is idempotent.

This patch series uses a new type, the struct folio, to manage memory.
It provides some basic infrastructure that's worthwhile in its own right,
shrinking the kernel by about 5kB of text.

Since v9:
- Rebase onto mmotm 2021-05-10-21-46
- Add folio_memcg() definition for !MEMCG (intel lkp)
- Change folio->private from an unsigned long to a void *
- Use folio_page() to implement folio_file_page()
- Add folio_try_get() and folio_try_get_rcu()
- Trim back down to just the first few patches, which are better-reviewed.
v9: https://lore.kernel.org/linux-mm/20210505150628.111735-1-willy@xxxxxxxxxxxxx/
v8: https://lore.kernel.org/linux-mm/20210430180740.2707166-1-willy@xxxxxxxxxxxxx/

Matthew Wilcox (Oracle) (33):
mm: Introduce struct folio
mm: Add folio_pgdat and folio_zone
mm/vmstat: Add functions to account folio statistics
mm/debug: Add VM_BUG_ON_FOLIO and VM_WARN_ON_ONCE_FOLIO
mm: Add folio reference count functions
mm: Add folio_put
mm: Add folio_get
mm: Add folio_try_get_rcu
mm: Add folio flag manipulation functions
mm: Add folio_young and folio_idle
mm: Handle per-folio private data
mm/filemap: Add folio_index, folio_file_page and folio_contains
mm/filemap: Add folio_next_index
mm/filemap: Add folio_offset and folio_file_offset
mm/util: Add folio_mapping and folio_file_mapping
mm: Add folio_mapcount
mm/memcg: Add folio wrappers for various functions
mm/filemap: Add folio_unlock
mm/filemap: Add folio_lock
mm/filemap: Add folio_lock_killable
mm/filemap: Add __folio_lock_async
mm/filemap: Add __folio_lock_or_retry
mm/filemap: Add folio_wait_locked
mm/swap: Add folio_rotate_reclaimable
mm/filemap: Add folio_end_writeback
mm/writeback: Add folio_wait_writeback
mm/writeback: Add folio_wait_stable
mm/filemap: Add folio_wait_bit
mm/filemap: Add folio_wake_bit
mm/filemap: Convert page wait queues to be folios
mm/filemap: Add folio private_2 functions
fs/netfs: Add folio fscache functions
mm: Add folio_mapped

Documentation/core-api/mm-api.rst | 4 +
Documentation/filesystems/netfs_library.rst | 2 +
fs/afs/write.c | 9 +-
fs/cachefiles/rdwr.c | 16 +-
fs/io_uring.c | 2 +-
include/linux/memcontrol.h | 63 ++++
include/linux/mm.h | 174 ++++++++--
include/linux/mm_types.h | 71 ++++
include/linux/mmdebug.h | 20 ++
include/linux/netfs.h | 77 +++--
include/linux/page-flags.h | 230 ++++++++++---
include/linux/page_idle.h | 99 +++---
include/linux/page_ref.h | 158 ++++++++-
include/linux/pagemap.h | 358 ++++++++++++--------
include/linux/swap.h | 7 +-
include/linux/vmstat.h | 107 ++++++
mm/Makefile | 2 +-
mm/filemap.c | 315 ++++++++---------
mm/folio-compat.c | 43 +++
mm/internal.h | 1 +
mm/memory.c | 8 +-
mm/page-writeback.c | 72 ++--
mm/page_io.c | 4 +-
mm/swap.c | 18 +-
mm/swapfile.c | 8 +-
mm/util.c | 59 ++--
26 files changed, 1374 insertions(+), 553 deletions(-)
create mode 100644 mm/folio-compat.c

--
2.30.2