[PATCH 0/1] ext4: fix NULL pointer dereference in ext4_mb_load_buddy_gfp()

[Aleksandr Panfilenko]

Hello,

We have hit a kernel NULL pointer dereference in ext4_mb_load_buddy_gfp()
twice in one day on a freshly installed server. Both crashes are identical
(same RIP and offset) and were captured by kdump. Patch 1 fixes it; this
cover letter contains the full crash data and the analysis.

Environment
-----------

Kernel:      6.1.192-1.el9.elrepo.x86_64 (vanilla linux-6.1.y built by ELRepo;
             the affected code is identical in current mainline)
Hardware:    ASUSTeK RS500-E9-PS4 (2 sockets, 64 GiB RAM)
Filesystem:  ext4 on LVM on md-raid1, 120 GiB, block size 4096,
             features: has_journal ext_attr resize_inode dir_index filetype
             extent 64bit flex_bg sparse_super large_file huge_file
             dir_nlink extra_isize metadata_csum
Mount:       rw,relatime
Workload:    fresh OS install followed by heavy package installation and
             database restore (lots of writes and page cache churn); both
             crashes happened in background writeback (kworker), not tied
             to any specific userspace task.

Crash 1 (kdump at 19:26:27, 4683s after boot)
---------------------------------------------

[ 4683.607255] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 4683.619530] #PF: supervisor read access in kernel mode
[ 4683.624757] #PF: error_code(0x0000) - not-present page
[ 4683.627387] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 4683.631834] CPU: 51 PID: 137989 Comm: kworker/u161:1 Kdump: loaded Tainted: G            E      6.1.192-1.el9.elrepo.x86_64 #1
[ 4683.652787] Workqueue: writeback wb_workfn (flush-253:1)
[ 4683.658194] RIP: 0010:ext4_mb_load_buddy_gfp+0xc0/0x3a0 [ext4]
[ 4683.664141] Code: 48 63 f0 89 54 24 14 ba 01 00 00 00 48 89 74 24 08 e8 44 49 ec f5 48 8b 74 24 08 48 3d 00 f0 ff ff 49 89 c7 0f 87 32 01 00 00 <48> 8b 00 a8 04 0f 84 1c 01 00 00 49 8b 07 a8 01 0f 85 11 01 00 00
[ 4683.688284] RAX: 0000000000000000 RBX: 00000000000000a2 RCX: 0000000000000022

[ 4683.765356] Call Trace:
[ 4683.767559]  <TASK>
[ 4683.773060]  ext4_mb_regular_allocator+0x1ba/0x600 [ext4]
[ 4683.777952]  ? ext4_find_extent+0x378/0x440 [ext4]
[ 4683.782848]  ext4_mb_new_blocks+0x3a9/0x530 [ext4]
[ 4683.787823]  ext4_ext_map_blocks+0x670/0x8a0 [ext4]
[ 4683.792460]  ext4_map_blocks+0x22b/0x5d0 [ext4]
[ 4683.797441]  mpage_map_one_extent+0x60/0x170 [ext4]
[ 4683.797441]  mpage_map_and_submit_extent+0x76/0x300 [ext4]
[ 4683.803020]  ext4_writepages+0x6f8/0x860 [ext4]
[ 4683.818105]  do_writepages+0xcc/0x1d0
[ 4683.821867]  __writeback_single_inode+0x41/0x270
[ 4683.826575]  writeback_sb_inodes+0x209/0x4c0
[ 4683.830937]  __writeback_inodes_wb+0x4c/0xe0
[ 4683.835300]  wb_writeback+0x1d7/0x2d0
[ 4683.839058]  wb_do_writeback+0x22a/0x2b0
[ 4683.843080]  wb_workfn+0x5d/0x250

Crash 2 (kdump at 20:19:39, 3005s after the following boot) is identical:
same RIP (ext4_mb_load_buddy_gfp+0xc0/0x3a0), same workqueue, same
instruction sequence, RAX=0, different block group (RBX=0xbe vs 0xa2)
and CPU.  (The only difference in the Code dump is the relative call
displacement - see the note under the disassembly below.)

Analysis
--------

The faulting instruction sequence decodes to (bytes from the Crash 1
dump above; in the Crash 2 dump the call displacement differs - the
call to __filemap_get_folio is stored as an R_X86_64_PLT32 relocation
against the vmlinux symbol in the module and is resolved at module
load time, so its encoded value depends on the kernel and module load
addresses of a given boot, while the instruction sequence is identical):

    e8 44 49 ec f5        call  __filemap_get_folio      ; fast path
    48 8b 74 24 08        mov   rsi, [rsp+0x8]
    48 3d 00 f0 ff ff     cmp   rax, 0xfffff000
    49 89 c7              mov   r15, rax
    0f 87 32 01 00 00     ja    err                      ; IS_ERR() only
    <48 8b 00>            mov   rax, [rax]               ; RAX == NULL -> #PF
    a8 04                 test  al, 0x4                  ; PG_uptodate
    0f 84 ...             jz    slow_path
    49 8b 07              mov   rax, [r15]
    a8 01                 test  al, 0x1                  ; PG_locked
    0f 85 ...             jnz   slow_path

This is the fast path in ext4_mb_load_buddy_gfp():

    folio = __filemap_get_folio(inode->i_mapping, pnum, FGP_ACCESSED, 0);
    if (IS_ERR(folio) || !folio_test_uptodate(folio) || folio_test_locked(folio)) {
            ...
    }

__filemap_get_folio() without FGP_CREAT legitimately returns NULL when
the folio is not in the page cache ("if (!folio) goto no_page;" in
mm/filemap.c).  The fast-path condition only checks IS_ERR(folio) before
dereferencing the folio, so a NULL return results in a NULL deref.

The ext4_mb_init_group() call at the top of the function is skipped for
groups that are already initialized (EXT4_MB_GRP_NEED_INIT not set), so
nothing guarantees the buddy cache folio is still in the page cache at
this point: the folios of the s_buddy_cache inode are clean and can be
reclaimed under memory pressure.  The folio conversion series dropped
the explicit NULL check that the old find_get_page_flags()-based code
had ("page == NULL ||" was the first condition) in both lookups: the
bitmap lookup was converted by commit 99b150d84e49 ("ext4: convert
bd_bitmap_page to bd_bitmap_folio") and the buddy lookup by commit
5eea586b47f0 ("ext4: convert bd_buddy_page to bd_buddy_folio"), which
is where the regression comes from.

The error-path fix from commit c6a6c9694aad ("ext4: fix error pointer
dereference in ext4_mb_load_buddy_gfp()") does not cover this case.

The patch switches the fast-path conditions (and the two folio_put()
guards) to IS_ERR_OR_NULL(), so a NULL folio falls back to the FGP_CREAT
path which re-creates it.

Deterministic reproducer
------------------------

Once the eviction mechanism was understood, the crash turned out to be
trivially reproducible on a test VM (same 6.1.192 kernel, stock ext4
module):

    # fallocate -l 1G /root/t.img
    # mkfs.ext4 -F /root/t.img
    # mount -o loop /root/t.img /mnt/t
    # dd if=/dev/urandom of=/mnt/t/f1 bs=1M count=200
    # sync
    # echo 3 > /proc/sys/vm/drop_caches
    # dd if=/dev/urandom of=/mnt/t/f2 bs=1M count=200

The drop_caches evicts the clean s_buddy_cache folios, and the next
block allocation in the writeback worker hits the NULL fast path within
seconds:

    BUG: kernel NULL pointer dereference, address: 0000000000000000
    CPU: 0 PID: 446 Comm: kworker/u4:4 Kdump: loaded
    Workqueue: writeback wb_workfn (flush-7:0)
    RIP: 0010:ext4_mb_load_buddy_gfp+0xc0/0x3a0 [ext4]
    Code: ... 48 3d 00 f0 ff ff 0f 87 ... <48> 8b 00 a8 04 ...
    RAX: 0000000000000000

Same RIP and the same instruction sequence as in both production crashes.

Verification
------------

I rebuilt the ext4 module against the same kernel and config with only
this fix applied (verified via srcversion that the patched module was
the one loaded) and re-ran the reproducer above for three full
write/drop_caches/write cycles: no oops, no ext4 warnings in dmesg.
The unpatched module panics on the first cycle.

Additional data
---------------

vmcores of both production crashes (captured by kdump) and of the
reproduced crash on the test VM are available on request.  No ext4
errors or warnings preceded the production oopses; the log simply ends
a few minutes before the kdump timestamp.

Aleksandr Panfilenko (1):
  ext4: fix NULL pointer dereference in ext4_mb_load_buddy_gfp()

 fs/ext4/mballoc.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

-- 
2.34.1