OOPS: v4.19.171 kernfs_sop_show_options - Invalid "root->syscall_ops" ?

From: Philipp Hahn
Date: Tue Jul 13 2021 - 06:47:37 EST

Hello,

Some time ago one of my VMs crashed running Debian-10-Buster using Linux kernel 4.19.171:

[Feb16 06:50] general protection fault: 0000 [#1] SMP PTI
[ +0,000065] CPU: 1 PID: 17450 Comm: nfsmounts Not tainted 4.19.0-14-amd64 #1 Debian 4.19.171-2
[ +0,000052] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.2-1 04/01/2014
[ +0,000067] RIP: 0010:kernfs_sop_show_options+0x2d/0x40
[ +0,000035] Code: 00 00 48 8b 46 30 48 85 c0 74 07 48 8b 80 50 02 00 00 48 8b 50 08 48 85 d2 48 0f 44 d0 48 8b 72 50 48 8b 46 30 48 85 c0 74 0e <48> 8b 40 08 48 85 c0 74 05 e9 05 57 72 00 31 c0 c3 66 90 0f 1f 44
[ +0,000106] RSP: 0018:ffff9e5f01fffe20 EFLAGS: 00010202
[ +0,000032] RAX: 0105028102951075 RBX: ffff8b0c7af66680 RCX: 00000000656f6e2c
[ +0,000041] RDX: ffff8b0c351de440 RSI: ffff8b0c34a74840 RDI: ffff8b0c7af66680
[ +0,000040] RBP: ffff8b0c34286c20 R08: 6d6974616c65722c R09: 656d6974616c6572
[ +0,000041] R10: ffff9e5f01fffddc R11: ffff8b0c75e123bc R12: ffff8b0c7cfec000
[ +0,000040] R13: ffff8b0c77e50e80 R14: ffff8b0c7abb4900 R15: ffff8b0c7af66680
[ +0,000042] FS: 00007ff4bb68e740(0000) GS:ffff8b0c7db00000(0000) knlGS:0000000000000000
[ +0,000045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0,000033] CR2: 00007ff4bb634e10 CR3: 000000007ac0c000 CR4: 00000000000006e0
[ +0,000069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ +0,000041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ +0,000041] Call Trace:
[ +0,000053] show_vfsmnt+0x124/0x170
[ +0,000038] seq_read+0x2e9/0x410
[ +0,000026] vfs_read+0x91/0x140
[ +0,000023] ksys_read+0x57/0xd0
[ +0,000058] do_syscall_64+0x53/0x110
[ +0,000051] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ +0,000053] RIP: 0033:0x7ff4bb77d461
[ +0,000058] Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8 e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54 49 89 d4 55 48
[ +0,000135] RSP: 002b:00007ffc1c076e68 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ +0,000067] RAX: ffffffffffffffda RBX: 000055f6fceab250 RCX: 00007ff4bb77d461
[ +0,000070] RDX: 0000000000002000 RSI: 000055f6fd162e10 RDI: 0000000000000004
[ +0,000096] RBP: 00007ff4bb84b2a0 R08: 0000000000000b40 R09: 000055f6fceab250
[ +0,000089] R10: 000055f6fcda0010 R11: 0000000000000246 R12: 0000000000002000
[ +0,000093] R13: 000055f6fd162e10 R14: 0000000000000d68 R15: 00007ff4bb84a760
[ +0,000100] Modules linked in: ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_state xt_conntrack nft_compat nft_counter nft_chain_route_ipv6 nft_chain_rou
[ +0,009114] raid1 raid0 multipath linear md_mod dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_mod hid_generic usbhid hid sr_mod cdrom sd_mod virtio_gpu ttm drm_kms_helper ahci libahci drm virtio_net net_failover failover virtio_scsi xhci_pci libata xhci_hcd [ +0,009397] ---[ end trace b63534c6f4770b73 ]---

Here's the dis-assembly of said code:

# scripts/decodecode
0: INCOMPLETE
2: 48 8b 46 30 mov 0x30(%rsi),%rax
6: 48 85 c0 test %rax,%rax
9: 74 07 je 0x12
b: 48 8b 80 50 02 00 00 mov 0x250(%rax),%rax
12: 48 8b 50 08 mov 0x8(%rax),%rdx
16: 48 85 d2 test %rdx,%rdx
19: 48 0f 44 d0 cmove %rax,%rdx
1d: 48 8b 72 50 mov 0x50(%rdx),%rsi
21: 48 8b 46 30 mov 0x30(%rsi),%rax
25: 48 85 c0 test %rax,%rax
28: 74 0e je 0x38
2a:* 48 8b 40 08 mov 0x8(%rax),%rax <-- trapping instruction
2e: 48 85 c0 test %rax,%rax
31: 74 05 je 0x38
33: e9 05 57 72 00 jmpq 0x72573d
38: 31 c0 xor %eax,%eax
3a: c3 retq 3b: STRIPPED

# objdump -S fs/kernfs/mount.o --start-address=0x30 --stop-address=0x60
0000000000000030 <kernfs_sop_show_options>:
static int kernfs_sop_show_options(struct seq_file *sf, struct dentry *dentry)
{
00: e8 00 00 00 00 callq 35 <kernfs_sop_show_options+0x5>
};
#define kernfs_info(SB) ((struct kernfs_super_info *)(SB->s_fs_info))
static inline struct kernfs_node *kernfs_dentry_node(struct dentry *dentry)
{
if (d_really_is_negative(dentry))
05: 48 8b 46 30 mov 0x30(%rsi),%rax
09: 48 85 c0 test %rax,%rax
0c: 74 07 je 45 <kernfs_sop_show_options+0x15>
return NULL;
return d_inode(dentry)->i_private;
0e: 48 8b 80 50 02 00 00 mov 0x250(%rax),%rax
if (kn->parent)
15: 48 8b 50 08 mov 0x8(%rax),%rdx
=== RDX: ffff8b0c351de440 -> kn->parent
19: 48 85 d2 test %rdx,%rdx
1c: 48 0f 44 d0 cmove %rax,%rdx
return kn->dir.root;
20: 48 8b 72 50 mov 0x50(%rdx),%rsi
=== RSI: ffff8b0c34a74840 -> root
struct kernfs_root *root = kernfs_root(kernfs_dentry_node(dentry));
struct kernfs_syscall_ops *scops = root->syscall_ops;
24: 48 8b 46 30 mov 0x30(%rsi),%rax
=== RAX: ffffffffffffffda -> root->syscall_ops ?= -ENOSYS ???
if (scops && scops->show_options)
28: 48 85 c0 test %rax,%rax
2b: 74 0e je 6b <kernfs_sop_show_options+0x3b>
2d: 48 8b 40 08 mov 0x8(%rax),%rax
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
31: 48 85 c0 test %rax,%rax
34: 74 05 je 6b <kernfs_sop_show_options+0x3b>
return scops->show_options(sf, root);
36: e9 00 00 00 00 jmpq 6b <kernfs_sop_show_options+0x3b>
return 0;
}
3b: 31 c0 xor %eax,%eax
3d: c3 retq


I don't know how to trigger that again, but here are my findings to far:

- The bug seems to be triggered by a Python script `/usr/lib/univention-directory-policy/nfsmounts` reading `/proc/self/mounts` AKA `/etc/mtab`.

- `syscall_ops` seems to be invalid, which is only ever initialized by fs/kernfs/dir.c:kernfs_create_root()

- This function seems to be used for every cgroup by kernel/cgroup/cgroup.c:cgroup_setup_root()


Does that ring any bells?

I had a look at some changes between v4.19.171...HEAD in that area and have found the following changes, which might be interesting - but I'm no Kernel guru
- 630faf81b3e6 cgroup: don't put ERR_PTR() into fc->root
- 35ac1184244f cgroup: saner refcounting for cgroup_root
- 399504e21a10 fix cgroup_do_mount() handling of failure exits
- 2fd60da46da7 kernfs: fix potential null pointer dereference

Debian and many other LTS users would probably like to get this fixed for stable/linux-4.19.y.
I'm willing to spend more time on this, but currently I'm out of ideads what else to do or where else to look.

Thank you in advance

Philipp Hahn