Re: kernel BUG at mm/huge_memory.c:2613!

From: Roman Gushchin
Date: Fri Jun 19 2020 - 14:56:05 EST

On Thu, Jun 18, 2020 at 10:40:26PM -0400, Andrea Arcangeli wrote:
> Hello,
>
> On Thu, Jun 18, 2020 at 06:14:49PM -0700, Roman Gushchin wrote:
> > I agree. The whole
> >
> > page = alloc_pages_node(nid, alloc_flags, order);
> > if (!page)
> > continue;
> > if (!order)
> > break;
> > if (!PageCompound(page)) {
> > split_page(page, order);
> > break;
> > } else if (!split_huge_page(page)) {
> > break;
> > }
> >
> > looks very suspicious to me.
> > My wild guess is that gfp flags changed somewhere above, so we hit
> > the branch which was never hit before.
>
> Right to be suspicious about the above: split_huge_page on a regular
> page allocated by a driver was never meant to work.
>
> The PageLocked BUG_ON is just a symptom of a bigger issue, basically
> split_huge_page it may survive, but it'll stay compound and in turn it
> must be freed as compound.
>
> The respective free method doesn't even contemplate freeing compound
> pages, the only way the free method can survive, is by removing
> __GFP_COMP forcefully in the allocation that was perhaps set here
> (there are that many __GFP_COMP in that directory):
>
> static void snd_malloc_dev_pages(struct snd_dma_buffer *dmab, size_t size)
> {
> gfp_t gfp_flags;
>
> gfp_flags = GFP_KERNEL
> | __GFP_COMP /* compound page lets parts be mapped */
>
> And I'm not sure what the comment means here, compound or non compound
> doesn't make a difference when you map it, it's not a THP, the
> mappings must be handled manually so nothing should check PG_compound
> anyway in the mapping code.
>
> Something like this may improve things, it's an untested quick hack,
> but this assumes it's always a bug to setup a compound page for these
> DMA allocations and given the API it's probably a correct
> assumption.. Compound is slower, unless you need it, you can avoid it
> and then split_page will give contiguous memory page granular. Ideally
> the code shouldn't call split_page at all and it should free it all at
> once by keeping track of the order and by returning the order to the
> caller, something the API can't do right now as it returns a plain
> array that can only represent individual small pages.
>
> Once this is resolved, you may want to check your config, iommu passthrough
> sounds more optimal for a soundcard.

It's based on the default Fedora 32 config + all defaults for the 5.6..5.8-rc1
difference. But thanks for the advice!

>
> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
> index f68a62c3c32b..3dfbc010fa83 100644
> --- a/drivers/iommu/dma-iommu.c
> +++ b/drivers/iommu/dma-iommu.c
> @@ -499,6 +499,10 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,
>
> /* IOMMU can map any pages, so himem can also be used here */
> gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
> + if (unlikely(gfp & __GFP_COMP)) {
> + WARN();
> + gfp &= ~__GFP_COMP;
> + }
>
> while (count) {
> struct page *page = NULL;
> @@ -522,13 +526,8 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,
> continue;
> if (!order)
> break;
> - if (!PageCompound(page)) {
> - split_page(page, order);
> - break;
> - } else if (!split_huge_page(page)) {
> - break;
> - }
> - __free_pages(page, order);
> + split_page(page, order);
> + break;
> }
> if (!page) {
> __iommu_dma_free_pages(pages, i);
> diff --git a/sound/core/memalloc.c b/sound/core/memalloc.c
> index 6850d13aa98c..378f5a36ec5f 100644
> --- a/sound/core/memalloc.c
> +++ b/sound/core/memalloc.c
> @@ -28,7 +28,6 @@ static void snd_malloc_dev_pages(struct snd_dma_buffer *dmab, size_t size)
> gfp_t gfp_flags;
>
> gfp_flags = GFP_KERNEL
> - | __GFP_COMP /* compound page lets parts be mapped */
> | __GFP_NORETRY /* don't trigger OOM-killer */
> | __GFP_NOWARN; /* no stack trace print - this call is non-critical */
> dmab->area = dma_alloc_coherent(dmab->dev.dev, size, &dmab->addr,
>
>

The patch looks very good to me, but unfortunately it seems to reveal
the next layer of problems:

[ 23.864671] page:ffffcd0a8fc3a3c0 refcount:0 mapcount:0
mapping:0000000000000000 index:0x0
[ 23.864674] flags: 0x17ffffc0000800(arch_1)
[ 23.864678] raw: 0017ffffc0000800 0000000000000000 ffffcd0a8fc3a388
0000000000000000
[ 23.864680] raw: 0000000000000000 0000000000000000 00000000ffffffff
0000000000000000
[ 23.864705] page dumped because: VM_BUG_ON_PAGE(((unsigned int)
page_ref_count(page) + 127u <= 127u))
[ 23.864771] ------------[ cut here ]------------
[ 23.864771] kernel BUG at include/linux/mm.h:1137!
[ 23.864783] invalid opcode: 0000 [#1] SMP NOPTI
[ 23.864787] CPU: 14 PID: 1876 Comm: alsa-sink-HDMI Not tainted
5.8.0-rc1+ #697
[ 23.864788] Hardware name: Gigabyte Technology Co., Ltd.
AB350-Gaming/AB350-Gaming-CF, BIOS F25 01/16/2019
[ 23.864799] RIP: 0010:snd_pcm_mmap_data_fault+0xe3/0x110 [snd_pcm]
[ 23.864802] Code: 00 00 80 48 2b 05 f5 01 7a e1 48 01 f8 48 c1 e8
0c 48 c1 e0 06 48 03 05 d3 01 7a e1 eb 83 48 c7 c6 88 a9 cb c0 e8 bd
4f 5e e0 <0f> 0b 4c 89 c7 e8 23 53 60 e0 e9 68 ff ff ff e8 b9 af 00 00
e9 5e
[ 23.864804] RSP: 0000:ffffa77b82477df0 EFLAGS: 00010286
[ 23.864808] RAX: 0000000000000000 RBX: ffffa77b82477e30 RCX: 0000000000000000
[ 23.864812] RDX: ffff90f78efa7060 RSI: 0000000000000000 RDI: ffff90f78ef98d00
[ 23.864813] RBP: ffff90f76d164bb8 R08: 0000000000000548 R09: 0000000000000003
[ 23.864815] R10: 0000000000000000 R11: 0000000000000001 R12: ffffa77b82477e30
[ 23.864817] R13: 00000000000007b8 R14: ffff90f76d164bb8 R15: ffff90f78d1eb300
[ 23.864820] FS: 00007f971efbf700(0000) GS:ffff90f78ef80000(0000)
knlGS:0000000000000000
[ 23.864821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 23.864823] CR2: 00007f971efcffa0 CR3: 00000003ffaa4000 CR4: 00000000003406e0
[ 23.864825] Call Trace:
[ 23.864834] __do_fault+0x36/0x100
[ 23.864838] handle_mm_fault+0x11e3/0x1970
[ 23.864846] do_user_addr_fault+0x1f9/0x490
[ 23.864853] exc_page_fault+0x81/0x1a0
[ 23.864858] ? asm_exc_page_fault+0x8/0x30
[ 23.864861] asm_exc_page_fault+0x1e/0x30
[ 23.864864] RIP: 0033:0x7f9730baba08
[ 23.864866] Code: Bad RIP value.
[ 23.864867] RSP: 002b:00007f971efbc538 EFLAGS: 00010202
[ 23.864869] RAX: 00007f971efc0000 RBX: 0000000000000001 RCX: 00007f971efc0080
[ 23.864871] RDX: 000000000000ffc0 RSI: 0000000000000000 RDI: 00007f971efc0000
[ 23.864872] RBP: 00007f971efbec90 R08: 0000000000000004 R09: 000000000000ffc0
[ 23.864874] R10: 0000000000020000 R11: 0000000000000000 R12: 00007f971efbec90
[ 23.864876] R13: 00005630003cc1b0 R14: 00007f971efbc640 R15: 00005630003cc140
[ 23.864880] Modules linked in: xt_CHECKSUM xt_MASQUERADE
xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp tun bridge stp
llc ccm nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat
nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle
ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw
iptable_security ip_set nfnetlink ebtable_filter ebtables
ip6table_filter ip6_tables iptable_filter cmac bnep sunrpc vfat fat
iwlmvm edac_mce_amd mac80211 kvm_amd snd_hda_codec_realtek libarc4
snd_hda_codec_generic snd_hda_codec_hdmi ledtrig_audio iwlwifi kvm
snd_hda_intel snd_intel_dspcfg irqbypass snd_usb_audio cfg80211
snd_hda_codec wmi_bmof snd_usbmidi_lib snd_rawmidi mc snd_hda_core
pcspkr snd_seq snd_hwdep snd_seq_device k10temp btusb snd_pcm
sp5100_tco btrtl i2c_piix4 btbcm btintel
[ 23.864918] bluetooth snd_timer snd soundcore ecdh_generic rfkill
ecc gpio_amdpt gpio_generic acpi_cpufreq ip_tables amdgpu iommu_v2
gpu_sched i2c_algo_bit ttm drm_kms_helper crct10dif_pclmul
crc32_pclmul drm ghash_clmulni_intel ccp r8169 nvme nvme_core wmi
pinctrl_amd btrfs blake2b_generic libcrc32c crc32c_intel xor raid6_pq
fuse
[ 23.864940] ---[ end trace a6b3dead26df473b ]---
[ 23.864947] RIP: 0010:snd_pcm_mmap_data_fault+0xe3/0x110 [snd_pcm]
[ 23.864951] Code: 00 00 80 48 2b 05 f5 01 7a e1 48 01 f8 48 c1 e8
0c 48 c1 e0 06 48 03 05 d3 01 7a e1 eb 83 48 c7 c6 88 a9 cb c0 e8 bd
4f 5e e0 <0f> 0b 4c 89 c7 e8 23 53 60 e0 e9 68 ff ff ff e8 b9 af 00 00
e9 5e
[ 23.864954] RSP: 0000:ffffa77b82477df0 EFLAGS: 00010286
[ 23.864956] RAX: 0000000000000000 RBX: ffffa77b82477e30 RCX: 0000000000000000
[ 23.864958] RDX: ffff90f78efa7060 RSI: 0000000000000000 RDI: ffff90f78ef98d00
[ 23.864960] RBP: ffff90f76d164bb8 R08: 0000000000000548 R09: 0000000000000003
[ 23.864961] R10: 0000000000000000 R11: 0000000000000001 R12: ffffa77b82477e30
[ 23.864963] R13: 00000000000007b8 R14: ffff90f76d164bb8 R15: ffff90f78d1eb300
[ 23.864965] FS: 00007f971efbf700(0000) GS:ffff90f78ef80000(0000)
knlGS:0000000000000000
[ 23.864967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 23.864968] CR2: 00007f971efcffa0 CR3: 00000003ffaa4000 CR4: 00000000003406e0
[ 39.037054] rfkill: input handler enabled
[ 40.287524] BUG: unable to handle page fault for address: ffffa77b833df000
[ 40.287529] #PF: supervisor write access in kernel mode
[ 40.287531] #PF: error_code(0x000b) - reserved bit violation
[ 40.287532] PGD 40d14e067 P4D 40d14e067 PUD 40d14f067 PMD 3ec54d067
PTE 80001688033d9163
[ 40.287538] Oops: 000b [#2] SMP NOPTI
[ 40.287542] CPU: 9 PID: 1986 Comm: pulseaudio Tainted: G D
5.8.0-rc1+ #697
[ 40.287544] Hardware name: Gigabyte Technology Co., Ltd.
AB350-Gaming/AB350-Gaming-CF, BIOS F25 01/16/2019
[ 40.287550] RIP: 0010:__memset+0x24/0x30
[ 40.287553] Code: cc cc cc cc cc cc 0f 1f 44 00 00 49 89 f9 48 89
d1 83 e2 07 48 c1 e9 03 40 0f b6 f6 48 b8 01 01 01 01 01 01 01 01 48
0f af c6 <f3> 48 ab 89 d1 f3 aa 4c 89 c8 c3 90 49 89 f9 40 88 f0 48 89
d1 f3
[ 40.287556] RSP: 0018:ffffa77b827a7e08 EFLAGS: 00010216
[ 40.287558] RAX: 0000000000000000 RBX: ffff90f77dced800 RCX: 00000000000008a0
[ 40.287560] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa77b833df000
[ 40.287561] RBP: ffff90f7898c7000 R08: ffff90f78c507768 R09: ffffa77b833df000
[ 40.287563] R10: ffffa77b833df000 R11: ffff90f7839f2d40 R12: 0000000000000000
[ 40.287564] R13: ffff90f76a802e00 R14: ffffffffc0cb8880 R15: ffff90f770f4e700
[ 40.287567] FS: 00007f3d8e8df880(0000) GS:ffff90f78ee40000(0000)
knlGS:0000000000000000
[ 40.287569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 40.287570] CR2: ffffa77b833df000 CR3: 00000003fa556000 CR4: 00000000003406e0
[ 40.287572] Call Trace:
[ 40.287584] snd_pcm_hw_params+0x3fd/0x490 [snd_pcm]
[ 40.287593] snd_pcm_common_ioctl+0x1c5/0x1110 [snd_pcm]
[ 40.287601] ? snd_pcm_info_user+0x64/0x80 [snd_pcm]
[ 40.287608] snd_pcm_ioctl+0x23/0x30 [snd_pcm]
[ 40.287613] ksys_ioctl+0x82/0xc0
[ 40.287617] __x64_sys_ioctl+0x16/0x20
[ 40.287622] do_syscall_64+0x4d/0x90
[ 40.287627] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 40.287630] RIP: 0033:0x7f3d8f21047b
[ 40.287631] Code: Bad RIP value.
[ 40.287632] RSP: 002b:00007ffe22938b08 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 40.287635] RAX: ffffffffffffffda RBX: 00007ffe22938d50 RCX: 00007f3d8f21047b
[ 40.287636] RDX: 00007ffe22938d50 RSI: 00000000c2604111 RDI: 000000000000001d
[ 40.287637] RBP: 000055dc9480fdf0 R08: 0000000000000000 R09: 0000000000000000
[ 40.287639] R10: 0000000000000004 R11: 0000000000000246 R12: 000055dc9480fd70
[ 40.287640] R13: 00007ffe22938b44 R14: 0000000000000000 R15: 00007ffe22938d50
[ 40.287644] Modules linked in: xt_CHECKSUM xt_MASQUERADE
xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp tun bridge stp
llc ccm nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat
nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle
ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw
iptable_security ip_set nfnetlink ebtable_filter ebtables
ip6table_filter ip6_tables iptable_filter cmac bnep sunrpc vfat fat
iwlmvm edac_mce_amd mac80211 kvm_amd snd_hda_codec_realtek libarc4
snd_hda_codec_generic snd_hda_codec_hdmi ledtrig_audio iwlwifi kvm
snd_hda_intel snd_intel_dspcfg irqbypass snd_usb_audio cfg80211
snd_hda_codec wmi_bmof snd_usbmidi_lib snd_rawmidi mc snd_hda_core
pcspkr snd_seq snd_hwdep snd_seq_device k10temp btusb snd_pcm
sp5100_tco btrtl i2c_piix4 btbcm btintel
[ 40.287688] bluetooth snd_timer snd soundcore ecdh_generic rfkill
ecc gpio_amdpt gpio_generic acpi_cpufreq ip_tables amdgpu iommu_v2
gpu_sched i2c_algo_bit ttm drm_kms_helper crct10dif_pclmul
crc32_pclmul drm ghash_clmulni_intel ccp r8169 nvme nvme_core wmi
pinctrl_amd btrfs blake2b_generic libcrc32c crc32c_intel xor raid6_pq
fuse
[ 40.287708] CR2: ffffa77b833df000
[ 40.287711] ---[ end trace a6b3dead26df473c ]---
[ 40.287717] RIP: 0010:snd_pcm_mmap_data_fault+0xe3/0x110 [snd_pcm]
[ 40.287720] Code: 00 00 80 48 2b 05 f5 01 7a e1 48 01 f8 48 c1 e8
0c 48 c1 e0 06 48 03 05 d3 01 7a e1 eb 83 48 c7 c6 88 a9 cb c0 e8 bd
4f 5e e0 <0f> 0b 4c 89 c7 e8 23 53 60 e0 e9 68 ff ff ff e8 b9 af 00 00
e9 5e
[ 40.287722] RSP: 0000:ffffa77b82477df0 EFLAGS: 00010286
[ 40.287724] RAX: 0000000000000000 RBX: ffffa77b82477e30 RCX: 0000000000000000
[ 40.287725] RDX: ffff90f78efa7060 RSI: 0000000000000000 RDI: ffff90f78ef98d00
[ 40.287726] RBP: ffff90f76d164bb8 R08: 0000000000000548 R09: 0000000000000003
[ 40.287728] R10: 0000000000000000 R11: 0000000000000001 R12: ffffa77b82477e30
[ 40.287729] R13: 00000000000007b8 R14: ffff90f76d164bb8 R15: ffff90f78d1eb300
[ 40.287731] FS: 00007f3d8e8df880(0000) GS:ffff90f78ee40000(0000)
knlGS:0000000000000000
[ 40.287733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 40.287734] CR2: ffffa77b833df000 CR3: 00000003fa556000 CR4: 00000000003406e0