Re: RFC: inet_timewait_sock->tw_timer list corruption

[Eric Dumazet]

On Thu, Aug 27, 2020 at 4:03 AM Wang Long <w@xxxxxxxxxxxxx> wrote:
>
> Hi，
>
> we encountered a kernel panic as following:
>
> [4394470.273792] general protection fault: 0000 [#1] SMP NOPTI
> [4394470.274038] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G
>       W     --------- -  - 4.18.0-80.el8.x86_64 #1
> [4394470.274477] Hardware name: Sugon I620-G30/60P24-US, BIOS MJGS1223
> 04/07/2020
> [4394470.274727] RIP: 0010:run_timer_softirq+0x34e/0x440
> [4394470.274957] Code: 84 3f ff ff ff 49 8b 04 24 48 85 c0 74 58 49 8b
> 1c 24 48 89 5d 08 0f 1f 44 00 00 48 8b 03 48 8b 53 08 48 85 c0 48 89 02
> 74 04 <48> 89 50 08 f6 43 22 20 48 c7 43 08 00 00 00 00 48 89 ef 4c 89 2b
> [4394470.275505] RSP: 0018:ffff88f000803ee0 EFLAGS: 00010086
> [4394470.275783] RAX: dead000000000200 RBX: ffff88e5e33ea078 RCX:
> 0000000000000100
> [4394470.276087] RDX: ffff88f000803ee8 RSI: 0000000000000000 RDI:
> ffff88f00081aa00
> [4394470.276391] RBP: ffff88f00081aa00 R08: 0000000000000001 R09:
> 0000000000000000
> [4394470.276697] R10: ffff88e5e33eb1f0 R11: 0000000000000000 R12:
> ffff88f000803ee8
> [4394470.277030] R13: dead000000000200 R14: ffff88f000803ee0 R15:
> 0000000000000000
> [4394470.277350] FS:  0000000000000000(0000) GS:ffff88f000800000(0000)
> knlGS:0000000000000000
> [4394470.277684] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [4394470.278020] CR2: 00007f200eddd160 CR3: 0000000e0b20a002 CR4:
> 00000000007606f0
> [4394470.278412] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [4394470.278799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [4394470.279194] PKRU: 55555554
> [4394470.279543] Call Trace:
> [4394470.279889]  <IRQ>
> [4394470.280237]  ? __hrtimer_init+0xb0/0xb0
> [4394470.280618]  ? sched_clock+0x5/0x10
> [4394470.281000]  __do_softirq+0xe8/0x2ef
> [4394470.281397]  irq_exit+0xf1/0x100
> [4394470.281761]  smp_apic_timer_interrupt+0x74/0x130
> [4394470.282132]  apic_timer_interrupt+0xf/0x20
> [4394470.282548]  </IRQ>
> [4394470.282954] RIP: 0010:cpuidle_enter_state+0xa0/0x2b0
> [4394470.283341] Code: 8b 3d 6c fb 59 4c e8 0f ed a6 ff 48 89 c3 0f 1f
> 44 00 00 31 ff e8 80 00 a7 ff 45 84 f6 0f 85 c3 01 00 00 fb 66 0f 1f 44
> 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
> [4394470.284219] RSP: 0018:ffffffffb4603e78 EFLAGS: 00000246 ORIG_RAX:
> ffffffffffffff13
> [4394470.284671] RAX: ffff88f000823080 RBX: 000f9cbf579e86c6 RCX:
> 000000000000001f
> [4394470.285129] RDX: 000f9cbf579e86c6 RSI: 0000000037a6f674 RDI:
> 0000000000000000
> [4394470.285623] RBP: 0000000000000002 R08: 00000000000000c4 R09:
> 0000000000000027
> [4394470.286088] R10: ffffffffb4603e58 R11: 000000000000004c R12:
> ffff88f00082df00
> [4394470.286566] R13: ffffffffb4724118 R14: 0000000000000000 R15:
> 000f9cbf579d44e0
> [4394470.287045]  ? cpuidle_enter_state+0x90/0x2b0
> [4394470.287527]  do_idle+0x200/0x280
> [4394470.288010]  cpu_startup_entry+0x6f/0x80
> [4394470.288501]  start_kernel+0x533/0x553
> [4394470.288994]  secondary_startup_64+0xb7/0xc0
>
>
> After analysis, we found that the timer which expires has
> timer->entry.next == POISON2 !(the list corruption )
>
> the crash scenario is the same as https://lkml.org/lkml/2017/3/21/732,
>
> I cannot reproduce this issue, but I found that the timer cause crash is
> the inet_timewait_sock->tw_timer(its callback function is
> tw_timer_handler), and the value of tcp_tw_reuse is 1.
>
> # cat /proc/sys/net/ipv4/tcp_tw_reuse
> 1
>
> In the production environment, we encountered this problem many times,
> and every time it was a problem with the inet_timewait_sock->tw_timer.
>
> Do anyone have any ideas for this issue? Thanks.
>

Nothing comes to mind, I am not aware of such crashes in stable linux kernels.