Re: Use-after-free in ipv4_conntrack_defrag

From: Dmitry Vyukov
Date: Thu Nov 05 2015 - 14:32:23 EST

Next message: Andy Shevchenko: "Re: [PATCH v6] i40e: Look up MAC address in Open Firmware or IDPROM"
Previous message: Andy Shevchenko: "Re: [PATCH 3/4] scsi: fix compiler warning for sg"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

+netdev, lkml
Adding public lists as requested.

Below is a use-after-free report in ipv4_conntrack_defrag and a fix from Eric.

On Thu, Nov 5, 2015 at 7:57 PM, Eric Dumazet <edumazet@xxxxxxxxxx> wrote:
> On Thu, Nov 5, 2015 at 10:33 AM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
>> Hello,
>>
>> I've updated from bcee19f424a0d8c26ecf2607b73c690802658b29 (Sep 21) to
>> 8e483ed1342a4ea45b70f0f33ac54eff7a33d918 (Nov 4) and started seeing
>> constant stream of the following reports:
>>
>> BUG: KASan: use after free in ipv4_conntrack_defrag+0x39c/0x550 at
>> addr ffff88003d4981b8
>> Read of size 1 by task swapper/1/0
>> =============================================================================
>> BUG skbuff_fclone_cache (Tainted: G B L ): kasan: bad access detected
>> -----------------------------------------------------------------------------
>>
>> CPU: 1 PID: 0 Comm: swapper/1 Tainted: G B L 4.3.0+ #27
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> 00000000ffffffff ffff88003ed06ce8 ffffffff81aada96 ffff88003e952640
>> ffff88003d498000 ffff88003d498000 ffff88003ed06d18 ffffffff814a6d54
>> ffff88003e952640 ffffea0000f52600 ffff88003d498000 ffff88003ed06fb0
>> Call Trace:
>> [<ffffffff814aedbe>] __asan_report_load1_noabort+0x3e/0x40
>> mm/kasan/report.c:248
>> [<ffffffff82a62eac>] ipv4_conntrack_defrag+0x39c/0x550
>> net/ipv4/netfilter/nf_defrag_ipv4.c:72
>> [<ffffffff8287b42d>] nf_iterate+0x15d/0x250 net/netfilter/core.c:274
>> [<ffffffff8287b6c1>] nf_hook_slow+0x1a1/0x300 net/netfilter/core.c:306
>> [< inline >] nf_hook_thresh include/linux/netfilter.h:187
>> [< inline >] nf_hook include/linux/netfilter.h:197
>> [<ffffffff82906593>] __ip_local_out+0x263/0x3c0 net/ipv4/ip_output.c:105
>> [<ffffffff8290671d>] ip_local_out+0x2d/0x1c0 net/ipv4/ip_output.c:114
>> [<ffffffff82906e56>] ip_build_and_send_pkt+0x5a6/0xa40 net/ipv4/ip_output.c:171
>> [<ffffffff82993add>] tcp_v4_send_synack+0x18d/0x270 net/ipv4/tcp_ipv4.c:841
>> [<ffffffff8294e18b>] tcp_conn_request+0x1f3b/0x2750 net/ipv4/tcp_input.c:6273
>> [<ffffffff8298d75e>] tcp_v4_conn_request+0x17e/0x240 net/ipv4/tcp_ipv4.c:1234
>> [<ffffffff829623ce>] tcp_rcv_state_process+0x6ae/0x4130
>> net/ipv4/tcp_input.c:5750
>> [<ffffffff82991a7b>] tcp_v4_do_rcv+0x2fb/0x9f0 net/ipv4/tcp_ipv4.c:1405
>> [<ffffffff82996bf2>] tcp_v4_rcv+0x2872/0x2f80 net/ipv4/tcp_ipv4.c:1630
>> [<ffffffff828ed369>] ip_local_deliver_finish+0x2a9/0xa30
>> net/ipv4/ip_input.c:216
>> [< inline >] NF_HOOK_THRESH include/linux/netfilter.h:226
>> [< inline >] NF_HOOK include/linux/netfilter.h:249
>> [<ffffffff828ef3c4>] ip_local_deliver+0x1c4/0x2f0 net/ipv4/ip_input.c:257
>> [< inline >] dst_input include/net/dst.h:465
>> [<ffffffff828ee104>] ip_rcv_finish+0x614/0x11d0 net/ipv4/ip_input.c:365
>> [< inline >] NF_HOOK_THRESH include/linux/netfilter.h:226
>> [< inline >] NF_HOOK include/linux/netfilter.h:249
>> [<ffffffff828eff66>] ip_rcv+0xa76/0x1470 net/ipv4/ip_input.c:455
>> [<ffffffff827c7379>] __netif_receive_skb_core+0x1cb9/0x38e0 net/core/dev.c:3940
>> [<ffffffff827c8fca>] __netif_receive_skb+0x2a/0x160 net/core/dev.c:3975
>> [<ffffffff827cb6a5>] netif_receive_skb_internal+0xe5/0x360 net/core/dev.c:4003
>> [< inline >] napi_skb_finish net/core/dev.c:4328
>> [<ffffffff827cfc70>] napi_gro_receive+0x1c0/0x260 net/core/dev.c:4357
>> [< inline >] e1000_receive_skb
>> drivers/net/ethernet/intel/e1000/e1000_main.c:4007
>> [<ffffffff823223cc>] e1000_clean_rx_irq+0x4ec/0x10c0
>> drivers/net/ethernet/intel/e1000/e1000_main.c:4459
>> [<ffffffff8231ffe6>] e1000_clean+0xa56/0x2520
>> drivers/net/ethernet/intel/e1000/e1000_main.c:3814
>> [< inline >] napi_poll net/core/dev.c:4793
>> [<ffffffff827cc9dd>] net_rx_action+0x74d/0xc70 net/core/dev.c:4858
>> [<ffffffff8110fdee>] __do_softirq+0x2ae/0x710 kernel/softirq.c:273
>> [< inline >] invoke_softirq kernel/softirq.c:350
>> [<ffffffff811104ed>] irq_exit+0x15d/0x190 kernel/softirq.c:391
>> [< inline >] exiting_irq ./arch/x86/include/asm/apic.h:653
>> [<ffffffff81013256>] do_IRQ+0x86/0x1a0 arch/x86/kernel/irq.c:252
>> [<ffffffff82f255c7>] common_interrupt+0x87/0x87 arch/x86/entry/entry_64.S:545
>> <EOI> [<ffffffff810d0706>] ? native_safe_halt+0x6/0x10
>> ./arch/x86/include/asm/irqflags.h:49
>> [< inline >] arch_safe_halt ./arch/x86/include/asm/paravirt.h:111
>> [<ffffffff81026e62>] default_idle+0x22/0x1e0 arch/x86/kernel/process.c:304
>> [<ffffffff81027f9a>] arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:295
>> [<ffffffff811d9c38>] default_idle_call+0x48/0x70 kernel/sched/idle.c:92
>> [< inline >] cpuidle_idle_call kernel/sched/idle.c:156
>> [< inline >] cpu_idle_loop kernel/sched/idle.c:251
>> [<ffffffff811da15d>] cpu_startup_entry+0x41d/0x570 kernel/sched/idle.c:299
>> [<ffffffff810ac8e3>] start_secondary+0x243/0x2d0 arch/x86/kernel/smpboot.c:251
>>
>> INFO: Allocated in __alloc_skb+0xba/0x5f0 age=3027 cpu=0 pid=2721
>> [< none >] __slab_alloc+0x23a/0x560 mm/slub.c:2402
>> [< inline >] slab_alloc_node mm/slub.c:2470
>> [< none >] kmem_cache_alloc_node+0x88/0x180 mm/slub.c:2540
>> [< none >] __alloc_skb+0xba/0x5f0 net/core/skbuff.c:216
>> [< inline >] alloc_skb_fclone include/linux/skbuff.h:855
>> [< none >] sk_stream_alloc_skb+0xa9/0x620 net/ipv4/tcp.c:827
>> [< none >] tcp_sendmsg+0x18c9/0x2c50 net/ipv4/tcp.c:1169
>> [< none >] inet_sendmsg+0x316/0x4f0 net/ipv4/af_inet.c:733
>> [< inline >] sock_sendmsg_nosec net/socket.c:610
>> [< none >] sock_sendmsg+0xca/0x110 net/socket.c:620
>> [< none >] sock_write_iter+0x216/0x3a0 net/socket.c:819
>> [< inline >] new_sync_write fs/read_write.c:478
>> [< none >] __vfs_write+0x2ed/0x3d0 fs/read_write.c:491
>> [< none >] vfs_write+0x16e/0x490 fs/read_write.c:538
>> [< inline >] SYSC_write fs/read_write.c:585
>> [< none >] SyS_write+0x111/0x220 fs/read_write.c:577
>> [< none >] entry_SYSCALL_64_fastpath+0x31/0x9a
>> arch/x86/entry/entry_64.S:187
>>
>> INFO: Freed in kfree_skbmem+0xca/0x100 age=3027 cpu=0 pid=2721
>> [< none >] __slab_free+0x1ec/0x350 mm/slub.c:2587
>> [< inline >] slab_free mm/slub.c:2736
>> [< none >] kmem_cache_free+0x1e5/0x200 mm/slub.c:2745
>> [< none >] kfree_skbmem+0xca/0x100 net/core/skbuff.c:633
>> [< none >] __kfree_skb+0x1d/0x20 net/core/skbuff.c:674
>> [< inline >] sk_wmem_free_skb include/net/sock.h:1431
>> [< inline >] tcp_clean_rtx_queue net/ipv4/tcp_input.c:3181
>> [< none >] tcp_ack+0x1a5d/0x4c60 net/ipv4/tcp_input.c:3611
>> [< none >] tcp_rcv_established+0xccd/0x20b0 net/ipv4/tcp_input.c:5284
>> [< none >] tcp_v4_do_rcv+0x573/0x9f0 net/ipv4/tcp_ipv4.c:1381
>> [< inline >] sk_backlog_rcv include/net/sock.h:866
>> [< inline >] __release_sock net/core/sock.c:2009
>> [< none >] release_sock+0x168/0x4f0 net/core/sock.c:2457
>> [< none >] tcp_sendmsg+0x1d9/0x2c50 net/ipv4/tcp.c:1307
>> [< none >] inet_sendmsg+0x316/0x4f0 net/ipv4/af_inet.c:733
>> [< inline >] sock_sendmsg_nosec net/socket.c:610
>> [< none >] sock_sendmsg+0xca/0x110 net/socket.c:620
>> [< none >] sock_write_iter+0x216/0x3a0 net/socket.c:819
>> [< inline >] new_sync_write fs/read_write.c:478
>> [< none >] __vfs_write+0x2ed/0x3d0 fs/read_write.c:491
>> [< none >] vfs_write+0x16e/0x490 fs/read_write.c:538
>> [< inline >] SYSC_write fs/read_write.c:585
>> [< none >] SyS_write+0x111/0x220 fs/read_write.c:577
>> [< none >] entry_SYSCALL_64_fastpath+0x31/0x9a
>> arch/x86/entry/entry_64.S:187
>>
>> INFO: Slab 0xffffea0000f52600 objects=19 used=0 fp=0xffff88003d498000
>> flags=0x100000000004080
>> INFO: Object 0xffff88003d498000 @offset=0 fp=0xffff88003d49ba80
>>
>>
>> They start happening during boot and there are so many of them that I
>> suspect that maybe it is something in my setup...
>> But the reports looks pretty legit. tcp_sendmsg allocates skb, frees
>> skb, but also enqueues it for transmission.
>> I've found only one recent commit to net/ipv4/tcp.c which is
>> 686a562449af96a0e8c18c6f1b87b47ff8c36de8, but it looks like no-op.
>>
>> What could cause these errors?
>
> Hi Dmitry
>
> Not sure why you spam so many people ?
> I can assure you netdev list is responsive, especially during merge
> window, where we expect bugs to surface.
>
> We had similar bugs caused by SYNACK attached to request sockets.
>
> Please try (sorry for bad formatting , but patch is trivial enough)
>
> Surely SYNACK messages fail ip_is_fragment() test anyway...
>
> diff --git a/net/ipv4/netfilter/nf_defrag_ipv4.c
> b/net/ipv4/netfilter/nf_defrag_ipv4.c
> index 0e5591c2ee9f..6fb869f646bf 100644
> --- a/net/ipv4/netfilter/nf_defrag_ipv4.c
> +++ b/net/ipv4/netfilter/nf_defrag_ipv4.c
> @@ -67,10 +67,9 @@ static unsigned int ipv4_conntrack_defrag(void *priv,
> const struct nf_hook_state *state)
> {
> struct sock *sk = skb->sk;
> - struct inet_sock *inet = inet_sk(skb->sk);
>
> - if (sk && (sk->sk_family == PF_INET) &&
> - inet->nodefrag)
> + if (sk && sk_fullsock(sk) && (sk->sk_family == PF_INET) &&
> + inet_sk(sk)->nodefrag)
> return NF_ACCEPT;
>
> #if IS_ENABLED(CONFIG_NF_CONNTRACK)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Andy Shevchenko: "Re: [PATCH v6] i40e: Look up MAC address in Open Firmware or IDPROM"
Previous message: Andy Shevchenko: "Re: [PATCH 3/4] scsi: fix compiler warning for sg"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]