Re: Scaling problem with a lot of AF_PACKET sockets on differentinterfaces

[Vitaly V. Bursov]

07.06.2013 16:05, Daniel Borkmann ÐÐÑÐÑ:
> On 06/07/2013 02:41 PM, Mike Galbraith wrote:
> > (CC's net-fu dojo)
> >
> > On Fri, 2013-06-07 at 14:56 +0300, Vitaly V. Bursov wrote:
> > > Hello,
> > >
> > > I have a Linux router with a lot of interfaces (hundreds or
> > > thousands of VLANs) and an application that creates AF_PACKET
> > > socket per interface and bind()s sockets to interfaces.
> > >
> > > Each socket has attached BPF filter too.
> > >
> > > The problem is observed on linux-3.8.13, but as far I can see
> > > from the source the latest version has alike behavior.
> > >
> > > I noticed that box has strange performance problems with
> > > most of the CPU time spent in __netif_receive_skb:
> > >    86.15%  [k] __netif_receive_skb
> > >     1.41%  [k] _raw_spin_lock
> > >     1.09%  [k] fib_table_lookup
> > >     0.99%  [k] local_bh_enable_ip
> > >
> > > and this the assembly with the "hot spot":
> > >          â       shr    $0x8,%r15w
> > >          â       and    $0xf,%r15d
> > >     0.00 â       shl    $0x4,%r15
> > >          â       add    $0xffffffff8165ec80,%r15
> > >          â       mov    (%r15),%rax
> > >     0.09 â       mov    %rax,0x28(%rsp)
> > >          â       mov    0x28(%rsp),%rbp
> > >     0.01 â       sub    $0x28,%rbp
> > >          â       jmp    5c7
> > >     1.72 â5b0:   mov    0x28(%rbp),%rax
> > >     0.05 â       mov    0x18(%rsp),%rbx
> > >     0.00 â       mov    %rax,0x28(%rsp)
> > >     0.03 â       mov    0x28(%rsp),%rbp
> > >     5.67 â       sub    $0x28,%rbp
> > >     1.71 â5c7:   lea    0x28(%rbp),%rax
> > >     1.73 â       cmp    %r15,%rax
> > >          â       je     640
> > >     1.74 â       cmp    %r14w,0x0(%rbp)
> > >          â       jne    5b0
> > >    81.36 â       mov    0x8(%rbp),%rax
> > >     2.74 â       cmp    %rax,%r8
> > >          â       je     5eb
> > >     1.37 â       cmp    0x20(%rbx),%rax
> > >          â       je     5eb
> > >     1.39 â       cmp    %r13,%rax
> > >          â       jne    5b0
> > >     0.04 â5eb:   test   %r12,%r12
> > >     0.04 â       je     6f4
> > >          â       mov    0xc0(%rbx),%eax
> > >          â       mov    0xc8(%rbx),%rdx
> > >          â       testb  $0x8,0x1(%rdx,%rax,1)
> > >          â       jne    6d5
> > >
> > > This corresponds to:
> > >
> > > net/core/dev.c:
> > >           type = skb->protocol;
> > >           list_for_each_entry_rcu(ptype,
> > >                           &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
> > >                   if (ptype->type == type &&
> > >                       (ptype->dev == null_or_dev || ptype->dev == skb->dev ||
> > >                        ptype->dev == orig_dev)) {
> > >                           if (pt_prev)
> > >                                   ret = deliver_skb(skb, pt_prev, orig_dev);
> > >                           pt_prev = ptype;
> > >                   }
> > >           }
> > >
> > > Which works perfectly OK until there are a lot of AF_PACKET sockets, since
> > > the socket adds a protocol to ptype list:
> > >
> > > # cat /proc/net/ptype
> > > Type Device      Function
> > > 0800 eth2.1989 packet_rcv+0x0/0x400
> > > 0800 eth2.1987 packet_rcv+0x0/0x400
> > > 0800 eth2.1986 packet_rcv+0x0/0x400
> > > 0800 eth2.1990 packet_rcv+0x0/0x400
> > > 0800 eth2.1995 packet_rcv+0x0/0x400
> > > 0800 eth2.1997 packet_rcv+0x0/0x400
> > > .......
> > > 0800 eth2.1004 packet_rcv+0x0/0x400
> > > 0800          ip_rcv+0x0/0x310
> > > 0011          llc_rcv+0x0/0x3a0
> > > 0004          llc_rcv+0x0/0x3a0
> > > 0806          arp_rcv+0x0/0x150
> > >
> > > And this obviously results in a huge performance penalty.
> > >
> > > ptype_all, by the looks, should be the same.
> > >
> > > Probably one way to fix this it to perform interface name matching in
> > > af_packet handler, but there could be other cases, other protocols.
> > >
> > > Ideas are welcome :)
>
> Probably, that depends on _your scenario_ and/or BPF filter, but would it be
> an alternative if you have only a few packet sockets (maybe one pinned to each
> cpu) and cluster/load-balance them together via packet fanout? (Where you
> bind the socket to ifindex 0, so that you get traffic from all devs...) That
> would at least avoid that "hot spot", and you could post-process the interface
> via sockaddr_ll. But I'd agree that this will not solve the actual problem you've
> observed. ;-)

I was't aware of the ifindex 0 thing, it can help, thanks! Of course, if it'll
work for me (applications is a custom DHCP server) it'll surely
increase the overhead of BPF (I don't need to tap the traffic from all
interfaces), there are vlans, bridges and bonds - likely the server will receive
same packets multiple times and replies must be sent too...
but it still should be faster.

I just checked isc-dhcpd-V3.1.3 running on multiple interfaces
(another system with 2.6.32):
$ cat /proc/net/ptype
Type Device      Function
ALL  eth0     packet_rcv_spkt+0x0/0x190
ALL  eth0.10  packet_rcv_spkt+0x0/0x190
ALL  eth0.11  packet_rcv_spkt+0x0/0x190
....

As I understand, it'll hit this code:
        list_for_each_entry_rcu(ptype, &ptype_all, list) {
                if (!ptype->dev || ptype->dev == skb->dev) {
                        if (pt_prev)
                                ret = deliver_skb(skb, pt_prev, orig_dev);
                        pt_prev = ptype;
                }
        }
which scales the same.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/