Re: [PATCH] softirq: let ksoftirqd do its job
From: Jesper Dangaard Brouer
Date: Thu Sep 01 2016 - 07:02:42 EST
On Wed, 31 Aug 2016 23:51:16 +0200
Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> wrote:
> On Wed, 31 Aug 2016 13:42:30 -0700
> Eric Dumazet <eric.dumazet@xxxxxxxxx> wrote:
>
> > On Wed, 2016-08-31 at 21:40 +0200, Jesper Dangaard Brouer wrote:
> >
> > > I can confirm the improvement of approx 900Kpps (no wonder people have
> > > been complaining about DoS against UDP/DNS servers).
> > >
> > > BUT during my extensive testing, of this patch, I also think that we
> > > have not gotten to the bottom of this. I was expecting to see a higher
> > > (collective) PPS number as I add more UDP servers, but I don't.
> > >
> > > Running many UDP netperf's with command:
> > > super_netperf 4 -H 198.18.50.3 -l 120 -t UDP_STREAM -T 0,0 -- -m 1472 -n -N
> >
> > Are you sure sender can send fast enough ?
>
> Yes, as I can see drops (overrun UDP limit UdpRcvbufErrors). Switching
> to pktgen and udp_sink to be sure.
>
> > >
> > > With 'top' I can see ksoftirq are still getting a higher %CPU time:
> > >
> > > PID %CPU TIME+ COMMAND
> > > 3 36.5 2:28.98 ksoftirqd/0
> > > 10724 9.6 0:01.05 netserver
> > > 10722 9.3 0:01.05 netserver
> > > 10723 9.3 0:01.05 netserver
> > > 10725 9.3 0:01.05 netserver
> >
> > Looks much better on my machine, with "udprcv -n 4" (using 4 threads,
> > and 4 sockets using SO_REUSEPORT)
> >
> > 10755 root 20 0 34948 4 0 S 79.7 0.0 0:33.66 udprcv
> > 3 root 20 0 0 0 0 R 19.9 0.0 0:25.49 ksoftirqd/0
> >
> > Pressing 'H' in top gives :
> >
> > 3 root 20 0 0 0 0 R 19.9 0.0 0:47.84 ksoftirqd/0
> > 10756 root 20 0 34948 4 0 R 19.9 0.0 0:30.76 udprcv
> > 10757 root 20 0 34948 4 0 R 19.9 0.0 0:30.76 udprcv
> > 10758 root 20 0 34948 4 0 S 19.9 0.0 0:30.76 udprcv
> > 10759 root 20 0 34948 4 0 S 19.9 0.0 0:30.76 udprcv
>
> Yes, I'm seeing the same when unning 5 instances my own udp_sink[1]:
> sudo taskset -c 0 ./udp_sink --port 10003 --recvmsg --reuse-port --count $((10**10))
>
> PID S %CPU TIME+ COMMAND
> 3 R 21.6 2:21.33 ksoftirqd/0
> 3838 R 15.9 0:02.18 udp_sink
> 3856 R 15.6 0:02.16 udp_sink
> 3862 R 15.6 0:02.16 udp_sink
> 3844 R 15.3 0:02.15 udp_sink
> 3850 S 15.3 0:02.15 udp_sink
>
> This is the expected result, that adding more userspace receivers
> scales up. I needed 5 udp_sink's before I don't see any drops, either
> this says the job performed by ksoftirqd is 5 times faster or the
> collective queue size of the programs was fast enough to absorb the
> scheduling jitter.
I need some help from scheduler people explaining this!
In above run of udp_sink (which had expected behavior), I ran udp_sink
in 5 different xterm/shells. Below, I'm running all 5 udp_sink
programs from the same bash shell (just backgrounding them).
PID S %CPU TIME+ COMMAND
3 R 50.0 29:02.23 ksoftirqd/0
10881 R 10.7 1:01.61 udp_sink
10837 R 10.0 1:05.20 udp_sink
10852 S 10.0 1:01.78 udp_sink
10862 R 10.0 1:05.19 udp_sink
10844 S 9.7 1:01.91 udp_sink
This is strange, why is ksoftirqd/0 getting 50% of the CPU time???
And I'm no-longer getting the full tput delivered into userspace (as I
did before with 5 receivers).
$ nstat > /dev/null && sleep 1 && nstat
#kernel
IpInReceives 1234368 0.0
IpInDelivers 1234368 0.0
UdpInDatagrams 1133971 0.0
UdpInErrors 80332 0.0
UdpRcvbufErrors 80332 0.0
IpExtInOctets 56792704 0.0
IpExtInNoECTPkts 1234624 0.0
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer