Re: Netperf UDP_STREAM regression due to not sending IPIs inttwu_queue()

From: Mel Gorman
Date: Wed Oct 10 2012 - 08:29:16 EST

On Wed, Oct 03, 2012 at 03:30:01PM +0200, Mike Galbraith wrote:
> On Wed, 2012-10-03 at 10:13 +0200, Mike Galbraith wrote:
>
> > Watching all cores instead.
> >
> > switch rate ~890KHz switch rate ~570KHz
> > NO_TTWU_QUEUE nohz=off TTWU_QUEUE nohz=off
> > 5.38% [kernel] [k] __schedule 4.81% [kernel] [k] _raw_spin_lock_irqsave
> > 4.29% [kernel] [k] _raw_spin_lock_irqsave 3.36% [kernel] [k] __skb_recv_datagram
> > 2.88% [kernel] [k] resched_task 2.71% [kernel] [k] copy_user_generic_string
> > 2.60% [kernel] [k] copy_user_generic_string 2.67% [kernel] [k] reschedule_interrupt
> > 2.38% [kernel] [k] __switch_to 2.62% [kernel] [k] sock_alloc_send_pskb
> > 2.15% [kernel] [k] sock_alloc_send_pskb 2.52% [kernel] [k] __schedule
> > 2.08% [kernel] [k] __skb_recv_datagram 2.31% [kernel] [k] try_to_wake_up
> > 1.81% [kernel] [k] udp_sendmsg 2.14% [kernel] [k] system_call
> > 1.76% [kernel] [k] system_call 1.98% [kernel] [k] udp_sendmsg
> > 1.73% [kernel] [k] __udp4_lib_lookup 1.96% [kernel] [k] __udp4_lib_lookup
> > 1.65% [kernel] [k] __slab_free.isra.42 1.78% [kernel] [k] sock_def_readable
> > 1.62% [kernel] [k] try_to_wake_up 1.63% [kernel] [k] __slab_free.isra.42
> > 1.43% [kernel] [k] update_rq_clock 1.60% [kernel] [k] __switch_to
> > 1.43% [kernel] [k] sock_def_readable 1.52% [kernel] [k] dma_issue_pending_all
> > 1.41% [kernel] [k] dma_issue_pending_all 1.48% [kernel] [k] __ip_append_data.isra.35
> > 1.40% [kernel] [k] menu_select 1.44% [kernel] [k] _raw_spin_lock
> > 1.36% [kernel] [k] finish_task_switch 1.38% [kernel] [k] _raw_spin_unlock_irqrestore
> > 1.30% [kernel] [k] ksize 1.33% [kernel] [k] __udp4_lib_rcv
> >
> > Strange.
>
> nohz=off, pipe-test with one half pinned to CPU0, the other to CPU1.
>
> procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
> r b swpd free buff cache si so bi bo in cs us sy id wa st
> TTW_QUEUE
> 1 0 0 3039488 50948 444720 0 0 0 0 539724 1013417 1 15 84 0 0
> 1 0 0 3039488 50956 444720 0 0 0 1 540853 1015679 1 15 84 0 0
> 1 0 0 3039364 50956 444720 0 0 0 0 541630 1017239 1 16 83 0 0
> 2 0 0 3038992 50956 444720 0 0 0 0 335550 1096569 4 20 76 0 0
> NO_TTWU_QUEUE
> 1 0 0 3038992 50956 444720 0 0 0 0 33100 1318984 1 27 71 0 0
> 1 0 0 3038868 50956 444720 0 0 0 0 33100 1319126 2 27 71 0 0
> 1 0 0 3038868 50956 444720 0 0 0 0 33097 1317968 1 27 72 0 0
> 2 0 0 3038868 50964 444720 0 0 0 1 33104 1318558 2 27 71 0 0
>
> We can switch faster with NO_TTWU_QUEUE, so we switch more, and that
> hurts netperf UDP_STREAM throughput.. somehow. Fatter is better is not
> the way context switch happy benchmarks usually work.
>

Do we really switch more though?

Look at the difference in interrupts vs context switch. IPIs are an interrupt
so if TTWU_QUEUE wakes process B using an IPI, does that count as a context
switch? It probably does not get accounted as a context switch even though
it's functionally similar in this case but I'd like to hear confirmation
of that.

If we did assume that these IPIs are effectively context switches then look
at the TTWU_QUEUE figures. There are 530K interrupts versus 33K interrupts
for NO_TTWU_QUEUE. If each one of those IPIs are effectively a context
switch then the actual switch rates are 1.5M switches versus 1.3 switches
and TTWU_QUEUE is actually switching faster.

--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/