Re: major network performance difference between 2.4 and 2.6.2-rc2

[Jim Faulkner]

On Wed, 4 Feb 2004, Andrew Morton wrote:

> Jim Faulkner <jfaulkne@xxxxxxxxxxx> wrote:
> >
> >
> > I am still experiencing severely degraded network performance under
> > 2.6.2-rc3 and 2.6.2-rc3-mm1.
>
> A kernel profile is needed.
>
> >  Based on some kernel output, I think this
> > problem may be related to Gerd Knorr's input patches, so I am CCing him on
> > this e-mail.
>
> Sounds unlikely.
>
> > Additionally, while large network transfers are going on, both ksoftirqd/0
> > and events/0 start going crazy, putting a huge load on my system:
> >
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >   3 root      35  19     0    0    0 S 45.9  0.0   0:46.98 ksoftirqd/0
> >   6 root       5 -10     0    0    0 S 43.3  0.0   1:56.63 events/0
> >   12008 dogshu 15   0  4800 2356 3828 S  5.3  0.2   0:05.98 proftpd
> >   12 root      15   0     0    0    0 S  0.3  0.0   0:00.41 pdflush
> >   9778 root    16   0  5888 1724 5516 R  0.3  0.2   0:00.12 sshd
> >
> > the load before that network transfer was 0.01, and the load after the
> > network transfer was 1.45.
>
> Could be a networking problem, but boy that's a lot of CPU time.
>

Thanks to Andrew's suggestion of profiling my kernel, I've figured out
what is happening here.  It is my fault, it is not a bug.

I use this iptables script generator:
http://ftp.berlios.de/pub/mldonkey/pango/goodies/ipblacklist_convert
in combination with this blacklist:
http://www.peerguardian.net/pgipdb/guarding.p2p

I had already modified the script so everything on my LAN interface was
accepted, however I didn't realize that the scipt was using "-I INPUT 1"
for all of its blacklist rules.  iptables was going through around 5300
rules for each and every packet that came in through my LAN interface,
which is definately not what I intended.

I fixed my firewall script, and my LAN throughput is back up at 10
megabytes per second, with nowhere near the load.

Sorry about that.  I'm not sure what the hauppauge messages are about, but
they're not the cause of my problem.

> > > Could be a networking problem, but boy that's a lot of CPU time.
> >
> > Andrew maybe something bolixed in the MAX_SOFTIRQ_RESTART stuff
> > we put into kernel/softirq.c?  Just a guess...
>
> Might be.  Jim, does a `patch -p1 -R' of the below help things?

I did try the patch before fixing my firewall rules.  ksoftirqd/0 was no
longer using alot of CPU time, however events/0 still was using alot of
CPU time, and of course my performance was still bad.

> Please, do this:
>
> - Boot with `profile=1' on the kernel command line
>
> sudo readprofile -r
> sudo readprofile -M10
> time <whatever command it is that is causing the problem>
> readprofile -n -v -m /boot/System.map | sort -n +2 | tail -40 | tee ~/log

If you're still curious, here is the output while I had my bad firewall
enabled:

c037a340 sock_def_readable                          1933  10.9830
c0149180 kmem_cache_free                            1954  20.3542
c03b3810 tcp_recvmsg                                2032   1.0672
c0174b20 do_select                                  2049   2.6680
c037aa30 skb_release_data                           2098  13.1125
c037a8e0 alloc_skb                                  2129   9.5045
c03ac190 ip_queue_xmit                              2181   1.6829
c01b62b0 is_leaf                                    2230   4.8060
c0174e50 sys_select                                 2274   1.7991
c03b1f70 tcp_sendmsg                                2679   0.5419
c037fb60 process_backlog                            2797   8.7406
c04110ff sysenter_past_esp                          2978  29.4851
c037f9b0 netif_receive_skb                          3115   7.2106
c03a61c0 ip_route_input                             3130   5.9280
c03b0810 tcp_poll                                   3208   8.3542
c012ca50 del_timer_sync                             3356  11.6528
c03a8700 ip_rcv                                     3429   2.6458
c03baf00 tcp_rcv_established                        3589   1.8538
c012c620 __mod_timer                                3835   6.3076
c037ab00 __kfree_skb                                3864  15.0938
c012c990 del_timer                                  3894  20.2812
c03a8c10 ip_local_deliver_finish                    4009   7.5928
c038a570 nf_iterate                                 4049  23.0057
c03bc6a0 tcp_transmit_skb                           4078   2.7114
c01491e0 kfree                                      4135  36.9196
c03e1b20 tcp_packet                                 4514   9.4042
c0163200 __find_get_block                           4607  22.1490
c0163b80 __block_prepare_write                      4860   4.5336
c0121640 __preempt_spin_lock                        4996  52.0417
c01b6590 search_by_key                              7268   1.9750
c02148f4 csum_partial                               8790  30.5208
c0215440 __copy_from_user_ll                       10875  84.9609
c03c4910 tcp_v4_rcv                                12975   5.4793
c011f760 __wake_up                                 15757 140.6875
c02153d0 __copy_to_user_ll                         17531 156.5268
c011efb0 schedule                                  18536  10.5318
c0116730 delay_tsc                                 25865 808.2812
c03e3730 ipt_do_table                             3933402 4552.5486
c0108c70 default_idle                             5633092 88017.0625
00000000 total                                    9915575   3.1080

sorry about the false alarm,
Jim Faulkner
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/