Re: [PATCH] af_packet: Don't use skb after dev_queue_xmit()
From: Michael Breuer
Date: Mon Jan 18 2010 - 11:30:40 EST
On 01/18/2010 02:30 AM, Jarek Poplawski wrote:
On Sun, Jan 17, 2010 at 06:15:22PM -0500, Michael Breuer wrote:Ok - up on the two patches, no DMAR. Some early observations:
On 1/17/2010 6:05 PM, Jarek Poplawski wrote:Exactly. Or if it's working for you already, the mainline (2.6.33-rc4)
On Sun, Jan 17, 2010 at 05:34:19PM -0500, Michael Breuer wrote:Ok - but absent the last patch, I think I still need the pskb_may_pull
On 1/17/2010 5:17 PM, Jarek Poplawski wrote:Hmm... OK, then let's wait with this report and go back to testing
On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:Not sure I agree. The symptoms are identical to those I hit without
On 01/13/2010 04:16 PM, Michael Breuer wrote:My proposal is to send some summary as a new thread, with dmar in the
On 1/13/2010 4:09 PM, Jarek Poplawski wrote:Update: after leaving the system up for a few days, I hit the DMAR
On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
subject, and cc-ed dmar maintainers.
DMAR earlier on. Also, as this issue only happens when there is high
receive load, I'm thinking there's some sort of race between TX and
RX within the sky2 driver, or hardware. I think that DMAR is
correctly catching the error.
it "really really long" ;-) without DMAR, and maybe without the
last Stephen's patch either? (So only the two things in the current
patch... so it'd be pskb_may_pull and afpacket v3 and no DMAR.
with the pskb_may_pull patch. And check for warnings from the latter.
Also - not sure if related, but there's still the odd tx side behaviorI'm not sure either, but until there is no crash it might be some
when RX is under load. That I CAN reproduce at will (yesterday's report
- no crash, but I confirmed that DHCPOFFER packets are being dropped
somewhere after wireshark sees them and before hitting the wire.
minor bug or/and missing stat. Btw, you could probably try alternative
test with ping from this overloaded box to the router and win7.
I am also wondering whether or not that testing I did yesterday set upMaybe, but you wrote earlier they had to fix something around this
today's hang - perhaps those lost TX packets are corrupting something
that manifests worse later.
DMAR in the meantime, because it triggered much faster during your
previous tests. So, I don't know why you assume this DMAR has to be
correct this time.
1. There's an early on MMAP oops (see below). This happens once, at the
completion of the transition to runlevel 5 (I've seen it entering
runlevel 3 as well). This does not recur when runlevels are subsequently
changed. I do not see this when running with DMAR enabled.
2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
still happens. Interestingly, I initially saw no dropped packets with
ping - but after I went the DCHP route and eventually reconnected, I
could then cause dropped tx packets with ping. To clarify:
a) start throughput
b) ping device - no packet loss - this was true for the entire test run.
c) start throughput again
d) ping - no loss.
e) drop wifi on the device & restart - first attempt worked. Repeat
attempt yielded the dropped DHCPOFFER packets. After about 6 tries, the
device reconnected to wifi.
f) ping again (after the reconnection) - packet loss rate about 80%.
g) simultaneously ping the wifi router - no loss.
h) After a while, packets are no longer dropped during ping. If I manage
to cause the dhcp drop again, and then ping after the device finally
reconnects, packet loss is significant for a while (maybe 30 sec to a
minute). Then things return to normal. Note that the packet loss
continues even if the reported throughput drops to nil.
i) I can't cause the initial packet loss at RX rates below about
30,000KBPS (as reported by nethogs). At rates over 40 I can reproduce
this on this set of patches & config about 60% of the time.
The initial sky2 oops:
Jan 18 10:42:43 mail kernel: ------------[ cut here ]------------
Jan 18 10:42:43 mail kernel: WARNING: at lib/dma-debug.c:898
Jan 18 10:42:43 mail kernel: Hardware name: System Product Name
Jan 18 10:42:43 mail kernel: sky2 0000:06:00.0: DMA-API: device driver
tries to sync DMA memory it has not allocated [device
address=0x00000003249b4022] [size=98 bytes]
Jan 18 10:42:43 mail kernel: Modules linked in: microcode(+)
ip6table_mangle ip6table_filter ip6_tables iptable_raw iptable_mangle
ipt_MASQUERADE iptable_nat nf_nat appletalk psnap llc nfsd lockd nfs_acl
auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit tunnel4
ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp nf_conntrack_ipv6
xt_multiport xt_DSCP xt_dscp xt_MARK ipv6 dm_multipath kvm_intel kvm
snd_hda_codec_analog snd_ens1371 gameport snd_hda_intel snd_rawmidi
snd_hda_codec snd_ac97_codec gspca_spca505 ac97_bus snd_hwdep snd_seq
gspca_main snd_seq_device firewire_ohci videodev firewire_core
v4l1_compat snd_pcm i2c_i801 pcspkr v4l2_compat_ioctl32 crc_itu_t
asus_atk0110 hwmon iTCO_wdt iTCO_vendor_support wmi snd_timer snd sky2
soundcore snd_page_alloc fbcon tileblit font bitblit softcursor raid456
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx
raid1 ata_generic pata_acpi pata_marvell nouveau ttm drm_kms_helper drm
agpgart fb i2c_algo_bit cfbcopyarea i2c_core cfbimgblt cfbf
Jan 18 10:42:43 mail kernel: illrect [last unloaded: ip6_tables]
Jan 18 10:42:43 mail kernel: Pid: 0, comm: swapper Not tainted
Jan 18 10:42:43 mail kernel: Call Trace:
Jan 18 10:42:43 mail kernel: <IRQ> [<ffffffff81053676>]
Jan 18 10:42:43 mail kernel: [<ffffffff810536e5>]
Jan 18 10:42:43 mail kernel: [<ffffffff8127ae7d>] check_sync+0xbd/0x426
Jan 18 10:42:43 mail kernel: [<ffffffff813c5b4c>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff8127b539>]
Jan 18 10:42:43 mail kernel: [<ffffffff812788d7>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff81278a33>] ?
Jan 18 10:42:43 mail kernel: [<ffffffffa015eed6>] sky2_poll+0x4c6/0xae1
Jan 18 10:42:43 mail kernel: [<ffffffff814673f2>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff813cc7ea>] net_rx_action+0xb5/0x1f3
Jan 18 10:42:43 mail kernel: [<ffffffff8105ae57>] __do_softirq+0xf8/0x1cd
Jan 18 10:42:43 mail kernel: [<ffffffff810a2e0e>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
Jan 18 10:42:43 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
Jan 18 10:42:43 mail kernel: [<ffffffff8105aa37>] irq_exit+0x4a/0x8c
Jan 18 10:42:43 mail kernel: [<ffffffff8146b445>] do_IRQ+0xa5/0xbc
Jan 18 10:42:43 mail kernel: [<ffffffff81012613>] ret_from_intr+0x0/0x16
Jan 18 10:42:43 mail kernel: <EOI> [<ffffffff812c251e>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff812c2517>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff813a1b78>] ?
Jan 18 10:42:43 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
Jan 18 10:42:43 mail kernel: [<ffffffff81460acf>] ?
Jan 18 10:42:43 mail kernel: ---[ end trace 188c0cdbace3665e ]---
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/