Re: 3.9.4 Oops running xfstests (WAS Re: 3.9.3: Oops runningxfstests)
From: CAI Qian
Date: Mon Jun 03 2013 - 23:15:39 EST
----- Original Message -----
> From: "Dave Chinner" <david@xxxxxxxxxxxxx>
> To: "CAI Qian" <caiqian@xxxxxxxxxx>
> Cc: xfs@xxxxxxxxxxx, stable@xxxxxxxxxxxxxxx, "LKML" <linux-kernel@xxxxxxxxxxxxxxx>, "linux-mm" <linux-mm@xxxxxxxxx>
> Sent: Monday, June 3, 2013 12:00:38 PM
> Subject: Re: 3.9.4 Oops running xfstests (WAS Re: 3.9.3: Oops running xfstests)
>
> On Sun, Jun 02, 2013 at 11:04:11PM -0400, CAI Qian wrote:
> >
> > > There's memory corruption all over the place. It is most likely
> > > that trinity is causing this - it's purpose is to trigger corruption
> > > issues, but they aren't always immediately seen. If you can trigger
> > > this xfs trace without trinity having been run and without all the
> > > RCU/idle/scheduler/cgroup issues occuring at the same time, then
> > > it's likely to be caused by XFS. But right now, I'd say XFS is just
> > > an innocent bystander caught in the crossfire. There's nothing I can
> > > do from an XFS persepctive to track this down...
> > OK, this can be reproduced by just running LTP and then xfstests without
> > trinity at all...
>
> Cai, can you be more precise about what is triggering it? LTP and
> xfstests do a large amount of stuff, and stack traces do not do not
> help narrow down the cause at all. Can you provide the follwoing
> information and perform the follwoing steps:
>
> 1. What xfstest is tripping over it?
Test #20.
> 2. Can you reproduce it just by running that one specific test
> on a pristine system (i.e. freshly mkfs'd filesystems,
> immediately after boot)
Yes, it was reproduced without LTP at all.
[ 98.534402] XFS (dm-0): Mounting Filesystem
[ 98.586673] XFS (dm-0): Ending clean mount
[ 99.741704] XFS (dm-2): Mounting Filesystem
[ 100.117248] XFS (dm-2): Ending clean mount
[ 100.723228] XFS (dm-0): Mounting Filesystem
[ 100.775965] XFS (dm-0): Ending clean mount
[ 101.980250] BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
[ 101.988136] IP: [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[ 101.993737] PGD 0
[ 101.995769] Oops: 0002 [#1] SMP
[ 101.999038] Modules linked in: lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm hp_wmi sparse_keymap rfkill iTCO_wdt e1000e pcspkr snd_page_alloc iTCO_vendor_support mei ptp pps_core lpc_ich i2c_i801 snd_timer mfd_core microcode(+) snd soundcore xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif nouveau video mxm_wmi i2c_algo_bit drm_kms_helper ahci ata_generic ttm libahci pata_acpi drm i2c_core libata wmi dm_mirror dm_region_hash dm_log dm_mod
[ 102.075355] CPU 2
[ 102.077197] Pid: 356, comm: kworker/2:2 Not tainted 3.9.4 #1 Hewlett-Packard HP Z210 Workstation/1587h
[ 102.086691] RIP: 0010:[<ffffffff81098cac>] [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[ 102.094705] RSP: 0018:ffff880078307c78 EFLAGS: 00010002
[ 102.100020] RAX: 0001f2b5a618ed0f RBX: 0000000000000001 RCX: 000000000000068a
[ 102.107157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8800772ceee8
[ 102.114293] RBP: ffff880078307c78 R08: 0000000000000008 R09: ffff88007d094400
[ 102.121422] R10: 0000000000000344 R11: 0000000000000001 R12: ffffffff81c78560
[ 102.128552] R13: ffffffff8108c460 R14: 0000000000000000 R15: ffff8800772ceee8
[ 102.135682] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:0000000000000000
[ 102.143776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 102.149524] CR2: 0000000000000098 CR3: 00000000018fa000 CR4: 00000000000407e0
[ 102.156654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 102.163785] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 102.170915] Process kworker/2:2 (pid: 356, threadinfo ffff880078306000, task ffff88007736b580)
[ 102.179527] Stack:
[ 102.181545] ffff880078307cc0 ffffffff810926b2 ffffffff81098c60 ffff8800772cf008
[ 102.189005] ffff880079b96f00 000000000000069c ffff880079b96ee8 0000000000014400
[ 102.196464] ffff88007d094400 ffff880078307db0 ffffffff8109f773 ffff88007cc10480
[ 102.203920] Call Trace:
[ 102.206372] [<ffffffff810926b2>] walk_tg_tree_from+0x32/0xe0
[ 102.212118] [<ffffffff81098c60>] ? task_waking_fair+0x20/0x20
[ 102.217955] [<ffffffff8109f773>] load_balance+0x2a3/0x7d0
[ 102.223444] [<ffffffff8108fa7c>] ? update_rq_clock.part.67+0x1c/0x170
[ 102.229977] [<ffffffff810a0142>] idle_balance+0x182/0x2f0
[ 102.235468] [<ffffffff8160f1ac>] __schedule+0x7bc/0x7d0
[ 102.240786] [<ffffffff8160f1e9>] schedule+0x29/0x70
[ 102.245756] [<ffffffff8107f404>] worker_thread+0x1b4/0x3d0
[ 102.251332] [<ffffffff8107f250>] ? __alloc_workqueue_key+0x500/0x500
[ 102.257777] [<ffffffff81084240>] kthread+0xc0/0xd0
[ 102.262662] [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[ 102.268761] [<ffffffff816192ac>] ret_from_fork+0x7c/0xb0
[ 102.274164] [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[ 102.280262] Code: 00 00 00 00 48 8b 14 f0 48 8b 0c f1 48 8b 82 98 00 00 00 48 0f af 01 48 8b 0a 31 d2 48 83 c1 01 48 f7 f1 48 8b 57 48 4a 8b 14 02 <48> 89 82 98 00 00 00 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 48
[ 102.300226] RIP [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[ 102.305906] RSP <ffff880078307c78>
[ 102.309394] CR2: 0000000000000098
[ 102.312710] ---[ end trace ba964230a74993fe ]---
[ 102.312714] BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
[ 102.312719] IP: [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[ 102.312720] PGD 0
[ 102.312723] Oops: 0002 [#2] SMP
[ 102.312752] Modules linked in: lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm hp_wmi sparse_keymap rfkill iTCO_wdt e1000e pcspkr snd_page_alloc iTCO_vendor_support mei ptp pps_core lpc_ich i2c_i801 snd_timer mfd_core microcode(+) snd soundcore xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif nouveau video mxm_wmi i2c_algo_bit drm_kms_helper ahci ata_generic ttm libahci pata_acpi drm i2c_core libata wmi dm_mirror dm_region_hash dm_log dm_mod
[ 102.312758] CPU 0
[ 102.312758] Pid: 78, comm: kworker/0:2 Tainted: G D 3.9.4 #1 Hewlett-Packard HP Z210 Workstation/1587h
[ 102.312762] RIP: 0010:[<ffffffff81098cac>] [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[ 102.312763] RSP: 0018:ffff880036c25c78 EFLAGS: 00010002
[ 102.312764] RAX: 0001f2b5a618ed0f RBX: 0000000000000001 RCX: 000000000000068a
[ 102.312765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8800772ceee8
[ 102.312766] RBP: ffff880036c25c78 R08: 0000000000000008 R09: ffff88007d094400
[ 102.312767] R10: 0000000000000344 R11: 0000000000000001 R12: ffffffff81c78560
[ 102.312768] R13: ffffffff8108c460 R14: 0000000000000000 R15: ffff8800772ceee8
[ 102.312770] FS: 0000000000000000(0000) GS:ffff88007d000000(0000) knlGS:0000000000000000
[ 102.312771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 102.312772] CR2: 0000000000000098 CR3: 00000000018fa000 CR4: 00000000000407f0
[ 102.312773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 102.312774] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 102.312776] Process kworker/0:2 (pid: 78, threadinfo ffff880036c24000, task ffff8800773c9ac0)
[ 102.312776] Stack:
[ 102.312779] ffff880036c25cc0 ffffffff810926b2 ffffffff81098c60 ffff8800772cf008
[ 102.312781] ffff880079b96f00 000000000000069c ffff880079b96ee8 0000000000014400
[ 102.312783] ffff88007d094400 ffff880036c25db0 ffffffff8109f773 ffff88007cc10080
[ 102.312783] Call Trace:
[ 102.312787] [<ffffffff810926b2>] walk_tg_tree_from+0x32/0xe0
[ 102.312789] [<ffffffff81098c60>] ? task_waking_fair+0x20/0x20
[ 102.312792] [<ffffffff8109f773>] load_balance+0x2a3/0x7d0
[ 102.312795] [<ffffffff8108fa7c>] ? update_rq_clock.part.67+0x1c/0x170
[ 102.312798] [<ffffffff810a0142>] idle_balance+0x182/0x2f0
[ 102.312801] [<ffffffff8160f1ac>] __schedule+0x7bc/0x7d0
[ 102.312803] [<ffffffff8160f1e9>] schedule+0x29/0x70
[ 102.312806] [<ffffffff8107f404>] worker_thread+0x1b4/0x3d0
[ 102.312809] [<ffffffff8107f250>] ? __alloc_workqueue_key+0x500/0x500
[ 102.312811] [<ffffffff81084240>] kthread+0xc0/0xd0
[ 102.312813] [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[ 102.312816] [<ffffffff816192ac>] ret_from_fork+0x7c/0xb0
[ 102.312818] [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[ 102.312837] Code: 00 00 00 00 48 8b 14 f0 48 8b 0c f1 48 8b 82 98 00 00 00 48 0f af 01 48 8b 0a 31 d2 48 83 c1 01 48 f7 f1 48 8b 57 48 4a 8b 14 02 <48> 89 82 98 00 00 00 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 48
[ 102.312839] RIP [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[ 102.312840] RSP <ffff880036c25c78>
[ 102.312840] CR2: 0000000000000098
[ 102.312842] ---[ end trace ba964230a74993ff ]---
[ 102.312866] general protection fault: 0000 [#3] SMP
[ 102.312896] Modules linked in: lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm hp_wmi sparse_keymap rfkill iTCO_wdt e1000e pcspkr snd_page_alloc iTCO_vendor_support mei ptp pps_core lpc_ich i2c_i801 snd_timer mfd_core microcode(+) snd soundcore xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif nouveau video mxm_wmi i2c_algo_bit drm_kms_helper ahci ata_generic ttm libahci pata_acpi drm i2c_core libata wmi dm_mirror dm_region_hash dm_log dm_mod
[ 102.312903] CPU 1
[ 102.312903] Pid: 1999, comm: attr Tainted: G D 3.9.4 #1 Hewlett-Packard HP Z210 Workstation/1587h
[ 102.312908] RIP: 0010:[<ffffffff810983a9>] [<ffffffff810983a9>] irqtime_account_process_tick.isra.2+0x239/0x3c0
[ 102.312909] =============================================================================
[ 102.312910] RSP: 0018:ffff88007d083e08 EFLAGS: 00010003
[ 102.312912] BUG kmalloc-1024 (Tainted: G D ): Padding overwritten. 0xffff88005b4e7ec0-0xffff88005b4e7fff
[ 102.312913] RAX: ffff88005b656288 RBX: ffff880079b43c80 RCX: 00000000000000a7
[ 102.312914] -----------------------------------------------------------------------------
[ 102.312914]
[ 102.312915] RDX: 6b6b6b6b6b6b6b6b RSI: 0000000000000000 RDI: 0000000000000086
[ 102.312917] INFO: Slab 0xffffea00016d3800 objects=24 used=24 fp=0x (null) flags=0x10000000004080
[ 102.312918] RBP: ffff88007d083e40 R08: 0000000000000000 R09: 0000000225c17d03
[ 102.312921] Pid: 518, comm: in:imklog Tainted: G B D 3.9.4 #1
[ 102.312922] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88007d08e800
[ 102.312922] Call Trace:
[ 102.312923] R13: ffff880058063580 R14: 0000000000000000 R15: ffff88007d094c88
[ 102.312926] FS: 00007f237404a740(0000) GS:ffff88007d080000(0000) knlGS:0000000000000000
[ 102.312931] [<ffffffff81181ed2>] slab_err+0xc2/0xf0
[ 102.312932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 102.312934] CR2: 00007f2373c352a0 CR3: 0000000063ea9000 CR4: 00000000000407e0
[ 102.312937] [<ffffffff81018018>] ? write_ok_or_segv+0x88/0x90
[ 102.312938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 102.312939] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 102.312943] [<ffffffff81020c2b>] ? save_stack_trace+0x2b/0x50
[ 102.312944] Process attr (pid: 1999, threadinfo ffff88005753e000, task ffff880058063580)
[ 102.312945] Stack:
[ 102.312947] [<ffffffff81180911>] ? set_track+0x61/0x1b0
[ 102.312949] ffff88007d08e800
[ 102.312952] [<ffffffff8118176d>] ? init_object+0x3d/0x70
[ 102.312953] 0000000000000086 ffff88007d094400
[ 102.312956] [<ffffffff81181ff5>] slab_pad_check.part.41+0xf5/0x170
[ 102.312957] 0000000000014400
[ 102.312967] [<ffffffff81063bdc>] ? do_syslog+0x23c/0x5c0
[ 102.312968] ffff880058063580 0000000000000000
[ 102.312969] [<ffffffff811820e3>] check_slab+0x73/0x100
[ 102.312970] 00000017979e99b3 ffff88007d083e78
[ 102.312970]
[ 102.312972] [<ffffffff81606b50>] alloc_debug_processing+0x21/0x118
[ 102.312973] ffffffff8109874c
[ 102.312974] [<ffffffff8160772f>] __slab_alloc+0x3b8/0x4a2
[ 102.312974] ffff880058063580 0000000000000000
[ 102.312976] [<ffffffff81063bdc>] ? do_syslog+0x23c/0x5c0
[ 102.312976] 0000000000000001
[ 102.312977] Call Trace:
[ 102.312978] [<ffffffff810915aa>] ? finish_task_switch+0xba/0xe0
[ 102.312979] <IRQ>
[ 102.312980] [<ffffffff81184dd1>] kmem_cache_alloc_trace+0x1b1/0x200
[ 102.312981] [<ffffffff8109874c>] account_process_tick+0x11c/0x1d0
[ 102.312983] [<ffffffff81063bdc>] do_syslog+0x23c/0x5c0
[ 102.312985] [<ffffffff81071f0d>] update_process_times+0x2d/0x80
[ 102.312986] [<ffffffff810850e0>] ? wake_up_bit+0x30/0x30
[ 102.312988] [<ffffffff810ba1c5>] tick_sched_handle.isra.13+0x25/0x60
[ 102.312991] [<ffffffff8120ca94>] kmsg_read+0x44/0x60
[ 102.312992] [<ffffffff810ba241>] tick_sched_timer+0x41/0x60
[ 102.312994] [<ffffffff811ffa9a>] proc_reg_read+0x6a/0xa0
[ 102.312996] [<ffffffff81087af4>] __run_hrtimer+0x74/0x1d0
[ 102.312997] [<ffffffff8119c56c>] vfs_read+0x9c/0x170
[ 102.312998] [<ffffffff810ba200>] ? tick_sched_handle.isra.13+0x60/0x60
[ 102.313000] [<ffffffff8119c939>] sys_read+0x49/0xa0
[ 102.313001] [<ffffffff810882d7>] hrtimer_interrupt+0xe7/0x220
[ 102.313004] [<ffffffff810e0ef6>] ? __audit_syscall_exit+0x1f6/0x2a0
[ 102.313005] [<ffffffff8161b0e9>] smp_apic_timer_interrupt+0x69/0x9c
[ 102.313007] [<ffffffff81619359>] system_call_fastpath+0x16/0x1b
[ 102.313008] [<ffffffff81619fdd>] apic_timer_interrupt+0x6d/0x80
[ 102.313009] Padding ffff88005b4e7ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313010] Padding ffff88005b4e7ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313010] <EOI>
[ 102.313011] Padding ffff88005b4e7ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313013] [<ffffffff813022f6>] ? memmove+0x46/0x1a0
[ 102.313013] Padding ffff88005b4e7ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 29 01 00 .............)..
[ 102.313014] Padding ffff88005b4e7f00: 07 1b 04 73 65 6c 69 6e 75 78 73 79 73 74 65 6d ...selinuxsystem
[ 102.313015] Padding ffff88005b4e7f10: 5f 75 3a 6f 62 6a 65 63 74 5f 72 3a 75 73 72 5f _u:object_r:usr_
[ 102.313032] [<ffffffffa02801a1>] ? xfs_attr_leaf_moveents.isra.2+0x91/0x280 [xfs]
[ 102.313032] Padding ffff88005b4e7f20: 74 3a 73 30 00 00 00 00 49 4e 81 a4 02 02 00 00 t:s0....IN......
[ 102.313033] Padding ffff88005b4e7f30: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 ................
[ 102.313033] Padding ffff88005b4e7f40: 00 00 00 00 00 00 00 02 51 47 09 00 00 00 00 00 ........QG......
[ 102.313042] [<ffffffffa0280467>] xfs_attr_leaf_compact+0xd7/0x130 [xfs]
[ 102.313043] Padding ffff88005b4e7f50: 51 47 09 00 00 00 00 00 51 ac 1e 27 21 f1 4e ad QG......Q..'!.N.
[ 102.313043] Padding ffff88005b4e7f60: 00 00 00 00 00 00 00 f2 00 00 00 00 00 00 00 01 ................
[ 102.313044] Padding ffff88005b4e7f70: 00 00 00 00 00 00 00 01 00 00 0e 01 00 00 00 00 ................
[ 102.313053] [<ffffffffa0281a2e>] xfs_attr_leaf_add+0xce/0x170 [xfs]
[ 102.313053] Padding ffff88005b4e7f80: 00 00 00 00 c1 6d 78 44 ff ff ff ff 00 00 00 00 .....mxD........
[ 102.313054] Padding ffff88005b4e7f90: 00 00 00 00 00 00 08 10 36 a0 00 01 00 00 00 00 ........6.......
[ 102.313062] [<ffffffffa027d850>] xfs_attr_leaf_addname+0xc0/0x3d0 [xfs]
[ 102.313062] Padding ffff88005b4e7fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313063] Padding ffff88005b4e7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313072] [<ffffffffa028bd4e>] ? xfs_bmap_one_block+0x3e/0xa0 [xfs]
[ 102.313072] Padding ffff88005b4e7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313073] Padding ffff88005b4e7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313074] Padding ffff88005b4e7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 102.313081] [<ffffffffa027e78c>] xfs_attr_set_int+0x30c/0x420 [xfs]
[ 102.313082] Padding ffff88005b4e7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 29 01 00 .............)..
[ 102.313085] [<ffffffff811be9f4>] ? setxattr+0xa4/0x1c0
[ 102.313085] FIX kmalloc-1024: Restoring 0xffff88005b4e7ec0-0xffff88005b4e7fff=0x5a
[ 102.313085]
[ 102.313086] ==========
CAI Qian
> 3. if you can't reproduce it like that, does it reproduce on
> an xfstest run on a pristine system? If so, what command
> line are you running, and what are the filesystem
> configurations?
> 4. if you cannot reproduce it just with xfstests and you need
> to run LTP first, then can you just run the xfstest that
> is failing after running LTP and see if that triggers the
> problem. If it does, please take a metadump of the
> filesystems after LTP has run, save them, and if the
> single test then fails send me the metadumps and your
> xfstests command line.
> 5. If all else fails, bisect the kernel to identify the
> commit that introduces the problem....
>
> Cheers,
>
> Dave.
>
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@xxxxxxxxxx For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/