Re: Re: Regression caused by commit 4bdc33ed("NFSDv4.2: Add NFS v4.2 support to the NFS server")

From: Jongman Heo
Date: Thu Sep 26 2013 - 00:23:19 EST


Hi,

>
>------- Original Message -------
>Sender : J. Bruce Fields<bfields@xxxxxxxxxxxx>
>Date : 2013-09-25 23:05 (GMT+09:00)
>Title : Re: Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
>
>On Wed, Sep 25, 2013 at 05:19:50AM +0000, Jongman Heo wrote:
>> My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
>>
>> Using git bisect, I found it is caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server").
>>
>>
>> 1. dmesg (NFS boot failure case)
>>
>> ...
>> [ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
>> [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [ 3.055023] IP-Config: Guessing netmask 255.255.0.0
>> [ 3.059979] IP-Config: Gateway not on directly connected network.
>> [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249
>> [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249
>> [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy.
>> [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0)
>> [ 3.135478] Please append a correct "root=" boot option; here are the available partitions:
>> [ 3.143831] 1f00 3072 mtdblock0 (driver?)
>> [ 3.148798] 1f01 64 mtdblock1 (driver?)
>> [ 3.153758] 1f02 64 mtdblock2 (driver?)
>> [ 3.158719] 1f03 64 mtdblock3 (driver?)
>> [ 3.163682] 1f04 64 mtdblock4 (driver?)
>> [ 3.168644] 1f05 64 mtdblock5 (driver?)
>> [ 3.173607] 1f06 64 mtdblock6 (driver?)
>> [ 3.178568] 0800 488386584 sda driver: sd
>> [ 3.183099] 0801 506016 sda1
>> [ 3.186927] 0802 4008217 sda2
>> [ 3.190755] 0803 483869767 sda3
>> [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk
>> [ 3.199802] b301 4096 mmcblk0p1
>> [ 3.204063] b302 102400 mmcblk0p2
>> [ 3.208330] b303 4096 mmcblk0p3
>> [ 3.212594] b304 1 mmcblk0p4
>> [ 3.216855] b305 2048 mmcblk0p5
>> [ 3.221116] b306 2048 mmcblk0p6
>> [ 3.225382] b307 2048 mmcblk0p7
>> [ 3.229644] b308 4096 mmcblk0p8
>> [ 3.233906] b309 12288 mmcblk0p9
>> [ 3.238176] b30a 16384 mmcblk0p10
>> [ 3.242524] b30b 142336 mmcblk0p11
>> [ 3.246869] b30c 1572864 mmcblk0p12
>> [ 3.251219] b320 12288 mmcblk0gp1 (driver?)
>> [ 3.256272] b310 12288 mmcblk0gp0 (driver?)
>> [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
>> [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1
>> [ 3.274776] Call Trace:
>> [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20
>> [ 3.281492] [<80d0dad1>] panic+0x65/0xd1
>> [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be
>> [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30
>> [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2
>> [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184
>> [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30
>> [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e
>> [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e
>> [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x10
>>
>>
>> 2. Client (my embedded box) configuration
>> It's kernel 2.6.35 based, and has following NFS kernel configs.
>>
>> # grep NFS .config
>> CONFIG_NFS_FS=y
>> CONFIG_NFS_V3=y
>> CONFIG_NFS_V3_ACL=y
>> CONFIG_NFS_V4=y
>> # CONFIG_NFS_V4_1 is not set
>> CONFIG_ROOT_NFS=y
>> # CONFIG_NFSD is not set
>> CONFIG_NFS_ACL_SUPPORT=y
>> CONFIG_NFS_COMMON=y
>>
>>
>> 3. Server (NFSD) configuration
>> Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit 22356f44, mm: Place preemption point in do_mlockall() loop)
>>
>>
>> 4. workaround
>>
>> Reverting the commit 4bdc33ed resolves my issue, NFS boot is working then.
>> I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
>
>So when you say you revert that commit, you mean you revert it on your
>*server*, right? You're not changing the client at all throughout these
>tests?

Right. I reverted the commit on my server, while client is same throughout the tests.

>
>A network trace might be interesting: so, on the server, run
>
>tcpdump -s0 -wtmp.pcap -ieth0
>
>(replace eth0 by the right network interface), then try booting the
>client and after the client fails, kill tcpdump and send us a copy of
>tmp.pcap.
>
>(And also you might want to fire up "wireshark tmp.pcap" and take a look
>yourself--you'll probably see something like a version mismatch error in
>the network traffic.)
>
>--b.

I've attached two tcpdump files.
In the dump, 165.213.88.238 is IP address for NFS client (embedded box with 2.6.35 kernel), and 192.168.64.128 is for NFS server (running latest git kernel with and without the commit revert)

* tmp_good_filtered.pcap
- latest linus git tree + commit 4bdc33ed reverted
- NFS boot is working

* tmp_bad_filtered.pcap
- latest linus git tree
- NFS boot doesn't work

In error case, I can see following message from wireshark packet window ;

Accept State: remote can't support version # (2)
Program Version (Minimum): 3
Program Version (Maximum): 4


And I forgot to attach my config of NFS server. Here it is.

# grep NFS .config
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
# CONFIG_NFS_V4_2 is not set
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_V4_SECURITY_LABEL is not set
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y

# rpm -qa|grep nfs
nfs-utils-1.2.8-4.0.fc19.i686
libnfsidmap-0.25-5.fc19.i686

# cat /etc/exports
/home/NFSROOT_mmc/ 165.213.88.238(rw,no_root_squash,sync)


Thanks,
Jongman Heo.

Attachment: tmp_good_filtered.pcap
Description: Binary data

Attachment: tmp_bad_filtered.pcap
Description: Binary data