[BUG BISECT] NFSv4 client fails on Flush Journal to Persistent Storage

[Krzysztof Kozlowski]

Hi,

When booting my boards under recent linux-next, I see failures of systemd:

[FAILED] Failed to start Flush Journal to Persistent Storage.
See 'systemctl status systemd-journal-flush.service' for details.
         Starting Create Volatile Files and Directories...
[**    ] A start job is running for Create Vâ [  223.209289] nfs:
server 192.168.1.10 not responding, still trying
[  223.209377] nfs: server 192.168.1.10 not responding, still trying

Effectively the boards fails to boot. Example is here:
https://krzk.eu/#/builders/1/builds/2157

This was bisected to:
commit 37ac86c3a76c113619b7d9afe0251bbfc04cb80a
Author: Chuck Lever <chuck.lever@xxxxxxxxxx>
Date:   Fri May 4 15:34:53 2018 -0400

    SUNRPC: Initialize rpc_rqst outside of xprt->reserve_lock

    alloc_slot is a transport-specific op, but initializing an rpc_rqst
    is common to all transports. In addition, the only part of initial-
    izing an rpc_rqst that needs serialization is getting a fresh XID.

    Move rpc_rqst initialization to common code in preparation for
    adding a transport-specific alloc_slot to xprtrdma.

    Signed-off-by: Chuck Lever <chuck.lever@xxxxxxxxxx>
    Signed-off-by: Anna Schumaker <Anna.Schumaker@xxxxxxxxxx>


Bisect log attached. Full configuration:
1. exynos_defconfig
2. ARMv7, octa-core, Exynos5422 and Exynos4412 (Odroid XU3, U3 and others)
3. NFSv4 client (from Raspberry Pi)

Let me know if you need any more information.

Best regards,
Krzysztof
git bisect start
# good: [c64d4419a17cfb39a5b573f9016cd02ade4c9a64] mtd: cfi_cmdset_0002: Change erase one block to enable XIP once
git bisect good c64d4419a17cfb39a5b573f9016cd02ade4c9a64
# good: [56c6855c81c8a6828b5d65aa974cd50f4b67760c] mtd: spi-nor: Add Micron MT25QU02 support
git bisect good 56c6855c81c8a6828b5d65aa974cd50f4b67760c
# bad: [2b70a7bd4673b6dcf2763888d0b172a40dd49434] Merge remote-tracking branch 'block/for-next'
git bisect bad 2b70a7bd4673b6dcf2763888d0b172a40dd49434
# bad: [25350cbeef4a3ec943754c4aa4c8ac1aaa64c7a2] Merge remote-tracking branch 'nand/nand/next'
git bisect bad 25350cbeef4a3ec943754c4aa4c8ac1aaa64c7a2
# good: [311da4975894aab7a4bb94aa83f38f052d7ffda4] Merge branch 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm
git bisect good 311da4975894aab7a4bb94aa83f38f052d7ffda4
# bad: [7d470f95b4ccd3244759524f6cc9a76a2a34a9c6] Merge remote-tracking branch 'printk/for-next'
git bisect bad 7d470f95b4ccd3244759524f6cc9a76a2a34a9c6
# good: [8a3ab2f38f1669e3be6433a1f6b82a077b38c4c7] brcmfmac: trigger memory dump upon firmware halt signal
git bisect good 8a3ab2f38f1669e3be6433a1f6b82a077b38c4c7
# good: [7fa76d777ec53eeece1546b737a3b93b37639575] netdevsim: Add extack error message for devlink reload
git bisect good 7fa76d777ec53eeece1546b737a3b93b37639575
# bad: [1fe087da59fde2cce4cf8e0a1c78b3279fb7ea44] Merge remote-tracking branch 'fbdev/fbdev-for-next'
git bisect bad 1fe087da59fde2cce4cf8e0a1c78b3279fb7ea44
# bad: [9d85b2fee6616994759eb4599c886af510ded175] Merge remote-tracking branch 'pci/next'
git bisect bad 9d85b2fee6616994759eb4599c886af510ded175
# good: [13fbadcd512c225c907d6e8147fb48a88114bf03] Merge branch 'pci/sparc'
git bisect good 13fbadcd512c225c907d6e8147fb48a88114bf03
# good: [741f8e7ecc2c6414cff442ec8eb07dcfe4481533] Merge branch 'lorenzo/pci/hv'
git bisect good 741f8e7ecc2c6414cff442ec8eb07dcfe4481533
# good: [1c2bef0a3fd14287a66edd7ead57fd2e439485a2] Merge branch 'lorenzo/pci/rcar'
git bisect good 1c2bef0a3fd14287a66edd7ead57fd2e439485a2
# good: [e52d38f4abf49f8b63a6ad0ce21e5f495c15897f] Merge branch 'lorenzo/pci/rockchip'
git bisect good e52d38f4abf49f8b63a6ad0ce21e5f495c15897f
# good: [73144d77cb87d60b4bcab6992a62d6787b09dcf0] Merge branch 'lorenzo/pci/vmd'
git bisect good 73144d77cb87d60b4bcab6992a62d6787b09dcf0
# good: [82e1719c4cd65bd7f7847d6c02376cfca3d5e793] PCI: Clean up whitespace in quirks.c
git bisect good 82e1719c4cd65bd7f7847d6c02376cfca3d5e793
# good: [0ecda3a087462eb89c1d9227deea998d8cd014e8] Merge branch 'pci/kconfig'
git bisect good 0ecda3a087462eb89c1d9227deea998d8cd014e8
# good: [488ad6d3678beee65bcd74e6a9764bd7cee9d3d3] Merge branch 'pci/trivial'
git bisect good 488ad6d3678beee65bcd74e6a9764bd7cee9d3d3
# good: [885892fb378dc096693557ba4f2b875188619b36] mlx4_core: restore optimal ICM memory allocation
git bisect good 885892fb378dc096693557ba4f2b875188619b36
# good: [488ad6d3678beee65bcd74e6a9764bd7cee9d3d3] Merge branch 'pci/trivial'
git bisect good 488ad6d3678beee65bcd74e6a9764bd7cee9d3d3
# good: [0ecda3a087462eb89c1d9227deea998d8cd014e8] Merge branch 'pci/kconfig'
git bisect good 0ecda3a087462eb89c1d9227deea998d8cd014e8
# good: [82e1719c4cd65bd7f7847d6c02376cfca3d5e793] PCI: Clean up whitespace in quirks.c
git bisect good 82e1719c4cd65bd7f7847d6c02376cfca3d5e793
# good: [1c2bef0a3fd14287a66edd7ead57fd2e439485a2] Merge branch 'lorenzo/pci/rcar'
git bisect good 1c2bef0a3fd14287a66edd7ead57fd2e439485a2
# good: [87cb5ac9cece9f0f75d0532fa2afcbf871f6b72e] Merge remote-tracking branch 'arm-soc/for-next'
git bisect good 87cb5ac9cece9f0f75d0532fa2afcbf871f6b72e
# good: [64eec192d609531b0c173c5b4885832372fb2a4c] Merge remote-tracking branch 'powerpc/next'
git bisect good 64eec192d609531b0c173c5b4885832372fb2a4c
# good: [dd8c1fd2071de3a02ea60e3fa68be24c1e89945e] Merge remote-tracking branch 'jfs/jfs-next'
git bisect good dd8c1fd2071de3a02ea60e3fa68be24c1e89945e
# bad: [6125a3dab21f1939dae7e836105dea0c9c465db4] Merge remote-tracking branch 'orangefs/for-next'
git bisect bad 6125a3dab21f1939dae7e836105dea0c9c465db4
# good: [3f0b3cf46e0542ac4b4241c579b944b755d11b67] NFS: Filter cache invalidation when holding a delegation
git bisect good 3f0b3cf46e0542ac4b4241c579b944b755d11b67
# good: [28771950c592482ee86cb1c3b661688aec3c0d7d] svcrdma: Fix incorrect return value/type in svc_rdma_post_recvs
git bisect good 28771950c592482ee86cb1c3b661688aec3c0d7d
# bad: [8335640cf89faa0f4e39e73e314f3f3a22d776f3] xprtrdma: Add trace_xprtrdma_dma_map(mr)
git bisect bad 8335640cf89faa0f4e39e73e314f3f3a22d776f3
# bad: [0e0b854cfb3302b1907e9d3a927469b95710238f] xprtrdma: Clean up Receive trace points
git bisect bad 0e0b854cfb3302b1907e9d3a927469b95710238f
# bad: [0e0b854cfb3302b1907e9d3a927469b95710238f] xprtrdma: Clean up Receive trace points
git bisect bad 0e0b854cfb3302b1907e9d3a927469b95710238f
# good: [75bc37fefc4471e718ba8e651aa74673d4e0a9eb] Linux 4.17-rc4
git bisect good 75bc37fefc4471e718ba8e651aa74673d4e0a9eb
# bad: [0e0b854cfb3302b1907e9d3a927469b95710238f] xprtrdma: Clean up Receive trace points
git bisect bad 0e0b854cfb3302b1907e9d3a927469b95710238f
# good: [914fcad9873cbd46e3a4c3c31551b98b15a49079] xprtrdma: Fix max_send_wr computation
git bisect good 914fcad9873cbd46e3a4c3c31551b98b15a49079
# bad: [a9cde23ab7cdf5e4e93432dffd0e734267f2b745] SUNRPC: Add a ->free_slot transport callout
git bisect bad a9cde23ab7cdf5e4e93432dffd0e734267f2b745
# bad: [37ac86c3a76c113619b7d9afe0251bbfc04cb80a] SUNRPC: Initialize rpc_rqst outside of xprt->reserve_lock
git bisect bad 37ac86c3a76c113619b7d9afe0251bbfc04cb80a
# first bad commit: [37ac86c3a76c113619b7d9afe0251bbfc04cb80a] SUNRPC: Initialize rpc_rqst outside of xprt->reserve_lock