Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot

[Rob Landley]

On 05/22/2017 07:05 AM, Yury Norov wrote:
> Hi Rob, 
> 
> I found that next-20170522 fails to boot on arm64 machine with the
> following log:

I don't know anything about your kernel config (is CONFIG_DEVTMPFS_MOUNT
enabled or disabled?) or what userspace you're booting with, but it
seems I can guess:

> [...]
> [4.179509] Freeing unused kernel memory: 1088K
> Loading, please wait...

At this point, the kernel has launched init and your userspace is
running. During that boot,the kernel mounted devtmpfs on /dev (you
edited the part where it did that out of your boot log), but the next line:

> mount: mounting udev on /dev failed: Device or resource busy

has an error that says you already have devtmpfs mounted on /dev, and
your userspace tries to mount devtmpfs on it _again_ and it fails
because you can't mount the exact same filesystem over itself due to a
sanity check in the kernel in fs/namespace.s line 2475 or so:

        /* Refuse the same filesystem on the same mount point */
        err = -EBUSY;
        if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb &&
            path->mnt->mnt_root == path->dentry)
                goto unlock;

> W: devtmpfs not available, falling back to tmpfs for /dev
> Couldn't get a file descriptor referring to the console

At which point your userspace does a "fixup" mounting something else
over the previously working devtmpfs, which succeeds (because you're
mounting a _different_ filesystem and not hitting the above sanity
test), thus breaking your userspace.

> Begin: Loading essential drivers ... done.
> Begin: Running /scripts/init-premount ... done.
> Begin: Mounting root file system ... Begin: Running
> /scripts/local-top ... done.
> chvt: can't open console

And then your userspace didn't notice for a while.

> Gave up waiting for root device.  Common problems:
>  - Boot args (cat /proc/cmdline)
>    - Check rootdelay= (did the system wait long enough?)
>    - Check root= (did the system wait for the right device?)
>  - Missing modules (cat /proc/modules; ls /dev)
> chvt: can't open console
> ALERT!  /dev/sda does not exist.  Dropping to a shell!
> Couldn't get a file descriptor referring to the console

And then it died.

> BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
> Enter 'help' for a list of built-in commands.
> 
> (initramfs)
> 
> Bisect points to your patch (attached below). If I revert it, everything
> becomes fine. If you need to know something more about my environment,
> feel free to ask me.

You were inappropriately specifying CONFIG_DEVTMPFS_MOUNT in your
config, now that it's no longer being ignored your init script is having
an allergic reaction to it. Either yank it from your config or fix your
userspace. It looks to me like my patch triggered a bug in your setup.

Your userspace mounted a tmpfs over /dev when it couldn't mount a second
identical instance of devtmpfs over itself. If you had a static /dev in
initramfs but didn't configure _in_ devtmpfs to your kernel, your broken
error path would have taken that out too with a pointless tmpfs mount.

By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_?
That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.)

Feel free to send more context if you think I'm wrong about this.

> Yury

Rob