Re: Patch 0727d35de ("Make initramfs honor CONFIG_DEVTMPFS_MOUNT") breaks boot

[Yury Norov]

On Mon, May 22, 2017 at 09:07:54PM -0500, Rob Landley wrote:
> On 05/22/2017 07:05 AM, Yury Norov wrote:
> > Hi Rob, 
> > 
> > I found that next-20170522 fails to boot on arm64 machine with the
> > following log:
> 
> I don't know anything about your kernel config (is CONFIG_DEVTMPFS_MOUNT
> enabled or disabled?) or what userspace you're booting with, but it
> seems I can guess:
> 
> > [...]
> > [4.179509] Freeing unused kernel memory: 1088K
> > Loading, please wait...
> 
> At this point, the kernel has launched init and your userspace is
> running. During that boot,the kernel mounted devtmpfs on /dev (you
> edited the part where it did that out of your boot log), but the next line:
> 
> > mount: mounting udev on /dev failed: Device or resource busy
> 
> has an error that says you already have devtmpfs mounted on /dev, and
> your userspace tries to mount devtmpfs on it _again_ and it fails
> because you can't mount the exact same filesystem over itself due to a
> sanity check in the kernel in fs/namespace.s line 2475 or so:
> 
>         /* Refuse the same filesystem on the same mount point */
>         err = -EBUSY;
>         if (path->mnt->mnt_sb == newmnt->mnt.mnt_sb &&
>             path->mnt->mnt_root == path->dentry)
>                 goto unlock;
> 
> > W: devtmpfs not available, falling back to tmpfs for /dev
> > Couldn't get a file descriptor referring to the console
> 
> At which point your userspace does a "fixup" mounting something else
> over the previously working devtmpfs, which succeeds (because you're
> mounting a _different_ filesystem and not hitting the above sanity
> test), thus breaking your userspace.
> 
> > Begin: Loading essential drivers ... done.
> > Begin: Running /scripts/init-premount ... done.
> > Begin: Mounting root file system ... Begin: Running
> > /scripts/local-top ... done.
> > chvt: can't open console
> 
> And then your userspace didn't notice for a while.
> 
> > Gave up waiting for root device.  Common problems:
> >  - Boot args (cat /proc/cmdline)
> >    - Check rootdelay= (did the system wait long enough?)
> >    - Check root= (did the system wait for the right device?)
> >  - Missing modules (cat /proc/modules; ls /dev)
> > chvt: can't open console
> > ALERT!  /dev/sda does not exist.  Dropping to a shell!
> > Couldn't get a file descriptor referring to the console
> 
> And then it died.
> 
> > BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
> > Enter 'help' for a list of built-in commands.
> > 
> > (initramfs)
> > 
> > Bisect points to your patch (attached below). If I revert it, everything
> > becomes fine. If you need to know something more about my environment,
> > feel free to ask me.
> 
> You were inappropriately specifying CONFIG_DEVTMPFS_MOUNT in your
> config, now that it's no longer being ignored your init script is having
> an allergic reaction to it. Either yank it from your config or fix your
> userspace. It looks to me like my patch triggered a bug in your setup.
> 
> Your userspace mounted a tmpfs over /dev when it couldn't mount a second
> identical instance of devtmpfs over itself. If you had a static /dev in
> initramfs but didn't configure _in_ devtmpfs to your kernel, your broken
> error path would have taken that out too with a pointless tmpfs mount.

CONFIG_DEVTMPFS_MOUNT is enabled on my machine, so I think your
suggestion is correct. But I didn't do that specifically - I run
almost default kernel based on Ubuntu 14.04 config and environment.

Grepping the kernel code shows that arc, arm, arm64, m86k, metag, 
mips, nios2, openrisc, parisc, powerpc, sh, tile, um, x86 and xetensa
enable it by default. So it means for me that (at least) users why run
Ubuntu 14.04 will have bricked system one day after updating the
kernel.

If you say that currently CONFIG_DEVTMPFS_MOUNT is ignored by kernel,
I think you cannot relay on it anymore because people may have it
enabled or disabled randomly. So the proper way is to remove broken
config option and introduce new one. BTW, I see it is used once in
drivers/base/devtmpfs.c.
 
> By the way, _why_ are you mounting a tmpfs over /dev on _initramfs_?
> That can already be tmpfs. (Commits 137fdcc18a59 through 6e19eded3684.)

Have no idea. This is how default Ubuntu works.

Yury