Re: Using overlay on top of overlay

[Miklos Szeredi]

On Wed, Dec 13, 2017 at 11:10 AM, RafaÅ MiÅecki <zajec5@xxxxxxxxx> wrote:
> Hi,
>
> For few kernel releases overlay supported using it on top of another
> overlay.
>
> FWIW there was a regression related to this "feature" in 4.2:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4bacc9c9234c7c8eec44f5ed4e960d9f96fa0f01
> but it was fixed:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c8a47df36d72ace8cf78eb6c228aa0f8027d3c2
>
>
> Starting with kernel 4.4 this "feature" seems to be disabled explicitly:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=76bc8e2843b66f8205026365966b49ec6da39ae7
>
> Disabling commit says that it "does not make sense" but actually there
> seem to be few projects that are looking for such a solution, see:
> 1) https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1629509
> 2) https://bugs.centos.org/view.php?id=12986
> 3) https://github.com/rkt/rkt/issues/1537
>
> I can't really elaborate on above cases but I have a similar need for
> my OpenWrt system running LXC (Linux Containers).
>
> First of all OpenWrt uses overlay to "merge" squashfs and some writable
> filesystem partition. It needs squashfs due to dealing with devices with
> really small flashes (down to 4 - 8 MiB). Then it stores all its LXC
> containers in the /srv/lxc/.
>
> Secondly LXC itself uses overlay feature in few places, e.g.
> 1) lxc.rootfs with the overlayfs:lower:upper syntax
> 2) lxc.mount.entry with the fstab syntax
>
> As you can guess LXC simply assumes overlay can be used and doesn't
> check if the host system uses it for directory holding container data.
> This breaks LXC because kernel refuses mounting dir with some message
> like:
> overlayfs: filesystem on '/foo' not supported as upperdir
>
>
> I managed to find a workaround for this. As OpenWrt uses:
> overlayfs:/overlay on / type overlay
> (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
> I can simply replace
> upperdir=/srv/lxc/foo/upper
> with
> upperdir=/overlay/upper/srv/lxc/foo/upper
> (same for workdir) but this isn't really a user friendly solution.
>
>
> I would like to ask if use cases I described above make any sense to
> you?
> Is there any chance someone could work on a proper support for overlay
> on top of overlay and enabling it again?

Overlayfs as upper layer never worked properly.  Think of what happens
when a file existing on a lower layer (in the outer overlay) is
removed.  We try to create a whiteout on the upper of the outer
overlay (which is the inner overlay).  That is denied, because the
representation would conflict with the representation of a whiteout on
the upper of the inner overlay.

We could do some "escaping" of whiteouts and "trusted.ovlerlay" xattrs
to represent these xattrs on the overlay, but this has not yet been
implemented.

Once that is done, I don't see other major obstacles to doing
overlayfs upper over overlayfs.

Thanks,
Miklos