Re: [PATCH 6/6] Add dockerfile

[Dmitry Monakhov]

Theodore Ts'o <tytso@xxxxxxx> writes:

> On Fri, Aug 19, 2016 at 12:27:43PM +0300, Dmitry Monakhov wrote:
>> Actually main idea is to have build environment for xfstests-bld development
>> plus automatic builds gives us CI. This is important because
>> xfstests-blk build is silently broken since May 30 due to 82c3a179
>> (fix proposed: https://github.com/dmonakhov/xfsprogs/commit/5dc7a0805d6b7a265863413a8e1b7acb191d5280)
>> But this was uncovered because most people do not build xfstests-bld
>> from scratch very often, we all too busy. Let's machine do that :)
>> In fact other good candidate for CI is https://travis-ci.com/
>
> Actually, I'm rebuilding all the time, but I'm not upgrading to the
> latest bleeding edge version of xfsprogs all the time.  In fact, I'm
> currently still using xfsprogs v4.5.0, since that's plenty for the
> testing I'm doing.  I have historically pinned the commit Id's for
> certain projects where newer versions were either (a) known to be
> flaky, or (b) known to be broken in some way -- that's why we
> currently have explicit commit ID's for fio and quota.  Perhaps I
> should add one for xfsprogs as well, so that we only upgrade to newer
> versions when it known to be needed and where it's helpful.
Yes. That is reasonable point.
My original idea was to check that any commit of xfstests-bld will
healthy with default config. And if github/docker build farm may do it
for us for free is it not bad.
>
>> Once we have Dockerfile which encapsulates all dependencies
>> one can create xfstets-blk from scratch simply typing
>> $ docker build -t my-xfstests-bld-image --file Dockerfile
>> Or use precreated environment
>> $ docker pull XXX/xfstests-bld
>
> If you use the precreated environment that will have a fixed commit
> ID.  So the advantage of doing continuous integration of the bleeding
> edge version of xfsprogs only works if you are using the "docker
> build" line, right.
>
>> Once image is created 
>> Log in to container and start hacking xfstest or e2fsprogs etc.
>> $ docker run -i -t XXX/xfstests-bld
>> $ emacs xfstest/tests/ext4/023 ....
>> $ kvm-xfstests --update-xfstest ext4/023
>> ...
>
> I'm not sure I see the advantage of doing this in a container, I
> guess.  I just do in my standard laptop environment today.
I can not because I laptop from famous thinkpad t430 series with
flaky SSD which starts to return EIO after intensive load.
https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T/T430s-Intel-SSD-520-180GB-issue/td-p/888083

Also. What about RH/SUSE/Fedora or even Gentoo people? They are exit :)
And if we give them just chance to try debian w/o complexities
maybe they will become debian adepts.
>
>> Other useful feature is ability to speed xfstests-bld by splitting
>> full test-set to groups and run kvm-xfstests instances in parallel.
>> For example:
>> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c 1k > 1k.log &
>> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c 4k > 4k.log &
>> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c dax > dax.log &
>> .....
>> w/o sand box they will conflict due to net ports (localhost:7500 etc)
>
> ... and because you will need separate scratch disks for each of the
> docker runs.
Space is not a problem for most people. Average disk size is more that 500Gb
>
> OK, I can see how that's certainly an advantage..  This is what I use
> gce-xfstests for, myself.  :-)
BTW: I would be good thing to have some sort of job preparation mode.
where test does not executed, but only listed. So one can get lists
of tests and then divide it to groups and run in parallel:
$ kvm-xfstests.sh -c 4k -g auto --list > all_tests.txt
# Next line divides all_tests.txt in 8 parts and run them parallel
$ parallel --joblog job.log -j 8 -m -a all_tests.txt "kvm-xfstests.sh -c 4k"
I'll send you a patch.
BTW: Sometimes many parallel jobs provoke BUGON on host kernel due to
some bug in KVM  :)
>
> The main thing I don't really like about the Dockerfile is that with
> the exception of the last use case, everyone would really need to have
> their own customized version of the Docker file, right?  And for
> someone who isn't Docker aware, I'm concerned that it might be very
> confusing to use, especially compared to all of the documentation I
> currently have for kvm-xfstests and gce-xfstests.
Absolutely no. Dockerfile is just script which used for container
creation(similar gen-image). As normal script it should be as generic
as possible. If I've embedded some ad-hoc references to my-config this
is just a common error than every scripts writers do :(
>
> Hmm, I'll have to think about this and play with it a bit more...
>
>      	       	  	      	       	    - Ted

Attachment: signature.asc
Description: PGP signature