Re: [PATCH v2] docs: Use make invocation's -j argument for parallelism

From: Mauro Carvalho Chehab
Date: Tue Sep 24 2019 - 13:02:19 EST


Em Mon, 23 Sep 2019 15:40:41 -0700
Kees Cook <keescook@xxxxxxxxxxxx> escreveu:

> On Sun, Sep 22, 2019 at 02:03:31PM -0600, Jonathan Corbet wrote:
> > On Thu, 19 Sep 2019 14:44:37 -0700
> > Kees Cook <keescook@xxxxxxxxxxxx> wrote:
> >
> > > While sphinx 1.7 and later supports "-jauto" for parallelism, this
> > > effectively ignores the "-j" flag used in the "make" invocation, which
> > > may cause confusion for build systems. Instead, extract the available
> >
> > What sort of confusion might we expect? Or, to channel akpm, "what are the
> > user-visible effects of this bug"?
>
> When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not
> parallelized.

Sphinx supports parallel builds for a while. With pre-1.7, you could do
something like:

make SPHINXOPTS="-j16" htmldocs

Yet, on my experiences on big machines (tested here with Xeon with 40 and 64
CPU threads), parallel build doesn't actually benefit with values higher than
-j5 to -j8, with pre-1.7.

Sphinx 1.7 and higher seem to have improved a lot with "-jauto"
(although I didn't time it comparing with -j5 or -j8 on a big server).

> When I run "make htmldocs -j8" with 1.7+ sphinx, it uses
> all my CPUs instead of 8. :)

This should do the trick:

make SPHINXOPTS="-j8" htmldocs

But yeah, IMHO, the best is if it could honor the Makefile flag:

make -j8 htmldocs

If SPHINXOPTS doesn't contain "-j".

> > > + -j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \
> >
> > This (and the shebang line in the script itself) will cause the docs build
> > to fail on systems lacking Python 3. While we have talked about requiring
> > Python 3 for the docs build, we have not actually taken that step yet. We
> > probably shouldn't sneak it in here. I don't see anything in the script
> > that should require a specific Python version, so I think it should be
> > tweaked to be version-independent and just invoke "python".
>
> Ah, no problem. I can fix this. In a quick scan it looked like sphinx
> was python3, but I see now that's just my install. :)

On Fedora 30, both python2 and python3 versions are available:

python2-sphinx.noarch : Python documentation generator
python3-sphinx.noarch : Python documentation generator

However, if one tries to install it without specifying "2" or "3", it
defaults to python2 version:

$ sudo dnf install python-sphinx
...
Installing:
python2-sphinx noarch 1:1.8.4-1.fc30 fedora 1.8 M

AFAIKT, this also applies when distro upgrades takes place: upgrading
a python2 sphinx from Fedora 30 to Rawhide will very likely keep
the python2 version.

Thanks,
Mauro