Re: Kernel docs: muddying the waters a bit

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 10:59:36 -0600
Jonathan Corbet <corbet@xxxxxxx> escreveu:

> On Wed, 4 May 2016 13:50:35 -0300
> Mauro Carvalho Chehab <mchehab@xxxxxxxxxxxxxxx> wrote:
> 
> > Em Wed, 4 May 2016 19:13:21 +0300
> > Jani Nikula <jani.nikula@xxxxxxxxx> escreveu:  
> > > I think we should go for vanilla sphinx at first, to make the setup step
> > > as easy as possible for everyone.    
> > 
> > Vanilla Sphinx doesn't work, as reST markup language is too limited
> > to support the docs we have. So, we should either push the needed
> > extensions to Sphinx reST or find a way to put it at Kernel tree
> > without causing too much pain for the developers, e. g. as something
> > that just doing "make htmldoc" would automatically use such extensions
> > without needing to actually install the extensions.  
> 
> It works for everything except the extended media book, right?  Or is there
> something I'm missing here?

As far as I know, yes, but I didn't check the other documentation
to be sure.

>  I am very much interested in having one system
> for *all* our docs, but we wouldn't attempt to convert a document can't be
> expressed in sphinx.

Looking from the other side, keeping only one document as DocBook
and having everything else converted to some markup solution seem
equally a bad solution, as, in the end of the day, we'll become
dependent of two different tool chains to produce document.

Also, media documentation is not just one more documentation. It is
the biggest one we have, and that has more changes than any other
documentation under Documentation/DocBook:

$ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
116
$ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l
179

It also is more than twice the size of the other DocBook docs:

$ wc -l $(ls *.tmpl|grep  media) `find media/ -type f`|tail -1
  82275 total
$ wc -l $(ls *.tmpl|grep -v media)|tail -1
  29568 total

E. g. media corresponds to 60% of the number of patches and 73% of
the DocBook stuff.

Not converting it to the new "official" Kernel markup language
would would mean that developers that work on more than the
media subsystem would need to know two different, incompatible
dialects to produce documentation (plus kernel-doc). That sounds
messy and it is an extra penalty to media developers. Some may
even start to think that that writing bad docs would be a good thing.

> > > Even if it means still doing that ugly
> > > docproc step to call kernel-doc. We can improve from there, and I
> > > definitely appreciate your work on making this work with sphinx
> > > extensions.    
> > 
> > I disagree: We should not cause documentation regressions by
> > producing incomplete documentation and losing tables because of
> > such conversion.  
> 
> > The quality of the documentation after the change should be *at least*
> > equal to the one we current produce, never worse.  
> 
> Agreed; that's why I think the existing DocBook mechanism should stay in
> place until that document will be improved by the change. But I'd rather
> not hold up the entire process for one book, especially since pushing that
> process forward can only help in dealing with those final problems.

Yeah, in practice, we won't be converting everything on just one
Kernel version. Yet, we need at least an strategy and a plan that would
allow converting everything to the new markup language.

Provided that we would merge Sphinx support on 4.8, I would like to convert
the media documentation just after merging the new toolchain upstream, 
in order to minimize the conversion impact and to not lose developers
and good developer's documentation in the process.

However, such work can only be started if we have a way to work with the
required extensions, as we're deciding to use a markup language that has
incomplete table support.

So, IMHO, we should have a way to run the required extensions in a
painless way before doing the merge.

> > > That said, how would it work to include the kernel-doc extension in the
> > > kernel source tree? Having things just work if sphinx is installed is
> > > preferred over requiring installation of something extra from pypi. (I
> > > know this may sound backwards for a lot of projects, but for kernel I'm
> > > pretty sure this is how it should be done.)    
> > 
> > Yeah, using pypi seems an additional undesired step. Aren't there
> > any way to make python to use an additional extension at the
> > Kernel tree without needing to install it?  
> 
> Well, sphinx has to come from somewhere too.  But yes, we should be able to
> carry extensions in the kernel tree.  But I would still rather upstream it
> (to sphinx) if possible, so we're not stuck trying to maintain it across
> several sphinx releases.  I don't know how volatile their interfaces are,
> but it would be, in any case, the analog of an out-of-tree module...

Yeah, the best would be to put everything we need at Sphinx upstream,
but assuming that they agree today to add the extensions we need,
we would still need to carry those extensions at the Kernel tree for a
while, in order to wait for Sphinx to release a version with those
versions, and such new version is not merged back at the distributions
we use.

Thanks,
Mauro