Re: [PATCH 00/10] Documentation/Sphinx

[Jani Nikula]

On Mon, 30 May 2016, Markus Heiser <markus.heiser@xxxxxxxxxxx> wrote:
> Here my 5cents about Jani's patch series:
>
> 1. Migration implementations should not be a part of the kernel tree

If you're referring to the conversion scripts, I don't care either
way. It's probably helpful to have them until everything is converted,
and we can dispose of them afterwards.

> 2. sed/pandoc migration fits not for all the XML documentation and has disadvantages.
>    Jonathan asked it before: http://article.gmane.org/gmane.linux.documentation/37533
>
>    I repeat myself: Summarize, why should one prefer this tools over pandoc + sed?
>
>    * Pandoc coverage is less on reading and writing, this is where 
>      dbxml comes into play
>
>      - reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23
>   
>      - writing reST has many bugs and leaks (you fixed some of them with sed)
>
>    * Pandoc does not support external entities (linux-tv), covered by dbxml
>
>    * dbxml brings the ability to chunk one large XML book into small 
>      reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking
>
>    * dbxml lets you manipulate the XML source before you convert it to reST
>
>      this might helpfull e.g. if you have to convert single-column informal-tables 
>      to lists or other things ... in short; dbxml and it's hooks are the key to hack
>      everything you need in a full automated DocBook-->reST migration workflow.

I am not proposing to merge the documents that I've converted mostly as
samples in the branch. I needed something to demonstrate the build is
sane.

The authors of the DocBook documents should make the conversions as they
see fit, when they see fit, with the tools they see fit, probably with
some manual work on top.

> 3. we also discussed before, that ASCII art tables are ugly, because the produce
>    confusing diffs, for this I wrote the flat-table directive (dbxml migrates tables
>    to flat-tables / pandoc can't). 
>    https://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table

See the above. Any authors that think the Sphinx support I've added is
good enough can go ahead and switch.

I think it's safe for me to say the GPU documents won't wait for further
extension directives or conversion tools. Some others will definitely
want to have the flat table extension before switching.

>> With this, we can put any .rst files (including ones that have
>> kernel-doc directives) anywhere under Documentation, add a link to them
>> in Documentation/index.rst table of contents, and it will just work. It
>> can't get much simpler than that.
>
> 4. We discussed it / I already mentioned that each document shipped in it's own
>    sphinx project. Bundling all documents into one sphinx-project will work for
>    4 or 5  small documents, but not for the whole documentation. BTW all XML 
>    documents are currently separated DocBook projects .. so why should we merge
>    them into one big project? Making one index-file for the different and small 
>    ".txt" files seems OK, but not for the XML docs.

FWIW I locally converted all the DocBook documents (except media) and it
works just fine, and to me it looks like exactly what we should
have. One of the goals was to have nice cross-referencing between the
documents (e.g. from GPU to kernel or device driver API). And it works.

This does not exclude having *additional* indexes or Sphinx config files
for subsystems or subprojects to build a subset of the documentation for
specific needs. It's up to the authors of the documents to decide.

For PDF documents, adding the documents separately in pdf_documents
seems to be the right thing to do.

> 5. In general, the markup of the linux kernel's source code comments remains
>    unchanged and the reST markup within the comments is passed through the
>    output. A closer lookup to the *kernel-doc* and *reST* markup revals, that
>    there are some conflicts between reST (inline) markup and kernel-doc
>    markup. Determined by the historical development of the kernel-doc comments, the
>    *classic* kernel-doc comments contain characters like ``*`` or strings with
>    e.g. leading/trailing underscore (``_``), which are inline markups in
>    reST. Here a schort example from a *classic* comment::
>
>         <SNIP> -----
>         * In contrast to the other drm_get_*_name functions this one here returns a
>         * const pointer and hence is threadsafe.
>         <SNAP> -----
>
>    In reST markup, the wildcard in the string ``drm_get_*_name`` has to be
>    masked: ``drm_get_\\*_name``. Some more examples from reST markup:
>
>    * Emphasis "*":  like ``*emphasis*`` or ``**emphasis strong**``
>    * Leading "_" :  is a *anchor* in reST markup (``_foo``).
>    * Trailing "_:  is a reference in reST markup (``foo_``).
>    * interpreted text: "`"
>    * inline literals: "``"
>    * substitution references: "|"
>
>     These special strings has to be masked in the output and can't be used as
>     *plain-text markup*.  To get in use of the fully reST markup (stop masking
>     special characters) we need some options in the sources documentation, comments
>     like
>
>        /* parse-markup: reST */
>
>     which influence the behavior.

I find it totally unacceptable to require explicitly marking kernel-doc
comments or source files as being reStructuredText.

Note that it's all opt-in already. If you add a .rst file that includes
kernel-doc via the kernel-doc extension, you better make sure the
comments parse as reStructuredText and render nicely. I'm willing to do
much of the job for all the things that I care about.

Besides, if you look at the results, you'll find it looks mostly good
without any fixes. In the sample documents, I've erred on the side of
having a few markup hickups here and there while most of it works
perfectly well as reStructuredText. I think this is exactly what we
should do, declare it all reStructuredText and fix issues as we go.

>> Sites like https://readthedocs.org/ can build the documentation,
>> including kernel-doc, without extra tweaks. As a whole, the build
>> becomes much simpler.
>
>
> 6. This fail assessments I also had before. RTD has limits in 
>    resources and in flexibility, that's why I moved to github pages
>
>    https://github.com/return42/sphkerneldoc/issues/1

I'm just personally using Read the Docs to ensure they can build the
documentation as "pure" Sphinx (they won't use the Makefiles), and so I
don't have to host the docs anywhere myself. Sure, I can't add *all* the
files there because it exceeds the build limits. They might cater for us
if they want to carry the kernel documentation going forward, but most
likely the output should be at kernel.org anyway. (Plus we'll have
freedesktop.org and 01.org for the GPU documentation too.)

> I will stop here ... I think it is good that everyone make its own
> experience, BUT ...
>
>   IMO it is a misjudgment to think that changing the 
>   markup and it's toolchain is only a series of patches.
>
> Sorry if my words are unpleasant, this is not my intend, but IMO *the 
> view of the whole* and concepts are missed.

I'm not sure what to say.

For one thing, a concrete series of patches to add Sphinx support to the
kernel, integrating it with the build system and kernel-doc and
everything is *exactly* what is needed. This is what I've done. It's
here now. It works. Anyone can take my git tree, run 'make htmldocs' and
see the results for themselves.

>From your message it remains unclear to me what "view of the whole" and
"concepts" and "comprehensive solution" I have missed.

> Many of the facts mentioned above have been covered in my POC at 
> https://github.com/return42/sphkerneldoc ... On others, 
> like 5. I'am working on ....
>
>> I've had a few moments of spare time to look into Sphinx. 
>
> ... a comprehensive solution needs time and will not be done in a
> hurry. Please give me a week or may be two, then I could present
> a much more comprehensive solution.

Please do not underestimate the productivity of my moments of spare
time. ;)

My view of the whole is that we've been talking about adding lightweight
markup support for the better part of a year now, and I'm getting pretty
tired of talking...


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center