Re: [PATCH 00/10] Documentation/Sphinx

[Markus Heiser]

Hi,

sorry for my temporary absence, I have been on holiday the last weeks :-)

Am 30.05.2016 um 11:10 schrieb Daniel Vetter <daniel.vetter@xxxxxxxx>:

> On Sun, May 29, 2016 at 10:33 PM, Jani Nikula <jani.nikula@xxxxxxxxx> wrote:
>> On Fri, 20 May 2016, Jani Nikula <jani.nikula@xxxxxxxxx> wrote:
>>> At this time I've put most effort into the configuration and build side
>>> of things, solving the problems described above, and handling missing
>>> tools and packages gracefully. There are still issues to be ironed out
>>> in a) the kernel-doc script rst output and b) the xml template to rst
>>> conversion. These are somewhat orthogonal from each other and the build,
>>> and I expect some hand-editing will be required in the end.
>> 
>> Jon -
>> 
>> I've added a significant amount of polish. It works and it looks
>> pretty. The code is in the "sphinx-wip" branch of [1], gitweb at [2],
>> and Read the Docs built copy at [3].
>> 
>> There's roughly 30 commits there that I think are getting ready for
>> merging. The rest is scripted conversion of some DocBook templates for
>> demonstration purposes; I think the final conversion should be done by
>> the authors of the documents when they are ready to ditch the DocBook
>> templates in the same commit.
>> 
>> I also think I've now spearheaded this pretty close to a point where
>> more benefits can be gained by merging the work to docs-next, and
>> collaborating there.
>> 
>> Let me know what you think.
> 
> I've looked both through the patches in Jani's branch and the
> resulting docs. Spotted 2 nitpicks, which are already fixed, and it
> looks imo _really_ good.
> 
> I think next steps is to get this merged into docs-next, with a stable
> tag, so that I can pull it into drm-misc. There's still piles of
> little polish to do to make it really pretty, but that's almost all in
> the kernel-doc comments or .rst files itself. I think for starting we
> could convert a few docs already, or just add the index.rst and leave
> conversions to maintainers. At least for gpu.tmpl I expect some need
> for cordination with ongoing work.
> -Daniel

Here my 5cents about Jani's patch series:

1. Migration implementations should not be a part of the kernel tree

2. sed/pandoc migration fits not for all the XML documentation and has disadvantages.
   Jonathan asked it before: http://article.gmane.org/gmane.linux.documentation/37533

   I repeat myself: Summarize, why should one prefer this tools over pandoc + sed?

   * Pandoc coverage is less on reading and writing, this is where 
     dbxml comes into play

     - reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23
  
     - writing reST has many bugs and leaks (you fixed some of them with sed)

   * Pandoc does not support external entities (linux-tv), covered by dbxml

   * dbxml brings the ability to chunk one large XML book into small 
     reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

   * dbxml lets you manipulate the XML source before you convert it to reST

     this might helpfull e.g. if you have to convert single-column informal-tables 
     to lists or other things ... in short; dbxml and it's hooks are the key to hack
     everything you need in a full automated DocBook-->reST migration workflow.

3. we also discussed before, that ASCII art tables are ugly, because the produce
   confusing diffs, for this I wrote the flat-table directive (dbxml migrates tables
   to flat-tables / pandoc can't). 
   https://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table

> With this, we can put any .rst files (including ones that have
> kernel-doc directives) anywhere under Documentation, add a link to them
> in Documentation/index.rst table of contents, and it will just work. It
> can't get much simpler than that.

4. We discussed it / I already mentioned that each document shipped in it's own
   sphinx project. Bundling all documents into one sphinx-project will work for
   4 or 5  small documents, but not for the whole documentation. BTW all XML 
   documents are currently separated DocBook projects .. so why should we merge
   them into one big project? Making one index-file for the different and small 
   ".txt" files seems OK, but not for the XML docs.

5. In general, the markup of the linux kernel's source code comments remains
   unchanged and the reST markup within the comments is passed through the
   output. A closer lookup to the *kernel-doc* and *reST* markup revals, that
   there are some conflicts between reST (inline) markup and kernel-doc
   markup. Determined by the historical development of the kernel-doc comments, the
   *classic* kernel-doc comments contain characters like ``*`` or strings with
   e.g. leading/trailing underscore (``_``), which are inline markups in
   reST. Here a schort example from a *classic* comment::

        <SNIP> -----
        * In contrast to the other drm_get_*_name functions this one here returns a
        * const pointer and hence is threadsafe.
        <SNAP> -----

   In reST markup, the wildcard in the string ``drm_get_*_name`` has to be
   masked: ``drm_get_\\*_name``. Some more examples from reST markup:

   * Emphasis "*":  like ``*emphasis*`` or ``**emphasis strong**``
   * Leading "_" :  is a *anchor* in reST markup (``_foo``).
   * Trailing "_:  is a reference in reST markup (``foo_``).
   * interpreted text: "`"
   * inline literals: "``"
   * substitution references: "|"

    These special strings has to be masked in the output and can't be used as
    *plain-text markup*.  To get in use of the fully reST markup (stop masking
    special characters) we need some options in the sources documentation, comments
    like

       /* parse-markup: reST */

    which influence the behavior.

> Sites like https://readthedocs.org/ can build the documentation,
> including kernel-doc, without extra tweaks. As a whole, the build
> becomes much simpler.


6. This fail assessments I also had before. RTD has limits in 
   resources and in flexibility, that's why I moved to github pages

   https://github.com/return42/sphkerneldoc/issues/1
   
I will stop here ... I think it is good that everyone make its own
experience, BUT ...

  IMO it is a misjudgment to think that changing the 
  markup and it's toolchain is only a series of patches.

Sorry if my words are unpleasant, this is not my intend, but IMO *the 
view of the whole* and concepts are missed.

Many of the facts mentioned above have been covered in my POC at 
https://github.com/return42/sphkerneldoc ... On others, 
like 5. I'am working on ....

> I've had a few moments of spare time to look into Sphinx. 

... a comprehensive solution needs time and will not be done in a
hurry. Please give me a week or may be two, then I could present
a much more comprehensive solution.

Again: my english is not the best, if some of my words has been 
unpleasant I'am sorry for this, it's not my intend.

--Markus--



> 
>> 
>> 
>> BR,
>> Jani.
>> 
>> PS. I never thought I'd do this much perl coding in my life. It didn't
>> kill me, I must be stronger now.
>> 
>> 
>> [1] git://people.freedesktop.org/~jani/drm
>> [2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-wip
>> [3] https://kernel.readthedocs.io/
>> 
>> 
>> --
>> Jani Nikula, Intel Open Source Technology Center
> 
> 
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch