Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next

[Markus Heiser]

Am 28.06.2016 um 21:05 schrieb Jani Nikula <jani.nikula@xxxxxxxxx>:

> On Tue, 28 Jun 2016, Markus Heiser <markus.heiser@xxxxxxxxxxx> wrote:
>> Hi Jonathan, hi Mauro,
>> 
>> here is my DocBook to reST movement on top of Jon's docs-next branch. It includes:
>> 
>> * kernel-doc parser & directive
>> * flat-table directive
>> * man page builder 'kernel-doc-man'
>> * the kernel-doc-HOWTO
>> * guides for starters (reST-nano-HOWTO.rst & template-book)
>> * A index-page which bundles the lose Documentation/*.rst files.
>> * sub sphinx-projects aka *books* under Documentation/*/conf.py
>> * a full DocBook-XML to reST migration (*books* matching
>> Documentation/books_migrated/*/conf.py) described in [2]. The migration is
>> based on the DocBook-XML content at commit 17defc2 from Johnatan's docs-next
>> branch.
>> 
>> I decided to comment out the pdf creation in the Makefile.reST, rst2pdf is to
>> fragil see [2] (pdf is WIP).
>> 
>> The output of html and man is included in the common %doc targets, to get
>> more information about reST builds use:
>> 
>>  make books-help 
>> 
>> Any comments are welcome.
> 
> Perhaps you misunderstood, I don't know. When we ask you to rebase your
> work on something, in this case docs-next, it generally means, accept
> what is there, and iteratively build and extend upon it, *not* rip out
> and rewrite everything from scratch.
> 
> Several people have spent a non-insignificant amount of time to review
> and polish what is in docs-next currently. We've converted gpu
> documentation on top, in drm-next, and set up autobuilders for it. We've
> ironed out python2 vs python3 issues. We've fixed kernel-doc comments
> here and there. Written documentation for the whole thing. Generally
> tested the stuff in various environments. Etc, etc. That is the baseline
> now, and it should be improved on iteratively, not destructively. That's
> just sane engineering.
> 
> On the actual content (and really, this is orthogonal to the above),
> I've repeatedly told you that I disagree with your approach to having
> several configuration files, having the distinction between "books" and
> other files, rewriting kernel-doc in python, having both rst and
> "vintage" kernel-doc comments, converting all the docbook files in one
> big lump. I won't repeat my rationale here, I've said it all before, but
> sadly I don't see any of that addressed.
> 
> IMHO the most productive thing you could do right now is to send out
> just the "flat table directive" patch. That's not controversial, and
> would still have a chance to make it to v4.8.
> 
> BR,
> Jani.


Take it as what it is: 

   a complete replacement of the XML toolchain.

IMHO, most of what you mentioned are assumptions, so my first 
question is:    

   have you pulled and tested?

To be more concrete, *here* is what I tested on drm-next:

  git checkout -b test123 airlied/drm-next 
  git pull  https://github.com/return42/linux.git docs-next/linux-doc-reST

To complete the merge:

  $ mv Documentation/books_migrated/gpu/conf.py Documentation/gpu
  $ rm -rf Documentation/books_migrated/gpu 
  
  In the Documentation/gpu/conf.py drop the line: --> kernel_doc_mode = "kernel-doc"
  In the Documentation/index.rst drop the line:   -->     gpu/index

  $ make books/gpu.html

... perfect ... 

and now some additional new features:

1.) If you want to see all the "Oops" of your document in one point,
    add the ".. todolist::" directive in the gpu/index.rst :


2.) If you want to create manpages, add option ":man-sect: <man sect-no>"
    to these ".. kernel-doc::" directives e.g:

  <reST> ---------
  .. kernel-doc:: drivers/gpu/drm/i915/i915_irq.c
     :functions: intel_irq_init intel_irq_init_hw intel_hpd_init
     :man-sect: 9
  <reST> ---------

  $ make books/gpu.man
  ...
  writing man pages ... intel_irq_init.9 
  build succeeded, 43 warnings.
  $ man -l Documentation/dist/books/gpu/man/intel_irq_init.9.gz 

  ... perfect ...

3.) If you want to suppress headings of "DOC:" sections, add 
    option ":no-header:" to the kernel-doc directives selecting "DOC:",
    do not suppress headings on declarations, this might end in unwanted
    man-pages constructs.

BTW: You have to accept, that reST is a structured markup, the readers,
writers, translators and builders working with these structures. Inline
bold-faces like the ones, the kernel-doc perl script produce:

    print "**Parameters**\n\n";
    ..
    print "**$section**\n\n";

is more "painting a document", where a section structure (heading) is required.

> I've repeatedly told you that I disagree with your approach to having
> several configuration files, having the distinction between "books" and
> other files,

You want to build one great sphinx-project, a monolith where all is in.
I build those monoliths in the past and it was my fail. The roundtrips 
of those monoliths increase, while the user acceptance decrease and 
this is not a assumption, this is my practical experience from the
last 8 years with sphinx and reST. 

> having both rst and
> "vintage" kernel-doc comments,

your scope is reduced on your single use-case, it is a parser option,
the default is "reST", so if you don't need a *vintage* mode simply
ignore it ... (see above and read the kernel-doc-HOWTO).

> converting all the docbook files in one
> big lump.

We don't have to. I shipped them in the "books_migration" folder for
your tests and to see how all current documentation will be build.
I also placed a README.txt in this folder ... read it.
John could drop this folder, but others like Mauro want to base on this
migration.

Now we can test the toolchain against the complete docs, have
you ever done this? ... now you could build your monolith (see below)
and test the roundtrips in practice ...

The py-parser you fear about, leafs the XML toolchain untouched, is more strict, 
then the perl one and gives more warnings about all these buggy source code 
comments ... and with the "NullTranslator" you get a great lint tool for free.

I studied all your's and John's patches on the perl one and checked if
the python one need a correction.

BTW: it parses only once and shows errors only ones (not n-times). 

> I've said it all before, but
> sadly I don't see any of that addressed.

I addressed all your points and tried to evaluate solutions that match
(e.g. loose-text files, reST mode as default etc), but this does not imply, 
that all your answers get a 1:1 implementation. 

I remember about chunking large files, your answers "no, it breaks the
story" and with commit 2fa91d1 you start to chunk ... 

now the discussion about sub-projects ... my recommendations comes from 
a 8 year ongoing practice so please honor them a bit.

I also ask you for a close collaboration, but you didn't want. I'am at
the end of my ideas ... 

> IMHO the most productive thing you could do right now is to send out
> just the "flat table directive" patch. That's not controversial, and
> would still have a chance to make it to v4.8.

Since your solution is not a full replacement (no man-page builder) and 
does not produce structural markup .... 

IMHO pull my solution, if you have any remarks, let me know. E.g if
you think it is better to use ":no-header:" as default on DOC: 
sections, I implement it and test it against the complete docs.

-- Markus --




> 
>> 
>> -- Markus --
>> 
>> [1] http://return42.github.io/sphkerneldoc/articles/dbtools.html
>> [2] https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542
>> 
>> 
>> The following changes since commit 17defc282fe6e6ac93edbad8873ce89ef86b2490:
>> 
>> Documentation: add meta-documentation for Sphinx and kernel-doc (2016-06-24 06:55:28 -0600)
>> 
>> are available in the git repository at:
>> 
>> https://github.com/return42/linux.git docs-next/linux-doc-reST
>> 
>> for you to fetch changes up to 81bd813a599f8582570a735302ab660e20c7f442:
>> 
>> doc-rst: full DocBook-XML to reST migration (docs-next) (2016-06-28 17:15:38 +0200)
>> 
>> ----------------------------------------------------------------
>> 
>> Markus Heiser (15):
>>     python: add scripts/site-packages
>>     kernel-doc-HOWTO: add kernel-doc specification
>>     doc-rst: add python package linuxdoc
>>     doc-rst: kernel-doc parser - inital python implementation
>>     doc-rst: kernel-doc directive - initial implementation
>>     doc-rst: flat-table directive - initial implementation
>>     doc-rst: kernel-doc-man sphinx builder - initial implementation
>>     doc-rst: add basic sphinx-build infrastructure
>>     doc-rst: add template book "Get started with reST"
>>     doc-rst: removed Jani's kernel-documentation.rst and index.rst
>>     doc-rst: infrastructure for *loose reST articles*
>>     doc-rst: add build-html decription to book-help.
>>     doc-rst: kernel-doc parser assert absolute pathname
>>     doc-rst: infrastructure for migrated DocBook-XML
>>     doc-rst: full DocBook-XML to reST migration (docs-next)
>> 
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center