Re: [PATCH 00/10] Documentation/Sphinx

[Markus Heiser]

Am 30.05.2016 um 16:46 schrieb Jani Nikula <jani.nikula@xxxxxxxxx>:

> On Mon, 30 May 2016, Markus Heiser <markus.heiser@xxxxxxxxxxx> wrote:
>> Here my 5cents about Jani's patch series:
>> 
>> 1. Migration implementations should not be a part of the kernel tree
> 
> If you're referring to the conversion scripts, I don't care either
> way. It's probably helpful to have them until everything is converted,
> and we can dispose of them afterwards.
> 
>> 2. sed/pandoc migration fits not for all the XML documentation and has disadvantages.
>>   Jonathan asked it before: http://article.gmane.org/gmane.linux.documentation/37533
>> 
>>   I repeat myself: Summarize, why should one prefer this tools over pandoc + sed?
>> 
>>   * Pandoc coverage is less on reading and writing, this is where 
>>     dbxml comes into play
>> 
>>     - reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23
>> 
>>     - writing reST has many bugs and leaks (you fixed some of them with sed)
>> 
>>   * Pandoc does not support external entities (linux-tv), covered by dbxml
>> 
>>   * dbxml brings the ability to chunk one large XML book into small 
>>     reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking
>> 
>>   * dbxml lets you manipulate the XML source before you convert it to reST
>> 
>>     this might helpfull e.g. if you have to convert single-column informal-tables 
>>     to lists or other things ... in short; dbxml and it's hooks are the key to hack
>>     everything you need in a full automated DocBook-->reST migration workflow.
> 
> I am not proposing to merge the documents that I've converted mostly as
> samples in the branch. I needed something to demonstrate the build is
> sane.

> The authors of the DocBook documents should make the conversions as they
> see fit, when they see fit, with the tools they see fit, probably with
> some manual work on top.

OK

> 
>> 3. we also discussed before, that ASCII art tables are ugly, because the produce
>>   confusing diffs, for this I wrote the flat-table directive (dbxml migrates tables
>>   to flat-tables / pandoc can't). 
>>   https://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
> 
> See the above. Any authors that think the Sphinx support I've added is
> good enough can go ahead and switch.
> 
> I think it's safe for me to say the GPU documents won't wait for further
> extension directives or conversion tools. Some others will definitely
> want to have the flat table extension before switching.

OK

> 
>>> With this, we can put any .rst files (including ones that have
>>> kernel-doc directives) anywhere under Documentation, add a link to them
>>> in Documentation/index.rst table of contents, and it will just work. It
>>> can't get much simpler than that.
>> 
>> 4. We discussed it / I already mentioned that each document shipped in it's own
>>   sphinx project. Bundling all documents into one sphinx-project will work for
>>   4 or 5  small documents, but not for the whole documentation. BTW all XML 
>>   documents are currently separated DocBook projects .. so why should we merge
>>   them into one big project? Making one index-file for the different and small 
>>   ".txt" files seems OK, but not for the XML docs.
> 
> FWIW I locally converted all the DocBook documents (except media) and it
> works just fine, and to me it looks like exactly what we should
> have.

With DocBook, it was hard to separate a file into small chunks (see media for
how it is done). With Sphinx, it is common to split a document in small chunks 
(along parts, chapters, sections ...) Thats why I recommend chunking documents
(from the beginning).

> One of the goals was to have nice cross-referencing between the
> documents (e.g. from GPU to kernel or device driver API). And it works.

For this, Sphinx-doc brings intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

> This does not exclude having *additional* indexes or Sphinx config files
> for subsystems or subprojects to build a subset of the documentation for
> specific needs. It's up to the authors of the documents to decide.

OK, I think we have a bit different point of view .. IMO it is better
to give the authors a *scalable structure* to place their's documentation. 

> For PDF documents, adding the documents separately in pdf_documents
> seems to be the right thing to do.

I can't recommend to use rst2pdf (it is less maintained), use default
sphinx LaTeX toolchain.

> 
>> 5. In general, the markup of the linux kernel's source code comments remains
>>   unchanged and the reST markup within the comments is passed through the
>>   output. A closer lookup to the *kernel-doc* and *reST* markup revals, that
>>   there are some conflicts between reST (inline) markup and kernel-doc
>>   markup. Determined by the historical development of the kernel-doc comments, the
>>   *classic* kernel-doc comments contain characters like ``*`` or strings with
>>   e.g. leading/trailing underscore (``_``), which are inline markups in
>>   reST. Here a schort example from a *classic* comment::
>> 
>>        <SNIP> -----
>>        * In contrast to the other drm_get_*_name functions this one here returns a
>>        * const pointer and hence is threadsafe.
>>        <SNAP> -----
>> 
>>   In reST markup, the wildcard in the string ``drm_get_*_name`` has to be
>>   masked: ``drm_get_\\*_name``. Some more examples from reST markup:
>> 
>>   * Emphasis "*":  like ``*emphasis*`` or ``**emphasis strong**``
>>   * Leading "_" :  is a *anchor* in reST markup (``_foo``).
>>   * Trailing "_:  is a reference in reST markup (``foo_``).
>>   * interpreted text: "`"
>>   * inline literals: "``"
>>   * substitution references: "|"
>> 
>>    These special strings has to be masked in the output and can't be used as
>>    *plain-text markup*.  To get in use of the fully reST markup (stop masking
>>    special characters) we need some options in the sources documentation, comments
>>    like
>> 
>>       /* parse-markup: reST */
>> 
>>    which influence the behavior.
> 
> I find it totally unacceptable to require explicitly marking kernel-doc
> comments or source files as being reStructuredText.
> Note that it's all opt-in already. If you add a .rst file that includes
> kernel-doc via the kernel-doc extension, you better make sure the
> comments parse as reStructuredText and render nicely. I'm willing to do
> much of the job for all the things that I care about.

We have a different POV ... I try to build up a documentation project,
which could use all given kernel-doc markups without any change, where
reST is an "addition". Your approach is to fix kernel-doc comments 
if they are referred by a kernl-doc directive in a .rst document.
There is nothing wrong about your approach, but I try to build
a whole source code documentation like the one I started here: 
http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

Benefit: Beside a kernel-doc directive, authors (from everywhere, not 
only within the kernel-src-tree) can refer kernel-docs with intersphinx
Here is a example, see "cross references" at: 
http://return42.github.io/sphkerneldoc/articles/linux_src_doc.html?cross-references


> Besides, if you look at the results, you'll find it looks mostly good
> without any fixes. In the sample documents, I've erred on the side of
> having a few markup hickups here and there while most of it works
> perfectly well as reStructuredText. I think this is exactly what we
> should do, declare it all reStructuredText and fix issues as we go.
> 
>>> Sites like https://readthedocs.org/ can build the documentation,
>>> including kernel-doc, without extra tweaks. As a whole, the build
>>> becomes much simpler.
>> 
>> 
>> 6. This fail assessments I also had before. RTD has limits in 
>>   resources and in flexibility, that's why I moved to github pages
>> 
>>   https://github.com/return42/sphkerneldoc/issues/1
> 
> I'm just personally using Read the Docs to ensure they can build the
> documentation as "pure" Sphinx (they won't use the Makefiles), and so I
> don't have to host the docs anywhere myself. Sure, I can't add *all* the
> files there because it exceeds the build limits. They might cater for us
> if they want to carry the kernel documentation going forward, but most
> likely the output should be at kernel.org anyway. (Plus we'll have
> freedesktop.org and 01.org for the GPU documentation too.)

There a some similar discussions on RTD ... yes, they make exceptions
for *special* users, but its all harry, use your default publishing 
sides and forget RTD ;-)

>> I will stop here ... I think it is good that everyone make its own
>> experience, BUT ...
>> 
>>  IMO it is a misjudgment to think that changing the 
>>  markup and it's toolchain is only a series of patches.
>> 
>> Sorry if my words are unpleasant, this is not my intend, but IMO *the 
>> view of the whole* and concepts are missed.
> 
> I'm not sure what to say.

As I said, we have a different POV, which is not in contradiction,
as your requirements are a subset of mine.

I worry a little bit in that reST will be only one more toolchain 
beside DocBook .. in the long term, kernel's documentation 
should get rid of all the DocBook artifacts and for this a more
comprehensive solution is needed.

> Please do not underestimate the productivity of my moments of spare
> time. ;)

1:0 for you ;-)

Am 30.05.2016 um 17:29 schrieb Daniel Vetter <daniel.vetter@xxxxxxxx>:

> I concur with Jani on all points, just want to follow-up here.
> 
> Tbh I'd like to get this in soonish, so that we have about 1 month of
> time before 4.8 for actually polishing gpu documentation. Maybe
> there's some room for more, better tables, other extensions, polish
> the kernel-doc script a bit more. But what we have now works, I think
> it does address everything we've discussed over the past year that we
> absolutely need. I think it makes it possible to extend for all the
> more optional use-cases we need to be able to convert, and we can add
> those extensions later on.

OK, I understand your needs, I don't wan't to stop you writing 
documentation with reST. May use Jani's approach for a first
toolchain 

> And with Jani's big pile of kernel-doc
> patches we now also have someone who understands that perl script,
> which is awesome.

? 

https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

> In short I want to go nuts improving the docs themselves and stop
> discussing the tooling to build them. Can we please make this happen?
> 
> Note that Jani's already started to throw out our old ascidoc hacks in
> the topic/kerneldoc branch in the drm-intel.git repo, and we'll switch
> over the autobuilder for the 01.org docs as soon as that's done. We're
> committed, I want this ;-)


Am 30.05.2016 um 16:46 schrieb Jani Nikula <jani.nikula@xxxxxxxxx>:

> I'm getting pretty tired of talking...

Yes, we are all tired in this discussions ... I see, you are in need 
to switch over ... for this, may use your toolchain ... but I have one 
please, do not change more than you really need and be open for
changing the toolchain later. 

I will inform you about any substantial progress within my 
solution ... may we can merge our experience soon or later, 
but don't stop writing documentation.

Thank's for your patience ... :-)

--Markus--