Re: Kernel docs: muddying the waters a bit

[Markus Heiser]

Hi all, hi Jonathan,

Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab <mchehab@xxxxxxxxxxxxxxx>:

> Em Fri, 6 May 2016 15:32:35 +0200
> Markus Heiser <markus.heiser@xxxxxxxxxxx> escreveu:
> 
>> Hi Mauro,
>> 
>> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab <mchehab@xxxxxxxxxxxxxxx>:
>> 
>>> Markus,
>>> 
>>> Em Fri, 6 May 2016 13:23:06 +0200
>>> Markus Heiser <markus.heiser@xxxxxxxxxxx> escreveu:
>>> 
>>>> 
>>>> In this conf.py you have to *register* your folder with the extensions.
>>>> A few words about the flat-table extension and a (future) kernel-doc one:  
>>> 
>>> ...
>>> 
>>>> The flat-table is a pure docutils (the layer below sphinx) extension which
>>>> is not application specific, so I will ask for moving it to the docutils 
>>>> upstream.   
>>> 
>>> So, if I understood well, your proposal is to have a conf.py and the
>>> flat-table (plus other extensions) at the Kernel tree.  
>> 
>> Each book (better call it root-node) is a Sphinx-project, each 
>> Sphinx-project need a conf.py file (the build configuration file)
>> in its reST-source tree.
>> 
>> * http://www.sphinx-doc.org/en/stable/config.html
>> 
>>> Assuming that docutils upstream receives the flat-table extension
>>> (and eventually modifies it), while the new version doesn't arrive
>>> all distros, we'll end by having some developers using the newer
>>> docutils with the extension, plus others using the in-tree one.
>>> 
>>> Is there a way to specify at the conf.py what extension variant
>>> should it use, in case of both the in-tree or the docutils have
>>> the same?  
>> 
>> The build configuration file is a regular python file, you can 
>> implement conditions whatever you want/need.
>> 
>>> This could be trickier if they end by modifying the extension,
>>> but we can always backport the latest version, if they change the
>>> API.  
>> 
>> As far as i know, the docutils API is stable since 2002. In the
>> meantime there has been so many application build on it that
>> it is not realistic, you will see a not backward compatibly 
>> change.
>> 
>> The docutils project is conservative, very conservative, IMO to
>> conservative.
>> 
>> Today I'am doubtful if it isn't better I would merge it sphinx
>> upstream. I have to discuss this with some maintainers, but 
>> before I have to persuade myself, that all aspects are covered
>> and the implementations are robust. We are at the beginning and
>> we should not fear about every bit which could happen in the future.
>> 
>> The sphinx / docutils bottom plate gives us a number of degrees 
>> of freedom to find answers to question we have not yet asked. ;-)
> 
> Ok. So, from what I understand, once Sphinx support is added at
> Kernel upstream, we could convert the media docbook to
> reST+flat-table extension, adding such extension either on a shared
> place or only for the media DocBook build, together with its
> conf.py.
> 

Yes, in your media-conf you could decide to use a extension.

> Once it reaches upstream (either sphinx or docutils), we can
> work to make it integrate better with upstream as needed.
> 
> Right?

yes, right :-)

> If so, I'm ok with merging it as soon as possible.

If we advice a merge of the flat-table directive we should 
bundle this with the (to implement) "kernel-doc" directive ...

> In reST the directive might look like:
> 
> <reST-SNIP> -----
> Device Instance and Driver Handling
> ===================================
> 
> .. kernel-doc::  drivers/gpu/drm/drm_drv.c
>   :doc:      driver instance overview
>   :exported:
> 
> <reST-SNAP> -----

and the patches from my kernel-doc perl script to produce
reST from source code comments.

With this bundle within the kernel tree we have a good starting
point to compose reST documents from scratch and to migrate book
by book from DocBook to reST.

I insist to migrate book by book, because there are some
broken books. Broken by that, that some sources have changed
but not the corresponding documentation which use the comments
of these sources ... grap "Ooops" in the builded (xml or rst) docs.

E.g. I greped the .rst file and found the following Oops in the migrated books:

./books/mtdnand/pubfunctions-000-012.rst:13:Oops
./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
./books/s390-drivers/ccw-000-004-003.rst:13:Oops
./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
./books/device-drivers/Basics-000-001-002.rst:13:Oops
./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
./books/device-drivers/Basics-000-009-032.rst:13:Oops
./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops
./books/gadget/api-000-011-005.rst:13:Oops
./books/gadget/api-000-011-009.rst:13:Oops
./books/gadget/api-000-011-007.rst:13:Oops
./books/gadget/api-000-011-003.rst:13:Oops
./books/gadget/api-000-011-011.rst:13:Oops
./books/genericirq/intfunctions-000-009.rst:13:Oops

----- Summarize ----

@Jonathan: what do you think? Should I prepare a patch
with a basic reST (sphinx) build infrastructure, including

* a folder for sphinx docs:

  ./Documentation/sphinx/

* flat-table & kernel-doc extension at

  ./Documentation/sphinx/extensions

* a patch with rst-Output for the kernel-doc perl script at

  ./scripts/kernel-doc

* An example document "HowTo document with reST" at

  ./Documentation/sphinx/kernel-doc-rst-howto

  which at minimum describes the "flat-table" and "kernel-doc" 
  directive and the requirements for a building docs.

* a make file which fit into the kernel Makefile infrastructure (not 
  the one created by sphinx-quickstart).

-- Markus --