Re: [RFC] A first shot at asciidoc-based formatted docs

From: Daniel Vetter
Date: Tue Jan 26 2016 - 07:17:13 EST


On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula <jani.nikula@xxxxxxxxx> wrote:
> On Tue, 26 Jan 2016, Jonathan Corbet <corbet@xxxxxxx> wrote:
>> So here is a proof-of-concept series showing how a fully asciidoc-based
>> toolchain might work. Lots of hackery here, this isn't meant to be applied
>> to anything at this point, but it's a good start. What this series has is:
>>
>> - Jani Nikula's patch adding asciidoc output to kernel-doc. Thanks for
>> doing this! It was the kickstart that was needed to get this process
>> going.
>
> Hooray! :)
>
>> - Tweak docproc to handle asciidoc template files. If a template ends in
>> ".adt", it's an asciidoc template; it's processed pretty much the same
>> way, except that kernel-doc gets the -asciidoc argument.
>>
>> - Bash on the Makefile to get it to process asciidoc templates into HTML.
>> Naturally this was where most of the time got spent. *Only* HTML output
>> works at the moment.
>>
>> - Convert tracepoints.html to tpoint.adt as a proof of concept. It works,
>> and the output is much pleasing, IMO.
>>
>> I'm sure there's a thousand details to deal with, and there is the issue of
>> the other output formats. asciidoctor claims to be able to create man
>> pages, but I've not tried that yet; neither tool will do PDF. Maybe we
>> could rely on pandoc to do that. Otherwise, getting to asciidoc to XML is
>> straightforward, so it should be possible to use xmlto as is done now.
>>
>> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
>> anybody wants to mess with it.
>>
>> Comments?
>
> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.
>
> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

This is a big bummer since with the parralized kernel-doc processing
using Makefiles and using asciidoctor even building something big like
the gpu docs is down to 2-3 seconds. From a clean tree, so not even
counting incremental speed-ups. Unfortunately asciidoc doesn't have an
built-in tooling (there's some experimental extensions) to split
things up.
-Daniel

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)
>
> One significant difference between our approaches is that I ditched
> docproc out of the equation. Instead of having the docproc ! directives
> in the asciidoc, I opted for using asciidoc's native include macro, with
> specially crafted filename suffixes to specify what parts of the source
> to include. Those files are then generated as intermediate asciidoc
> files using kernel-doc, with dependencies set right. There's a bunch of
> scripting involved, but it's pretty straight forward.
>
> So you'd have, for example,
>
> include::drivers/gpu/drm/drm_ioctl.c,export[]
>
> to include all the exported functions, or
>
> include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]
>
> to include the DOC: section. I think I'd like some version of this,
> regardless of whether asciidoc generates HTML or DocBook. It lets make
> parallelize all of kernel-doc processing for free, which is a big win.
>
> Patches follow, also available in kernel-asciidoc branch of
> git://people.freedesktop.org/~jani/drm (web interface
> http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)
>
> BR,
> Jani.
>
>
> Jani Nikula (10):
> kernel-doc: rewrite usage description, remove duplicated comments
> kernel-doc: add support for asciidoc output
> kernel-doc: support printing exported and non-exported symbols
> kernel-doc: add support for printing DOC: comments with escaped names
> scripts: add asciidoc-includes to extract includes from asciidoc
> scripts: add a kernel-doc helper for special invocation
> scripts: add tool for generating asciidoc dependencies and rules
> scripts: add a crude converter from DocBook tmpl to asciidoc
> Documentation: convert gpu.tmpl to gpu.txt
> Documentation: build asciidoc documentation
>
> Documentation/DocBook/Makefile | 37 +-
> Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
> Documentation/DocBook/gpu.txt | 1183 ++++++++++++++
> scripts/asciidoc-includes | 6 +
> scripts/kernel-doc | 365 ++++-
> scripts/kernel-doc-deps | 66 +
> scripts/kernel-doc-helper | 70 +
> scripts/tmpl2asciidoc | 39 +
> 8 files changed, 1709 insertions(+), 3556 deletions(-)
> delete mode 100644 Documentation/DocBook/gpu.tmpl
> create mode 100644 Documentation/DocBook/gpu.txt
> create mode 100755 scripts/asciidoc-includes
> create mode 100755 scripts/kernel-doc-deps
> create mode 100755 scripts/kernel-doc-helper
> create mode 100755 scripts/tmpl2asciidoc
>
> --
> 2.1.4
>



--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch