[PATCH v2 00/25] kernel-doc: make it parse new functions and structs

[Mauro Carvalho Chehab]

Hi Jon,

It is impressive how a single patch became a series with 25 ones ;-)

This version was rebased on next-20260127, which is after docs-mw
got merged with my doc patches, solving some merge conflicts.

It should now apply cleanly on the top of upstream and it will address
the kernel-doc issues with context analysis patches that are already
at linux-next.

On this version, I opted to do something I always wanted when
porting from the venerable kernel-doc perl logic: properly handle
macros without writing really complex regular expressions.

After those changes, the replacement logic becomes a lot more
easier to write, as we can have a cleaner logic to handle
function calls:

	function_xforms = [
	    (KernRe(r"^static +"), ""),
	    (KernRe(r"^extern +"), ""),
	...
	    (NestedMatch(r"__cond_acquires\s*\("), ""),
	    (NestedMatch(r"__cond_releases\s*\("), ""),
	...
	}

It can even be used to replace comma-separated arguments like here:

	struct_xforms = [
	...
	    (NestedMatch(r'\bstruct_group\s*\('), r'\2'),
	    (NestedMatch(r'\bstruct_group_attr\s*\('), r'\3'),
	    (NestedMatch(r'\bstruct_group_tagged\s*\('), r'struct \1 { \3 } \2;'),
	    (NestedMatch(r'\b__struct_group\s*\('), r'\4'),
	...
	]

As both KernRe and NestedMatch have sub methods, they can be placed
altogether at the same transforms tables.

The regex syntax for them are quite simple: they should end with a
delimiter (usually open parenthesis, but brackets and square
brackets are also supported).

The first 15 patches on this series were co-developed with Randy,
with came up with the original patch. I ended helping identifying
kernel-doc issues and doing some changes to make the parser more
reliable.

Then, I added 10 other patches to improve NestedMatch.

The final two patches on it fixes an issue when parsing
include/net/page_pool/types.h. Right now, it doesn't parse
well struct page_pool_params, as the second struct_group_tagged()
has a /* private */ on it, causing the end parenthesis to be
commented out. After the series, the documentation for it
should be ok.

-

Even NestedMatch being more complex than KernRe, on my machine,
parsing all files is 5% faster than before, because we're not
parsing anymore macro definitions.

Ah, due to the complexity of NestedMatch, I opted to write
some unit tests to verify that the logic there is correct.
We can use it to add other border cases.

Using it is as easy as running:

	$ tools/unittests/nested_match.py

(I opted to create a separate directory for it, as this
is not really documentation)

Mauro Carvalho Chehab (23):
  docs: kdoc_re: add support for groups()
  docs: kdoc_re: don't go past the end of a line
  docs: kdoc_parser: move var transformers to the beginning
  docs: kdoc_parser: don't mangle with function defines
  docs: kdoc_parser: add functions support for NestedMatch
  docs: kdoc_parser: use NestedMatch to handle __attribute__ on
    functions
  docs: kdoc_parser: fix variable regexes to work with size_t
  docs: kdoc_parser: fix the default_value logic for variables
  docs: kdoc_parser: add some debug for variable parsing
  docs: kdoc_parser: don't exclude defaults from prototype
  docs: kdoc_parser: fix parser to support multi-word types
  docs: kdoc_parser: add support for LIST_HEAD
  docs: kdoc_re: properly handle strings and escape chars on it
  docs: kdoc_re: better show KernRe() at documentation
  docs: kdoc_re: don't recompile NextMatch regex every time
  docs: kdoc_re: Change NestedMath args replacement to \0
  docs: kdoc_re: make NextedMatch use KernRe
  tools: kdoc_re: add support on NestedMatch for argument replacement
  tools: python: add helpers to run unit tests
  unittests: add tests for NestedMatch class
  tools/lib/python/unittest_helper.py
  docs: kdoc_parser: better handle struct_group macros
  docs: kdoc_re: fix a parse bug on struct page_pool_params

Randy Dunlap (2):
  docs: kdoc_parser: ignore context analysis and lock attributes
  kdoc_parser: handle struct member macro VIRTIO_DECLARE_FEATURES(name)

 tools/lib/python/kdoc/kdoc_parser.py | 109 +++--
 tools/lib/python/kdoc/kdoc_re.py     | 154 +++++--
 tools/lib/python/unittest_helper.py  | 279 +++++++++++++
 tools/unittests/nested_match.py      | 589 +++++++++++++++++++++++++++
 4 files changed, 1063 insertions(+), 68 deletions(-)
 create mode 100755 tools/lib/python/unittest_helper.py
 create mode 100755 tools/unittests/nested_match.py

-- 
2.52.0