Re: [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper
From: Simon Glass
Date: Fri Nov 20 2026 - 10:22:58 EST
Hi Douglas,
On 2026-11-16T20:52:57, Douglas Anderson <dianders@xxxxxxxxxxxx> wrote:
> pylibfdt: Speed up FdtRo.properties() with a C helper
>
> Iterating over a node's properties via `FdtRo.properties()` currently
> crosses the Python/SWIG boundary `1 + 2 * num_props` times per node
> (`first_property_offset()`, `get_property_by_offset()`, and
> `next_property_offset()`, along with their Python `check_err()` and
> `check_err_null()` wrappers). When walking every property across large
> device trees from Python, those per-property SWIG round-trips dominate
> runtime.
>
> Add a C helper `fdt_node_properties_stub()` in `pylibfdt/libfdt.i`
> backing `FdtRo.properties()` so all `Property` objects for a node are
> collected in a single C call per node, interning property name strings
> via `PyUnicode_InternFromString()`. Also add an invalid-offset exception
> check to `testPropertiesIterator` in `tests/pylibfdt_tests.py`.
>
> On one stress test (walking all ~573k nodes and ~2.06M properties across
> 438 arm64 Qualcomm DTBs) this reduced `FdtRo.properties()` time by
> ~1.8x.
>
> [...]
>
> pylibfdt/libfdt.i | 64 +++++++++++++++++++++++++++++++++++++++++++++----
> tests/pylibfdt_tests.py | 3 +++
> 2 files changed, 63 insertions(+), 4 deletions(-)
> diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
> @@ -24,6 +24,61 @@ static int fdt_property_stub(void *fdt, const char *name, const void *val,
> + name_obj = PyUnicode_InternFromString(pname);
> + if (!name_obj) {
> + Py_DECREF(list);
> + return NULL;
> + }
Just to check, how much of the 1.8x comes from interning? My
understanding is that interned strings are immortal in Python 3.12 (I
am not sure about later versions), so every distinct property name
seen in any blob stays allocated for the life of the process. If the
gain is small, PyUnicode_FromString() would be safer.
> diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
> @@ -477,10 +532,9 @@ class FdtRo(object):
> - poff = self.first_property_offset(nodeoffset, quiet=QUIET_NOTFOUND)
> - while poff >= 0:
> - yield self.get_property_by_offset(poff)
> - poff = self.next_property_offset(poff, quiet=QUIET_NOTFOUND)
> + return iter(check_err(
> + fdt_node_properties_stub(self._fdt, nodeoffset, Property)
> + ))
This changes the behaviour a little, since properties() is no longer a
generator. The FdtException is now raised by the call itself rather
than on the first next(), and an error part-way through a node (e.g. a
truncated blob) now produces no properties at all, where before the
caller saw the earlier ones first.
I think that is OK, but please can you mention it in the commit
message and update the docstring, which still says 'Yields'.
> diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
> @@ -270,6 +270,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
> + with self.assertRaises(FdtException) as e:
> + list(self.fdt.properties(-1))
> + self.assertEqual(e.exception.err, -libfdt.BADOFFSET)
There is already an identical check a little further up in this file
(just before testGetName()) and, since it wraps the call in list(), it
passes with both the old and new code.
It would be worth replacing it with a test for the new error path,
i.e. a failure after the first property, such as a blob truncated in
the middle of a node's properties.
Regards,
Simon