Re: [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper

From: Doug Anderson

Date: Tue Nov 17 2026 - 11:19:32 EST


Hi David,

On Mon, Nov 16, 2026 at 7:47 PM David Gibson
<david@xxxxxxxxxxxxxxxxxxxxx> wrote:
>
> > On one stress test (walking all ~573k nodes and ~2.06M properties across
> > 438 arm64 Qualcomm DTBs) this reduced `FdtRo.properties()` time by
> > ~1.8x.
>
> Hm, optimizing pylibfdt has never been a focus, on the general
> assumption that if speed is really important to you, you shouldn't be
> using Python. I'm happy enough with patch 4, because it's a speed win
> with a very small and simple change. This optimization is a much
> larger chunk of code, so I'm not so convinced this is worth the source
> complexity we pay for it.
>
> That's not an outright rejection, but I'd need convincing this is
> worth it.

Right, if we're trying to get the absolute maximum speed then of course
we wouldn't choose Python. For a tool like `fdtfactor`, speed is
certainly not the top priority and Python is the superior choice.
Writing it in C would have been a much bigger pain.

That all being said, just because speed isn't the top priority doesn't
mean it doesn't matter at all. When I started looking at optimizations I
was seeing my 438-DTB run take ~25 seconds, which is just a bit too
much. With all the optimizations I ended up with ~12 seconds. That's
still a lot and it would be nice to make it even better, but it also
starts getting into the realm of acceptable.

`FdtRo.properties()` stood out in profiling because
`DeviceTree.unflatten()` calls it on every node (~573k nodes / ~2.06M
properties across the 438 Qualcomm DTBs). In the pure-Python generator,
each node makes `1 + 2 * num_props` SWIG calls (~4.7M boundary
crossings, each going through Python `check_err` / `check_err_null`
wrappers and generator suspension/resumption) and allocates a fresh
non-interned `str` for repeated property names like `compatible`, `reg`,
`status`, and `phandle`.

Below is a small standalone script (`bench_properties.py`) that builds a
1,300-node (~140 KB) DTB in memory via `FdtSw` and walks it 400 times
(~520k nodes, ~2.08M properties) with patch 06/14 applied. On my
machine:
* Default `meson setup build` (`-O0`):
- `FdtRo.properties()`: `2.52s -> 1.50s` (1.68x faster)
- Full tree walk: `3.35s -> 2.25s` (1.49x faster)
* `meson setup --buildtype=release` (`-O3`):
- `FdtRo.properties()`: `1.76s -> 0.98s` (1.81x faster)
- Full tree walk: `2.24s -> 1.43s` (1.57x faster)

On the source-complexity side, roughly half of
`fdt_node_properties_stub()` in v4 is repetitive `Py_DECREF`
error-cleanup boilerplate -- using `PyObject_CallFunction(prop_cls,
"Oy#", ...)` and a single `err:` label trims it from 55 lines down to
~30 lines. That said, as noted in the patch notes, ~1 second on a
400-DTB run isn't make-or-break, so if you'd still rather not carry the
C helper in `pylibfdt/libfdt.i` I'm totally fine dropping this patch in
v5 (or trimming it to the ~30-line version if that tips the balance).

---8<---
#!/usr/bin/env python3
# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
# Run from the dtc repo root after applying patch 06/14 and building:
# meson compile -C build && python3 bench_properties.py
import os
import sys
import time

if os.path.isdir('build/pylibfdt') and 'build/pylibfdt' not in sys.path:
sys.path.insert(0, 'build/pylibfdt')

import libfdt
from libfdt import QUIET_NOTFOUND


def old_properties(fdt, nodeoffset):
poff = fdt.first_property_offset(nodeoffset, quiet=QUIET_NOTFOUND)
while poff >= 0:
yield fdt.get_property_by_offset(poff)
poff = fdt.next_property_offset(poff, quiet=QUIET_NOTFOUND)


def unflatten_walk(fdt, props_fn):
off, depth = 0, 0
while depth >= 0:
props = {p.name: p for p in props_fn(fdt, off)}
if depth > 0:
fdt.get_name(off)
off, depth = fdt.next_node(off, depth)
return props


def make_sample_dtb(num_nodes=1300):
sw = libfdt.FdtSw()
sw.finish_reservemap()
with sw.add_node(''):
sw.property_string('compatible', 'example,board')
sw.property_u32('#address-cells', 2)
sw.property_u32('#size-cells', 2)
with sw.add_node('soc@0'):
for i in range(num_nodes):
with sw.add_node(f'dev@{i:x}'):
sw.property_string('compatible', 'vendor,periph')
sw.property('reg', b'\x00\x00\x00\x00' * 4)
sw.property_string('status', 'okay')
sw.property_u32('phandle', i + 1)
return sw.as_fdt()


def main():
if not hasattr(libfdt, 'fdt_node_properties_stub'):
sys.exit('Apply patch 06/14 and rebuild pylibfdt first')

fdt = make_sample_dtb()
offsets = []
off, depth = 0, 0
while depth >= 0:
offsets.append(off)
off, depth = fdt.next_node(off, depth)

iters = 400 # ~520k nodes, ~2.08M props (similar to 438 arm64 DTBs)

t0 = time.perf_counter()
for _ in range(iters):
for o in offsets:
for _ in old_properties(fdt, o):
pass
dt_old = time.perf_counter() - t0

t0 = time.perf_counter()
for _ in range(iters):
for o in offsets:
for _ in fdt.properties(o):
pass
dt_new = time.perf_counter() - t0

t0 = time.perf_counter()
for _ in range(iters):
unflatten_walk(fdt, old_properties)
dt_unf_old = time.perf_counter() - t0

t0 = time.perf_counter()
for _ in range(iters):
unflatten_walk(fdt, libfdt.FdtRo.properties)
dt_unf_new = time.perf_counter() - t0

sz = fdt.totalsize()
print(f'{len(offsets)} nodes ({sz:,} B) x {iters} trees:')
print(f' FdtRo.properties() old (SWIG loop): {dt_old:.3f} s')
print(f' FdtRo.properties() new (C helper): {dt_new:.3f} s '
f'({dt_old / dt_new:.2f}x faster)')
print(f' Full tree walk old: {dt_unf_old:.3f} s')
print(f' Full tree walk new: {dt_unf_new:.3f} s '
f'({dt_unf_old / dt_unf_new:.2f}x faster)')


if __name__ == '__main__':
main()
--->8---

-Doug