Re: [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression

From: Simon Glass

Date: Fri Nov 20 2026 - 10:30:04 EST


Hi Douglas,

On 2026-11-16T20:52:57, Douglas Anderson <dianders@xxxxxxxxxxxx> wrote:
>
> Because Python's `zlib` and `_lzma` C extensions and `subprocess.run()`
> for `lz4` all release the GIL, add a small `_map_blobs()` helper using
> `concurrent.futures.ThreadPoolExecutor` to decompress input blobs and
> compress output base/overlay blobs across worker threads in
> deterministic input order whenever multiple compressed blobs are
> processed.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@xxxxxxxxxxxx>
>
> fdtfactor | 190 +++++++++++++++++++++++++++++++++++++++++++++-----------------
> 1 file changed, 138 insertions(+), 52 deletions(-)

> diff --git a/fdtfactor b/fdtfactor
> @@ -140,6 +141,32 @@ def compress_blob(raw_bytes, comp, filename=''):
> + if pool is not None:
> + return list(pool.map(lambda t: func(*t), tasks))
> + with concurrent.futures.ThreadPoolExecutor() as new_pool:
> + return list(new_pool.map(lambda t: func(*t), tasks))

Is the shared pool worth it? Creating a ThreadPoolExecutor is cheap
next to compressing even one blob, and factor_dtb_files() already
relies on the pool being created here. If _map_blobs() always creates
its own, the 'pool' argument can be dropped from
_extract_fit_fdt_entries(), _strip_fit_unfactored_local_fixups() and
_factor_fit_group(), and factor_fit() doesn't need to be re-indented.
What do you think?

BTW the default is up to 32 workers, so with lz4 this can start up to
32 processes at once, regardless of any -j limit in the build that
calls the tool. It would be worth a -j/--jobs option, perhaps as a
follow-up.

> diff --git a/fdtfactor b/fdtfactor
> @@ -1668,7 +1731,24 @@ def factor_dtb_files(args):
> + dtbs = [
> + ParsedDtb(
> + p,
> + comp=comp,
> + fdt=_open_fdt(raw, p),
> + uncompressed_size=len(raw),
> + compressed_size=len(fblob),
> + )
> + for p, fblob, raw in zip(input_dtb_paths, file_blobs, raw_blobs)
> + ]

This was the last caller which relied on ParsedDtb reading and
decompressing the file itself, so the 'if fdt is None' path in
ParsedDtb.__init__() is now dead code. Please can you drop it, make
'fdt' a required argument and update the docstring?

Regards,
Simon