Re: [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression

From: Doug Anderson

Date: Fri Nov 20 2026 - 12:53:30 EST


Hi Simon,

On Fri, Nov 20, 2026 at 7:29 AM Simon Glass <sjg@xxxxxxxxxxxx> wrote:
>
> > diff --git a/fdtfactor b/fdtfactor
> > @@ -140,6 +141,32 @@ def compress_blob(raw_bytes, comp, filename=''):
> > + if pool is not None:
> > + return list(pool.map(lambda t: func(*t), tasks))
> > + with concurrent.futures.ThreadPoolExecutor() as new_pool:
> > + return list(new_pool.map(lambda t: func(*t), tasks))
>
> Is the shared pool worth it? Creating a ThreadPoolExecutor is cheap
> next to compressing even one blob, and factor_dtb_files() already
> relies on the pool being created here. If _map_blobs() always creates
> its own, the 'pool' argument can be dropped from
> _extract_fit_fdt_entries(), _strip_fit_unfactored_local_fixups() and
> _factor_fit_group(), and factor_fit() doesn't need to be re-indented.
> What do you think?

Makes sense. Creating a `ThreadPoolExecutor` inside `_map_blobs()` is
in the noise next to blob compression, and dropping `pool` keeps
`factor_fit()` and its helpers much cleaner. Done for v5.

> BTW the default is up to 32 workers, so with lz4 this can start up to
> 32 processes at once, regardless of any -j limit in the build that
> calls the tool. It would be worth a -j/--jobs option, perhaps as a
> follow-up.

Agreed, adding `-j`/`--jobs` (or jobserver integration) as a follow-up
sounds worthwhile.

> > diff --git a/fdtfactor b/fdtfactor
> > @@ -1668,7 +1731,24 @@ def factor_dtb_files(args):
> > + dtbs = [
> > + ParsedDtb(
> > + p,
> > + comp=comp,
> > + fdt=_open_fdt(raw, p),
> > + uncompressed_size=len(raw),
> > + compressed_size=len(fblob),
> > + )
> > + for p, fblob, raw in zip(input_dtb_paths, file_blobs, raw_blobs)
> > + ]
>
> This was the last caller which relied on ParsedDtb reading and
> decompressing the file itself, so the 'if fdt is None' path in
> ParsedDtb.__init__() is now dead code. Please can you drop it, make
> 'fdt' a required argument and update the docstring?

Done for v5.

-Doug