Re: [PROBLEM] Very long .deb package build times for bindeb-pkg build target

[Masahiro Yamada]

On Sat, Jan 13, 2024 at 8:28 PM Mirsad Todorovac
<mirsad.todorovac@xxxxxxxxxxxx> wrote:
>
> On 1/13/24 06:40, Masahiro Yamada wrote:
> > On Fri, Jan 12, 2024 at 7:48 AM Mirsad Todorovac
> > <mirsad.todorovac@xxxxxxxxxxxx> wrote:
> >>
> >> On 11. 01. 2024. 16:37, Nicolas Schier wrote:
> >>> Hi Mirsad,
> >>>
> >>> On Thu 11 Jan 2024 13:22:39 GMT, Mirsad Todorovac wrote:
> >>>> Hi,
> >>>>
> >>>> With this new release, it seems that Debian kernel build uses "xz" in single-
> >>>> threaded mode:
>
> Hi, Masahiro,
>
> Thank you for your reply.
>
> > New release of what?
>
> Forgive me for not being precise. It was the new release of torvalds tree mainline
> kernel 6.7+ (with merge commits up-to-date). I sort of mistakenly assumed that
> I wrote it, but it was declared in some mail on LKML.



When you report a regression in the kernel code in the future,
please try to do "git bisect" and pin-point an offending commit.

That is more helpful to figure out how to fix the issue.

And, you will be sure whether or not the root cause exists
in the kernel.





>
> One environment was Ubuntu 23.10 Mantic Minotaur on the desktop at work.
>
> The laptop also has Mantic Minotaur, but for some reason it spawns multithreaded
> dpkg-deb when building packages, unlike desktop which spawned single-threaded "xz".
>
> This is at least what the "top" displayed when turning on "H" (show threads).
>
> On one system (upgraded 23.10 from 22.04 LTS) the same dpkg-deb shows as multi-threaded
> dpkg-deb, while on the other it calls xz which didn't respect XT_OPT=-T0 (visible
> from the copy+paste from top output).
>
> I am perplexed myself.
>
> >>>> Tasks: 484 total,   2 running, 481 sleeping,   0 stopped,   1 zombie
> >>>> %Cpu(s):  2.5 us,  2.2 sy,  6.3 ni, 85.1 id,  2.3 wa,  0.0 hi,  1.7 si,  0.0 st
> >>>> MiB Mem :  64128.3 total,    524.3 free,   5832.0 used,  58540.9 buff/cache
> >>>> MiB Swap:  32760.0 total,  32758.7 free,      1.2 used.  58296.3 avail Mem
> >>>>
> >>>>      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> >>>> COMMAND
> >>>>
> >>>>   978084 marvin    30  10  112440  97792   2432 R 100.0   0.1  29:3023 xz
> >>>>
> >>>>
> >>>> Before dpkg-deb was using up to 3200% of CPU time on a 16 core SMT CPU.
> >>>>
> >>>> Can it be something with dpkg-deb --thread-max=%n option?
> >>>
> >>> I cannot find any --thread-max option in Linux tree.  Do you call
> >>> dpkg-deb manually or somehow induce a thread maximum?
> >>>
> >>>> Waiting for half an hour just for the build of linux-image-...-dbg package
> >>>> seems like an overkill ...
> >>>
> >>> With current v6.7 release tree I do not see the reported slow-downs
> >>> when building bindeb-pkg; I tested by cross-compiling for arm64 on
> >>> amd64 with CONFIG_MODULE_COMPRESS_XZ=y and =n).
> >>>
> >>> Both take roughly 5mins on my 24-core i9 system.
> >>>
> >>> Kind regards,
> >>> Nicolas
> >>
> >> I am perplexed too, but you can see from the top output the
> >> single-threaded xz with 29:30m processor time.
> >>
> >> On my laptop with the sam Ubuntu 23.10 mantic minotaur, I have
> >> dpkg-deb version 1.20.12 and it shows things like 400% and 3200%
> >> CPU time, so it is working multithreaded.
> >>
> >> On desktop machine with the same Ubuntu 23.10 and the same git
> >> torvalds tree, it starts single-threaded xz from dpkg-deb instead.
> >
> > You built the same mainline git tree on your laptop and desktop.
> > The former runs xz multi-threaded, the latter single-threaded.
> >
> > So, this is not about the kernel code, but about your environment,
> > isn't it?
>
> It should be. Somehow it doesn't behave rationally. To be true to the facts,
> one is 23.10 Mantic Minotaur upgraded from 22.04 LTS, and the other is
> fresh Mantic from install.
>
> > You mentioned you were using Ubuntu 23.10, where
> > dpkg-deb version is 1.22.0
>
> This is correct.
>
> > Your dpkg-deb version, 1.20.12, is too old for Ubuntu 23.10.
> > Is it a self-built one?
>
> Yes, it is the build from Debian 11 source package.
>
> This had the similar rationale because Ubuntu version ran single-threaded.
>
> dpkg-deb from Debian 11 source package worked well for months.
>
> > dpkg-deb usually compresses binary packages, but the default
> > compression type depends on the distro.
> > (It is determined at "./configure" time)
>
> I see. But the home-built dpkg-deb also resorted to running "xz", and
> xz did not recognise XZ_OPT environment variable. Very odd. :-/



It depends on the environment whether or not dpkg-deb
spawns the external 'xz' command.


With WITH_LIBLZMA defined, dpkg-deb uses internal code
to compress the data with xz.  [1]

Without WITH_LIBLZMA, dpkg-deb sparns "xz" processes. [2]


[1]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/1.20.12/lib/dpkg/compress.c?ref_type=tags#L412
[2]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/1.20.12/lib/dpkg/compress.c?ref_type=tags#L663


Since you said you saw "xz" in the "ps" command output,
I believe your case is the latter.


When dpkg-deb swarns the external "xz" command,
dpkg-deb unsets "XZ_OPT". [3]

I believe that's why you do not see XZ_OPT propagated.

[3]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/1.20.12/lib/dpkg/compress.c?ref_type=tags#L643




>
> > On Ubuntu, the default compression type for dpkg-deb is "zstd"
> > (while it is "xz" on Debian)
> >
> > Check "man dpkg-deb" on your Ubuntu machine.
> >
> >    -Zcompress‐type
> >        Specify which compression type to use when building a package.
> >        Allowed values are gzip, xz (since dpkg 1.15.6),
> >        zstd (since dpkg 1.21.18) and none (default is zstd).
>
> Verified. "man dpkg-deb" confirms that.
>
> > You are still allowed to use xz with "make KDEB_COMPRESS=xz deb-pkg".
> > Is this your case?
>
> Somehow, my Mantic's "xz" did not react to the environment variables. Maybe it is
> enchanted? :-/
> > Overall, your report is not sensible.
>
> > You should check what you are seeing.
>
> I was seeing exactly what I copy+pasted (top output). Give me the benefit of doubt
> that I did not falsify copy+paste. Ideally, I should have submitted JPG terminal
> screenshot.
>
> Home-built dpkg-deb worked well on Ubuntu 22.04 LTS and 23.10 (upgraded from 22.04).
> So it was a surprise that it started showing quirks on this new box with fresh 23.10
> from .iso.
>
> But I am only 16 months in the Linux kernel development, so it is probably my fault.
>
> I saw from the output of "ps xewww" that "xz" had "XZ_OPT=-T0", but it refused to obey
> the environment variable.
>
> This is probably worth digging and delving into to find the culprit, but eventually the
> workaround was the script manually prepending "--threads=0" to the parameter list and
> calling system xz:
>
>         /usr/bin/xz --threads=0 "$@"
>
> Certainly, I need to establish "normality" before building complex kernels and applying
> patches or the results will be non-reproducible and useless.
>
> I did the background search on your valuable contributions to the LK, but as we have this
> problem with the vanilla installation of Ubuntu, maybe we can think of something to
> optimise the LK build time?


I do not think so.

There already exists a solution to control the number of threads.



See "man dpkg-deb"

   DPKG_DEB_THREADS_MAX
       Sets the maximum number of threads allowed for compressors that
       support multi‐threaded operations (since dpkg 1.21.9).

       The --threads-max option overrides this value.




By setting this env variable, you should be able to control the multi-threading.


However, your dpkg-deb is 1.20.12, so you need to use a newer version.



You locally hacked builddeb to add --threads-max, but it does not
work for you for the same reason; it requires dpkg 1.21.9+


  --threads-max=threads
     Sets the maximum number of threads allowed for compressors that support
     multi‐threaded operations (since dpkg 1.21.9).




Why do you stick to the old home-built dpkg-deb?

If you use the default dpkg-deb bundled with Ubuntu 23.10,
you will have a better experience.

Or, if you still insist on a home-built dpkg-deb,
why don't you build a newer version?








>
> To recall better from Thursday, I actually tried to prepend "XZ_OPT=--threads=0" to
> system xz, and it worked. But when called from dpkg-deb, it suddenly got amnesia, despite
> XZ_OPT=-T0 being seen in "ps xewww" output :-/
>
> I believe there should be a rational explanation rather than black or red magic.
>
> Thank you again for considering this problem report and have a nice weekend!
>
> Best regards,
> Mirsad Todorovac
>
> >> I tried things like this:
> >>
> >> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> >> index d7dd0d04c70c..b2319c23db34 100755
> >> --- a/scripts/package/builddeb
> >> +++ b/scripts/package/builddeb
> >> @@ -38,7 +38,7 @@ create_package() {
> >>
> >>          # Fix ownership and permissions
> >>          if [ "$DEB_RULES_REQUIRES_ROOT" = "no" ]; then
> >> -               dpkg_deb_opts="--root-owner-group"
> >> +               dpkg_deb_opts="--threads-max=0 --root-owner-group"
> >>          else
> >>                  chown -R root:root "$pdir"
> >>          fi
> >>
> >> and it didn't work either - dpkg-deb --threads-max=0 still spawned a
> >> single-threaded xz that ran 30 minutes.
> >>
> >> Then the workaround was a very simple xz shell script that adds option --threads=0
> >> and calls system xz:
> >>
> >> ~/bin/xz:
> >> ----------------------------------------------------------------
> >> #!/bin/bash -f
> >>
> >> /usr/bin/xz --threads=0 "$@"
> >> ----------------------------------------------------------------
> >>
> >> This finally worked, but sometimes I get:
> >>
> >> marvin@defiant:~/linux/kernel$ xz -9 --memlimit-compress=8000MiB linux-image-6.7.0-rc8-dbg_6.7.0-rc8-6_amd64.deb
> >> /usr/bin/xz: Reduced the number of threads from 8 to 6 to not exceed the memory usage limit of 8000 MiB
> >>
> >> (This is of course just an example of compressing a large file, as .deb is already compressed.)
> >>
> >> I used the default Ubuntu 23.10 config, with .pems excluded, and I think module compression
> >> did not work either. I had to turn it off ...
> >>
> >> Hope this helps.
> >>
> >> Regards,
> >> Mirsad
> >>
> >> --
> >> Mirsad Goran Todorovac
> >> Sistem inženjer
> >> Grafički fakultet | Akademija likovnih umjetnosti
> >> Sveučilište u Zagrebu
> >>
> >> System engineer
> >> Faculty of Graphic Arts | Academy of Fine Arts
> >> University of Zagreb, Republic of Croatia
> >> The European Union
> >>
> >> "I see something approaching fast ... Will it be friends with me?"
> >>
> >>
> >
> >



-- 
Best Regards
Masahiro Yamada