Re: [PATCH 2/2] Makefile: clang-tools: Omit printing stack trace when KeyboardInterrupt is raised

[Maciej Falkowski]

On 5/22/21 5:03 AM, Masahiro Yamada wrote:
> On Sat, May 22, 2021 at 2:18 AM Nick Desaulniers
> <ndesaulniers@xxxxxxxxxx> wrote:
> > On Thu, May 20, 2021 at 4:18 PM Maciej Falkowski
> > <maciej.falkowski9@xxxxxxxxx> wrote:
> > > When user terminates the script (also implicitly through for example
> > > `make clang-analyzer`) by using
> > > Ctrl+C (or sending SIGINT more generally) the KeyboardInterrupt
> > > is raised printing stack trace of the execution as shown below:
> > >
> > > $ ./scripts/clang-tools/run-clang-tools.py clang-tidy ./compile_commands.json
> > > ^CTraceback (most recent call last):
> > >    File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
> > >      main()
> > >    File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
> > >      pool.map(run_analysis, datastore)
> > >    File "/usr/lib64/python3.8/multiprocessing/pool.py", line 364, in map
> > >      return self._map_async(func, iterable, mapstar, chunksize).get()
> > >    File "/usr/lib64/python3.8/multiprocessing/pool.py", line 765, in get
> > >      self.wait(timeout)
> > >    File "/usr/lib64/python3.8/multiprocessing/pool.py", line 762, in wait
> > >      self._event.wait(timeout)
> > >    File "/usr/lib64/python3.8/threading.py", line 558, in wait
> > > Process ForkPoolWorker-6:
> > > Process ForkPoolWorker-1:
> > > Process ForkPoolWorker-5:
> > > Process ForkPoolWorker-7:
> > > Process ForkPoolWorker-2:
> > > Process ForkPoolWorker-3:
> > > Process ForkPoolWorker-4:
> > > Process ForkPoolWorker-8:
> > >      signaled = self._cond.wait(timeout)
> > >    File "/usr/lib64/python3.8/threading.py", line 302, in wait
> > >      waiter.acquire()
> > > KeyboardInterrupt
> > With this applied,
> > $ make LLVM=1 LLVM_IAS=1 -j72 clang-analyzer
> > ^C
> > Process ForkPoolWorker-5:
> > make: *** [Makefile:1902: clang-analyzer] Error 130
> >
> > Reviewed-by: Nick Desaulniers <ndesaulniers@xxxxxxxxxx>
> > Tested-by: Nick Desaulniers <ndesaulniers@xxxxxxxxxx>
> >
> > Thanks for the patch!
>
> I am not a python expert, but is
> "Let's suppress back-trace when a user presses an interrupt"
> the common coding style?
I think that is not a common coding style.
In this case, the back-trace is verbose and it is caused
by a bug in multiprocessing Pool.

> If really so, do we need to do something similar in all python scripts?
> I do not know what is special about run-clang-tools.py.
I think no, I misunderstood the nature of the problem.
The verbose back-trace is a result of the multiprocessing Pool
not handling KeyboardInterrupt properly.
It is the old bug(bugs.python.org/issue8296), yet to be 
solved(bugs.python.org/issue22393).
For more info please see:
https://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool

This issue is already once solved in a Linux source tree in a 
scripts/checkkconfigsymbols.py:

scripts/checkkconfigsymbols.py:312
    try:
        return check_symbols_helper(pool, ignore)
    except KeyboardInterrupt:
        pool.terminate()
        pool.join()
        sys.exit(1)

Also the comment:
scripts/checkkconfigsymbols.py:321
"""Helper method for check_symbols(). Used to catch keyboard interrupts in
check_symbols() in order to properly terminate running worker processes."""

> For example, if I press Ctrl-C while building Clang
> by using tc-build, I see a back-trace.
> I have never thought back-tracing was annoying.
>
> The exit code is 130 regardless of this patch.
Okay.
> BTW, I prefer not having "Makefile:" in the patch subject
> since this is not touching Makefile at all.
>
> I rather like "clang-tools:" or "scripts/clang-tools:".
Okay.

The solution should cover the whole bug, I think this patch
may be dismissed then.
Thank you for your feedback!

Best regards,
Maciej Falkowski

> > > The patch handles the raise of the KeyboardInterrupt and exits when occurred
> > > with code 130 as documented in: https://tldp.org/LDP/abs/html/exitcodes.html
> > >
> > > Signed-off-by: Maciej Falkowski <maciej.falkowski9@xxxxxxxxx>
> > > ---
> > >   scripts/clang-tools/run-clang-tools.py | 5 ++++-
> > >   1 file changed, 4 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py
> > > index 38fc311d2e03..eb0e0ecfce24 100755
> > > --- a/scripts/clang-tools/run-clang-tools.py
> > > +++ b/scripts/clang-tools/run-clang-tools.py
> > > @@ -77,4 +77,7 @@ def main():
> > >
> > >
> > >   if __name__ == "__main__":
> > > -    main()
> > > +    try:
> > > +        main()
> > > +    except KeyboardInterrupt:
> > > +        sys.exit(130)
> > > --
> > > 2.26.3
> >
> > --
> > Thanks,
> > ~Nick Desaulniers
>
>
> --
> Best Regards
> Masahiro Yamada