Re: [PATCH] x86_64: fix delayed signals

From: Török Edwin
Date: Sat Jul 12 2008 - 06:33:09 EST

Next message: Török Edwin: "Re: [PATCH] x86_64: fix delayed signals"
Previous message: John Kacur: "Re: [PATCH -v2] ftrace: Documentation"
In reply to: Linus Torvalds: "Re: [PATCH] x86_64: fix delayed signals"
Next in thread: Török Edwin: "Re: [PATCH] x86_64: fix delayed signals"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On 2008-07-12 02:22, Linus Torvalds wrote:
> On Fri, 11 Jul 2008, Linus Torvalds wrote:
>
>> No, after testing more, I think Edwin is right.
>>
>
> Naah. It is true that there is a huge delay in doing
>
> ls -l /usr/bin
>
> and then pressing ^C, but after having hit my head against this for a
> while, I realized that it has nothing to do with the kernel.
>
> Doing an "strace ls" showed that ls doesn't play any games with signals
> etc, which fooled me into looking for a kernel reason.
>
> HOWEVER, it looks like at least fedora does a
>
> alias ls='ls --color=auto'
>
> and it turns out that if you do that --color=auto, then ls will indeed
> catch all normal signals and set a "please stop now" flag, instead of
> dying immediately. The reason is probably to avoiding leaving the terminal
> with some odd color if interrupted in an inconvenient place.
>
> So I was chasing this latency thing totally unnecessarily. It's in user
> space (or at least _my_ particular issue was).
>
> Other user cases may obviously be elsewhere.

Thanks for looking into the problem.
I did some more testing, and here are my results so far (with my
testcase of running dd and find):

On my 32-bit box (slow disks, SMP, XFS filesystem) 2.6.26-rc9 behaves
the same as 2.6.26-rc8, I can reliably reproduce a 2-3 second latency
[1] between pressing ^C the first time, and the shell returning (on the
text console too).
Using ftrace available from tip/master, I see up to 3 seconds of delay
between kill_pgrp and detach_pid (and during that time I can press ^C
again, leading to 2-3 kill_pgrp calls)

On my 64-bit box (2 disks in raid-0, UP, reiserfs filesystem) 2.6.25 and
2.6.26-rc9 behave the same, and most of the time (10-20 times in a row)
find responds to ^C instantly.
However in _some_ cases find doesn't respond to ^C for a very long time
(~30 seconds), and when this happens I can't do anything else but switch
consoles,
starting another process (latencytop -d) hangs, and so does any other
external command.
I have tried to get a latencytop output from the 64-bit box, but I
couldn't reproduce the long latency when latencytop was running.
I have reproduced the latency after launching/killing find 20+ times,
but after I started latencytop I couldn't reproduce.
But I can no longer reproduce it now, even if I stop latencytop (and
latencytop sysctl _is_ 0).

I haven't yet tried ftrace on this box, and neither did I try Roland's
patch yet. I will try that now, and hopefuly come back with some numbers
shortly.

[1]
I switch to text console, then I run this script in the background:

cd /debug/tracing
echo kill_pgrp detach_pid\
>set_ftrace_filter
echo ftrace >current_tracer
echo 1 >tracing_enabled
sleep 60
echo 0 >tracing_enabled
cat trace >/tmp/trace.out
cat latency_trace >/tmp/latencytrace.out

Then I run this script in the background, and wait until in shows
'Waiting for dd to finish':
dd if=/dev/zero of=xt bs=100M count=4&
dd if=/dev/zero of=yt bs=100M count=4&
rm xt
rm yt
echo "Waiting for dd to finish..."
wait %1 %2
echo "Done"

Then I start 'find / >/dev/null 2>/dev/null, and type ^C after about a
second'.
trace and latencytrace outputs below (for the 32-bit box):

# tracer: ftrace
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
events/0-9 [00] 450.146629: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 450.386653: detach_pid <-release_task
events/0-9 [00] 451.631327: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 452.394820: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 453.127932: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 453.143686: detach_pid <-release_task
bash-5821 [00] 453.143686: detach_pid <-release_task
bash-5821 [00] 453.143686: detach_pid <-release_task
^^^^^^^^^^^^ I had to press ^C 3 times here, and there is a ~2 sec delay

events/0-9 [00] 454.164452: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 455.027774: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 455.762679: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 455.770332: detach_pid <-release_task
bash-5821 [00] 455.770332: detach_pid <-release_task
bash-5821 [00] 455.770332: detach_pid <-release_task
^^^^^^^ again, a 1 second delay
events/0-9 [00] 456.718620: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 456.862658: detach_pid <-release_task
bash-5821 [00] 456.862658: detach_pid <-release_task
bash-5821 [00] 456.862658: detach_pid <-release_task
events/0-9 [00] 457.880204: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 457.908467: detach_pid <-release_task
bash-5821 [00] 457.908467: detach_pid <-release_task
bash-5821 [00] 457.908467: detach_pid <-release_task
events/0-9 [00] 459.238903: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 460.081542: detach_pid <-release_task
bash-5821 [00] 460.081542: detach_pid <-release_task
bash-5821 [00] 460.081542: detach_pid <-release_task
events/0-9 [00] 460.083701: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 462.246407: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 462.975581: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 463.583139: detach_pid <-release_task
^^^^^^^^^^^^^^^^^^^ 3.5 second delay here
bash-5821 [00] 463.583139: detach_pid <-release_task
bash-5821 [00] 463.583139: detach_pid <-release_task
events/0-9 [00] 463.699181: kill_pgrp <-n_tty_receive_buf
events/0-9 [00] 464.735022: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 465.094654: detach_pid <-release_task
bash-5821 [00] 465.094654: detach_pid <-release_task
bash-5821 [00] 465.094654: detach_pid <-release_task
events/0-9 [00] 466.537663: kill_pgrp <-n_tty_receive_buf
bash-5821 [00] 466.945909: detach_pid <-release_task

# tracer: ftrace
#
ftrace latency trace v1.1.5 on 2.6.26-rc9-tip-01669-g639a570
--------------------------------------------------------------------
latency: 0 us, #210/210, CPU#0 | (M:unknown VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
-----------------

# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| /
# ||||| delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
kstopmac-6225 0d..3 10us : detach_pid (release_task)
kstopmac-6225 0d..3 10us : detach_pid (release_task)
kstopmac-6225 0d..3 10us : detach_pid (release_task)
init-1 0d..3 10us : detach_pid (release_task)
init-1 0d..3 10us : detach_pid (release_task)
init-1 0d..3 10us!: detach_pid (release_task)
sh-6228 1d..3 3944272us+: detach_pid (release_task)
sh-6228 1d..3 3944273us : detach_pid (release_task)
sh-6228 1d..3 3944274us!: detach_pid (release_task)
sh-6228 1d..3 3945744us+: detach_pid (release_task)
sh-6228 1d..3 3945745us : detach_pid (release_task)
sh-6228 1d..3 3945746us!: detach_pid (release_task)
hal-syst-6233 0d..3 4156674us : detach_pid (release_task)
hal-syst-6233 0d..3 4156674us : detach_pid (release_task)
hal-syst-6233 0d..3 4156674us!: detach_pid (release_task)
hald-run-5416 0d..3 4160008us : detach_pid (release_task)
hald-run-5416 0d..3 4160008us : detach_pid (release_task)
hald-run-5416 0d..3 4160008us!: detach_pid (release_task)
events/0-9 0.... 9523335us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 9763359us : detach_pid (release_task)
bash-5821 0d..3 9763359us : detach_pid (release_task)
bash-5821 0d..3 9763359us!: detach_pid (release_task)
hal-syst-6236 0d..3 10167777us : detach_pid (release_task)
hal-syst-6236 0d..3 10167779us : detach_pid (release_task)
hal-syst-6236 0d..3 10167779us!: detach_pid (release_task)
hald-run-5416 0d..3 10172764us : detach_pid (release_task)
hald-run-5416 0d..3 10172765us : detach_pid (release_task)
hald-run-5416 0d..3 10172765us!: detach_pid (release_task)
events/0-9 0.... 11008033us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 11771525us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 12504637us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 12520392us : detach_pid (release_task)
bash-5821 0d..3 12520392us : detach_pid (release_task)
bash-5821 0d..3 12520392us!: detach_pid (release_task)
events/0-9 0.... 13541157us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 14404479us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 15139385us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 15147037us : detach_pid (release_task)
bash-5821 0d..3 15147037us : detach_pid (release_task)
bash-5821 0d..3 15147037us!: detach_pid (release_task)
events/0-9 0.... 16095325us!: kill_pgrp (n_tty_receive_buf)
hal-syst-6242 0d..3 16180193us : detach_pid (release_task)
hal-syst-6242 0d..3 16180193us : detach_pid (release_task)
hal-syst-6242 0d..3 16180193us!: detach_pid (release_task)
hald-run-5416 0d..3 16183420us : detach_pid (release_task)
hald-run-5416 0d..3 16183420us : detach_pid (release_task)
hald-run-5416 0d..3 16183420us!: detach_pid (release_task)
bash-5821 0d..3 16239363us : detach_pid (release_task)
bash-5821 0d..3 16239363us : detach_pid (release_task)
bash-5821 0d..3 16239363us!: detach_pid (release_task)
events/0-9 0.... 17256910us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 17285172us : detach_pid (release_task)
bash-5821 0d..3 17285172us : detach_pid (release_task)
bash-5821 0d..3 17285172us!: detach_pid (release_task)
events/0-9 0.... 18615608us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 19458248us : detach_pid (release_task)
bash-5821 0d..3 19458248us : detach_pid (release_task)
bash-5821 0d..3 19458248us!: detach_pid (release_task)
events/0-9 0.... 19460406us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 21623112us!: kill_pgrp (n_tty_receive_buf)
hal-syst-6247 0d..3 22194451us : detach_pid (release_task)
hal-syst-6247 0d..3 22194453us : detach_pid (release_task)
hal-syst-6247 0d..3 22194453us!: detach_pid (release_task)
hald-run-5416 0d..3 22199518us : detach_pid (release_task)
hald-run-5416 0d..3 22199520us : detach_pid (release_task)
hald-run-5416 0d..3 22199520us!: detach_pid (release_task)
events/0-9 0.... 22352287us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 22959844us : detach_pid (release_task)
bash-5821 0d..3 22959844us : detach_pid (release_task)
bash-5821 0d..3 22959844us!: detach_pid (release_task)
events/0-9 0.... 23075886us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 24111727us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 24471359us : detach_pid (release_task)
bash-5821 0d..3 24471359us : detach_pid (release_task)
bash-5821 0d..3 24471359us!: detach_pid (release_task)
events/0-9 0.... 25914368us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 26322615us : detach_pid (release_task)
bash-5821 0d..3 26322615us : detach_pid (release_task)
bash-5821 0d..3 26322615us!: detach_pid (release_task)
hal-syst-6252 0d..3 28207618us : detach_pid (release_task)
hal-syst-6252 0d..3 28207619us : detach_pid (release_task)
hal-syst-6252 0d..3 28207620us!: detach_pid (release_task)
hald-run-5416 0d..3 28212746us : detach_pid (release_task)
hald-run-5416 0d..3 28212747us : detach_pid (release_task)
hald-run-5416 0d..3 28212748us!: detach_pid (release_task)
events/0-9 0.... 28473354us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 29208378us!: kill_pgrp (n_tty_receive_buf)
pdflush-236 1d..3 29463682us+: detach_pid (release_task)
pdflush-236 1d..3 29463684us : detach_pid (release_task)
pdflush-236 1d..3 29463685us!: detach_pid (release_task)
bash-5821 0d..3 29633203us : detach_pid (release_task)
bash-5821 0d..3 29633203us : detach_pid (release_task)
bash-5821 0d..3 29633203us!: detach_pid (release_task)
events/0-9 0.... 31177193us!: kill_pgrp (n_tty_receive_buf)
events/0-9 0.... 32081149us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 32096726us : detach_pid (release_task)
bash-5821 0d..3 32096726us : detach_pid (release_task)
bash-5821 0d..3 32096726us!: detach_pid (release_task)
hal-syst-6255 0d..3 34221092us : detach_pid (release_task)
hal-syst-6255 0d..3 34221094us : detach_pid (release_task)
hal-syst-6255 0d..3 34221095us!: detach_pid (release_task)
hald-run-5416 0d..3 34226008us+: detach_pid (release_task)
hald-run-5416 0d..3 34226009us : detach_pid (release_task)
hald-run-5416 0d..3 34226010us!: detach_pid (release_task)
events/0-9 0.... 34943377us!: kill_pgrp (n_tty_receive_buf)
bash-5821 0d..3 34953441us : detach_pid (release_task)
bash-5821 0d..3 34953441us : detach_pid (release_task)
bash-5821 0d..3 34953441us!: detach_pid (release_task)
sh-6228 1d..3 38434166us : detach_pid (release_task)
sh-6228 1d..3 38434166us : detach_pid (release_task)
sh-6228 1d..3 38434166us!: detach_pid (release_task)
sh-6228 1d..3 38740305us+: detach_pid (release_task)
sh-6228 1d..3 38740307us : detach_pid (release_task)
sh-6228 1d..3 38740308us!: detach_pid (release_task)
bash-5821 0d..3 38744188us : detach_pid (release_task)
bash-5821 0d..3 38744188us : detach_pid (release_task)
bash-5821 0d..3 38744188us!: detach_pid (release_task)
bash-5821 0d..3 40161326us : detach_pid (release_task)
bash-5821 0d..3 40161327us : detach_pid (release_task)
bash-5821 0d..3 40161328us!: detach_pid (release_task)
hal-syst-6259 0d..3 40233791us : detach_pid (release_task)
hal-syst-6259 0d..3 40233792us : detach_pid (release_task)
hal-syst-6259 0d..3 40233793us!: detach_pid (release_task)
hald-run-5416 0d..3 40238677us : detach_pid (release_task)
hald-run-5416 0d..3 40238678us+: detach_pid (release_task)
hald-run-5416 0d..3 40238680us!: detach_pid (release_task)
bash-5821 1d..3 43913077us : detach_pid (release_task)
bash-5821 1d..3 43913077us : detach_pid (release_task)
bash-5821 1d..3 43913077us!: detach_pid (release_task)
bash-5821 0d..3 44968815us : detach_pid (release_task)
bash-5821 0d..3 44968815us : detach_pid (release_task)
bash-5821 0d..3 44968815us!: detach_pid (release_task)
hal-syst-6263 0d..3 46247842us : detach_pid (release_task)
hal-syst-6263 0d..3 46247844us : detach_pid (release_task)
hal-syst-6263 0d..3 46247845us!: detach_pid (release_task)
hald-run-5416 0d..3 46252860us : detach_pid (release_task)
hald-run-5416 0d..3 46252861us : detach_pid (release_task)
hald-run-5416 0d..3 46252862us!: detach_pid (release_task)
nullmail-5260 1d..3 46591091us : detach_pid (release_task)
nullmail-5260 1d..3 46591092us : detach_pid (release_task)
nullmail-5260 1d..3 46591093us!: detach_pid (release_task)
nullmail-5260 1d..3 46594439us : detach_pid (release_task)
nullmail-5260 1d..3 46594439us : detach_pid (release_task)
nullmail-5260 1d..3 46594439us!: detach_pid (release_task)
nullmail-5260 1d..3 46596485us : detach_pid (release_task)
nullmail-5260 1d..3 46596486us : detach_pid (release_task)
nullmail-5260 1d..3 46596487us!: detach_pid (release_task)
nullmail-5260 1d..3 46599932us : detach_pid (release_task)
nullmail-5260 1d..3 46599933us : detach_pid (release_task)
nullmail-5260 1d..3 46599934us!: detach_pid (release_task)
nullmail-5260 0d..3 46627792us : detach_pid (release_task)
nullmail-5260 0d..3 46627792us : detach_pid (release_task)
nullmail-5260 0d..3 46627792us!: detach_pid (release_task)
nullmail-5260 0d..3 46636851us : detach_pid (release_task)
nullmail-5260 0d..3 46636851us : detach_pid (release_task)
nullmail-5260 0d..3 46636851us!: detach_pid (release_task)
nullmail-5260 0d..3 46647738us : detach_pid (release_task)
nullmail-5260 0d..3 46647740us : detach_pid (release_task)
nullmail-5260 0d..3 46647740us!: detach_pid (release_task)
nullmail-5260 0d..3 46656857us : detach_pid (release_task)
nullmail-5260 0d..3 46656857us : detach_pid (release_task)
nullmail-5260 0d..3 46656857us!: detach_pid (release_task)
nullmail-5260 0d..3 46666791us : detach_pid (release_task)
nullmail-5260 0d..3 46666791us : detach_pid (release_task)
nullmail-5260 0d..3 46666791us!: detach_pid (release_task)
nullmail-5260 0d..3 46680090us : detach_pid (release_task)
nullmail-5260 0d..3 46680090us : detach_pid (release_task)
nullmail-5260 0d..3 46680090us!: detach_pid (release_task)
nullmail-5260 0d..3 46683330us : detach_pid (release_task)
nullmail-5260 0d..3 46683330us : detach_pid (release_task)
nullmail-5260 0d..3 46683330us!: detach_pid (release_task)
nullmail-5260 1d..3 46684182us : detach_pid (release_task)
nullmail-5260 1d..3 46684182us : detach_pid (release_task)
nullmail-5260 1d..3 46684182us!: detach_pid (release_task)
nullmail-5260 1d..3 46687500us : detach_pid (release_task)
nullmail-5260 1d..3 46687500us : detach_pid (release_task)
nullmail-5260 1d..3 46687500us!: detach_pid (release_task)
nullmail-5260 1d..3 46692670us : detach_pid (release_task)
nullmail-5260 1d..3 46692670us : detach_pid (release_task)
nullmail-5260 1d..3 46692670us!: detach_pid (release_task)
bash-5821 1d..3 47327646us : detach_pid (release_task)
bash-5821 1d..3 47327647us : detach_pid (release_task)
bash-5821 1d..3 47327648us!: detach_pid (release_task)
bash-6280 1d..3 47331181us : detach_pid (release_task)
bash-6280 1d..3 47331181us : detach_pid (release_task)
bash-6280 1d..3 47331181us!: detach_pid (release_task)
bash-5821 1d..3 47331699us : detach_pid (release_task)
bash-5821 1d..3 47331700us : detach_pid (release_task)
bash-5821 1d..3 47331701us!: detach_pid (release_task)
bash-6282 1d..3 47334166us : detach_pid (release_task)
bash-6282 1d..3 47334166us : detach_pid (release_task)
bash-6282 1d..3 47334166us : detach_pid (release_task)
bash-5821 1d..3 47334166us : detach_pid (release_task)
bash-5821 1d..3 47334166us : detach_pid (release_task)
bash-5821 1d..3 47334166us!: detach_pid (release_task)
hal-syst-6284 0d..3 52261078us : detach_pid (release_task)
hal-syst-6284 0d..3 52261079us : detach_pid (release_task)
hal-syst-6284 0d..3 52261080us!: detach_pid (release_task)
hald-run-5416 0d..3 52266033us+: detach_pid (release_task)
hald-run-5416 0d..3 52266034us : detach_pid (release_task)
hald-run-5416 0d..3 52266035us!: detach_pid (release_task)
hal-syst-6286 0d..3 58273784us : detach_pid (release_task)
hal-syst-6286 0d..3 58273786us : detach_pid (release_task)
hal-syst-6286 0d..3 58273786us!: detach_pid (release_task)
hald-run-5416 0d..3 58278681us : detach_pid (release_task)
hald-run-5416 0d..3 58278682us : detach_pid (release_task)
hald-run-5416 0d..3 58278683us!: detach_pid (release_task)
sh-6224 0d..3 60000916us+: detach_pid (release_task)
sh-6224 0d..3 60000918us : detach_pid (release_task)
sh-6224 0d..3 60000919us : detach_pid (release_task)

vim:ft=help

Best regards,
--Edwin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Török Edwin: "Re: [PATCH] x86_64: fix delayed signals"
Previous message: John Kacur: "Re: [PATCH -v2] ftrace: Documentation"
In reply to: Linus Torvalds: "Re: [PATCH] x86_64: fix delayed signals"
Next in thread: Török Edwin: "Re: [PATCH] x86_64: fix delayed signals"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]