Re: 2.6.14-rc4-rt7
From: George Anzinger
Date: Tue Oct 25 2005 - 15:13:12 EST
john stultz wrote:
On Tue, 2005-10-25 at 17:44 +0200, Ingo Molnar wrote:
John
i found one source of timekeeping bugs on SMP boxes, it's the
non-monotonicity of the TSC:
... time warped from 1270809453 to 1270808096.
... MTSC warped from 0000000a731a8c3c [0] to 0000000a731a899c [2].
... MTSC warped from 0000000a7c93baec [0] to 0000000a7c93b7a8 [3].
... MTSC warped from 0000000a881d6afc [0] to 0000000a881d67d0 [2].
... MTSC warped from 0000000a924217a0 [0] to 0000000a924216ac [3].
... MTSC warped from 0000000a9c592788 [0] to 0000000a9c59232c [2].
... MTSC warped from 0000000aa7aa95c8 [0] to 0000000aa7aa9338 [3].
... MTSC warped from 0000000b33206d60 [0] to 0000000b33206a48 [3].
... time warped from 26699635824 to 26699633144.
... MTSC warped from 00000013f379cb88 [0] to 00000013f379c7e0 [3].
... MTSC warped from 0000001413df8660 [0] to 0000001413df8200 [3].
... MTSC warped from 00000014194f5360 [1] to 00000014194f51b0 [2].
... time warped from 60775269225 to 60775266727.
the number in square brackets is the CPU#. I.e. CPUs on this 4-CPU box
have small TSC differences, which ends up leaking into the generic TOD
code, causing real time warps, which causes ktimer weirdnesses (timers
failed to expire, etc.).
(the above output tracks TSC results globally, under a spinlock. It also
detects time-warps that propagate into the monotonic clock output.)
unfortunately, there's no easy solution for this. We could make
cycle_last per-CPU, but that again brings up the question of how to set
up the per-CPU 'TSC offset' values - those would need similar technique
that the current clear-all-TSCs-on-all-CPUs code does - which as we can
see failed ...
Indeed. This is a nasty issue can affect a number of different systems.
The best solution in my mind is to utilize alternative clocksources when
necessary (one of the main reasons for creating the flexible clocksource
interface: so we can easily use something else).
The TSC is such a fast and, usually, accurate answer, I think it deserves a little effort to save
it. With your new clock code I think we could use per cpu TSC counters, read the full 64 bits and,
in real corner cases, even per cpu conversion "constants" and solve this problem.
George
In my patches, I have a function mark_tsc_unstable(), when called will
drop the tsc's rating value and will cause another clocksource to be
chosen (as long as one is available). Right now we call it when we know
the TSC is going to have problems. But maybe we should be more dynamic
in our detection.
Do you have any details about the hardware? Are the TSCs not being
synced well enough, or are they falling out of sync? i386 is a bit more
aggressive about using the TSC in SMP systems, where x86-64 has more
conditionals. Maybe some of the x86-64 logic should be moved to i386 as
well.
thanks
-john
--
George Anzinger george@xxxxxxxxxx
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/