Re: RFD: nanoseconds, rdtsc and SMP

Colin Plumb (colin@nyx.net)
Thu, 25 Feb 1999 04:17:48 -0700 (MST)

Ulrich Windl wrote:
> Hello,
>
> another request for discussion:
>
> With microseconds resolution there were many CPU cycles per
> microsecond, but for nanoseconds there are multiple nanoseconds per
> CPU cycle (at least this year).
>
> Therefore variations when reading the cycle counter result in time
> jitter at least, eventually maybe even in time running virtually
> backwards occasionally.
>
> The question is: "what assumptions can be made?" On the i386
> architecture, will all cycle counters start at the same moment, and
> will they be bound to the same oscillator? If not one has to
> calibrate each CPU, and remember the cycle counter of each CPU during
> timer interrupt. When getting the time one must find the cycle
> counter of the own CPU and subtract that counter at the last
> interrupt to get the difference. Other architectures maybe even worse.
>
> Colin Plumb suggested to synchronize the cycle counters on i386
> architecture, assuming they'll remain in sync. This would make the
> time code much easier, but break things terrible, if the counters
> drift apart.

That was actually someone else's suggestion, but I'm inclined to agree.
Currently they are always synchronous on x86, but there is sometimes
an offset. Since the counters can be written, that can be fixed
at boot time and forgotten about. (Until someone starts doing evil things
with APM on multiprocessors.)

The problem is that some architectures might, and the Alpha currently
*does*, use asynchronous clocks. This makes things much hairier, because
the phases drift as the system runs, and you need to know the processor
you're running on to interpret the cycle counter.

On an x86, I've thought of just exporting the conversion factors
(from cycle counter to real time) to userland through shared memory
and letting the library implement gettimeofday() without a system call.
(There would, of course, be an exported "valid" flag without which
the library would do an old-fashioned system call.)

This doesn't work, however, unless you can determine the cycle counter
and the processor number atomically with respect to process scheduling.
(The Alpha only provides a 32-bit cycle counter; the upper 32 bits of
the register are software controllable, which permits a fix for it,
but other systems may have problems.)

My ambition is to phase-lock the counters almost exactly, to within 1
tick (3 ns or less). This requires symmetric round-trip message timing,
like what NTP does over the net, but over the bus. That's the only way to
correct for the ~50-clock one-way multiprocessor synchronization overhead.

This doesn't take a lot of wall-clock time to do once at boot time, but
it requires a lot of blocking on spin locks, which is extremely painful
on a running system. I'm not sure how to accurately measure the
interprocessor phases on a running system.

(Measuring phases without bidirectional communication is *very*
difficult. I'm working on algorithms for it, but they're a pain
and I'm not sure about their stability.)

-- 
	-Colin


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/