Re: [PATCH RFC] x86, tsc: Allow for high latency in quick_pit_calibrate()
From: Ingo Molnar
Date: Thu Jun 04 2015 - 08:32:54 EST
* H. Peter Anvin <hpa@xxxxxxxxx> wrote:
> [...]
>
> The RTC is probably the most reliable reference clock, in part because 32 kHz
> crystals are generally calibrated and extremely stable. However, to get more
> than 1 Hz frequency out of it you have to enable interrupts (which gets you to
> 8192 Hz). I have heard claims that this is actually what Windows uses for
> calibration, but I don't know.
So I ran a few tests to determine how reliable the RTC clock is at 8192 Hz, and
running it during early bootup gives this type of jitter:
[ 0.000000] tsc: RTC IRQ 0, at 30566853736, delta: 30, jitter: 30
[ 0.000000] tsc: RTC IRQ 1, at 30567333663, delta: 479927, jitter: 479897
[ 0.000000] tsc: RTC IRQ 2, at 30567578651, delta: 244988, jitter: -234939
[ 0.000000] tsc: RTC IRQ 3, at 30567825147, delta: 246496, jitter: 1508
[ 0.000000] tsc: RTC IRQ 4, at 30568070180, delta: 245033, jitter: -1463
[ 0.000000] tsc: RTC IRQ 5, at 30568315220, delta: 245040, jitter: 7
[ 0.000000] tsc: RTC IRQ 6, at 30568561747, delta: 246527, jitter: 1487
[ 0.000000] tsc: RTC IRQ 7, at 30568806794, delta: 245047, jitter: -1480
[ 0.000000] tsc: RTC IRQ 8, at 30569051789, delta: 244995, jitter: -52
[ 0.000000] tsc: RTC IRQ 9, at 30569296824, delta: 245035, jitter: 40
[ 0.000000] tsc: RTC IRQ 10, at 30569543304, delta: 246480, jitter: 1445
[ 0.000000] tsc: RTC IRQ 11, at 30569788346, delta: 245042, jitter: -1438
[ 0.000000] tsc: RTC IRQ 12, at 30570033393, delta: 245047, jitter: 5
[ 0.000000] tsc: RTC IRQ 13, at 30570278433, delta: 245040, jitter: -7
[ 0.000000] tsc: RTC IRQ 14, at 30570524894, delta: 246461, jitter: 1421
[ 0.000000] tsc: RTC IRQ 15, at 30570769950, delta: 245056, jitter: -1405
[ 0.000000] tsc: RTC IRQ 16, at 30571014993, delta: 245043, jitter: -13
[ 0.000000] tsc: RTC IRQ 17, at 30571260024, delta: 245031, jitter: -12
[ 0.000000] tsc: RTC IRQ 18, at 30571506504, delta: 246480, jitter: 1449
[ 0.000000] tsc: RTC IRQ 19, at 30571751560, delta: 245056, jitter: -1424
[ 0.000000] tsc: RTC IRQ 20, at 30571996577, delta: 245017, jitter: -39
[ 0.000000] tsc: RTC IRQ 21, at 30572243072, delta: 246495, jitter: 1478
[ 0.000000] tsc: RTC IRQ 22, at 30572488121, delta: 245049, jitter: -1446
[ 0.000000] tsc: RTC IRQ 23, at 30572733125, delta: 245004, jitter: -45
[ 0.000000] tsc: RTC IRQ 24, at 30572978181, delta: 245056, jitter: 52
[ 0.000000] tsc: RTC IRQ 25, at 30573224673, delta: 246492, jitter: 1436
[ 0.000000] tsc: RTC IRQ 26, at 30573469712, delta: 245039, jitter: -1453
[ 0.000000] tsc: RTC IRQ 27, at 30573714760, delta: 245048, jitter: 9
[ 0.000000] tsc: RTC IRQ 28, at 30573959800, delta: 245040, jitter: -8
[ 0.000000] tsc: RTC IRQ 29, at 30574206272, delta: 246472, jitter: 1432
[ 0.000000] tsc: RTC IRQ 30, at 30574451320, delta: 245048, jitter: -1424
[ 0.000000] tsc: RTC IRQ 31, at 30574696335, delta: 245015, jitter: -33
[ 0.000000] tsc: RTC IRQ 32, at 30574941390, delta: 245055, jitter: 40
[ 0.000000] tsc: RTC IRQ 33, at 30575187872, delta: 246482, jitter: 1427
[ 0.000000] tsc: RTC IRQ 34, at 30575432905, delta: 245033, jitter: -1449
[ 0.000000] tsc: RTC IRQ 35, at 30575677944, delta: 245039, jitter: 6
[ 0.000000] tsc: RTC IRQ 36, at 30575924434, delta: 246490, jitter: 1451
[ 0.000000] tsc: RTC IRQ 37, at 30576169465, delta: 245031, jitter: -1459
[ 0.000000] tsc: RTC IRQ 38, at 30576414504, delta: 245039, jitter: 8
[ 0.000000] tsc: RTC IRQ 39, at 30576659542, delta: 245038, jitter: -1
[ 0.000000] tsc: RTC IRQ 40, at 30576906030, delta: 246488, jitter: 1450
[ 0.000000] tsc: RTC IRQ 41, at 30577151072, delta: 245042, jitter: -1446
[ 0.000000] tsc: RTC IRQ 42, at 30577396105, delta: 245033, jitter: -9
[ 0.000000] tsc: RTC IRQ 43, at 30577641144, delta: 245039, jitter: 6
[ 0.000000] tsc: RTC IRQ 44, at 30577887632, delta: 246488, jitter: 1449
[ 0.000000] tsc: RTC IRQ 45, at 30578132665, delta: 245033, jitter: -1455
[ 0.000000] tsc: RTC IRQ 46, at 30578377704, delta: 245039, jitter: 6
[ 0.000000] tsc: RTC IRQ 47, at 30578622749, delta: 245045, jitter: 6
[ 0.000000] tsc: RTC IRQ 48, at 30578869230, delta: 246481, jitter: 1436
[ 0.000000] tsc: RTC IRQ 49, at 30579114272, delta: 245042, jitter: -1439
[ 0.000000] tsc: RTC IRQ 50, at 30581814501, delta: 2700229, jitter: 2455187
[ 0.000000] tsc: RTC IRQ 51, at 30582059549, delta: 245048, jitter: -2455181
[ 0.000000] tsc: RTC IRQ 52, at 30582304584, delta: 245035, jitter: -13
[ 0.000000] tsc: RTC IRQ 53, at 30582549631, delta: 245047, jitter: 12
[ 0.000000] tsc: RTC IRQ 54, at 30582796117, delta: 246486, jitter: 1439
[ 0.000000] tsc: RTC IRQ 55, at 30583041136, delta: 245019, jitter: -1467
[ 0.000000] tsc: RTC IRQ 56, at 30583286184, delta: 245048, jitter: 29
[ 0.000000] tsc: RTC IRQ 57, at 30583532682, delta: 246498, jitter: 1450
[ 0.000000] tsc: RTC IRQ 58, at 30583777720, delta: 245038, jitter: -1460
[ 0.000000] tsc: RTC IRQ 59, at 30584022745, delta: 245025, jitter: -13
[ 0.000000] tsc: RTC IRQ 60, at 30584267784, delta: 245039, jitter: 14
[ 0.000000] tsc: RTC IRQ 61, at 30584514273, delta: 246489, jitter: 1450
[ 0.000000] tsc: RTC IRQ 62, at 30584759311, delta: 245038, jitter: -1451
[ 0.000000] tsc: RTC IRQ 63, at 30585004357, delta: 245046, jitter: 8
[ 0.000000] tsc: RTC IRQ 64, at 30585249385, delta: 245028, jitter: -18
[ 0.000000] tsc: RTC IRQ 65, at 30585495880, delta: 246495, jitter: 1467
[ 0.000000] tsc: RTC IRQ 66, at 30585740908, delta: 245028, jitter: -1467
[ 0.000000] tsc: RTC IRQ 67, at 30585985953, delta: 245045, jitter: 17
[ 0.000000] tsc: RTC IRQ 68, at 30586232425, delta: 246472, jitter: 1427
[ 0.000000] tsc: RTC IRQ 69, at 30586477464, delta: 245039, jitter: -1433
[ 0.000000] tsc: RTC IRQ 70, at 30586722493, delta: 245029, jitter: -10
[ 0.000000] tsc: RTC IRQ 71, at 30586967516, delta: 245023, jitter: -6
[ 0.000000] tsc: RTC IRQ 72, at 30587213989, delta: 246473, jitter: 1450
[ 0.000000] tsc: RTC IRQ 73, at 30587459029, delta: 245040, jitter: -1433
[ 0.000000] tsc: RTC IRQ 74, at 30587704077, delta: 245048, jitter: 8
[ 0.000000] tsc: RTC IRQ 75, at 30587949125, delta: 245048, jitter: 0
[ 0.000000] tsc: RTC IRQ 76, at 30588195597, delta: 246472, jitter: 1424
[ 0.000000] tsc: RTC IRQ 77, at 30588440629, delta: 245032, jitter: -1440
[ 0.000000] tsc: RTC IRQ 78, at 30588685660, delta: 245031, jitter: -1
[ 0.000000] tsc: RTC IRQ 79, at 30588930716, delta: 245056, jitter: 25
[ 0.000000] tsc: RTC IRQ 80, at 30589177189, delta: 246473, jitter: 1417
[ 0.000000] tsc: RTC IRQ 81, at 30589422229, delta: 245040, jitter: -1433
[ 0.000000] tsc: RTC IRQ 82, at 30589667285, delta: 245056, jitter: 16
[ 0.000000] tsc: RTC IRQ 83, at 30589913760, delta: 246475, jitter: 1419
[ 0.000000] tsc: RTC IRQ 84, at 30590158798, delta: 245038, jitter: -1437
[ 0.000000] tsc: RTC IRQ 85, at 30590403837, delta: 245039, jitter: 1
[ 0.000000] tsc: RTC IRQ 86, at 30590648866, delta: 245029, jitter: -10
[ 0.000000] tsc: RTC IRQ 87, at 30590895346, delta: 246480, jitter: 1451
[ 0.000000] tsc: RTC IRQ 88, at 30591140397, delta: 245051, jitter: -1429
[ 0.000000] tsc: RTC IRQ 89, at 30591385446, delta: 245049, jitter: -2
[ 0.000000] tsc: RTC IRQ 90, at 30591630477, delta: 245031, jitter: -18
[ 0.000000] tsc: RTC IRQ 91, at 30591876949, delta: 246472, jitter: 1441
[ 0.000000] tsc: RTC IRQ 92, at 30592121998, delta: 245049, jitter: -1423
[ 0.000000] tsc: RTC IRQ 93, at 30592367029, delta: 245031, jitter: -18
[ 0.000000] tsc: RTC IRQ 94, at 30592612075, delta: 245046, jitter: 15
[ 0.000000] tsc: RTC IRQ 95, at 30592858555, delta: 246480, jitter: 1434
[ 0.000000] tsc: RTC IRQ 96, at 30593103597, delta: 245042, jitter: -1438
[ 0.000000] tsc: RTC IRQ 97, at 30593348630, delta: 245033, jitter: -9
[ 0.000000] tsc: RTC IRQ 98, at 30593595101, delta: 246471, jitter: 1438
[ 0.000000] tsc: RTC IRQ 99, at 30593840157, delta: 245056, jitter: -1415
[ 0.000000] tsc: RTC IRQ 100, at 30594085197, delta: 245040, jitter: -16
The time measurements are in cycles, the TSC on this box runs at 2.010 GHz.
What you can see is that the jitter is pretty stable, with some harmonics in it.
There's outliners that occur at about 80 Hz frequency:
[ 0.000000] tsc: RTC IRQ 50, at 30581814501, delta: 2700229, jitter: 2455187
[ 0.000000] tsc: RTC IRQ 51, at 30582059549, delta: 245048, jitter: -2455181
That's too frequent to be the RTC update delay - but its length, 1.3 msecs, comes
close to that delay - so maybe that is it. We could filter out these outliner
during calibration.
With that I think we could get down to a jitter of around 10 ppm with just a few
dozen samples - i.e. collect it all in just 2-3 milliseconds.
So that seems desirable and in would bring our calibration accuracy to the same
level as our delayed-work based 'refined TSC' calibration method is.
The big practical problem I found is implementational. Boot dependencies are
horrible:
- one problem is that the IO-APIC is not enabled yet when we calibrate these
values. With vile hacks I was able to enable the RTC during early boot by
forcing an early init of the IO-APIC, but I found no robust method.
- there's also a catch-22 with time init: IO-APIC init relies on time init. I
wasn't successful at untangling it.
- I also tried to activate the RTC IRQ in the i8259 PIC during early boot, so
that we can just use it to sample the RTC and then forget about that state, but
didn't succeed.
all in one, receiving RTC IRQs so early in the bootup was a pretty fragile affair.
- Alternatively, I also tried a different method: to set up the RTC periodic IRQ
during early boot, but not have an IRQ handler, polling RTC_PF in the
rtc_cmos_read(RTC_INTR_FLAGS) IRQ status byte.
Unfortunately when I do this then PIO based RTC accesses can take tens of
thousands of cycles, and the resulting jitter is pretty bad and hard to filter:
[ 0.000000] tsc: RTC edge 57 from 0 to 64, at 29694678517, delta: 246360, jitter: 2456, loops: 7, 35194 cycles/loop
[ 0.000000] tsc: RTC edge 58 from 0 to 64, at 29695169485, delta: 490968, jitter: 244608, loops: 118, 4160 cycles/loop
[ 0.000000] tsc: RTC edge 59 from 0 to 64, at 29695413981, delta: 244496, jitter: -246472, loops: 6, 40749 cycles/loop
[ 0.000000] tsc: RTC edge 60 from 0 to 64, at 29695660661, delta: 246680, jitter: 2184, loops: 7, 35240 cycles/loop
[ 0.000000] tsc: RTC edge 61 from 0 to 64, at 29695904853, delta: 244192, jitter: -2488, loops: 6, 40698 cycles/loop
[ 0.000000] tsc: RTC edge 62 from 0 to 64, at 29696151141, delta: 246288, jitter: 2096, loops: 7, 35184 cycles/loop
[ 0.000000] tsc: RTC edge 63 from 0 to 64, at 29696396445, delta: 245304, jitter: -984, loops: 6, 40884 cycles/loop
[ 0.000000] tsc: RTC edge 64 from 0 to 64, at 29696642669, delta: 246224, jitter: 920, loops: 7, 35174 cycles/loop
[ 0.000000] tsc: RTC edge 65 from 0 to 64, at 29696887245, delta: 244576, jitter: -1648, loops: 6, 40762 cycles/loop
[ 0.000000] tsc: RTC edge 66 from 0 to 64, at 29697377909, delta: 490664, jitter: 246088, loops: 117, 4193 cycles/loop
[ 0.000000] tsc: RTC edge 67 from 0 to 64, at 29697622701, delta: 244792, jitter: -245872, loops: 6, 40798 cycles/loop
[ 0.000000] tsc: RTC edge 68 from 0 to 64, at 29697868773, delta: 246072, jitter: 1280, loops: 7, 35153 cycles/loop
[ 0.000000] tsc: RTC edge 69 from 0 to 64, at 29700569301, delta: 2700528, jitter: 2454456, loops: 13, 207732 cycles/loop
[ 0.000000] tsc: RTC edge 70 from 0 to 64, at 29700813805, delta: 244504, jitter: -2456024, loops: 6, 40750 cycles/loop
[ 0.000000] tsc: RTC edge 71 from 0 to 64, at 29701060125, delta: 246320, jitter: 1816, loops: 7, 35188 cycles/loop
[ 0.000000] tsc: RTC edge 72 from 0 to 64, at 29701550189, delta: 490064, jitter: 243744, loops: 117, 4188 cycles/loop
[ 0.000000] tsc: RTC edge 73 from 0 to 64, at 29701796677, delta: 246488, jitter: -243576, loops: 7, 35212 cycles/loop
[ 0.000000] tsc: RTC edge 74 from 0 to 64, at 29702040829, delta: 244152, jitter: -2336, loops: 6, 40692 cycles/loop
[ 0.000000] tsc: RTC edge 75 from 0 to 64, at 29702287597, delta: 246768, jitter: 2616, loops: 7, 35252 cycles/loop
[ 0.000000] tsc: RTC edge 76 from 0 to 64, at 29702531741, delta: 244144, jitter: -2624, loops: 6, 40690 cycles/loop
[ 0.000000] tsc: RTC edge 77 from 0 to 64, at 29702778341, delta: 246600, jitter: 2456, loops: 7, 35228 cycles/loop
[ 0.000000] tsc: RTC edge 78 from 0 to 64, at 29703022661, delta: 244320, jitter: -2280, loops: 6, 40720 cycles/loop
[ 0.000000] tsc: RTC edge 79 from 0 to 64, at 29703514245, delta: 491584, jitter: 247264, loops: 118, 4165 cycles/loop
[ 0.000000] tsc: RTC edge 80 from 0 to 64, at 29703759165, delta: 244920, jitter: -246664, loops: 6, 40820 cycles/loop
[ 0.000000] tsc: RTC edge 81 from 0 to 64, at 29704005397, delta: 246232, jitter: 1312, loops: 7, 35176 cycles/loop
[ 0.000000] tsc: RTC edge 82 from 0 to 64, at 29704249589, delta: 244192, jitter: -2040, loops: 6, 40698 cycles/loop
and this was on a pretty standard whitebox PC, I'm pessimistic about how this
would work on more exotic machines.
So I think we should stay with the PIT fast-calibration method, it's very fast in
most cases.
Thanks,
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/