isolcpus jitter performance

[Keith Kong]

Hi everyone,
Â
Newbie here to this list, so please be gentle 8-)
Â
Iâm doing some latency-sensitive work that requires isolating a CPU and running stuff on it with as little perturbance as possible. Someone pointed me to isolcpus and Max Krasnyanskyâs work on preventing IRQs from being handled by isolated CPUs. Exactly what I needed, so I went ahead and ftped kernel 2.6.25.6 and patched it with this: http://kernel.org/pub/linux/kernel/people/maxk/cpuisol/patches/v2.6.25.6.cpuisol1.patch
Â
But the measured jitter is much bigger than âsingle digit usec worst case and around 200nsec ave response timeâ Â(http://readlist.com/lists/vger.kernel.org/linux-kernel/89/447300.html).
Â
So a couple of questions:

1. ***Has anyone gotten the kinds of results above? If so, how did you do it, along with your definition of âworst case and ave response time.â*** If your source code is not too big, attaching it would be good. 

2. The methodology I used is below. *** Any issues with it? ***

  1. Patch kernel 2.6.25.6 with v2.6.25.6.cpuisol1.patch, RedHat Enterprise AS4, on 8GB dual Xeon quad core (8 cores) 2.67GHz machine. No load on the machine (according to âtopâ).

  2. Add kernel parms: isolcpus=4,5,6,7 

  3. Ensure /etc/sysconfig/syspart has PAR0_CPUS=â0-3â so that CPUs 0-3 handle general kernel and user tasks, and CPUs 4-7 are isolated 
Ensure /etc/init.d/syspart has âactivatedâ the partition 

  4. Collected Pentium clock register values via rdtscll() using following code (ensuring cpu affinity set correctly): 
Â
ÂÂÂÂÂ #define BUFSIZE 1000000
ÂÂÂÂÂ #define LOOPCNT 10000
      for ( i = 0; i < BUFSIZE; i++ ) {
ÂÂÂÂÂÂÂ for ( j = 0; j < LOOPCNT; j++ ) {}
ÂÂÂÂÂÂÂ rdtscll(gullStartCpuTime); /* Pentium clock register */
ÂÂÂÂÂÂÂ ullTimeStampBuf[i] = gullStartCpuTime;
ÂÂÂ Â }

  5. Dumped the ullTimeStampBuf[] data into a file and do post-process. Definition of jitter and details of post-processing below. 
Â
The kinds of results Iâm getting are:

Max jitter (max diff between consecutive Pentium counter values):
*** ~ 460 usec ***

Ave. jitter (std. deviation of diff between consecutive Pentium counters): *** ~ 5.7 usec ***
Â
Â
Thanks!
Â
Â
Keith

----- detailed explanation of methodology below --- 
All the data is based on computing statistics on âdiff and diff of diffâ between successive Pentium counters. For example:
Â
83228160 71336 0 0
83299496 71336 0 0
83370832 71336 0 0
83442168 71336 0 0
83513528 71360 24 0.00900003262511827
83584864 71336 -24 -0.00900003262511827
83656200 71336 0 0
83727536 71336 0 0
83798872 71336 0 0
Â
The first column is Pentium counter, second column is diff of previous and current counter, third column is diff of this diff. Last column is simply third column converted into microseconds.
Â
The third column is added to some array, which is then fed as input to some statistics package to provide results. Note that STDDEV (usec): 5.7567 means standard deviation of the third column.
Â
Â
Â
SAMPLE INFO FILE
------------------------------
STATISTICS
----------
ÂDIFF SAMPLE COUNT: 999998
ÂÂ DIFF MIN (usec):ÂÂ 26.7449164221
ÂÂ DIFF MAX (usec):Â 462.6135543326
 DIFF MEAN (usec): 26.8811017636
DIFF MEDIAN (usec):ÂÂ 26.7509164034
DIFF STDDEV (usec):ÂÂÂ 5.7567862963
ÂÂ DIFF VAR (usec):ÂÂ 33.1405884609
Â
Â
ÂDIFFDIFF SAMPLE COUNT: 999997
ÂÂ DIFFDIFF MIN (usec): -435.1036403011
ÂÂ DIFFDIFF MAX (usec):Â 435.8626379293
 DIFFDIFF MEAN (usec): -0.0000089190
DIFFDIFF MEDIAN (usec):ÂÂÂ 0.0000000000
DIFFDIFF STDDEV (usec):ÂÂÂ 8.1034032634
ÂÂ DIFFDIFF VAR (usec):ÂÂ 65.6651444491
Â
Â
PENTIUM COUNTER ROLLOVER SANITY CHECK
-------------------------------------
ÂÂÂÂÂÂÂÂÂ GETTIMEOFDAY LAPSED TIME (usec): 26881133
ÂTOTAL PENTIUM COUNTER LAPSED TIME (usec): 26881048.00135
ÂÂÂÂÂÂÂÂÂÂÂÂÂ DIFF OF LAPSED TIMES (usec): 84.9986500032246
[DIFF TIME OF 84.9986500032246 usec < 10 ms (resolution of gettimeofday() on linux), SO GOOD CONFIDENCE ROLLOVERS ACCOUNTED FOR PROPERLY]
Â
Â
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/