user/sys ticks for process exceeds overall user/sys ticks

[Peter Hofer]

Hello everyone,

 I am trying to determine what fraction of the whole system's user and
sys time are consumed by a specific process, over a period of time.
Hence, I read the respective tick counts from the first lines
of /proc/stat and /proc/<pid>/stat at the start and end of that period
and compute deltas.

However, on an otherwise idle system, I often see the ticks consumed by
a threaded process exceed those of the whole system. The difference is
not just a few ticks, but can amount to over 10%, regardless of how
long the observed time period is. There is no virtualization involved.
I have been able to reproduce this with the current openSUSE and Ubuntu
releases on different x86_64 machines.

How can this be explained? Intuitively, I would expect that a tick
accounted to a running task is at the same time accounted to the CPU it
is running on, and so a process can never be accounted more ticks than
the whole system, i.e. all CPUs.

I have attached a simple Python script to reproduce this behavior.
Example output:

> ./ticks.py 
delta process: user: 2495 sys: 181
delta all: user: 2283 sys: 258

Thanks,
 Peter#!/usr/bin/python
import operator
import hashlib
from threading import Thread

def ticks_all():
  with open('/proc/stat') as f:
    cpu = f.readline().split()
    return (int(cpu[1]), int(cpu[3]))

def ticks_process():
  with open('/proc/self/stat') as f:
    cpu = f.readline().split()
    return (int(cpu[13]), int(cpu[14]))

def do_work():
  d = hashlib.md5()
  d.update('nobody inspects')
  for i in xrange(0, 10000000):
    d.update(' the spammish repetition')

before_all_user, before_all_sys = ticks_all()
before_process_user, before_process_sys = ticks_process()

threads = []
for i in xrange(0, 8):
  t = Thread(target=do_work)
  threads.append(t)
  t.start()
for t in threads:
  t.join()

after_process_user, after_process_sys = ticks_process()
after_all_user, after_all_sys = ticks_all()

print 'delta process: user:', after_process_user - before_process_user, 'sys:', after_process_sys - before_process_sys
print 'delta all: user:', after_all_user - before_all_user, 'sys:', after_all_sys - before_all_sys