Very HIGH system load on 4.4.x kernels ?

[Vellemans, Noel]

Hi all, 

I'm facing a problem , for multiple weeks already.

I have ported a recent kernel (4.4.x branch) to one of our custom ARM ( imx53-based ) boards. 
This board has been running kernel 2.6.35.x for some years, and at this time we decided we needed newer kernel ( for wifi operation), so far so good.


When running this a NEW (recent) kernel 4.4.61 (or even newer 4.4.x Lets say) I see a HIGH CPU load compared to the older 2.6.35 kernel ( when running the same applications) 

Comparing systems is difficult I know.. but... What has been changed ? only the KERNEL ( same hardware, same gcc-compiler, same C-lib, same user-space applications) only the KERNEL has been upgraded (nothing else) , both boards I compare are identical .. running at the same clock-speed... same amount of ram / storage. ! 100% sure on this ! . 

When comparing 2.6.35.x to 4.4.x kernels I see that a LARGE fraction of the CPU power is consumed into system ( looking at TOP)


I know that I moved couple of versions ( 2.6.35 -> 4.4.x is quite a big step there is no discussion about that, we all know that, and want to understand that ) 

BUT I did not expect the same 'user application' (for example TOP / HTOP , or my custom application) to use that MUCH CPU ( running on 100% the same hardware configuration ). 
{ did not recompile any application, the only thing that has been upgraded is the kernel nothing else , same compiler, same Clib, etc..} 

Reaction speed/time of my custom app is very bad.. when I compare .. 2.6.35 to 4.4.x ( even with the same (aprox the same) kernel configuration)

I do compare the reaction speed of both kernels on the same user application, and I can't understand that an 'empty/clean' Linux system (custom app being removed) just running TOP (or HTOP) on a ARM CPU (running at 1GHZ) is causing 31% of cpu load on a 4.4 l kernel and 1% on 2.6.35-kernel.  { I show you here the results on top/htop because everyone know these, but the 'real' issue I'm fighting with is the fact that all is responding very badly on those 4.4.x kernels ! }

whatever you seem to do on a 4.4.x kernel .. it seems to be using that much cpu very quickly, and due to this high cpu load all applications ( even bare basic system application like top/htop / watch .. etc ... ) are NOT running smooth !


Both system I compare have been stripped down to a 'bare' minimum required.

( I could send you a screenshot .. of both if you want.. 100% the same processes are running... same builds, same versions, build with same compiler, running on the same HARDWARE , the SINGLE change is the kernel-version 2.6.35 vs 4.4.x ! )

I have exactly the same number of TASKS/processes (in htop) and I have exactly the same process layout (when comparing both).

On the 4.4.x kernel I see HIGH CPU load, 'system load' ( what means kernel load) on the 2.6.35 kernel.. I have way less cpu load. ( Running exactly the same number of applications)

On the 4.4.x system I see a load up to +31% (system) on HTOP/TOP while I see for the 2.6.35 kernel... a load of 1% ... with the SAME processes running ! ( you can count on that )

Spending 31% load on a 1GHZ CPU ... with a system that is nearly 'empty' while on the same hardware and a 2.6.35. kernel ( 100% same setup , same process tree ) I nearly have 1% CPU load.


And the load is that different on both kernels you can 'feel' this ( you can see this in almost every action you take, for example on display drawing, reaction on key-input.. etc etc...)

(since I can not include pictures here I've include some 'live' textual info)


LOOK at the results below ....

Best Regards 

ANY HINTS.. I've been digging for a long time.. whatever I tried... cpu load stays high !



====================================4.4.x=========================

Kernel 4.4.x

uname -a
Linux DU11 4.4.73 #3 PREEMPT Tue Jul 4 09:06:46

#top
Mem: 176076K used, 844800K free, 0K shrd, 44K buff, 44K cached
CPU: 0% usr 31% sys 0% nic 68% idle 0% io 0% irq 0% sirq
Load average: 1.31 1.41 1.43 1/85 12125
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
12125 21246 root R 1184 0% 23% top
12142 1 root S 3728 0% 0% /usr/sbin/mosquitto -c /etc/mosquitto/

# htop
CPU[##********** 40.0%] Tasks: 32, 0 thr; 1 running
Mem[|||||** 142/978MB] Load average: 7.20 7.18 7.12
Swp[ 0/0MB] Uptime: 16:22:48

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
30886 root 20 0 1064 576 432 R 33.0 0.1 0:00.06 htop

====================================2.6.35=========================


Kernel 2.6.35
uname -a
Linux OLD 2.6.35.3 #1 PREEMPT Tue Jun 14 13:45:24 

#top
Mem: 97064K used, 904376K free, 0K shrd, 0K buff, 46160K cached
CPU: 0% usr 0% sys 0% nic 99% idle 0% io 0% irq 0% sirq
Load average: 1.41 1.27 1.26 2/70 18808
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
18696 30503 root R 1176 0% 0% top
842 1 root S 3928 0% 0% /usr/sbin/mosquitto -c /etc/mosquitto/



#htop
CPU[#* 1.5%] Tasks: 32, 0 thr; 1 running
Mem[||||** 126/996MB] Load average: 1.21 1.23 1.30
Swp[ 0/0MB] Uptime: 2 days, 00:05:53

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
13431 root 20 0 1076 772 628 R 1.0 0.1 0:00.66 htop