Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Ingo Molnar <mingo@xxxxxxxxxx> wrote:

> With nproc set to 120 it seems to be chugging along at about 25% system 
> utilization:
> 
> Tasks: 1271 total,   1 running, 1270 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  4.5 us, 35.3 sy,  0.0 ni, 55.2 id,  0.5 wa,  4.5 hi,  0.0 si,  0.0 st
> KiB Mem : 26401230+total, 25401017+free,  1624640 used,  8377496 buff/cache
> KiB Swap:        0 total,        0 free,        0 used. 26143329+avail Mem 
> 
>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                          
>  87593 mingo     20   0 7850436 162284  11672 S  2941  0.1  33:01.83 syz-fuzzer                                                                                                       
>    923 root      20   0   84772  44344  43840 S  22.8  0.0   1:23.86 systemd-journal                                                                                                  
>   1369 root      16  -4  114636   3256   2832 S  15.7  0.0   0:29.03 auditd                                                                                                           
>   1379 root      12  -8   80236   1764   1432 S   8.3  0.0   0:15.79 audispd                                                                                                          
>    878 root      20   0       0      0      0 S   6.9  0.0   0:16.55 jbd2/sda1-8                                                                                                      
>   1381 root      16  -4   52216   3232   2892 S   3.8  0.0   0:07.08 sedispatch              
> 
> Even that one is not ideal - obviously there's way too much systemd-journal 
> overhead, but I'm unable to turn the darn thing off ...

The journal.conf man page suggests that putting 'Storage=none' into 
/etc/systemd/journal.conf disables journalling - but that's not true.

It's apparently impossible to disable systemd logging via any normal means ...

So the workaround for all that is a brutal:

   mv /var/log/journal /var/log/journal.dontuse

after that there does not seem to be systemd logging anymore.

... but there is tons of auditd logging now! ;-)

Fortunately that's easily stopped via:

  service stop auditd

with that there's no log IO anymore during fuzzing (yay!).

Except that systemd journald rears its ugly head back:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                          
111464 mingo     20   0 7249024 143348  11476 S  2741  0.1   7:41.31 syz-fuzzer                                                                                                       
122632 root      20   0   29200   2768   2524 S   9.5  0.0   0:28.74 systemd-journal                                                                                                  
111463 root      20   0  165596   5772   3840 R   4.8  0.0   0:00.80 top                                                                                                              
112078 mingo     20   0   20928    704    644 S   2.9  0.0   0:00.17 syz-executor                                                                                                     
112516 mingo     20   0   20928    708    644 S   2.9  0.0   0:00.16 syz-executor                                                                                                     
   878 root      20   0       0      0      0 S   1.9  0.0   0:53.83 jbd2/sda1-8                                                                                                      
111711 mingo     20   0   20928    704    644 S   1.9  0.0   0:00.16 syz-executor                                                                                                     
111901 mingo     20   0   20928    704    644 S   1.9  0.0   0:00.15 syz-executor                                                                                                     
     8 root      20   0       0      0      0 S   1.0  0.0   1:00.55 rcu_sched                       

so it's eating about 10% of system overhead despite doing nothing (!).

The workaround for that systemd bug is a brutal:

 [root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
 [root@fomalhaut ~]# 

Thanks,

	Ingo