Re: [PATCH] nohz: don't kick non-idle CPUs in tick_nohz_full_kick_cpu()

[Yury Norov]

On Mon, Jul 16, 2018 at 05:31:10PM +0200, Frederic Weisbecker wrote:
> External Email
> 
> On Thu, Jul 12, 2018 at 09:19:22PM +0300, Yury Norov wrote:
> > IIUC, tick_nohz_full_kick_cpu() is intended to wakeup idle CPUs
> > that will not be poked by scheduler because they are actually
> > nohz_full.
> 
> Not exactly. It is intended to trigger an interrupt on a nohz_full
> CPU that may be running in userspace without any tick. The irq_exit()
> code let us reprogramm the tick with the latest dependency updates.
> 
> >
> > But in fact this function kicks all CPUs listed in tick_nohz_full_mask,
> > namely:
> >  - idle CPUs;
> >  - CPUs runnung normal tasks;
> >  - CPUs running isolated tasks [1];
> >
> > For normal tasks it introduces unneeded latency, and for isolated tasks
> > it's fatal because isolation gets broken and task receives SIGKILL.
> 
> So this patch applies on Chris series right?

This patch may be applied on master. That's why I sent it to you.

> For now there is no such
> distinction between normal and isolated tasks. Any task running in a
> nohz_full CPU is considered to be isolated.
>
> > The patch below makes tick_nohz_full_kick_cpu() kicking only idle CPUs.
> > Non-idle nohz_full CPUs will observe changed system settings just like
> > non-idle normal (i.e. not nohz_full) CPUs, at next reschedule.
> 
> That's not exactly what we want. In fact when a task runs in a nohz_full CPU,
> it may not meet any reschedule interrupt for a long while. This is why we have
> tick_nohz_full_kick_cpu() in order to force a nohz_full CPU to see the latest
> changes.

OK, got it.

So if my understanding correct, there is 'soft isolation' which is
nohz_full, and 'hard isolation' which is Chris' task_isonation feature. For
soft isolation, the desirable behavior is to receive interrupts generated 
by tick_nohz_full_kick_cpu(), and for hard isolation it's obviously not
desirable because it kills application. 

The patch below adds check against task isolation in tick_nohz_full_kick_cpu().
It is on top of Chris' series. Is it OK from nohz point of view?

---

While here. I just wonder, on my system IRQs are sent to nohz_full CPUs
at every incoming ssh connection. The trace is like this:
[  206.835533] Call trace:
[  206.848411] [<ffff00000889f984>] dump_stack+0x84/0xa8
[  206.853455] [<ffff0000081ea308>] _task_isolation_remote+0x130/0x140
[  206.859714] [<ffff0000081bf5ec>] irq_work_queue_on+0xcc/0xfc
[  206.865365] [<ffff0000081478ac>] tick_nohz_full_kick_cpu+0x88/0x94
[  206.871536] [<ffff000008147930>] tick_nohz_dep_set_all+0x78/0xa8
[  206.877533] [<ffff000008147b58>] tick_nohz_dep_set_signal+0x28/0x34
[  206.883792] [<ffff0000081421fc>] set_process_cpu_timer+0xd0/0x128
[  206.889876] [<ffff0000081422ac>] update_rlimit_cpu+0x58/0x7c
[  206.895528] [<ffff0000083aa3d0>] selinux_bprm_committing_creds+0x180/0x1fc
[  206.902394] [<ffff00000839e394>] security_bprm_committing_creds+0x40/0x5c
[  206.909173] [<ffff00000828c4a0>] install_exec_creds+0x20/0x6c
[  206.914911] [<ffff0000082e15b0>] load_elf_binary+0x368/0xbb8
[  206.920561] [<ffff00000828d09c>] search_binary_handler+0xb8/0x224
[  206.926645] [<ffff00000828d99c>] do_execveat_common+0x44c/0x5f0
[  206.932555] [<ffff00000828db78>] do_execve+0x38/0x44
[  206.937510] [<ffff00000828dd74>] SyS_execve+0x34/0x44

I suspect that scp, ssh tunneling and similar network activities will source 
ticks on nohz_full CPUs as well. On high-loaded server it may generate
significant interrupt traffic on nohz_full CPUs. Is it desirable behavior?

---
Yury