Re: [patch v12 00/13] extensible prctl task isolation interface and vmstat sync

[Marcelo Tosatti]

Hi Thomas,

On Wed, May 04, 2022 at 10:15:14PM +0200, Thomas Gleixner wrote:
> On Wed, May 04 2022 at 15:56, Marcelo Tosatti wrote:
> > On Wed, May 04, 2022 at 03:20:03PM +0200, Thomas Gleixner wrote:
> >> Can we please focus on the initial problem of
> >> providing a sensible isolation mechanism with well defined semantics?
> >
> > Case 2, however, was implicitly suggested by you (or at least i
> > understood that):
> >
> > "Summary: The problem to be solved cannot be restricted to
> >
> >     self_defined_important_task(OWN_WORLD);
> >
> > Policy is not a binary on/off problem. It's manifold across all levels
> > of the stack and only a kernel problem when it comes down to the last
> > line of defence.
> >
> > Up to the point where the kernel puts the line of last defence, policy
> > is defined by the user/admin via mechanims provided by the kernel.
> >
> > Emphasis on "mechanims provided by the kernel", aka. user API.
> >
> > Just in case, I hope that I don't have to explain what level of scrunity
> > and thought this requires."
> 
> Correct. This reasoning is still valid and I haven't changed my opinion
> on that since then.
> 
> My main objections against the proposed solution back then were the all
> or nothing approach and the implicit hard coded policies.
> 
> > The idea, as i understood was that certain task isolation features (or
> > they parameters) might have to be changed at runtime (which depends on
> > the task isolation features themselves, and the plan is to create
> > an extensible interface).
> 
> Again. I'm not against useful controls to select the isolation an
> application requires. I'm neither against extensible interfaces.
> 
> But I'm against overengineered implementations which lack any form of
> sensible design and have ill defined semantics at the user ABI.
> 
> Designing user space ABI is _hard_ and needs a lot of thoughts. It's not
> done with throwing something 'extensible' at the kernel and hope it
> sticks. As I showed you in the review, the ABI is inconsistent in
> itself, it has ill defined semantics and lacks any form of justification
> of the approach taken.
> 
> Can we please take a step back and:
> 
>   1) Define what is trying to be solved

Avoid interruptions to application code execution on isolated CPUs.

Different use-cases might accept different length/frequencies
of interruptions (including no interruptions).

>      and what are the pieces known
>      today which need to be controlled in order to achieve the desired
>      isolation properties.

I hope you don't mean the current CPU isolation features which have to
be enabled, but only the ones which are not enabled today:

"Isolation of the threads was done through the following kernel parameters:

nohz_full=8-15,24-31 rcu_nocbs=8-15,24-31 poll_spectre_v2=off
numa_balancing=disable rcutree.kthread_prio=3 intel_pstate=disable nosmt

And systemd was configured with the following affinites:

system.conf:CPUAffinity=0-7,16-23

This means that the second socket will be generally free of tasks and   
kernel threads."

So here are some features which could be written on top of the proposed
task isolation via prctl:

1) 

Enable or disable the following optional behaviour

A.
if (cpu->isolated_avoid_queue_work)
	return -EBUSY;

queue_work_on(cpu, workfn);

(for the functions that can handle errors gracefully).

B.
if (cpu->isolated_avoid_function_ipi)
	return -EBUSY;

smp_call_function_single(cpu, fn);
(for the functions that can handle errors gracefully).
Those that can't handle errors gracefully should be changed 
to either handle errors or to remote work.

Not certain if this should be on per-case basis: say
"avoid action1|avoid action2|avoid action3|..." (bit per
action) and a "ALL" control, where actionZ is an action
that triggers an IPI or remote work (then you would check
for whether to fail not at smp_call_function_single 
time but before the action starts).

Also, one might use something such as stalld (that schedules 
tasks in/out for a short amount of time every given time window),
which might be acceptable for his workload, so he'd disable
cpu->isolated_avoid_queue_work (or expose this on per-case basis,
unsure which is better).

As for IPIs, whether to block a function call to an isolated
CPU depends on whether that function call (and its frequency) 
will cause the latency sensitive application to violate its "latency" 
requirements.

Perhaps "ALL - action1, action2, action3" is useful.

=======================================

2)

In general, avoiding (or uncaching on return to userspace) 
a CPU from caching per-CPU data (which might require an 
IPI to invalidate later on) (see point [1] below for more thoughts
on this issue).


For example, for KVM:

/*
 * MMU notifier 'invalidate_range_start' hook.
 */
void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
                                       unsigned long end, bool may_block)
{
        DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS);
        struct gfn_to_pfn_cache *gpc;
        bool wake_vcpus = false;
	...
	called = kvm_make_vcpus_request_mask(kvm, req, vcpu_bitmap);

	which will
	smp_call_function_many(cpus, ack_flush, NULL, wait);
...


====================================================

3) Enabling a kernel warning when a task switch happens on a CPU
which runs a task isolated thread?

>From Christoph:

Special handling when the scheduler
switches a task? If tasks are being switched that requires them to be low
latency and undisturbed then something went very very wrong with the
system configuration and the only thing I would suggest is to issue some
kernel warning that this is not the way one should configure the system.

====================================================

4) Sending a signal whenever an application is interrupted
(hum, this could be done via BPF).

Those are the ones i can think of at the moment. 
Not sure what other people can think of.

>   2) Describe the usage scenarios and the resulting constraints.

Well the constraints should be in the form

	"In a given window of time, there should be no more than N
	 CPU interruptions of length L each."

	(should be more complicated due to cache effects, but choosing
	 a lower N and L one is able to correct that)

I believe?

Also some memory bandwidth must be available to the application
(or data/code in shared caches).
Which depends on what other CPUs in the system are doing, the
cache hierarchy, the application, etc.

[1]: There is also a question of whether to focus only on 
applications that do not perform system calls on their latency 
sensitive path, and applications that perform system calls. 

Because some CPU interruptions can't be avoided if the application 
is in the kernel: for example instruction cache flushes due to 
static_key rewrites or kernel TLB flushes (well they could be avoided 
with more infrastructure, but there is no such infrastructure at
the moment).

>   3) Describe the requirements for features on top, e.g. inheritance
>      or external control.

1) Be able to use unmodified applications (as long as the features
to be enabled are compatible with such usage, for example "killing 
/ sending signal to application if task is interrupted" is obviously
incompatible with unmodified applications).

2) External control: be able to modify what task isolation features are
enabled externally (not within the application itself). The latency
sensitive application should inform the kernel the beginning of 
the latency sensitive section (at this time, the task isolation 
features configured externally will be activated).

3) One-shot mode: be able to quiesce certain kernel activities
only on the first time a syscall is made (because the overhead
of subsequent quiescing, for the subsequent system calls, is
undesired).

> Once we have that, we can have a discussion about the desired control
> granularity and how to support the extra features in a consistent and
> well defined way.
> 
> A good and extensible UABI design comes with well defined functionality
> for the start and an obvious and maintainable extension path. The most
> important part is the well defined functionality.
> 
> There have been enough examples in the past how well received approaches
> are, which lack the well defined part. Linus really loves to get a pull
> request for something which cannot be described what it does, but could
> be used for cool things in the future.
> 
> > So for case 2, all you'd have to do is to modify the application only
> > once and allow the admin to configure the features.
> 
> That's still an orthogonal problem, which can be solved once a sensible
> mechanism to control the isolation and handle it at the transition
> points is in place. You surely want to consider it when designing the
> UABI, but it's not required to create the real isolation mechanism in
> the first place.

Ok, can drop all of that for smaller patches with the handling 
of transition points only (then later add oneshot mode, inheritance,
external control).

But might wait for discussion of requirements that you raise 
first.

> Problem decomposition is not an entirely new concept, really.

Sure, thanks.