Re: [PATCH 0/6] mm, sched, cgroup: LRU Cache never drains when isolcpus are used with spinning SCHED_FIFO tasks causing page migration stalls

[Michal Hocko]

On Tue 03-11-26 20:59:34, Matthew Ruffell wrote:
> What happens is that the spinning SCHED_FIFO processes running on isolcpus are
> constantly dirtying the LRU cache, and when we attempt NUMA migration of the
> 50GB of pages, systemd calls flush_migrate_mm_task_workfn() which creates a
> kworker that calls cpuset_migrate_mm_workfn() then do_migrate_pages() then 
> lru_cache_disable(), and then __lru_add_drain_all(). __lru_add_drain_all() then
> creates a kworker on each CPU to drain the LRU cache, and systemd and the
> initial kworker wait until this is done on all CPUs.
> 
> Since the isolcpus are running SCHED_FIFO tasks, they never get preempted,
> and kworkers never get a chance to run until they get an opportunity to by
> returning from a syscall, and it takes a very long time before all 192 CPUs
> actually drain their LRU cache before the page migration can continue.

The idea is that they are running on isolated cpus should they shouldn't
be preempted by unrelated work. But at the same time those
isolated/cached pages shouldn't be blocking the rest of the system. IIRC
there was a proposal to flush all pages cached on pcp on return to the
userspace. What has happened with that work? I do not think this has
been merged. This should deal with the underlying problem without
introducing lock contention that might be a result dopped caching.
-- 
Michal Hocko
SUSE Labs