Re: [RFC] workqueue: allow system workqueue be used in memory reclaim

[Tejun Heo]

Hello,

On Tue, Nov 07, 2023 at 05:28:21PM -0800, Junxiao Bi wrote:
> The following deadlock was triggered on Intel IMSM raid1 volumes.
> 
> The sequence of the event is this:
> 
> 1. memory reclaim was waiting xfs journal flushing and get stucked by
> md flush work.
> 
> 2. md flush work was queued into "md" workqueue, but never get executed,
> kworker thread can not be created and also the rescuer thread was executing
> md flush work for another md disk and get stuck because
> "MD_SB_CHANGE_PENDING" flag was set.
> 
> 3. That flag should be set by some md write process which was asking to
> update md superblock to change in_sync status to 0, and then it used
> kernfs_notify to ask "mdmon" process to update superblock, after that,
> write process waited that flag to be cleared.
> 
> 4. But "mdmon" was never wake up, because kernfs_notify() depended on
> system wide workqueue "system_wq" to do the notify, but since that
> workqueue doesn't have a rescuer thread, notify will not happen.

Things like this can't be fixed by adding RECLAIM to system_wq because
system_wq is shared and someone else might occupy that rescuer thread. The
flag doesn't guarantee unlimited forward progress. It only guarantees
forward progress of one work item.

That seems to be where the problem is in #2 in the first place. If a work
item is required during memory reclaim, it must have guaranteed forward
progress but it looks like that's waiting for someone else who can end up
waiting for userspace?

You'll need to untangle the dependencies earlier.

Thanks.

-- 
tejun