Re: [PATCH] kernfs: support kernfs notify in memory recliam context

[junxiao . bi]

On 11/14/23 11:06 AM, Tejun Heo wrote:

> Hello,
>
> On Tue, Nov 14, 2023 at 10:59:47AM -0800, Junxiao Bi wrote:
> > kernfs notify is used in write path of md (md_write_start) to wake up
> > userspace daemon, like "mdmon" for updating md superblock of imsm raid,
> > md write will wait for that update done before issuing the write, if this
> How is forward progress guarnateed for that userspace daemon? This sounds
> like a really fragile setup.

For imsm raid, userspace daemon "mdmon" is responsible for updating raid 
metadata, kernel will use kernfs_notify to wake up the daemon anywhere 
metadata update is required. If the daemon can't move forward, write may 
hung, but that will be a bug in the daemon?

> > write is used for memory reclaim, the system may hung due to kernel notify
> > can't be executed, that's because kernel notify is executed by "system_wq"
> > which doesn't have a rescuer thread and kworker thread may not be created
> > due to memory pressure, then userspace daemon can't be woke up and md write
> > will hung.
> >
> > According Tejun, this can't be fixed by add RECLAIM to "system_wq" because
> > that workqueue is shared and someone else might occupy that rescuer thread,
> > to fix this from md side, have to replace kernfs notify with other way to
> > communite with userspace daemon, that will break userspace interface,
> > so use a separated workqueue for kernefs notify to allow it be used in
> > memory reclaim context.
> I'm not necessarily against the change but please go into a bit more details
> on how and why it's structured this way and add a comment explaining
> explaining who's depending on kernfs notify for reclaim forward progress.

"kthreadd" was doing memory reclaim and stuck by md flush work, md flush 
work was stuck by md_write_start, where it was waiting 
"MD_SB_CHANGE_PENDING" flag to be cleared, before waiting, it invoked 
kernefs_notify to wake up userspace daemon which should update the 
meatadata and clear the flag.



PID: 2        TASK: ffff8df829539e40  CPU: 103  COMMAND: "kthreadd"
 #0 [ffffaf14800f3220] __schedule at ffffffff9488cbac
 #1 [ffffaf14800f32c0] schedule at ffffffff9488d1c6
 #2 [ffffaf14800f32d8] schedule_timeout at ffffffff948916e6
 #3 [ffffaf14800f3360] wait_for_completion at ffffffff9488ddeb
 #4 [ffffaf14800f33c8] flush_work at ffffffff940b5103
 #5 [ffffaf14800f3448] xlog_cil_force_lsn at ffffffffc0571791 [xfs]
 #6 [ffffaf14800f34e8] _xfs_log_force_lsn at ffffffffc056f79f [xfs]
 #7 [ffffaf14800f3570] xfs_log_force_lsn at ffffffffc056fa8c [xfs]
 #8 [ffffaf14800f35a8] __dta___xfs_iunpin_wait_3444 at ffffffffc05595c4 
[xfs]
 #9 [ffffaf14800f3620] xfs_iunpin_wait at ffffffffc055c229 [xfs]
#10 [ffffaf14800f3630] __dta_xfs_reclaim_inode_3358 at ffffffffc054f8cc 
[xfs]
#11 [ffffaf14800f3680] xfs_reclaim_inodes_ag at ffffffffc054fd56 [xfs]
#12 [ffffaf14800f3818] xfs_reclaim_inodes_nr at ffffffffc0551013 [xfs]
#13 [ffffaf14800f3838] xfs_fs_free_cached_objects at ffffffffc0565469 [xfs]
#14 [ffffaf14800f3848] super_cache_scan at ffffffff942959a7
#15 [ffffaf14800f38a0] shrink_slab at ffffffff941fa935
#16 [ffffaf14800f3988] shrink_node at ffffffff942005d8
#17 [ffffaf14800f3a10] do_try_to_free_pages at ffffffff94200ae2
#18 [ffffaf14800f3a78] try_to_free_pages at ffffffff94200e89
#19 [ffffaf14800f3b00] __alloc_pages_slowpath at ffffffff941ed82c
#20 [ffffaf14800f3c20] __alloc_pages_nodemask at ffffffff941ee191
#21 [ffffaf14800f3c90] __vmalloc_node_range at ffffffff9423a8e7
#22 [ffffaf14800f3d00] copy_process at ffffffff94096670
#23 [ffffaf14800f3de8] _do_fork at ffffffff94097f30
#24 [ffffaf14800f3e68] kernel_thread at ffffffff94098219
#25 [ffffaf14800f3e78] kthreadd at ffffffff940bd4e5
#26 [ffffaf14800f3f50] ret_from_fork at ffffffff94a00354


PID: 852      TASK: ffff8e351fc51e40  CPU: 77   COMMAND: "md"
 #0 [ffffaf148e983c68] __schedule at ffffffff9488cbac
 #1 [ffffaf148e983d08] schedule at ffffffff9488d1c6
 #2 [ffffaf148e983d20] md_write_start at ffffffff9469fc75
 #3 [ffffaf148e983d80] raid1_make_request at ffffffffc038d8bd [raid1]
 #4 [ffffaf148e983da8] md_handle_request at ffffffff9469cc24
 #5 [ffffaf148e983e18] md_submit_flush_data at ffffffff9469cce1
 #6 [ffffaf148e983e38] process_one_work at ffffffff940b5bd9
 #7 [ffffaf148e983e80] rescuer_thread at ffffffff940b6334
 #8 [ffffaf148e983f08] kthread at ffffffff940bc245
 #9 [ffffaf148e983f50] ret_from_fork at ffffffff94a00354


bool md_write_start(struct mddev *mddev, struct bio *bi)
{
    ...

   >>> process 852 go into the "if" and set "MD_SB_CHANGE_PENDING"

    if (mddev->in_sync || mddev->sync_checkers) {
        spin_lock(&mddev->lock);
        if (mddev->in_sync) {
            mddev->in_sync = 0;
            set_bit(MD_SB_CHANGE_CLEAN, &mddev->sb_flags);
            set_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags);
            md_wakeup_thread(mddev->thread);
            did_change = 1;
        }
        spin_unlock(&mddev->lock);
    }
    rcu_read_unlock();

    >>> invoke kernfs_notify to wake up userspace daemon

    if (did_change)
        sysfs_notify_dirent_safe(mddev->sysfs_state);
    if (!mddev->has_superblocks)
        return true;

   >>>> hung here waiting userspace daemon clear that flag.

    wait_event(mddev->sb_wait,
           !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags) ||
           is_md_suspended(mddev));
    if (test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags)) {
        percpu_ref_put(&mddev->writes_pending);
        return false;
    }
    return true;
}

Thanks,

Junxiao.

> Thanks.