Re: Infiniate systemd loop when power off the machine with multiple MD RAIDs

[Hannes Reinecke]

On 8/23/23 03:28, Yu Kuai wrote:
> Hi,
>
> 在 2023/08/23 3:13, Carlos Carvalho 写道:
> > Song Liu (song@xxxxxxxxxx) wrote on Tue, Aug 22, 2023 at 03:56:04PM -03:
> > > >From systemd code, i.e. function delete_md(), this error:
> > >
> > > [ 205.957004] systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:6).
> > > [ 205.964177] systemd-shutdown[1]: Could not stop MD /dev/md124p1:
> > > Device or resource busy
> > >
> > > is most likely triggered by ioctl(STOP_ARRAY).
> > >
> > > And based on the code, I think the ioctl fails here:
> > >
> > >          if (cmd == STOP_ARRAY || cmd == STOP_ARRAY_RO) {
> > >                  /* Need to flush page cache, and ensure no-one else 
> > > opens
> > >                   * and writes
> > >                   */
> > >                  mutex_lock(&mddev->open_mutex);
> > >                  if (mddev->pers && atomic_read(&mddev->openers) > 1) {
> > >                          mutex_unlock(&mddev->open_mutex);
> > >                          err = -EBUSY;
> > >                          goto out;        ////////////////////// HERE
>
> Yes, I suspect here as well, but I do some test with error injection to
> gurantee that ioctl(STOP_ARRAY) always return -EBUSY, but I found that
> system reboot didn't hang, it'll try a few times but eventually reboot
> finished.
>
> > >                  }
> > >                  if (test_and_set_bit(MD_CLOSING, &mddev->flags)) {
> > >                          mutex_unlock(&mddev->open_mutex);
> > >                          err = -EBUSY;
> > >                          goto out;
> > >                  }
> > >                  did_set_md_closing = true;
> > >                  mutex_unlock(&mddev->open_mutex);
> > >                  sync_blockdev(bdev);
> > >          }
> >
> > Probably. The problem is why doesn't it manage to flush the page 
> > cache? I find
> > strange that the problem appears only when trying to stop the array, I 
> > get it
> > when trying to umount the filesystem, where it also hangs because of 
> > the same
> > reason. The kworker thread runs continuously using 100% cpu of only 1 
> > core.
>
> The kworker do you mean the daemon thread or the sync thread?  runs
> continuously using 100% cpu doen't sounds correct to me.
That sounds more like kworker waiting on a mutex to be released.
Can you trigger a kdump while in that state?
Or do a sysrq-T to get a stack trace of all running processes?
That should tell you where exactly the 100% cpu time is being
wasted.

Cheers,

Hannes
--
Dr. Hannes Reinecke                Kernel Storage Architect
hare@xxxxxxx                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman