Re: v6.12-rc workqueue lockups

[Jan Kara]

Hi!

On Wed 23-10-24 11:19:24, John Garry wrote:
> I have been seeing lockups reliably occur on v6.12-rc1, 3, 4 and linus'
> master branch:
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:07:15 ...
>  kernel:watchdog: BUG: soft lockup - CPU#12 stuck for 26s! [khugepaged:154]
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
>  kernel:BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck
> for 44s!
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
>  kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
> for 35s!
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 22 09:08:07 ...
>  kernel:BUG: workqueue lockup - pool cpus=10 node=0 flags=0x0 nice=0 stuck
> for 33s!
> 
> This is while doing some server MySQL performance testing. v6.11 has no such
> issue.
> 
> I added some debug, and we seem to be spending a lot of time in FS
> writeback, specifically wb_workfn() -> wb_do_writeback() - ring any bells?

Thanks for report. This doesn't remind me of anything but checking the
writeback changes we have merged 532980cb1bf ("inode: port __I_SYNC to var
event") which could have odd consequences if we made mistake somewhere...
It probably won't be easy to revert but you could try whether the problem
reproduces before / after this commit.

								Honza
-- 
Jan Kara <jack@xxxxxxxx>
SUSE Labs, CR