Re: [PATCH v4] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Aaron Tomlin]

On Thu, Jul 02, 2026 at 03:19:18PM +0200, Petr Mladek wrote:
> On Sun 2026-06-28 16:56:39, Aaron Tomlin wrote:
> > On Sun, Jun 28, 2026 at 12:47:50PM +0800, Lance Yang wrote:
> > To step back and address your question directly regarding the real-world
> > problem this patch aims to solve:
> 
> Thanks a lot for slowing down. It allows people to think more about
> the problem and get feedback from more poeple. And it reduces the risk
> of burn out of maintainers and reviewers.
> 
> > In large-scale, multi-tenant, production environments, lock contention is a
> > frequent reality. When a core resource (e.g., a heavily contended rwsem or
> > mutex) blocks, it does not just hang one task; it causes a cascading
> > failure that halts hundreds of tasks simultaneously.
> > 
> > When khungtaskd runs its scan during such an outage, it often reports
> > identical stack traces into the kernel ring buffer, which is not entirely
> > useful.
> > 
> > The global sysctl_hung_task_warnings budget is instantly exhausted by a
> > single lock storm. Consequently, the kernel is left entirely blind to
> > subsequent, completely unrelated deadlocks occurring elsewhere in the
> > system hours later.
> > 
> > The changes introduced to date, moving away from the heuristic wchan
> > approach to a more deterministic t->blocker tracking as per Petr's
> > feedback, were an attempt to solve this without introducing complex
> > heuristics or dangerous blind spots.
> 
> I would split this into two problems:
> 
> 1. A single lock contention might trigger hung_report for many tasks
>    waiting for the same lock. It bloats the kernel log and messages
>    might even get lost.
> 
> 2. The number of printed backtraces can be reduced by a global limit.
>    But the limit silences the hung task detector and system
>    administrators are blind once the limit is reached.
> 
> IMHO, the global limit "sysctl_hung_task_warnings" has been introduced
> because of the 1st problem but it caused the 2nd problem.
> 
> My proposal:
> ------------
> 
> a) We could Change the semantic of "sysctl_hung_task_warnings". It could newly
>    limit the number of printed backtraces in a single hung-system
>    situations. I mean to reset it when the check_hung_uninterruptible_tasks()
>    does not detect any blocked tasks.
> 
>    We could even print a message in this case. Something like:
> 
> 	if (atomic_long_read(&sysctl_hung_task_detect_count) && !this_round_count) {
> 		pr_err("INFO: Tasks are not blocked by sleeping locks any longer.\n");
> 		atomic_long_set(&sysctl_hung_task_detect_count, 0);
> 	}
> 
>    This would keep it working for 1st problem and solve the 2nd problem.
> 
> 
> b) I like the check of task->blocker when it is available. But it
>    depends on CONFIG_DETECT_HUNG_TASK_BLOCKER. Also the hash array
>    looks like an overkill to me.
> 
>    I would replace the hash array with a simple array[10]. It
>    should be enough in practice. Also it would be much easier
>    to clear it when the hung situation has gone.
> 
>    That said, I am not sure if it is worth it.
> 
> 
> c) Also storing the info about printed backtraces into struct
>    task_struct is interesting idea.
> 
>    But again, I am not sure if it is worth it.
> 
> 
> My opinion:
> -----------
> 
> I would start with a). It is trivial. It solves the regression caused
> by the current global limit. And the message about that
> the hung-situation has been resolved is useful. So it
> looks like win-win solution.
> 
> I would do b) and/or c) only when a) is not enough in practice.
> 
> That said:
> ----------
> 
> IMHO, CONFIG_DETECT_HUNG_TASK_BLOCKER is a rather cheap feature.
> I believe that the overhead is small especially when we are
> talking about sleeping locks. It is even enabled by default.
> 
> Adding the filtering by the blocker might be more effective
> in practice than the "sysctl_hung_task_warnings" global
> limit.
> 
> Best Regards,

Hi Petr,

Thank you for the detailed breakdown and for taking the time to review the
underlying issues. I appreciate the advice regarding the development
cadence; taking a step back to consider broader feedback is certainly
prudent, and I am grateful for your perspective.

Your proposal (a) is indeed elegant. Resetting sysctl_hung_task_warnings
when check_hung_uninterruptible_tasks() detects zero blocked tasks provides
a clean, deterministic way to recover from the "blind spot" regression
without adding architectural complexity.

Regarding the blocker tracking, I take your point that a hash array is
likely overkill for the common scenarios encountered in production. I
suggest we combine your proposal (a) with a simplified version of (b):
implementing a fixed-size array (e.g., array[10]) to track unique blockers.

This hybrid approach would function as follows:

    1.  When khungtaskd identifies a hung task, it compares the blocker
        against the array. If the blocker is already tracked, we suppress
        the warning, which keeps our sysctl_hung_task_warnings budget
        intact. If the blocker is new, we print the warning, add it to the
        array, and decrement the budget.

    2.  As you suggested, we reset the budget and clear the array[10] once
        the hang resolves, accompanied by your proposed recovery message in
        the ring buffer.

This seems to offer the best of both worlds: it provides the necessary
filtering to prevent log bloat during lock storms while ensuring the system
remains observable once the contention clears. It also adheres to the
principle of addressing the primary regression while keeping the
implementation overhead minimal.

Regarding the size of the array, I initially considered a size of 10 to
keep the footprint minimal. However, I am happy to increase this to 32 to
ensure we have sufficient coverage for complex multi-lock contention
scenarios without incurring significant cache overhead. Does 32 strike you
as a reasonable middle ground, or would you prefer I stick to a smaller
fixed size?

I am happy to draft a v5 patch that combines these elements, should you
agree that this remains a sensible direction.


Kind regards,
-- 
Aaron Tomlin