Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Petr Mladek]

On Tue 2026-07-21 13:08:32, Lance Yang wrote:
> 
> On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
> >Currently, during severe lock contention, multiple tasks can hang while
> >waiting on the exact same resource. The khungtaskd kthread
> >indiscriminately reports every single instance with a stack trace.
> >This can roll the kernel ring buffer and prematurely exhaust the
> >kernel.hung_task_warnings budget. Consequently, the kernel is left
> >entirely blind to subsequent, unrelated deadlocks.
> >
> >To preserve the warning budget and ring buffer without sacrificing
> >observability, this patch introduces a two-tier deduplication mechanism:
> >
> >    1. Introduces a hung_task_reported in task_struct. If a task remains
> >       hung across multiple check intervals, khungtaskd suppresses
> >       redundant stack traces for that specific task until it makes
> >       progress (verified via context switch counters). Furthermore, it
> >       is packed into an existing compiler alignment hole, consuming
> >       zero additional memory
> >
> >    2. For tasks blocked on a lock, we leverage the
> >       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
> >       exact memory address of the blocker lock in a persistent array of
> >       size 32, we deduplicate warning reports for tasks waiting on the
> >       same resource across check intervals. If the blocker is already
> >       tracked, the warning is suppressed and the warning budget is
> >       preserved
> >
> >    3. For duplicate tasks, we still print the single-line "INFO: task
> >       ..." message and trigger tracepoint trace_sched_process_hang().
> >       It merely skips calling sched_show_task() and
> >       debug_show_blocker(), printing a concise suppression notice
> >       instead
> >
> >    4. Once the hang resolves (i.e., no hung tasks are detected in a
> >       check round), the budget is restored to the value captured at the
> >       start of the hang, and the blocker array is cleared, accompanied
> >       by a recovery message in the ring buffer
> >
> >Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxxx>
> >---
> 
> Sorry for the late reply.

Don't worry. IMHO, a reply within one week is perfectly
fine. Sometimes even this is not possible.

> Still, v6 looks far too complicated ... and touches too much core code
> for the problem you're trying to address ...

I agree that it looks too complicated. I see several problems:

  1. Many changes are added in a single patch. I suggest to split
     it next time into more patches and add the features from
     the most important one, ...

  2. The code seems to be a bit over-engineered. There are too many
     variables hung_task_blockers_count, hung_task_has_active,
     warnings_decremented, skip_show_task, scan_completed,
     did_report, seen_blockers_mask, ... I believe that it might
     be done an easier way.

  3. Too much spaghetti code is added to the already large
     check_hung_uninterruptible_tasks() function. It might look
     better when the detection of the duplicated task is handled
     in a separate function, ...


> Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
> problems are real:
> 
> "
> I would split this into two problems:
> 
> 1. A single lock contention might trigger hung_report for many tasks
>    waiting for the same lock. It bloats the kernel log and messages
>    might even get lost.
> 
> 2. The number of printed backtraces can be reduced by a global limit.
>    But the limit silences the hung task detector and system
>    administrators are blind once the limit is reached.
> "
> 
> IMO, that's the tradeoff: keep log noise down without leaving users
> blind for good once budget runs out ...
> 
> Still not sold on dedup, though. Sure, duplicate reports happen, but how
> often, really? Doesn't seem common enough to need this much code :)
> 
> If we really want to handle this in kernel, how about a simple mode
> switch for hung_task_warnings?
> 
>   0 = keep the current global budget
>   1 = allow up to hung_task_warnings reports in each scan

Honestly, I do not think that this is the right way to go. The more I think
about it, the more I believe that the global budget is simply wrong.

IMHO, we should do the following:

1. Reset the global budget when the situation is resolved and
   there is no hung task any longer.

2. The global budget should limit only printing all the details,
   especially backtraces. We should always print at least
   the task name, ...

3. We could prevent printing the same process again and again
   by adding "hung_task_reported" into struct task_struct.

   But then we should print some summary about how many hung tasks
   were found in the last round at least. So that we know that
   the problem persists.

   This should still be rather easy. But I am not sure if it is
   worth it.

4. We could add the filtering of the duplicated backtraces. But
   it has to be handled by some helper function.

   It looks like to most complicated part. I would personally skip
   it for now. We could always add it later when the above proposed
   changes are not enough to tune the kernel.hung_task_warnings
   value in the real life.


I tried to implement the 1st and 2nd change using gemini just to
see how it would look. And it looks good to me: