Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Aaron Tomlin]

On Tue, Jul 21, 2026 at 01:08:32PM +0800, Lance Yang wrote:
> 
> On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
> >Currently, during severe lock contention, multiple tasks can hang while
> >waiting on the exact same resource. The khungtaskd kthread
> >indiscriminately reports every single instance with a stack trace.
> >This can roll the kernel ring buffer and prematurely exhaust the
> >kernel.hung_task_warnings budget. Consequently, the kernel is left
> >entirely blind to subsequent, unrelated deadlocks.
> >
> >To preserve the warning budget and ring buffer without sacrificing
> >observability, this patch introduces a two-tier deduplication mechanism:
> >
> >    1. Introduces a hung_task_reported in task_struct. If a task remains
> >       hung across multiple check intervals, khungtaskd suppresses
> >       redundant stack traces for that specific task until it makes
> >       progress (verified via context switch counters). Furthermore, it
> >       is packed into an existing compiler alignment hole, consuming
> >       zero additional memory
> >
> >    2. For tasks blocked on a lock, we leverage the
> >       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
> >       exact memory address of the blocker lock in a persistent array of
> >       size 32, we deduplicate warning reports for tasks waiting on the
> >       same resource across check intervals. If the blocker is already
> >       tracked, the warning is suppressed and the warning budget is
> >       preserved
> >
> >    3. For duplicate tasks, we still print the single-line "INFO: task
> >       ..." message and trigger tracepoint trace_sched_process_hang().
> >       It merely skips calling sched_show_task() and
> >       debug_show_blocker(), printing a concise suppression notice
> >       instead
> >
> >    4. Once the hang resolves (i.e., no hung tasks are detected in a
> >       check round), the budget is restored to the value captured at the
> >       start of the hang, and the blocker array is cleared, accompanied
> >       by a recovery message in the ring buffer
> >
> >Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxxx>
> >---
> 
> Sorry for the late reply.
> 
> Still, v6 looks far too complicated ... and touches too much core code
> for the problem you're trying to address ...
> 
> Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
> problems are real:
> 
> "
> I would split this into two problems:
> 
> 1. A single lock contention might trigger hung_report for many tasks
>    waiting for the same lock. It bloats the kernel log and messages
>    might even get lost.
> 
> 2. The number of printed backtraces can be reduced by a global limit.
>    But the limit silences the hung task detector and system
>    administrators are blind once the limit is reached.
> "
> 
> IMO, that's the tradeoff: keep log noise down without leaving users
> blind for good once budget runs out ...
> 
> Still not sold on dedup, though. Sure, duplicate reports happen, but how
> often, really? Doesn't seem common enough to need this much code :)
> 
> If we really want to handle this in kernel, how about a simple mode
> switch for hung_task_warnings?
> 
>   0 = keep the current global budget
>   1 = allow up to hung_task_warnings reports in each scan
> 
> ---8<---
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 6fcc94ce4ca9..4ec2937a98e6 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -58,6 +58,7 @@ unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_
>  static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
> 
>  static int __read_mostly sysctl_hung_task_warnings = 10;
> +static int __read_mostly sysctl_hung_task_warnings_mode;
> 
>  static int __read_mostly did_panic;
>  static bool hung_task_call_panic;
> @@ -228,12 +229,13 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
>   * @t: Pointer to the detected hung task.
>   * @timeout: Timeout threshold for detecting hung tasks
>   * @this_round_count: Count of hung tasks detected in the current iteration
> + * @warning_budget: Warning budget for reporting hung tasks
>   *
>   * Print structured information about the specified hung task, if warnings
>   * are enabled or if the panic batch threshold is exceeded.
>   */
>  static void hung_task_info(struct task_struct *t, unsigned long timeout,
> -			   unsigned long this_round_count)
> +			   unsigned long this_round_count, int *warning_budget)
>  {
>  	trace_sched_process_hang(t);
> 
> @@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>  	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
>  	 * accordingly
>  	 */
> -	if (sysctl_hung_task_warnings || hung_task_call_panic) {
> -		if (sysctl_hung_task_warnings > 0)
> -			sysctl_hung_task_warnings--;
> +	if (*warning_budget || hung_task_call_panic) {
> +		if (*warning_budget > 0)
> +			(*warning_budget)--;
>  		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
>  		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
>  		       (jiffies - t->last_switch_time) / HZ);
> @@ -264,8 +266,8 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>  		sched_show_task(t);
>  		debug_show_blocker(t, timeout);
> 
> -		if (!sysctl_hung_task_warnings)
> -			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
> +		if (!*warning_budget)
> +			pr_info("Hung task warning budget exhausted, see sysctl kernel.hung_task_warnings\n");
>  	}
> 
>  	touch_nmi_watchdog();
> @@ -305,6 +307,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
>  	struct task_struct *g, *t;
>  	unsigned long this_round_count;
>  	int need_warning = sysctl_hung_task_warnings;
> +	int warnings_left = need_warning;
> +	int *warning_budget = sysctl_hung_task_warnings_mode ?
> +			      &warnings_left : &sysctl_hung_task_warnings;
>  	unsigned long si_mask = hung_task_si_mask;
> 
>  	/*
> @@ -334,7 +339,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
>  			 */
>  			atomic_long_inc(&sysctl_hung_task_detect_count);
>  			this_round_count++;
> -			hung_task_info(t, timeout, this_round_count);
> +			hung_task_info(t, timeout, this_round_count,
> +				       warning_budget);
>  		}
>  	}
>   unlock:
> @@ -483,6 +489,15 @@ static const struct ctl_table hung_task_sysctls[] = {
>  		.proc_handler	= proc_dointvec_minmax,
>  		.extra1		= SYSCTL_NEG_ONE,
>  	},
> +	{
> +		.procname	= "hung_task_warnings_mode",
> +		.data		= &sysctl_hung_task_warnings_mode,
> +		.maxlen		= sizeof(int),
> +		.mode		= 0644,
> +		.proc_handler	= proc_dointvec_minmax,
> +		.extra1		= SYSCTL_ZERO,
> +		.extra2		= SYSCTL_ONE,
> +	},
>  	{
>  		.procname	= "hung_task_detect_count",
>  		.maxlen		= sizeof(unsigned long),
> ---
> 
> Per-scan mode avoids the permanent blind spot, while still letting users
> pick whichever behavior fits their setup.
> 
> Keeps the change local, too. If that turns out not to be enough, sure,
> revisit dedup then. That's the direction I'd support :D
> 
> [1] https://lore.kernel.org/lkml/akZlVgbImH6Jqy55@xxxxxxxxxxxxxxx/
> 
> Thanks, Lance

Hi Lance,

Thank you for taking the time to review the patch and share your thoughts.

While I understand your reservations regarding the complexity, I must
respectfully point out that implementation [1] (v6), coupled with the minor
adjustment below, already directly addresses both of the precise issues
Petr outlined, and does so without requiring a new sysctl.

[1]: https://lore.kernel.org/lkml/20260719161305.428947-1-atomlin@xxxxxxxxxxx/

To Petr's first point, log bloat from multiple tasks waiting on a single
lock, your proposed hung_task_warnings_mode does not mitigate the redundant
stack traces generated within a single scan. If dozens of tasks are hung on
the identical lock, the ring buffer will still be severely flooded. The
deduplication in v6 is specifically designed to solve this fundamental
issue.

To Petr's second point, the permanent blind spot, v6 already implements a
native solution that eliminates the need for an additional mode switch. As
noted in the original commit message, the warning budget is automatically
restored once the hang resolves (i.e., when no active hung tasks are
detected in a round). This ensures system administrators are never left
permanently blind.

The following minor adjustment to implementation [1] simply relocates the
panic evaluation back to the main loop, ensuring it operates correctly
prior to the automated budget restoration logic:


diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6bb2b26693a1..2a11d5ebcdf1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -252,11 +252,6 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 {
 	trace_sched_process_hang(t);
 
-	if (sysctl_hung_task_panic && this_round_count >= sysctl_hung_task_panic) {
-		console_verbose();
-		hung_task_call_panic = true;
-	}
-
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
@@ -371,6 +366,12 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 			atomic_long_inc(&sysctl_hung_task_detect_count);
 			this_round_count++;
 
+			if (sysctl_hung_task_panic &&
+			    this_round_count >= sysctl_hung_task_panic) {
+				console_verbose();
+				hung_task_call_panic = true;
+			}
+
 #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
 			if (!hung_task_has_active) {
 				warnings_reset_value = sysctl_hung_task_warnings;


By automatically restoring the budget upon resolution, we achieve the
desired persistent visibility without burdening users to configure a new
sysctl, while crucially preventing log saturation during a massive lock
contention event.

I believe this approach provides the comprehensive fix to both of Petr's
concerns.


Kind regards,
-- 
Aaron Tomlin