Re: [PATCH v2] hung_task: Add per-round stack trace deduplication

[Aaron Tomlin]

On Sat, Jun 20, 2026 at 11:37:15AM +0800, Lance Yang wrote:
> Hi Aaron,
> 
> On Fri, Jun 19, 2026 at 09:35:59PM -0400, Aaron Tomlin wrote:
> >Currently, when multiple tasks hang in the exact same location (e.g.,
> >such as severe contention for a mutex), khungtaskd indiscriminately
> >reports every single instance. This wastes ring buffer space with
> >identical stack traces up to the defined warning limit (i.e.,
> >kernel.hung_task_warnings), obscuring the root cause without providing
> >any additional diagnostic value.
> >
> >Introduce a lightweight, hash-based stack trace deduplicator for
> >khungtaskd to ensure only unique stack traces are reported during
> >a single detection interval.
> >
> >Technical details of the implementation:
> > - Uses a 12-bit hash table (4096 slots), consuming just 16 KB of
> >   static memory to prevent cache thrashing during massive hangs.
> >
> > - Operates purely serially within the single khungtaskd thread,
> >   requiring zero atomic operations or concurrent locking overhead.
> >
> > - Flushes the lossy cache via memset() at the beginning of each
> >   detection round. This ensures the immediate "thundering herd" of
> >   duplicates is suppressed, but guarantees the system will not
> >   permanently suppress identical hangs that occur in future rounds.
> >
> > - Introduces a new sysctl, kernel.hung_task_dedup, which defaults to 1
> >   (enabled). The sysctl is locally cached at the outset of each
> >   interval to prevent tearing caused by concurrent userspace toggling.
> >
> 
> Thanks for working on this, but ... guess I'll be the bad guy here, not
> convinced this should go in ...
> 
> When khungtaskd fires, somthing is already wrong, no? I don't see why it
> should grow a new sysctl, a stack hash table, and extra filtering logic
> just ot hide part of the report ...
> 
> Emm ... do you have real cases where duplicate hung-task stacks caused
> serious pain?
> 
> If many tasks hang at once, usually one root cause, not a bunch of
> different bugs. At least from what I've seen, any one of those stacks is
> enough to start debugging ...
> 
> We already have hung_task_detect_count and trace_sched_process_hang() for
> basic counting/observability. Even if hung_task_warnings is finite and
> the warning budget runs out, we still don't lose detections: counter gets
> bumped and tracepoint fires before printk output is gated :)
> 
> If someone wants stack grouping, I'd rather leave that to a tool than add
> another policy knob to khungtaskd. Once it lands, maintainers have to
> carry it forever. Not every nice-to-have feature is worth that cost, IMHO
> 
> And if someone really wants more hung-task stacks in the log, we already
> have hung_task_warnings for that. Raise it, or set it to -1.
> 
> Also, looking at the v1 thread, I don't think the concerns there have
> really settled yet ... If nobody replies, maybe give it a week before
> sending a new version.

Hi Lance,

Thank you for taking the time to review the patch and for your candour.

You raise an entirely fair point regarding maintainability; every new
control knob indeed carries a permanent cost for the maintainers, and I
respect your caution.

To answer your question regarding real world pain: the primary issue is not
merely visual clutter, but the premature exhaustion of the warning budget
and the preservation of the kernel ring buffer during cascading failures.

In our production environments, we typically leave
kernel.hung_task_warnings at its default value of 10. If a severe lock
contention occurs, a single bottleneck can easily cause 10 tasks to hang
simultaneously with the exact same stack trace. Under the current logic,
those 10 identical traces will completely exhaust the warning budget.
Consequently, the kernel is left entirely blind to any subsequent or
completely unrelated deadlocks that might be occurring concurrently, as all
further reports are silenced.

Furthermore, dumping a full stack trace for every duplicate rapidly injects
several of lines of identical noise into dmesg. We have found that this
sudden burst frequently rolls the circular ring buffer.

Userspace tooling is unfortunately unable to group or analyse logs that
have already been evicted before the tool could read them, nor can it
recover traces the kernel silently dropped due to an exhausted budget.

The deduplicator acts as a telemetry filter, ensuring that the limited
warning budget is spent strictly on unique traces rather than redundant
noise, thereby preserving the history of the crash and ensuring secondary
failures are not obscured.

I wanted to clarify the exact operational context and the limitation of
relying on userspace. Please let me know if this operational context alters
your perspective at all.

Please note Sashiko [1] flagged a few issues which will be addressed by the
following changes:

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index ccd42f0a27ab..51f6540b70b3 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -77,6 +77,8 @@ static u32 hung_task_hash_table[HUNG_TASK_HT_SIZE];
  * Defaults to 1 (enabled).
  */
 static int __read_mostly sysctl_hung_task_dedup = 1;
+#else
+#define sysctl_hung_task_dedup 0
 #endif
 
 static int __read_mostly did_panic;
@@ -258,8 +260,18 @@ static bool hung_task_stack_is_unique(struct task_struct *t)
 	unsigned long entries[64];
 	unsigned int nr_entries;
 	u32 hash, idx;
+	void *stack;
+
+	/* Pin the stack safely before unwinding */
+	stack = try_get_task_stack(t);
+	if (!stack)
+		/* Task is exiting; pretend it is unique it */
+		return true;
 
 	nr_entries = stack_trace_save_tsk(t, entries, ARRAY_SIZE(entries), 0);
+
+	put_task_stack(t);
+
 	hash = jhash2((u32 *)entries, nr_entries * (sizeof(unsigned long) /
 		      sizeof(u32)), JHASH_INITVAL);
 


[1]: https://sashiko.dev/#/patchset/20260620013559.1537893-1-atomlin%40atomlin.com

Kind regards,
-- 
Aaron Tomlin

Attachment: signature.asc
Description: PGP signature