Re: [BUG] kernel freezes with latest tree

[Eric Dumazet]

Le mercredi 11 janvier 2012 Ã 13:25 +0100, Peter Zijlstra a Ãcrit :
> On Wed, 2012-01-11 at 10:04 +0100, Peter Zijlstra wrote:
> > Maybe adding a few more NEED_BREAK bits
> > and making it a counter and overflowing it into ABORT might be good.
> > 
> > 
> I could reproduce and confirm something like the below makes the hang
> go-away. I haven't managed to fully understand why we're stuck though
> because we do release the runqueue locks and re-enable IRQs on this
> lock-break.
> 
> My stuck machine had several CPUs stuck in a load-balance pass, so it
> could be they're bouncing tasks back and forth without actually making
> any progress what so ever.
> 
> I reproduced with hackbench 500, which results in 20000 tasks, spread
> over 24 cpus that gives some 833 tasks per runqueue on average, easily
> overflowing that lock-break scanning limit.
> 
> ---
> Subject: sched: Limit load-balance retries on lock-break
> From: Peter Zijlstra <a.p.zijlstra@xxxxxxxxx>
> Date: Wed Jan 11 13:11:12 CET 2012
> 
> Eric and David reported dead machines and traced it to commit a195f004 ("sched:
> Fix load-balance lock-breaking"), it turns out there's still a
> scenario where we can end up re-trying forever.
> 
> Limit the number of retries and simply abort.
> 
> Reported-by: Eric Dumazet <eric.dumazet@xxxxxxxxx>
> Reported-by: David Ahern <dsahern@xxxxxxxxx>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@xxxxxxxxx>
> ---
>  kernel/sched/fair.c |    9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3130,8 +3130,10 @@ task_hot(struct task_struct *p, u64 now,
>  }
>  
>  #define LBF_ALL_PINNED	0x01
> -#define LBF_NEED_BREAK	0x02
> -#define LBF_ABORT	0x04
> +#define LBF_NEED_BREAK	0x02	/* clears into HAD_BREAK */
> +#define LBF_HAD_BREAK	0x04
> +#define LBF_HAD_BREAKS	0x0C	/* count HAD_BREAKs overflows into ABORT */
> +#define LBF_ABORT	0x10
>  
>  /*
>   * can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
> @@ -4509,6 +4511,9 @@ static int load_balance(int this_cpu, st
>  
>  		if (lb_flags & LBF_NEED_BREAK) {
>  			lb_flags &= ~LBF_NEED_BREAK;
> +			lb_flags += LBF_HAD_BREAK;
> +			if (lb_flags & LBF_ABORT)
> +				goto out_balanced;
>  			goto redo;
>  		}
>  
> 

The two lines :
	lb_flags &= ~LBF_NEED_BREAK;
	lb_flags += LBF_HAD_BREAK;

could be combined to :
	lb_flags += LBF_HAD_BREAK - LBF_NEED_BREAK;

Anyway, your patch solved the problem on my machines, thanks a lot.

$ uptime
 14:20:32 up 31 min,  2 users,  load average: 170.18, 183.00, 156.53

Tested-by: Eric Dumazet <eric.dumazet@xxxxxxxxx>

One perf top output, on the 32bit kernel, to check scheduler functions
dont use too many cycles ;)

 12.31%        hackbench  [kernel.kallsyms]             [k] __slab_free
  8.98%        hackbench  [kernel.kallsyms]             [k] _raw_spin_lock_irqsave
  6.52%        hackbench  [kernel.kallsyms]             [k] sock_alloc_send_pskb
  6.25%        hackbench  [kernel.kallsyms]             [k] _raw_spin_lock
  5.33%        hackbench  [kernel.kallsyms]             [k] fget_light
  5.04%        hackbench  [kernel.kallsyms]             [k] unix_stream_recvmsg
  4.01%        hackbench  [kernel.kallsyms]             [k] __copy_user_intel
  3.14%        hackbench  [kernel.kallsyms]             [k] __kmalloc_node_track_caller
  2.61%        hackbench  [kernel.kallsyms]             [k] __alloc_skb
  2.55%        hackbench  [kernel.kallsyms]             [k] skb_release_head_state
  2.18%        hackbench  [kernel.kallsyms]             [k] sock_wfree
  2.00%        hackbench  [kernel.kallsyms]             [k] cred_to_ucred
  1.68%        hackbench  [kernel.kallsyms]             [k] kfree
  1.65%        hackbench  [kernel.kallsyms]             [k] unix_stream_sendmsg
  1.62%        hackbench  [kernel.kallsyms]             [k] kmem_cache_alloc_node
  1.55%        hackbench  [kernel.kallsyms]             [k] unix_destruct_scm
  1.43%        hackbench  [kernel.kallsyms]             [k] __schedule
  1.39%        hackbench  [kernel.kallsyms]             [k] sched_clock_cpu
  1.37%        hackbench  [kernel.kallsyms]             [k] kmem_cache_free
  1.30%        hackbench  [kernel.kallsyms]             [k] sock_def_readable
  1.25%        hackbench  [kernel.kallsyms]             [k] __slab_alloc
  1.19%        hackbench  [kernel.kallsyms]             [k] sched_clock_local
  1.16%        hackbench  [kernel.kallsyms]             [k] fput
  1.11%        hackbench  [kernel.kallsyms]             [k] sysenter_past_esp
  0.98%        hackbench  [kernel.kallsyms]             [k] get_partial_node.isra.43
  0.93%        hackbench  [kernel.kallsyms]             [k] select_task_rq_fair
  0.81%        hackbench  [unknown]                     [.] 0xffffe424
  0.80%        hackbench  [kernel.kallsyms]             [k] update_curr
  0.75%        hackbench  [kernel.kallsyms]             [k] skb_release_data
  0.60%        hackbench  [kernel.kallsyms]             [k] try_to_wake_up
  0.56%        hackbench  [kernel.kallsyms]             [k] vfs_write
  0.54%        hackbench  libpthread-2.3.4.so           [.] __pthread_disable_asynccancel
  0.52%        hackbench  [kernel.kallsyms]             [k] update_rq_clock
  0.47%        hackbench  [kernel.kallsyms]             [k] ktime_get
  0.45%        hackbench  libpthread-2.3.4.so           [.] __pthread_enable_asynccancel
  0.45%        hackbench  [kernel.kallsyms]             [k] skb_dequeue


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/