Re: [PATCH v2] x86/alternatives: remove false sharing in poke_int3_handler()

[Peter Zijlstra]

On Mon, Mar 24, 2025 at 08:30:51AM +0000, Eric Dumazet wrote:
> eBPF programs can be run 50,000,000 times per second on busy servers.
> 
> Whenever /proc/sys/kernel/bpf_stats_enabled is turned off,
> hundreds of calls sites are patched from text_poke_bp_batch()
> and we see a huge loss of performance due to false sharing
> on bp_desc.refs lasting up to three seconds.
> 
>    51.30%  server_bin       [kernel.kallsyms]           [k] poke_int3_handler
>             |
>             |--46.45%--poke_int3_handler
>             |          exc_int3
>             |          asm_exc_int3
>             |          |
>             |          |--24.26%--cls_bpf_classify
>             |          |          tcf_classify
>             |          |          __dev_queue_xmit
>             |          |          ip6_finish_output2
>             |          |          ip6_output
>             |          |          ip6_xmit
>             |          |          inet6_csk_xmit
>             |          |          __tcp_transmit_skb
>             |          |          |
>             |          |          |--9.00%--tcp_v6_do_rcv
>             |          |          |          tcp_v6_rcv
>             |          |          |          ip6_protocol_deliver_rcu
>             |          |          |          ip6_rcv_finish
>             |          |          |          ipv6_rcv
>             |          |          |          __netif_receive_skb
>             |          |          |          process_backlog
>             |          |          |          __napi_poll
>             |          |          |          net_rx_action
>             |          |          |          __softirqentry_text_start
>             |          |          |          asm_call_sysvec_on_stack
>             |          |          |          do_softirq_own_stack
> 
> Fix this by replacing bp_desc.refs with a per-cpu bp_refs.
> 
> Before the patch, on a host with 240 cores (480 threads):
> 
> $ bpftool prog | grep run_time_ns
> ...
> 105: sched_cls  name hn_egress  tag 699fc5eea64144e3  gpl run_time_ns
> 3009063719 run_cnt 82757845
> 
> -> average cost is 36 nsec per call
> 
> echo 0 >/proc/sys/kernel/bpf_stats_enabled
> text_poke_bp_batch(nr_entries=2)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 324 usec
> 
> text_poke_bp_batch(nr_entries=164)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 2655300 usec
> 
> After this patch:
> 
> $ bpftool prog | grep run_time_ns
> ...
> 105: sched_cls  name hn_egress  tag 699fc5eea64144e3  gpl run_time_ns
> 1928223019 run_cnt 67682728
> 
>  -> average cost is 28 nsec per call
> 
> echo 0 >/proc/sys/kernel/bpf_stats_enabled
> text_poke_bp_batch(nr_entries=2)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 519 usec
> 
> text_poke_bp_batch(nr_entries=164)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 702 usec

This is unreadable due to the amount of pointless repetition.

Also, urgh.