Re: [PATCH] net: use sync wakeups for socket error reports

[Usama Arif]

On 08/07/2026 16:25, Breno Leitao wrote:
> On Wed, Jul 08, 2026 at 06:38:15AM -0700, Usama Arif wrote:
>> Measured on a 176-core EPYC 9D64 host running a Meta production
>> workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
>> attributed the sock_def_error_report -> ep_poll_callback ->
>> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
>> chain to 16,326 IPIs/min.
> 
> I am interested in why so many sock_def_error_report().
> 
> That's seems a lot for genuine socket errors (RST/ICMP) on a healthy
> host, so I suspect these aren't errors at all?
> 
> Can you share the full stack above sock_def_error_report()?


I ran this bpftrace script the host now (results added at the end):

sudo bpftrace -e '
  kprobe:sock_def_error_report
  {
    @wake_src[kstack()] = count();
  }
  interval:s:60
  {
    print(@wake_src, 5);
    exit();
  }'


The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said
is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it
with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues
it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report
so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE)
to get the SND/ACK timestamp.

So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every
packet completion and every ACK triggers a timestamp delivery through the
error-queue path, which is why sock_def_error_report fires.



Attached 2 probes
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        tcp_ack+3399
        tcp_rcv_established+1630
        tcp_v6_do_rcv+372
        tcp_v6_rcv+4748
        ip6_protocol_deliver_rcu+653
        ip6_input_finish+79
        ip6_input+43
        ipv6_list_rcv+4339
        __netif_receive_skb_list_core+244
        netif_receive_skb_list_internal+433
        napi_complete_done+149
        bnxt_poll_p5+499
        net_rx_action+513
        irq_exit_rcu+312
        common_interrupt+62
        asm_common_interrupt+34
]: 1812
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_dequeue_err_skb+194
        ipv6_recv_error+74
        bpf_trampoline_6442598004+73
        ____sys_recvmsg.llvm.18251018526254450710+168
        ___sys_recvmsg+312
        __x64_sys_recvmsg+95
        do_syscall_64+316
        entry_SYSCALL_64_after_hwframe+75
]: 8045
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        tcp_ack+3399
        tcp_rcv_established+1258
        tcp_v6_do_rcv+372
        tcp_v6_rcv+4748
        ip6_protocol_deliver_rcu+653
        ip6_input_finish+79
        ip6_input+43
        ipv6_list_rcv+4339
        __netif_receive_skb_list_core+244
        netif_receive_skb_list_internal+433
        napi_complete_done+149
        bnxt_poll_p5+499
        net_rx_action+513
        irq_exit_rcu+312
        common_interrupt+125
        asm_common_interrupt+34
        cpuidle_enter_state+202
        cpuidle_enter+40
        cpu_startup_entry+497
        ap_starting+0
        common_startup_64+318
]: 12603
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        tcp_ack+3399
        tcp_rcv_established+1258
        tcp_v6_do_rcv+372
        tcp_v6_rcv+4748
        ip6_protocol_deliver_rcu+653
        ip6_input_finish+79
        ip6_input+43
        ipv6_list_rcv+4339
        __netif_receive_skb_list_core+244
        netif_receive_skb_list_internal+433
        napi_complete_done+149
        bnxt_poll_p5+499
        net_rx_action+513
        irq_exit_rcu+312
        common_interrupt+62
        asm_common_interrupt+34
]: 19314
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        bnxt_start_xmit+1769
        dev_hard_start_xmit+160
        sch_direct_xmit+165
        __qdisc_run+714
        __dev_queue_xmit+2052
        skb_do_redirect+2531
        netkit_xmit+715
        dev_hard_start_xmit+160
        __dev_queue_xmit+1049
        ip6_finish_output2+848
        ip6_finish_output+213
        ip6_output+86
        ip6_xmit+933
        inet6_csk_xmit+163
        __tcp_transmit_skb+2733
        tcp_write_xmit+2948
        __tcp_push_pending_frames+46
        tcp_sendmsg_locked+4187
        tcp_sendmsg+40
        __x64_sys_sendmsg+567
        do_syscall_64+316
        entry_SYSCALL_64_after_hwframe+75
]: 33514