Re: [PATCH net v2] wireguard: wait for per-peer crypto during removal

[Jason A. Donenfeld]

On Fri, Sep 18, 2026 at 03:48:39PM -0500, Chris Arges wrote:
> On 2026-09-16 12:06:08, Jason A. Donenfeld wrote:
> > On Sun, Sep 13, 2026 at 07:19:08AM -0500, Chris J Arges wrote:
> > > Calling peer_remove_after_dead() currently flushes device-wide packet
> > > crypto and handshake workqueues while holding RTNL. This is problematic as
> > > unrelated peers can continue adding work to those queues, blocking other
> > > tasks that want to take the RTNL lock.
> > > 
> > > Instead, this patch tracks pending crypto handoffs for each peer using a
> > > counter. After marking the peer dead, synchronize_net() prevents new
> > > submissions. Next, wait for pending crypto workers to schedule TX work and
> > > for RX NAPI to drain the peer's RX queue. Then flush only the peer's TX
> > > packet and handshake work.
> > > 
> > > This scopes teardown synchronization to the removed peer and prevents
> > > unrelated peers from extending the RTNL hold time.
> > >  struct wg_device;
> > > @@ -161,6 +162,7 @@ static inline int wg_queue_enqueue_per_device_and_peer(
> > >  	 */
> > >  	if (unlikely(!wg_prev_queue_enqueue(peer_queue, skb)))
> > >  		return -ENOSPC;
> > > +	atomic_inc(&PACKET_PEER(skb)->packet_crypt_pending);
> > >  
> > >  	/* Then we queue it up in the device queue, which consumes the
> > >  	 * packet as soon as it can.
> > > @@ -182,6 +184,8 @@ static inline void wg_queue_enqueue_per_peer_tx(struct sk_buff *skb, enum packet
> > >  	atomic_set_release(&PACKET_CB(skb)->state, state);
> > >  	queue_work_on(wg_cpumask_choose_online(&peer->serial_work_cpu, peer->internal_id),
> > >  		      peer->device->packet_crypt_wq, &peer->transmit_packet_work);
> > > +	if (atomic_dec_and_test(&peer->packet_crypt_pending))
> > > +		wake_up_var(&peer->packet_crypt_pending);
> > >  	wg_peer_put(peer);
> > >  }
> > >  
> > > diff --git a/drivers/net/wireguard/receive.c b/drivers/net/wireguard/receive.c
> > > index 824bbefce61c..bb35e3205491 100644
> > > --- a/drivers/net/wireguard/receive.c
> > > +++ b/drivers/net/wireguard/receive.c
> > > @@ -476,9 +476,11 @@ int wg_packet_rx_poll(struct napi_struct *napi, int budget)
> > >  
> > >  next:
> > >  		wg_noise_keypair_put(keypair, false);
> > > -		wg_peer_put(peer);
> > >  		if (unlikely(free))
> > >  			dev_kfree_skb(skb);
> > > +		if (atomic_dec_and_test(&peer->packet_crypt_pending))
> > > +			wake_up_var(&peer->packet_crypt_pending);
> > > +		wg_peer_put(peer);
> > 
> > This adds two atomic updates to a per-peer counter for every RX packet
> > and TX batch. That could introduce contention across crypto workers. I
> > suppose it'd be good to see some measurements in if this changes
> > anything. Certainly it should change _something_. Question is by how
> > much.
> > 
> > Jason
> 
> Jason,
> 
> I benchmarked a wg peer on a 2 vCPU and 8 vCPU setup with 1200 byte
> UDP payloads. Goal was stressing the path where these atomics are getting
> incremented.
> - patched 2 vCPU test showed ~1-2% reduction in throughput
> - patched 8 vCPU test showed ~6% reduction in throughput (more contention)
> 
> My main goal is reducing the amount of time holding rtnl_lock when we remove a
> peer. In our systems we frequently get hangs due to workloads bringing up and
> tearing down wg peers. So perhaps I'll need to look into another approach
> where we're not introducing something like a counter into the hotpath.

Did you have any luck with other approaches? I'm still interested in
this.

Thanks,
Jason