Re: [PATCH net] virtio-net: enable all napis before scheduling refill work

[Bui Quang Minh]

On 12/11/25 14:27, Jason Wang wrote:
> On Wed, Dec 10, 2025 at 11:33 PM Bui Quang Minh
> <minhquangbui99@xxxxxxxxx> wrote:
> > On 12/10/25 12:45, Jason Wang wrote:
> > > On Tue, Dec 9, 2025 at 11:23 PM Bui Quang Minh <minhquangbui99@xxxxxxxxx> wrote:
> > > > On 12/9/25 11:30, Jason Wang wrote:
> > > > > On Mon, Dec 8, 2025 at 11:35 PM Bui Quang Minh <minhquangbui99@xxxxxxxxx> wrote:
> > > > > > Calling napi_disable() on an already disabled napi can cause the
> > > > > > deadlock. In commit 4bc12818b363 ("virtio-net: disable delayed refill
> > > > > > when pausing rx"), to avoid the deadlock, when pausing the RX in
> > > > > > virtnet_rx_pause[_all](), we disable and cancel the delayed refill work.
> > > > > > However, in the virtnet_rx_resume_all(), we enable the delayed refill
> > > > > > work too early before enabling all the receive queue napis.
> > > > > >
> > > > > > The deadlock can be reproduced by running
> > > > > > selftests/drivers/net/hw/xsk_reconfig.py with multiqueue virtio-net
> > > > > > device and inserting a cond_resched() inside the for loop in
> > > > > > virtnet_rx_resume_all() to increase the success rate. Because the worker
> > > > > > processing the delayed refilled work runs on the same CPU as
> > > > > > virtnet_rx_resume_all(), a reschedule is needed to cause the deadlock.
> > > > > > In real scenario, the contention on netdev_lock can cause the
> > > > > > reschedule.
> > > > > >
> > > > > > This fixes the deadlock by ensuring all receive queue's napis are
> > > > > > enabled before we enable the delayed refill work in
> > > > > > virtnet_rx_resume_all() and virtnet_open().
> > > > > >
> > > > > > Fixes: 4bc12818b363 ("virtio-net: disable delayed refill when pausing rx")
> > > > > > Reported-by: Paolo Abeni <pabeni@xxxxxxxxxx>
> > > > > > Closes: https://netdev-ctrl.bots.linux.dev/logs/vmksft/drv-hw-dbg/results/400961/3-xdp-py/stderr
> > > > > > Signed-off-by: Bui Quang Minh <minhquangbui99@xxxxxxxxx>
> > > > > > ---
> > > > > >     drivers/net/virtio_net.c | 59 +++++++++++++++++++---------------------
> > > > > >     1 file changed, 28 insertions(+), 31 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > > index 8e04adb57f52..f2b1ea65767d 100644
> > > > > > --- a/drivers/net/virtio_net.c
> > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > @@ -2858,6 +2858,20 @@ static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq,
> > > > > >            return err != -ENOMEM;
> > > > > >     }
> > > > > >
> > > > > > +static void virtnet_rx_refill_all(struct virtnet_info *vi)
> > > > > > +{
> > > > > > +       bool schedule_refill = false;
> > > > > > +       int i;
> > > > > > +
> > > > > > +       enable_delayed_refill(vi);
> > > > > This seems to be still racy?
> > > > >
> > > > > For example, in virtnet_open() we had:
> > > > >
> > > > > static int virtnet_open(struct net_device *dev)
> > > > > {
> > > > >            struct virtnet_info *vi = netdev_priv(dev);
> > > > >            int i, err;
> > > > >
> > > > >            for (i = 0; i < vi->max_queue_pairs; i++) {
> > > > >                    err = virtnet_enable_queue_pair(vi, i);
> > > > >                    if (err < 0)
> > > > >                            goto err_enable_qp;
> > > > >            }
> > > > >
> > > > >            virtnet_rx_refill_all(vi);
> > > > >
> > > > > So NAPI and refill work is enabled in this case, so the refill work
> > > > > could be scheduled and run at the same time?
> > > > Yes, that's what we expect. We must ensure that refill work is scheduled
> > > > only when all NAPIs are enabled. The deadlock happens when refill work
> > > > is scheduled but there are still disabled RX NAPIs.
> > > Just to make sure we are on the same page, I meant, after refill work
> > > is enabled, rq0 is NAPI is enabled, in this case the refill work could
> > > be triggered by the rq0's NAPI so we may end up in the refill work
> > > that it tries to disable rq1's NAPI while holding the netdev lock.
> > I don't quite get your point. The current deadlock scenario is this
> >
> > virtnet_rx_resume_all
> > napi_enable(rq0) (the rq1 napi is still disabled)
> > enable_refill_work
> >
> > refill_work
> > napi_disable(rq0) -> still okay
> > napi_enable(rq0) -> still okay
> > napi_disable(rq1)
> > -> hold netdev_lock
> >       -> stuck inside the while loop in napi_disable_locked
> >               while (val & (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC)) {
> >                   usleep_range(20, 200);
> >                   val = READ_ONCE(n->state);
> >               }
> >
> >
> > napi_enable(rq1)
> > -> stuck while trying to acquire the netdev_lock
> >
> > The problem is that we must not call napi_disable() on an already
> > disabled NAPI (rq1's NAPI in the example).
> >
> > In the new virtnet_open
> >
> > static int virtnet_open(struct net_device *dev)
> > {
> >            struct virtnet_info *vi = netdev_priv(dev);
> >            int i, err;
> >
> >            // Note that at this point, refill work is still disabled, vi->refill_enabled == false,
> >            // so even if virtnet_receive is called, the refill_work will not be scheduled.
> >            for (i = 0; i < vi->max_queue_pairs; i++) {
> >                    err = virtnet_enable_queue_pair(vi, i);
> >                    if (err < 0)
> >                            goto err_enable_qp;
> >            }
> >
> >            // Here all RX NAPIs are enabled so it's safe to enable refill work again
> >            virtnet_rx_refill_all(vi);
> I meant this part:
>
> +static void virtnet_rx_refill_all(struct virtnet_info *vi)
> +{
> +       bool schedule_refill = false;
> +       int i;
> +
> +       enable_delayed_refill(vi);
>
> refill_work could run here.

I don't see how this can trigger the current deadlock race. However, I 
see that this code is racy, the try_fill_recv function is not safe to 
concurrently executed on the same receive queue. So there is a 
requirement that we need to call try_fill_recv before enabling napi. Is 
it what you mean?

> +       for (i = 0; i < vi->curr_queue_pairs; i++)
> +               if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL))
> +                       schedule_refill = true;
> +
>
> I think it can be fixed by moving enable_delayed_refill() here.
>
> +       if (schedule_refill)
> +               schedule_delayed_work(&vi->refill, 0);
> +}

Thanks,
Quang Minh.