Re: [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery

From: Sahil Chandna

Date: Wed Nov 18 2026 - 04:25:13 EST


On 13-11-2026 15:46, Breno Leitao wrote:
> On Fri, Nov 13, 2026 at 02:04:08PM +0530, Sahil Chandna wrote:
>> On 12-11-2026 17:37, Breno Leitao wrote:
>>> Hello Sahil,
>>>
>>> On Thu, Nov 12, 2026 at 04:57:18AM -0600, Sahil Chandna wrote:
>>>> Implement a graduated recovery approach: on the first TX timeout (or
>>>> when the previous timeout was more than 2 minutes ago), schedule NAPI on
>>>> the stalled queue's TX CQ via napi_schedule(). This triggers
>>>> mana_poll_tx_cq() through the normal NAPI path, processing any pending
>>>> completions without relying on the interrupt and recovering
>>>> missed-interrupt scenarios without queue reset.
>>>
>>> Which interrupt is being missed here?
>>>
>> Hi Breno,
>>
>> It is the TX CQ completion interrupt. After NAPI finishes, it re-arms
>> the TX CQ. When a new CQE arrives, the hardware post EQE, which raises
>> the MSI-X and schedules NAPI. If that notification for the TX CQ is lost
>> , NAPI never runs for the TX CQ and pending_sends never drains and the
>> queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs
>> and re-arms it.
>
> Thanks. Which one gets lost, the EQE or the MSI-X?
Hi Breno,

Sorry for the delay, i was checking on these with my team.
Regarding what is lost here. The patch is trying to solve the scenario
where TX CQ never arrive due to some reason, it is not handling the
MSI-X lost scenario and one of the netdev bot comment highlights the
same. Though i will work on framing commit message more clearly in next
version. Thanks !
>
>>>> If any subsequent TX timeout fires on any queue, within 2 minutes of
>>>> the previous one, escalate to the existing port-level detach/attach
>>>> reset, as the CQE poll did not recover the queue.
>>>
>>> How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no?
>> The watchdog_timeo is TX stall timeout setting i.e if a queue remains
>> stalled for this much time, it trigger watchdog timeout. Here, 2 minute
>> is not timeout/wait. The 2 minutes here is for deciding whether a new
>> queue stall event is a repeat of the previous incident or a new,
>> unrelated one that should get the cheap poll first. For e.g. if a queue
>> gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger
>> manual NAPI, if another stall occurs (it can be on any queue) within 2
>> minutes of last one, the existing recovery mechanism of mana
>> detach/attach will be tried.The patch is adding a first level less
>> destructive recovery first.
>> For selecting 2 minutes, I do not have any hardware basis for it afaik,
>> but I can confirm and check with our host team as well.
>
> So the goal is to tell an isolated stall from a wider problem.
>
Yes, and take just a minimal step to solve it rather than entire port
detach/attach.
> A port-wide timestamp only approximates that. It cannot tell the same
> queue from another one, and since dev_watchdog() reports one queue per
> run, two queues that stall together show up ~15s apart: the first gets
> a kick, the second gets the port reset anyway, no?
>
yes, that is correct. The approach in this patch solves first instance
of stall on any queue with CQE polling and if that does not solve it
(based on another queue stall observed) go for existing mechanism
(detach/attach), but I agree to your point here that another queue which
stall after one queue had level 1 CQE polling would have a full
detach/attach rather than level 1 polling. I will work on this in v2.
> Could mana_tx_timeout() check this directly? Look at the other TX queues
> the way dev_watchdog() does (stopped, with trans_start older than
> watchdog_timeo). If more than one is stalled, reset right away.
>
> Otherwise kick this queue, and reset if the same queue times out again
> within a couple of watchdog periods, as gve_tx_timeout_try_q_kick()
> does. That also gets rid of the 2 minutes.
Thanks, I will go through this approach and update in v2. I am currently
discussing this internally.

Regards,
Sahil