RE: [PATCH V1] block: Fix null pointer dereference issue on struct io_cq

[Pradeep Pragallapati (QUIC)]

Hi,

-----Original Message-----
From: Damien Le Moal <dlemoal@xxxxxxxxxx> 
Sent: Wednesday, May 17, 2023 3:02 PM
To: Christoph Hellwig <hch@xxxxxxxxxxxxx>
Cc: Yu Kuai <yukuai1@xxxxxxxxxxxxxxx>; Pradeep Pragallapati (QUIC) <quic_pragalla@xxxxxxxxxxx>; axboe@xxxxxxxxx; linux-block@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; yukuai (C) <yukuai3@xxxxxxxxxx>
Subject: Re: [PATCH V1] block: Fix null pointer dereference issue on struct io_cq

On 5/17/23 18:21, Christoph Hellwig wrote:
> On Wed, May 17, 2023 at 06:20:19PM +0900, Damien Le Moal wrote:
>> twice for the same icq. The missing rcu lock in ioc_exit_icqs() 
>> already was in itself a bug, and the missing flag check is another.
> 
> spinlocks imply a rcu critical section, no need to duplicate it.

Right. And I misread the code. As Yu said, given that ioc_exit_icqs() iterates the list of icqs under ioc->lock and the ioc is removed from that list under the same lock, ioc_exit_icqs() should never see an icq that went through ioc_destroy_icq()...
Very weird.

This weird can be possible 
1. updating icq_hint which is annotated as __rcu type without RCU-protected context in ioc_destroy_icq().
Moreover, this was taken care in else part of ioc_release_fn() by rcu_read_lock/unlock() but missed in if statement which can lead to this weird.

2. extracting icq from hlist/list elements are done using rcu locks protected in ioc_clear_queue() but same was not at ioc_exit_icqs().

So, far we have seen 10+ instances of this crash on 6.1 kernel during stability testing (Involves IO, reboots, device suspend/resume, and few more).
With the V1 patch, we didn't observe the issue for at least 48hrs+ of stability testing.


--
Damien Le Moal
Western Digital Research