Re: 4.14: WARNING: CPU: 4 PID: 2895 at block/blk-mq.c:1144 with virtio-blk (also 4.12 stable)

[Jens Axboe]

On 11/21/2017 11:12 AM, Christian Borntraeger wrote:
> 
> 
> On 11/21/2017 07:09 PM, Jens Axboe wrote:
>> On 11/21/2017 10:27 AM, Jens Axboe wrote:
>>> On 11/21/2017 03:14 AM, Christian Borntraeger wrote:
>>>> Bisect points to
>>>>
>>>> 1b5a7455d345b223d3a4658a9e5fce985b7998c1 is the first bad commit
>>>> commit 1b5a7455d345b223d3a4658a9e5fce985b7998c1
>>>> Author: Christoph Hellwig <hch@xxxxxx>
>>>> Date:   Mon Jun 26 12:20:57 2017 +0200
>>>>
>>>>     blk-mq: Create hctx for each present CPU
>>>>     
>>>>     commit 4b855ad37194f7bdbb200ce7a1c7051fecb56a08 upstream.
>>>>     
>>>>     Currently we only create hctx for online CPUs, which can lead to a lot
>>>>     of churn due to frequent soft offline / online operations.  Instead
>>>>     allocate one for each present CPU to avoid this and dramatically simplify
>>>>     the code.
>>>>     
>>>>     Signed-off-by: Christoph Hellwig <hch@xxxxxx>
>>>>     Reviewed-by: Jens Axboe <axboe@xxxxxxxxx>
>>>>     Cc: Keith Busch <keith.busch@xxxxxxxxx>
>>>>     Cc: linux-block@xxxxxxxxxxxxxxx
>>>>     Cc: linux-nvme@xxxxxxxxxxxxxxxxxxx
>>>>     Link: http://lkml.kernel.org/r/20170626102058.10200-3-hch@xxxxxx
>>>>     Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
>>>>     Cc: Oleksandr Natalenko <oleksandr@xxxxxxxxxxxxxx>
>>>>     Cc: Mike Galbraith <efault@xxxxxx>
>>>>     Signed-off-by: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
>>>
>>> I wonder if we're simply not getting the masks updated correctly. I'll
>>> take a look.
>>
>> Can't make it trigger here. We do init for each present CPU, which means
>> that if I offline a few CPUs here and register a queue, those still show
>> up as present (just offline) and get mapped accordingly.
>>
>> From the looks of it, your setup is different. If the CPU doesn't show
>> up as present and it gets hotplugged, then I can see how this condition
>> would trigger. What environment are you running this in? We might have
>> to re-introduce the cpu hotplug notifier, right now we just monitor
>> for a dead cpu and handle that.
> 
> I am not doing a hot unplug and the replug, I use KVM and add a previously
> not available CPU.
> 
> in libvirt/virsh speak:
>   <vcpu placement='static' current='1'>4</vcpu>

So that's why we run into problems. It's not present when we load the device,
but becomes present and online afterwards.

Christoph, we used to handle this just fine, your patch broke it.

I'll see if I can come up with an appropriate fix.

-- 
Jens Axboe