Re: [PATCH V3 1/2] nvme: split resetting state into reset_prepate and resetting

[jianchao.wang]

Hi keith

Thanks for your kindly review and response.

On 01/14/2018 05:48 PM, Sagi Grimberg wrote:
> 
>> Currently, the ctrl->state will be changed to NVME_CTRL_RESETTING
>> before queue the reset work. This is not so strict. There could be
>> a big gap before the reset_work callback is invoked. In addition,
>> there is some disable work in the reset_work callback, strictly
>> speaking, not part of reset work, and could lead to some confusion.
>>
>> This patch splits the NVME_CTRL_RESETTING into NVME_CTRL_RESET_PREPARE
>> and NVME_CTRL_RESETTING. Before queue the reset work, changes state
>> to NVME_CTRL_RESET_PREPARE, after disable work completes, changes
>> state to NVME_CTRL_RESETTING.
> 
> What guarantees that nvme_timeout will do the right thing? If it
> is relying on the fact that nvme-pci sets the state to RESETTING
> only _after_ quiescing the requests queues it needs to be documented
> as its a real delicate dependency.
nvme_dev_disable will iterate all the outstanding request with nvme_cancel_request to requeue
the IO requests and complete admin requests. This is the boundary.
Transit the state from RESET_PREPARE to RESETTING after nmve_dev_disable.
So when the state is RESETTING, all the previous outstanding requests all has been handled by
nvme_cancel_request, the only requests can expire at this moment is from the adminq of initializing
procedure. 

> 
> 
> 
>>
>> Suggested-by: Christoph Hellwig <hch@xxxxxx>
>> Signed-off-by: Jianchao Wang <jianchao.w.wang@xxxxxxxxxx>
>> ---
>> Â drivers/nvme/host/core.cÂÂ | 17 +++++++++++++++--
>> Â drivers/nvme/host/fc.cÂÂÂÂ |Â 2 ++
>> Â drivers/nvme/host/nvme.hÂÂ |Â 1 +
>> Â drivers/nvme/host/pci.cÂÂÂ | 28 ++++++++++++++++++++++------
>> Â drivers/nvme/host/rdma.cÂÂ |Â 8 ++++++++
>> Â drivers/nvme/target/loop.c |Â 5 +++++
>> Â 6 files changed, 53 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 1e46e60..106a437 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -87,7 +87,7 @@ static __le32 nvme_get_log_dw10(u8 lid, size_t size)
>> Â Â int nvme_reset_ctrl(struct nvme_ctrl *ctrl)
>> Â {
>> -ÂÂÂ if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
>> +ÂÂÂ if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESET_PREPARE))
>> ÂÂÂÂÂÂÂÂÂ return -EBUSY;
>> ÂÂÂÂÂ if (!queue_work(nvme_wq, &ctrl->reset_work))
>> ÂÂÂÂÂÂÂÂÂ return -EBUSY;
>> @@ -243,7 +243,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
>> ÂÂÂÂÂÂÂÂÂ }
>> ÂÂÂÂÂÂÂÂÂ break;
>> -ÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> ÂÂÂÂÂÂÂÂÂ switch (old_state) {
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_NEW:
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_LIVE:
>> @@ -253,10 +253,21 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
>> ÂÂÂÂÂÂÂÂÂ }
>> ÂÂÂÂÂÂÂÂÂ break;
>> +
>> +ÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂÂÂÂÂ switch (old_state) {
>> +ÂÂÂÂÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> +ÂÂÂÂÂÂÂÂÂÂÂ changed = true;
>> +ÂÂÂÂÂÂÂÂÂÂÂ /* FALLTHRU */
>> +ÂÂÂÂÂÂÂ default:
>> +ÂÂÂÂÂÂÂÂÂÂÂ break;
>> +ÂÂÂÂÂÂÂ }
>> +ÂÂÂÂÂÂÂ break;
>> ÂÂÂÂÂ case NVME_CTRL_RECONNECTING:
>> ÂÂÂÂÂÂÂÂÂ switch (old_state) {
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_LIVE:
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂÂÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ changed = true;
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ /* FALLTHRU */
>> ÂÂÂÂÂÂÂÂÂ default:
>> @@ -267,6 +278,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
>> ÂÂÂÂÂÂÂÂÂ switch (old_state) {
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_LIVE:
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂÂÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> ÂÂÂÂÂÂÂÂÂ case NVME_CTRL_RECONNECTING:
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ changed = true;
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ /* FALLTHRU */
>> @@ -2603,6 +2615,7 @@ static ssize_t nvme_sysfs_show_state(struct device *dev,
>> ÂÂÂÂÂÂÂÂÂ [NVME_CTRL_NEW]ÂÂÂÂÂÂÂ = "new",
>> ÂÂÂÂÂÂÂÂÂ [NVME_CTRL_LIVE]ÂÂÂ = "live",
>> ÂÂÂÂÂÂÂÂÂ [NVME_CTRL_RESETTING]ÂÂÂ = "resetting",
>> +ÂÂÂÂÂÂÂ [NVME_CTRL_RESET_PREPARE]ÂÂÂ = "reset-prepare",
>> ÂÂÂÂÂÂÂÂÂ [NVME_CTRL_RECONNECTING]= "reconnecting",
>> ÂÂÂÂÂÂÂÂÂ [NVME_CTRL_DELETING]ÂÂÂ = "deleting",
>> ÂÂÂÂÂÂÂÂÂ [NVME_CTRL_DEAD]ÂÂÂ = "dead",
>> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
>> index 794e66e..516c1ea 100644
>> --- a/drivers/nvme/host/fc.c
>> +++ b/drivers/nvme/host/fc.c
>> @@ -547,6 +547,7 @@ nvme_fc_resume_controller(struct nvme_fc_ctrl *ctrl)
>> ÂÂÂÂÂÂÂÂÂ break;
>> Â ÂÂÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> ÂÂÂÂÂÂÂÂÂ /*
>> ÂÂÂÂÂÂÂÂÂÂ * Controller is already in the process of terminating the
>> ÂÂÂÂÂÂÂÂÂÂ * association. No need to do anything further. The reconnect
>> @@ -790,6 +791,7 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl)
>> ÂÂÂÂÂÂÂÂÂ break;
>> Â ÂÂÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> ÂÂÂÂÂÂÂÂÂ /*
>> ÂÂÂÂÂÂÂÂÂÂ * Controller is already in the process of terminating the
>>  * association. No need to do anything further. The reconnect
> 
> James, can you take a look at this?
> 
>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>> index f5800c3..e477c35 100644
>> --- a/drivers/nvme/host/pci.c
>> +++ b/drivers/nvme/host/pci.c
>> @@ -1141,8 +1141,13 @@ static bool nvme_should_reset(struct nvme_dev *dev, u32 csts)
>> ÂÂÂÂÂ bool nssro = dev->subsystem && (csts & NVME_CSTS_NSSRO);
>> Â ÂÂÂÂÂ /* If there is a reset ongoing, we shouldn't reset again. */
>> -ÂÂÂ if (dev->ctrl.state == NVME_CTRL_RESETTING)
>> +ÂÂÂ switch (dev->ctrl.state) {
>> +ÂÂÂ case NVME_CTRL_RESETTING:
>> +ÂÂÂ case NVME_CTRL_RESET_PREPARE:
>> ÂÂÂÂÂÂÂÂÂ return false;
>> +ÂÂÂ default:
>> +ÂÂÂÂÂÂÂ break;
>> +ÂÂÂ }
> 
> Isn't the indentation off? also in other places...
I see all the "switch" in core.c and pci.c is this style.
switch(x) {
case xxx:
...
case xxx:
...
}
So I followed.... :)
> 
>> @@ -2292,7 +2303,7 @@ static void nvme_reset_work(struct work_struct *work)
>> ÂÂÂÂÂ bool was_suspend = !!(dev->ctrl.ctrl_config & NVME_CC_SHN_NORMAL);
>> ÂÂÂÂÂ int result = -ENODEV;
>> Â -ÂÂÂ if (WARN_ON(dev->ctrl.state != NVME_CTRL_RESETTING))
>> +ÂÂÂ if (WARN_ON(dev->ctrl.state != NVME_CTRL_RESET_PREPARE))
>> ÂÂÂÂÂÂÂÂÂ goto out;
>> Â ÂÂÂÂÂ /*
>> @@ -2302,6 +2313,11 @@ static void nvme_reset_work(struct work_struct *work)
>> ÂÂÂÂÂ if (dev->ctrl.ctrl_config & NVME_CC_ENABLE)
>> ÂÂÂÂÂÂÂÂÂ nvme_dev_disable(dev, false);
>> Â +ÂÂÂ if (!nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESETTING)) {
>> +ÂÂÂÂÂÂÂ WARN_ON_ONCE(dev->ctrl.state != NVME_CTRL_DELETING);
>> +ÂÂÂÂÂÂÂ goto out;
>> +ÂÂÂ }
>> +
> 
> This needs documentation on _why_ here.
OK, sure.
> 
>> -ÂÂÂ nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESETTING);
>> +ÂÂÂ nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESET_PREPARE);
>> ÂÂÂÂÂ dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
> 
> Can you rebase on top of my commit:
> 4caff8fc19f1 nvme-pci: don't open-code nvme_reset_ctrl
OK, I will
> 
>> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
>> index 37af565..8ae073e 100644
>> --- a/drivers/nvme/host/rdma.c
>> +++ b/drivers/nvme/host/rdma.c
>> @@ -1753,6 +1753,14 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
>> ÂÂÂÂÂ nvme_stop_ctrl(&ctrl->ctrl);
>> ÂÂÂÂÂ nvme_rdma_shutdown_ctrl(ctrl, false);
>> Â +ÂÂÂ changed = nvme_change_ctrl_state(&ctrl->ctrl,
>> +ÂÂÂÂÂÂÂÂÂÂÂ NVME_CTRL_RESET_PREPARE);
>> +ÂÂÂ if (!changed) {
>> +ÂÂÂÂÂÂÂ /* state change failure is ok if we're in DELETING state */
>> +ÂÂÂÂÂÂÂ WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING);
>> +ÂÂÂÂÂÂÂ return;
>> +ÂÂÂ }
>> +
> 
> setting RESET_PREPARE here??
This is my fault. I checked the code time by time, but still a stupid fault here.
> 
> Also, the error recovery code is mutually excluded from reset_work
> by trying to set the same state which is protected by the ctrl state
> machine, so a similar change is needed there.
> 
> So in order not to break rdma ctrl reset (again) please look at:
> bcdacf7fc3d8 nvme-rdma: fix concurrent reset and reconnect
> 
Ok, I will look into it.
Thanks for your directive. That's really appreciated.

Thanks
Jianchao