Re: [PATCH v2 00/13] mpt3sas driver NVMe support:

[James Bottomley]

On Mon, 2017-08-07 at 20:01 +0530, Kashyap Desai wrote:
> > 
> > -----Original Message-----
> > From: James Bottomley [mailto:James.Bottomley@xxxxxxxxxxxxxxxxxxxxx
> > ]
> > Sent: Monday, August 07, 2017 7:48 PM
> > To: Kashyap Desai; Christoph Hellwig; Hannes Reinecke
> > Cc: Suganath Prabu Subramani; martin.petersen@xxxxxxxxxx; linux-
> > scsi@xxxxxxxxxxxxxxx; Sathya Prakash Veerichetty; linux-
> > kernel@xxxxxxxxxxxxxxx; Chaitra Basappa; Sreekanth Reddy; linux-
> > nvme@xxxxxxxxxxxxxxxxxxx
> > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > 
> > On Mon, 2017-08-07 at 19:26 +0530, Kashyap Desai wrote:
> > > 
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: James Bottomley [mailto:James.Bottomley@HansenPartnership
> > > > .com
> > > > ]
> > > > Sent: Saturday, August 05, 2017 8:12 PM
> > > > To: Christoph Hellwig; Hannes Reinecke
> > > > Cc: Suganath Prabu S; martin.petersen@xxxxxxxxxx; linux-
> > > > scsi@xxxxxxxxxxxxxxx; Sathya.Prakash@xxxxxxxxxxxx;
> > > > kashyap.desai@xxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> > > > chaitra.basappa@xxxxxxxxxxxx; sreekanth.reddy@xxxxxxxxxxxx;
> > > > linux-
> > > > nvme@xxxxxxxxxxxxxxxxxxx
> > > > Subject: Re: [PATCH v2 00/13] mpt3sas driver NVMe support:
> > > > 
> > > > On Sat, 2017-08-05 at 06:53 -0700, Christoph Hellwig wrote:
> > > > > 
> > > > > 
> > > > > On Wed, Aug 02, 2017 at 10:14:40AM +0200, Hannes Reinecke
> > > > > wrote:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > I'm not happy with this approach.
> > > > > > NVMe devices should _not_ appear as SCSI devices; this will
> > > > > > just
> > > > > > confuse matters _and_ will be incompatible with 'normal'
> > > > > > NVMe
> > > > > > devices.
> > > > > > 
> > > > > > Rather I would like to see the driver to hook into the
> > > > > > existing
> > > > > > NVMe framework (which essentially means to treat the
> > > > > > mpt3sas as
> > > > > > a weird NVMe-over-Fabrics HBA), and expose the NVMe devices
> > > > > > like
> > > > > > any other NVMe HBA.
> > > > > 
> > > > > That doesn't make any sense.ÂÂThe devices behind the mpt
> > > > > adapter
> > > > > don't look like NVMe devices at all for the hosts - there are
> > > > > no
> > > > > NVMe commands or queues involved at all, they hide behind the
> > > > > same
> > > > > somewhat leaky scsi abstraction as other devices behind the
> > > > > mpt
> > > > > controller.
> > > > 
> > > > You might think about what we did for SAS: split the generic
> > > > handler
> > > > into two pieces, libsas for driving the devices, which mpt
> > > > didn't
> > > > need because of the fat firmware and the SAS transport class so
> > > > mpt
> > > > could at least show the same sysfs files as everything else for
> > > > SAS
> > > > devices.
> > > 
> > > ÂVentura generation of controllers are adding connectivity of
> > > NVME
> > > Âdrives seamlessly and protocol handling is in Firmware.
> > > ÂSame as SCSI to ATA translation done in firmware, Ventura
> > > controller
> > > Âis doing SCSI to NVME translation and for end user protocol
> > > handling
> > > Âis abstracted.
> > > 
> > > ÂThis product handles new Transport protocol (NVME) same as ATA
> > > and
> > > Âtransport is abstracted for end user.
> > > 
> > > NVME pass-through related driver code, it is just a big tunnel
> > > for
> > > user space application. It is just a basic framework like SATA
> > > PASS-
> > > Through in existing mpt3sas driver.
> > 
> > I know how it works ... and I'm on record as not liking your SATL
> > approach
> > because we keep tripping across bugs in the SATL that we have to
> > fix in
> > the
> > driver.
> 
> We discussed about NVME device support behind <mpt3sas> to Hannes and
> he suggested to describe to product behavior to wider audience to be
> aware. Just wanted to share the notes.

Great, thanks! it's certainly better than releasing the product and
having us have to support it.

> > However, at least for bot SAS and SATA they appear to the system as
> > SCSI
> > devices regardless of HBA, so we've largely smoothed over any
> > problems if
> > you
> > transfer from mp3sas to another SAS/SATA controller.
> > 
> > I believe your current proposal is to have NVMe devices appear as
> > SCSI,
> > which
> > isn't how the native NVMe driver handles them at all.ÂÂThis is
> > going to
> > have to
> > be special cased in any tool designed to handle nvme devices and
> > it's
> > going to
> > cause big problems if someone changes controller (or moves the
> > device).ÂÂWhat's the proposal for making this as painless as
> > possible?
> 
> We have to attempt this use case and see how it behaves. I have not
> tried this, so not sure if things are really bad or just some tuning
> may be helpful. I will revert back to you on this.
> 
> I understood request as -ÂÂWe need some udev rules to be working well
> for *same* NVME drives if it is behind <mpt3sas> or native <nvme>.
> Example - If user has OS installed on NVME drive which is behind
> <mpt3sas> driver as SCSI disk should be able to boot if he/she hooked
> same NVME drive which is detected by native <nvme> driver (and vice
> versa.)

It's not just the udev rules, it's the tools as well; possibly things
like that nvme-cli toolkit Intel is doing.

James