Re: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal memory to dma coherent memory

[Dutt, Sudeep]

On Thu, 2020-10-29 at 01:51 +0000, Sherry Sun wrote:
> Hi Greg,
> 
> > Subject: Re: [EXT] Re: [PATCH V5 0/2] Change vring space from nomal
> > memory to dma coherent memory
> > 
> > On Wed, Oct 28, 2020 at 03:11:15PM +0000, Andy Duan wrote:
> > > From: Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> Sent: Wednesday,
> > > October
> > > 28, 2020 7:14 PM
> > > > On Wed, Oct 28, 2020 at 10:17:39AM +0000, Andy Duan wrote:
> > > > > From: Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> Sent: Wednesday,
> > > > > October 28, 2020 3:07 PM
> > > > > > On Wed, Oct 28, 2020 at 06:05:28AM +0000, Sherry Sun wrote:
> > > > > > > Hi Greg,
> > > > > > > 
> > > > > > > > Subject: Re: [PATCH V5 0/2] Change vring space from
> > > > > > > > nomal
> > > > > > > > memory to dma coherent memory
> > > > > > > > 
> > > > > > > > On Wed, Oct 28, 2020 at 10:03:03AM +0800, Sherry Sun
> > > > > > > > wrote:
> > > > > > > > > Changes in V5:
> > > > > > > > > 1. Reorganize the vop_mmap function code in patch 1,
> > > > > > > > > which
> > > > > > > > > is done by
> > > > > > > > 
> > > > > > > > Christoph.
> > > > > > > > > 2. Completely remove the unnecessary code related to
> > > > > > > > > reassign the used ring for card in patch 2.
> > > > > > > > > 
> > > > > > > > > The original vop driver only supports dma coherent
> > > > > > > > > device,
> > > > > > > > > as it allocates and maps vring by _get_free_pages and
> > > > > > > > > dma_map_single, but not use
> > > > > > > > > dma_sync_single_for_cpu/device
> > > > > > > > > to sync the updates of device_page/vring between EP
> > > > > > > > > and
> > > > > > > > > RC, which will cause memory synchronization problem
> > > > > > > > > for
> > > > > > > > > device don't support
> > > > 
> > > > hardware dma coherent.
> > > > > > > > > 
> > > > > > > > > And allocate vrings use dma_alloc_coherent is a
> > > > > > > > > common way
> > > > > > > > > in kernel, as the memory interacted between two
> > > > > > > > > systems
> > > > > > > > > should use consistent memory to avoid caching
> > > > > > > > > effects. So
> > > > > > > > > here add noncoherent platform
> > > > > > > > 
> > > > > > > > support for vop driver.
> > > > > > > > > Also add some related dma changes to make sure
> > > > > > > > > noncoherent
> > > > > > > > > platform works well.
> > > > > > > > > 
> > > > > > > > > Sherry Sun (2):
> > > > > > > > >   misc: vop: change the way of allocating vrings and
> > > > > > > > > device page
> > > > > > > > >   misc: vop: do not allocate and reassign the used
> > > > > > > > > ring
> > > > > > > > > 
> > > > > > > > >  drivers/misc/mic/bus/vop_bus.h     |   2 +
> > > > > > > > >  drivers/misc/mic/host/mic_boot.c   |   9 ++
> > > > > > > > >  drivers/misc/mic/host/mic_main.c   |  43 ++------
> > > > > > > > >  drivers/misc/mic/vop/vop_debugfs.c |   4 -
> > > > > > > > >  drivers/misc/mic/vop/vop_main.c    |  70 +--------
> > > > > > > > > ---
> > > > > > > > >  drivers/misc/mic/vop/vop_vringh.c  | 166 ++++++++++-
> > > > > > > > > ------------
> > 
> > ------
> > > > > > > > >  include/uapi/linux/mic_common.h    |   9 +-
> > > > > > > > >  7 files changed, 85 insertions(+), 218 deletions(-)
> > > > > > > > 
> > > > > > > > Have you all seen:
> > > > > > > > 
> > > > > > > > 
https://eur01.safelinks.protection.outlook.com/?url=https%3A
> > > > > > > > %2F%25
> > > > > > > > 25
> > > > > > > > 
> > 
> > 2Flore.kernel.org%2Fr%2F8c1443136563de34699d2c084df478181c205db4.16
> > > > > > > > 
> > 
> > 03854416.git.sudeep.dutt%40intel.com&amp;data=04%7C01%7Csherry.sun%
> > > > > > > > 
> > 
> > 40nxp.com%7Cc19c987667434969847e08d87b0685e8%7C686ea1d3bc2b4c6f
> > > > > > > > 
> > 
> > a92cd99c5c301635%7C0%7C0%7C637394615238940323%7CUnknown%7CTW
> > > > > > > > 
> > 
> > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> > > > > > > > 
> > 
> > VCI6Mn0%3D%7C1000&amp;sdata=Zq%2FtHWTq%2BuIVBYXFGoeBmq0JJzYd
> > > > > > > > 9zDyv4NVN4TpC%2FU%3D&amp;reserved=0
> > > > > > > > 
> > > > > > > > Looks like this code is asking to just be deleted, is
> > > > > > > > that ok with you?
> > > > > > > 
> > > > > > > Yes, I saw that patch. I'm ok with it.
> > > > > > 
> > > > > > Great, can you please provide a "Reviewed-by:" or "Acked-
> > > > > > by:" for it?
> > > > > > 
> > > > > > thanks,
> > > > > > 
> > > > > > greg k-h
> > > > > 
> > > > > Sherry took much effort on the features support on i.MX
> > > > > series
> > > > > like
> > > > 
> > > > i.MX8QM/i.MX8QXP/i.MX8MM.
> > > > > 
> > > > > Now it is a pity to delete the vop code.
> > > > > 
> > > > > One question,
> > > > > can we resubmit vop code by clean up, now only for i.MX
> > > > > series as
> > > > > Dutt's
> > > > 
> > > > suggestion ?

Resubmitting the VOP code with cleanups tailored for i.MX makes sense
to me.

> > > > > Or we have to drop the design and switch to select other
> > > > > solutions ?
> > > 
> > > Okay, we plan to switch to NTB solution.
> > 
> > What is a "NTB solution" exactly?
> 
> The driver located at drivers/ntb/, it also can setup a point-to-
> point PCI-E bus connecting between two systems.
> But we haven't got a deep look of this driver yet, so we are not sure
> whether it can replace the vop framework.
> 
> > 
> > > 
> > > > If this whole subsystem is being deleted because it is not used
> > > > and
> > > > never shipped, yes, please use a different solution.
> > > > 
> > > > I don't understand why you were trying to piggy-back on this
> > > > codebase if the hardware was totally different, for some reason
> > > > I
> > > > thought this was the same hardware.  What exactly is this?
> > > 
> > > Not the whole codebase, just the vop framework.
> > 
> > That didn't answer the question at all, what are you all trying to
> > do here, with
> > what hardware, that the VOP code seemed like a good fit?
> 
> Vop is a common framework which is independent of the Intel MIC
> hardware.
> We planed to reuse vop framework on two arm64 architecture devices,
> to setup the connection between two systems based on virtio over
> PCIE.

Yes, we wanted Virtio Over PCIe (VOP) to be independent of the hardware
as much as possible. It did end up under the mic/ driver subsystem
though so it would be good to attempt placing it in a generic folder
which is not tied to a specific hardware layer this time around.

Regards,
Sudeep Dutt