Re: [RFC PATCH v4 0/2] RDMA/rxe: Add dma-buf support

[Christian König]

Am 13.12.21 um 12:18 schrieb Shunsuke Mie:
> 2021年12月10日(金) 22:29 Christian König <christian.koenig@xxxxxxx>:
> > Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:
> > > On Fri, Dec 10, 2021 at 01:47:37PM +0100, Christian König wrote:
> > > > Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
> > > > > On Fri, Dec 10, 2021 at 08:29:24PM +0900, Shunsuke Mie wrote:
> > > > > > Hi Jason,
> > > > > > Thank you for replying.
> > > > > >
> > > > > > 2021年12月8日(水) 2:14 Jason Gunthorpe <jgg@xxxxxxxx>:
> > > > > > > On Fri, Dec 03, 2021 at 12:51:44PM +0900, Shunsuke Mie wrote:
> > > > > > > > Hi maintainers,
> > > > > > > >
> > > > > > > > Could you please review this patch series?
> > > > > > > Why is it RFC?
> > > > > > >
> > > > > > > I'm confused why this is useful?
> > > > > > >
> > > > > > > This can't do copy from MMIO memory, so it shouldn't be compatible
> > > > > > > with things like Gaudi - does something prevent this?
> > > > > > I think if an export of the dma-buf supports vmap, CPU is able to access the
> > > > > > mmio memory.
> > > > > >
> > > > > > Is it wrong? If this is wrong, there is no advantages this changes..
> > > > > I don't know what the dmabuf folks did, but yes, it is wrong.
> > > > >
> > > > > IOMEM must be touched using only special accessors, some platforms
> > > > > crash if you don't do this. Even x86 will crash if you touch it with
> > > > > something like an XMM optimized memcpy.
> > > > >
> > > > > Christian? If the vmap succeeds what rules must the caller use to
> > > > > access the memory?
> > > > See dma-buf-map.h and especially struct dma_buf_map.
> > > >
> > > > MMIO memory is perfectly supported here and actually the most common case.
> > > Okay that looks sane, but this rxe RFC seems to ignore this
> > > completely. It stuffs the vaddr directly into a umem which goes to all
> > > manner of places in the driver.
> > >
> > > ??
> > Well, yes that can go boom pretty quickly.
> Sorry, I was wrong. The dma_buf_map treats both iomem and vaddr region, but
> this RFC only supports vaddr. Advantage of the partial support is we can use the
> vaddr dma-buf in RXE without changing a rxe data copy implementation.

Well that is most likely not a good idea.

For example buffers for GPU drivers can be placed in both MMIO memory 
and system memory.

If you don't want to provoke random failures you *MUST* be able to 
handle both if you want to use this.

Regards,
Christian.

> An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for
> VRAM and we can get dma-buf for the region that indicates vaddr regions.
> Specifically, the gpu driver using gpu/drm/drm_gem_cma_helper.c is one such
> example.
>
> > Not sure what they want to use this for.
> I'd like to use RDMA with RXE for that memory region.
>
> Best,
> Shunsuke
> > Christian.
> >
> > > Jason