Re: [PATCH v4 00/15] Habana Labs kernel driver

[Oded Gabbay]

On Thu, Feb 14, 2019 at 1:04 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
>
> On Thu, Feb 14, 2019 at 12:45:00PM +0200, Oded Gabbay wrote:
> > On Thu, Feb 14, 2019 at 12:37 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> > >
> > > On Thu, Feb 14, 2019 at 12:15:19PM +0200, Oded Gabbay wrote:
> > > > On Thu, Feb 14, 2019 at 12:07 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > On Thu, Feb 14, 2019 at 11:58:41AM +0200, Oded Gabbay wrote:
> > > > > > On Thu, Feb 14, 2019 at 9:13 AM Oded Gabbay <oded.gabbay@xxxxxxxxx> wrote:
> > > > > > >
> > > > > > > On Thu, Feb 14, 2019 at 9:11 AM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> > > > > > > >
> > > > > > > > On Mon, Feb 11, 2019 at 05:17:36PM +0200, Oded Gabbay wrote:
> > > > > > > > > Hello,
> > > > > > > > > This is v4 of the Habana Labs kernel driver patch-set. It contains fixes
> > > > > > > > > according to reviews done on v3, mainly for the command buffer, sysfs and MMU
> > > > > > > > > patches. In addition, patch 2/15 was reduced in size from 4.3MB to 1.4MB.
> > > > > > > > >
> > > > > > > > > The patch-set is rebased on v5.0-rc6.
> > > > > > > > >
> > > > > > > > > Link to v3 cover letter: https://lkml.org/lkml/2019/2/4/1033
> > > > > > > > >
> > > > > > > > > Link to v2 cover letter: https://lkml.org/lkml/2019/1/30/1003
> > > > > > > > >
> > > > > > > > > Link to v1 cover letter: https://lwn.net/Articles/777342/
> > > > > > > > >
> > > > > > > > > I would appricate any feedback, question and/or review.
> > > > > > > >
> > > > > > > > There's been some 0-day bot feedback on some of these patches now that I
> > > > > > > > put them in my -testing branch.  So I'm going to drop the patch series
> > > > > > > > from there now and wait for a v5 of the series that hopefully will have
> > > > > > > > those issues fixed :)
> > > > > > > >
> > > > > > Hi Greg,
> > > > > > I looked at the 4 warnings I received from your emails, and they all
> > > > > > appear in i386 architecture.
> > > > > > I don't want to support 32-bit kernel and I don't intend to support it.
> > > > > > Can we just specify in kconfig that we don't support it, and then you
> > > > > > won't get these warnings ?
> > > > >
> > > > > No, if you use the correct kernel types and castings, you should be
> > > > > fine.
> > > > >
> > > > > > I initially set in kconfig to support only x86_64, and you told me
> > > > > > (and you were right) not to limit to that. But I do think I would like
> > > > > > to disable the driver on i386.
> > > > >
> > > > > You might want to not support it on 32bit kernels, but even then, I
> > > > > think all you need to do here is use the proper kernel types and you
> > > > > will be ok.
> > > > >
> > > > > As an example:
> > > > >    drivers/misc/habanalabs/goya/goya.c: In function 'goya_early_init':
> > > > >    drivers/misc/habanalabs/goya/goya.c:404:4: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 4 has type 'resource_size_t' {aka 'unsigned int'} [-Wformat=]
> > > > >        "Not " HL_NAME "? BAR %d size %llu, expecting %llu\n",
> > > > >        ^~~~~~
> > > > >
> > > > > Use the correct printk type for a resource_size_t.
> > > > >
> > > > > You got that warning twice.
> > > > >
> > > > > Another one is:
> > > > > >> drivers/misc/habanalabs/device.c:283:24: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
> > > > >      volatile u32 *paddr = (volatile u32 *) addr;
> > > > >
> > > > > Now using a volatile makes me want to say "you are doing it wrong!", as
> > > > > yes, you shouldn't be reading directly from a memory pointer, you need
> > > > > to use the correct iomem accessors, right?
> > > > >
> > > > > So I think just fixing this stuff up should be simple, the
> > > > > resource_size_t fix is needed no matter what size kernel you run on.
> > > > >
> > > > > thanks,
> > > > >
> > > > > greg k-h
> > > >
> > > > ok, got it, will be fixed.
> > > >
> > > > Regarding the volatile, this is not an I/O memory. This is host memory
> > > > that is changed by the device. That's why I wrote in the comment
> > > > there:
> > > > /*
> > > > * paddr is defined as volatile because it points to HOST memory,
> > > > * which is being written to by the device. Therefore, we can't use
> > > > * locks to synchronize it and it is not a memory-mapped register space
> > >
> > > What do you mean by "HOST" memory?  The memory that the processor is
> > > running on?
> > >
> > Yes, exactly. The memory of the server. Not a memory on my device.
> >
> > > > */
> > > >
> > > > Am I missing something here ? I don't think I should use the iomem
> > > > accessors on host memory, right ? Assuming I'm right, is there another
> > > > way to ensure the compiler won't optimize this without using the
> > > > volatile keyword ?
> > >
> > > What are you trying to prevent from being "optimized" here?
> > >
> > > Are you sure you just don't need a correct memory barrier?  That's the
> > > only way to ensure that if you write to a location from one thread/cpu,
> > > it will show up to the other thread/cpu correctly.  volatile will not
> > > ensure that for you at all (hint, the compiler just ignores it for the
> > > most part.)
> >
> > But the writing entity in this case is NOT another thread/cpu. The
> > writing entity is the device. So a memory barrier, IMO, won't help me
> > here, because memory barriers affect only on the CPU. Not on external
> > initiators.
> >
> > AFAIK, the volatile keyword tells the compiler that the value of the
> > variable may change at any time--without any action being taken by the
> > code the compiler finds nearby. And this is exactly what happens here.
> > I poll on a memory location of the CPU, and that memory can change at
> > any time by the device.
>
> And how is that memory location mapped into the device memory?  As such,
> it's iomemory, right?
>
> volatile doesn't tell the compiler much, if anything, anymore.
>
> I can't seem to trace back the code here (it's split across multiple
> emails), but it seems that the memory location is coming from struct
> armcp_packet in one location, and a dma pool in another one.  I don't
> know what the rules are for dma mapped memory, but it feels like
> 'volatile' is not the way to use it :)
>
Yes, this is dma coherent memory.

And I actually found one call to this function where we use this
function for polling on host memory that is changed by another
thread/core.
Therefore, I replaced the volatile with an mb() to flush the CPU read
and write buffers. I believe that is enough and I hope this closes
this issue :)

Oded


> thanks,
>
> greg k-h