Re: [PATCH v5 0/4] Raspberry Pi 4 DMA addressing support

[Matthias Brugger]

On 13/09/2019 12:08, Stefan Wahren wrote:
> Am 13.09.19 um 11:25 schrieb Matthias Brugger:
>>
>> On 13/09/2019 10:50, Stefan Wahren wrote:
>>> Am 13.09.19 um 10:09 schrieb Matthias Brugger:
>>>> On 12/09/2019 21:32, Stefan Wahren wrote:
>>>>> Am 12.09.19 um 19:18 schrieb Matthias Brugger:
>>>>>> On 10/09/2019 11:27, Matthias Brugger wrote:
>>>>>>> On 09/09/2019 21:33, Stefan Wahren wrote:
>>>>>>>> Hi Nicolas,
>>>>>>>>
>>>>>>>> Am 09.09.19 um 11:58 schrieb Nicolas Saenz Julienne:
>>>>>>>>> Hi all,
>>>>>>>>> this series attempts to address some issues we found while bringing up
>>>>>>>>> the new Raspberry Pi 4 in arm64 and it's intended to serve as a follow
>>>>>>>>> up of these discussions:
>>>>>>>>> v4: https://lkml.org/lkml/2019/9/6/352
>>>>>>>>> v3: https://lkml.org/lkml/2019/9/2/589
>>>>>>>>> v2: https://lkml.org/lkml/2019/8/20/767
>>>>>>>>> v1: https://lkml.org/lkml/2019/7/31/922
>>>>>>>>> RFC: https://lkml.org/lkml/2019/7/17/476
>>>>>>>>>
>>>>>>>>> The new Raspberry Pi 4 has up to 4GB of memory but most peripherals can
>>>>>>>>> only address the first GB: their DMA address range is
>>>>>>>>> 0xc0000000-0xfc000000 which is aliased to the first GB of physical
>>>>>>>>> memory 0x00000000-0x3c000000. Note that only some peripherals have these
>>>>>>>>> limitations: the PCIe, V3D, GENET, and 40-bit DMA channels have a wider
>>>>>>>>> view of the address space by virtue of being hooked up trough a second
>>>>>>>>> interconnect.
>>>>>>>>>
>>>>>>>>> Part of this is solved on arm32 by setting up the machine specific
>>>>>>>>> '.dma_zone_size = SZ_1G', which takes care of reserving the coherent
>>>>>>>>> memory area at the right spot. That said no buffer bouncing (needed for
>>>>>>>>> dma streaming) is available at the moment, but that's a story for
>>>>>>>>> another series.
>>>>>>>>>
>>>>>>>>> Unfortunately there is no such thing as 'dma_zone_size' in arm64. Only
>>>>>>>>> ZONE_DMA32 is created which is interpreted by dma-direct and the arm64
>>>>>>>>> arch code as if all peripherals where be able to address the first 4GB
>>>>>>>>> of memory.
>>>>>>>>>
>>>>>>>>> In the light of this, the series implements the following changes:
>>>>>>>>>
>>>>>>>>> - Create both DMA zones in arm64, ZONE_DMA will contain the first 1G
>>>>>>>>>   area and ZONE_DMA32 the rest of the 32 bit addressable memory. So far
>>>>>>>>>   the RPi4 is the only arm64 device with such DMA addressing limitations
>>>>>>>>>   so this hardcoded solution was deemed preferable.
>>>>>>>>>
>>>>>>>>> - Properly set ARCH_ZONE_DMA_BITS.
>>>>>>>>>
>>>>>>>>> - Reserve the CMA area in a place suitable for all peripherals.
>>>>>>>>>
>>>>>>>>> This series has been tested on multiple devices both by checking the
>>>>>>>>> zones setup matches the expectations and by double-checking physical
>>>>>>>>> addresses on pages allocated on the three relevant areas GFP_DMA,
>>>>>>>>> GFP_DMA32, GFP_KERNEL:
>>>>>>>>>
>>>>>>>>> - On an RPi4 with variations on the ram memory size. But also forcing
>>>>>>>>>   the situation where all three memory zones are nonempty by setting a 3G
>>>>>>>>>   ZONE_DMA32 ceiling on a 4G setup. Both with and without NUMA support.
>>>>>>>>>
>>>>>>>> i like to test this series on Raspberry Pi 4 and i have some questions
>>>>>>>> to get arm64 running:
>>>>>>>>
>>>>>>>> Do you use U-Boot? Which tree?
>>>>>>> If you want to use U-Boot, try v2019.10-rc4, it should have everything you need
>>>>>>> to boot your kernel.
>>>>>>>
>>>>>> Ok, here is a thing. In the linux kernel we now use bcm2711 as SoC name, but the
>>>>>> RPi4 devicetree provided by the FW uses mostly bcm2838.
>>>>> Do you mean the DTB provided at runtime?
>>>>>
>>>>> You mean the merged U-Boot changes, doesn't work with my Raspberry Pi
>>>>> series?
>>>>>
>>>>>>  U-Boot in its default
>>>>>> config uses the devicetree provided by the FW, mostly because this way you don't
>>>>>> have to do anything to find out how many RAM you really have. Secondly because
>>>>>> this will allow us, in the near future, to have one U-boot binary for both RPi3
>>>>>> and RPi4 (and as a side effect one binary for RPi1 and RPi2).
>>>>>>
>>>>>> Anyway, I found at least, that the following compatibles need to be added:
>>>>>>
>>>>>> "brcm,bcm2838-cprman"
>>>>>> "brcm,bcm2838-gpio"
>>>>>>
>>>>>> Without at least the cprman driver update, you won't see anything.
>>>>>>
>>>>>> "brcm,bcm2838-rng200" is also a candidate.
>>>>>>
>>>>>> I also suppose we will need to add "brcm,bcm2838" to
>>>>>> arch/arm/mach-bcm/bcm2711.c, but I haven't verified this.
>>>>> How about changing this in the downstream kernel? Which is much easier.
>>>> I'm not sure I understand what you want to say. My goal is to use the upstream
>>>> kernel with the device tree blob provided by the FW.
>>> The device tree blob you are talking is defined in this repository:
>>>
>>> https://github.com/raspberrypi/linux
>>>
>>> So the word FW is misleading to me.
>>>
>> No, it's part of
>> https://github.com/raspberrypi/firmware.git
>> file boot/bcm2711-rpi-4-b.dtb
> The compiled DT blobs incl. the kernel image are stored in this artifact
> repository. But the sources for the kernel and the DT are in the Linux
> repo. This is necessary to be compliant to the GPL.

Got it, thanks for clarifying.

>>
>>>>  If you talk about the
>>>> downstream kernel, I suppose you mean we should change this in the FW DT blob
>>>> and in the downstream kernel. That would work for me.
>>>>
>>>> Did I understand you correctly?
>>> Yes
>>>
>>> So i suggest to add the upstream compatibles into the repo mentioned above.
>>>
>>> Sorry, but in case you decided as a U-Boot developer to be compatible
>>> with a unreviewed DT, we also need to make U-Boot compatible with
>>> upstream and downstream DT blobs.
>>>
>> Well RPi3 is working with the DT blob provided by the FW, as I mentioned earlier
>> if we can use this DTB we can work towards one binary that can boot both RPi3
>> and RPi4. On the other hand we can rely on the FW to detect the amount of memory
>> our RPi4 has.
>>
>> That said, I agree that we should make sure that U-Boot can boot with both DTBs,
>> the upstream one and the downstream. Now the question is how to get to this. I'm
>> a bit puzzled that by talking about "unreviewed DT" you insinuate that bcm2711
>> compatible is already reviewed and can't be changed. From what I can see none of
>> these compatibles got merged for now, so we are still at time to change them.
> 
> Stephen Boyd was okay with clk changes except of a small nit. So i fixed
> this is as he suggested in a separate series. Unfortunately this hasn't
> be applied yet [1].
> 
> The i2c, pinctrl and the sdhci changes has been applied yet.
> 
> In my opinion it isn't the job of the mainline kernel to adapt to a
> vendor device tree. It's the vendor device tree which needs to be fixed.
> 

I agree with that. But if we can make this easier by choosing a compatible which
fits downstream without violating upstream and it makes sense with the naming
scheme of the RPi, I think that's a good argument.

> Sorry, but this is my holiday. I will back after the weekend.
> 

Sure, enjoy. I'll be on travel for the next two weeks but will try to keep up
with emails.

Regards,
Matthias

> Best regards
> Stefan
> 
> [1] - https://www.spinics.net/lists/linux-clk/msg40534.html
> 
>>
>> Apart from the point Florian made, to stay consistent with the RPi SoC naming,
>> it will save us work, both in the kernel and in U-Boot, as we would need to add
>> both compatibles to the code-base.
>>
>> Regards,
>> Matthias
>>
>>>>>> Regards,
>>>>>> Matthias
>>>>>>
>>>>>>> Regards,
>>>>>>> Matthias
>>>>>>>
>>>>>>>> Are there any config.txt tweaks necessary?
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> linux-arm-kernel mailing list
>>>>>>> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
>>>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>>>>
>>>>>> _______________________________________________
>>>>>> linux-arm-kernel mailing list
>>>>>> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
>>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel