Re: Dealing with holes in CPU address space
From: Florian Fainelli
Date: Thu Apr 09 2020 - 00:03:19 EST
On 4/8/2020 2:33 PM, Chris Packham wrote:
> On Wed, 2020-04-08 at 15:29 +0800, Jiaxun Yang wrote:
>> On Wed, 8 Apr 2020 05:14:22 +0000
>> Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi All,
>>>
>>> I'm trying to port an old Broadcom MIPS CPU (BCM53003) to a shiny
>>> new
>>> kernel. I have some old historic source from a long forgotten
>>> Broadcom
>>> LDK but I'd prefer to do things the modern way with device-trees.
>>>
>>> The problem I've been grappling with is trying to open up access to
>>> all of the RAM on the board. It has 512MB of DDR2. The CPU has two
>>> areas where this appears. The first 128MB is from 0 to 0x07ffffff
>>> the
>>> second area is from 0x88000000 to 0x9fffffff.
>>>
>>> SoC peripherals are at 0x18000000 and there is an IO window for
>>> flash
>>> at 0x20000000.
>>>
>>> The old code has some custom tlb initialisation to deal with this
>>> but
>>> I figured it should be possible with the following dts snippet.
>>>
>>> memory@0 {
>>> device_type = "memory";
>>> reg = <0x00000000 0x08000000
>>> 0x88000000 0x18000000>;
>>> };
>>>
>>> I end up with only 128MB available. This appears to be
>>> because the default HIGHMEM_START of 0x20000000 stops the rest from
>>> being made available. If I add an override of HIGHMEM_START to
>>> 0xffffffff I seem to have the full 512MB avaiable but then I get a
>>> kernel panic
>>
>> Hi,
>>
>> Have you tried to enable CONFIG_HIGHMEM?
>>
>
> I have but that didn't seem to help. As I understand it HIGHMEM is
> intended for situations when you have more physical RAM that can be
> addressed (e.g. >4GB on a 32-bit system).
On MIPS you may have to enable HIGHMEM as soon as you run out of virtual
kernel address space to map the entire amount of memory that is
populated AFAICT. The kernel has a little under 1GB of virtual address
space that can be mapped via the TLB since the first 512MB are occupied
by KSEG0/1.
>
>>>
>>> CPU 0 Unable to handle kernel paging request at virtual address
>>> 1fc00000, epc == 800167b8, ra == 800e2860
>>>
>>> 0x1fc00000 is in the range where the SoC peripherals are so I'm
>>> thinking that is the problem. But then again that is a virtual
>>> address
>>> so maybe it's just a co-incidence.
>>
>> 0x1fc00000 should be the Boot ROM's physical address. Probably you
>> forgot to convert it into virtual address in your platform code?
>
> Yes that's right it's the bootroms PA. I'm not intitentionally doing
> anything with it but maybe that's the problem.
If you were accessing this as a virtual address then it would be either
via KSEG0/1 and the addresses would be 0x1fc0_0000 + 0x8000_0000 or
0x1fc0_0000 + 0xa000_0000 but here it looks like the raw physical
address is being accessed which suggests the TLB is incorrectly
programmed somehow.
>
>>
>> Check the EPC of exception in vmlinux with addr2line may help. (Don't
>> forget to compile your kernel with debuginfo).
>>
>
> The full panic is
>
> CPU 0 Unable to handle kernel paging request at virtual address 1fc00000, epc == 800167b8, ra == 800e2860
> Oops[#1]:
> CPU: 0 PID: 0 Comm: swapper Not tainted 5.5.0-at1 #46
> $ 0 : 00000000 00000001 00000001 807e4270
> $ 4 : 1fc00000 00000000 1fc00f80 00000001
> $ 8 : 83e52a00 83e52a24 807f628c 8080fa00
> $12 : ffffffff 00000001 0000000b ffffffff
> $16 : 83e52a20 80000000 80870000 83e52a00
> $20 : 8080fdd4 807dde00 00000000 8080f9bc
> $24 : 8080fb68 ffffff7f
> $28 : 807dc000 807ddd38 83e52a00 800e2860
> Hi : 00000000
> Lo : 00000000
> epc : 800167b8 clear_page+0x18/0x128
> ra : 800e2860 get_page_from_freelist+0xa94/0xdd4
> Status: 11000002 KERNEL EXL
> Cause : 4080000c (ExcCode 03)
> BadVA : 1fc00000
> PrId : 00019749 (MIPS 74Kc)
> Modules linked in:
> Process swapper (pid: 0, threadinfo=(ptrval), task=(ptrval), tls=00000000)
> *HwTLS: e19f7d3a
> Stack : 807e0000 807dde98 80867cd8 80791814 00000001 80687974 807dde18 00000000
> 807f5ed0 807f628c 807f628c 8080fdd4 80870000 00000001 807e0000 00000044
> 0000005c 807dde00 00000000 00000001 80810000 8075a660 80870001 00000000
> 00000100 00000000 807e0000 00000000 00000000 00000001 80860000 808492b4
> 87d75608 800e3028 00000100 00000000 00000001 80058c08 80870000 80870000
> ...
> Call Trace:
> [<800167b8>] clear_page+0x18/0x128
> [<800e2860>] get_page_from_freelist+0xa94/0xdd4
> [<800e3028>] __alloc_pages_nodemask+0xf4/0xbb8
> [<800e3b08>] __get_free_pages+0x1c/0x58
> [<80013430>] setup_zero_pages+0x3c/0xe4
> [<80826eac>] mem_init+0x40/0x50
> [<808219c0>] start_kernel+0x250/0x510
> Code: cc9e0040 cc9e0060 cc9e0080 <ac800000> ac800004 ac800008 ac80000c 24840020 ac80fff0
>
> I think this is just the early setup of the pages.
>
>>>
>>> Anyway I'd really appreciate any guidance that anyone could provide
>>> on
>>> this. Even if it's just "go look at this SoC".
>>>
>>> Thanks,
>>> Chris
>>>
>>>
>>
>> --
>> Jiaxun Yang
--
Florian