Re: Dealing with holes in CPU address space

From: Chris Packham
Date: Wed Apr 08 2020 - 17:33:15 EST


On Wed, 2020-04-08 at 15:29 +0800, Jiaxun Yang wrote:
> On Wed, 8 Apr 2020 05:14:22 +0000
> Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> wrote:
>
> > Hi All,
> >
> > I'm trying to port an old Broadcom MIPS CPU (BCM53003) to a shiny
> > new
> > kernel. I have some old historic source from a long forgotten
> > Broadcom
> > LDK but I'd prefer to do things the modern way with device-trees.
> >
> > The problem I've been grappling with is trying to open up access to
> > all of the RAM on the board. It has 512MB of DDR2. The CPU has two
> > areas where this appears. The first 128MB is from 0 to 0x07ffffff
> > the
> > second area is from 0x88000000 to 0x9fffffff.
> >
> > SoC peripherals are at 0x18000000 and there is an IO window for
> > flash
> > at 0x20000000.
> >
> > The old code has some custom tlb initialisation to deal with this
> > but
> > I figured it should be possible with the following dts snippet.
> >
> > memory@0 {
> > device_type = "memory";
> > reg = <0x00000000 0x08000000
> > 0x88000000 0x18000000>;
> > };
> >
> > I end up with only 128MB available. This appears to be
> > because the default HIGHMEM_START of 0x20000000 stops the rest from
> > being made available. If I add an override of HIGHMEM_START to
> > 0xffffffff I seem to have the full 512MB avaiable but then I get a
> > kernel panic
>
> Hi,
>
> Have you tried to enable CONFIG_HIGHMEM?
>

I have but that didn't seem to help. As I understand it HIGHMEM is
intended for situations when you have more physical RAM that can be
addressed (e.g. >4GB on a 32-bit system).

> >
> > CPU 0 Unable to handle kernel paging request at virtual address
> > 1fc00000, epc == 800167b8, ra == 800e2860
> >
> > 0x1fc00000 is in the range where the SoC peripherals are so I'm
> > thinking that is the problem. But then again that is a virtual
> > address
> > so maybe it's just a co-incidence.
>
> 0x1fc00000 should be the Boot ROM's physical address. Probably you
> forgot to convert it into virtual address in your platform code?

Yes that's right it's the bootroms PA. I'm not intitentionally doing
anything with it but maybe that's the problem.

>
> Check the EPC of exception in vmlinux with addr2line may help. (Don't
> forget to compile your kernel with debuginfo).
>

The full panic is

CPU 0 Unable to handle kernel paging request at virtual address 1fc00000, epc == 800167b8, ra == 800e2860
Oops[#1]:
CPU: 0 PID: 0 Comm: swapper Not tainted 5.5.0-at1 #46
$ 0 : 00000000 00000001 00000001 807e4270
$ 4 : 1fc00000 00000000 1fc00f80 00000001
$ 8 : 83e52a00 83e52a24 807f628c 8080fa00
$12 : ffffffff 00000001 0000000b ffffffff
$16 : 83e52a20 80000000 80870000 83e52a00
$20 : 8080fdd4 807dde00 00000000 8080f9bc
$24 : 8080fb68 ffffff7f
$28 : 807dc000 807ddd38 83e52a00 800e2860
Hi : 00000000
Lo : 00000000
epc : 800167b8 clear_page+0x18/0x128
ra : 800e2860 get_page_from_freelist+0xa94/0xdd4
Status: 11000002 KERNEL EXL
Cause : 4080000c (ExcCode 03)
BadVA : 1fc00000
PrId : 00019749 (MIPS 74Kc)
Modules linked in:
Process swapper (pid: 0, threadinfo=(ptrval), task=(ptrval), tls=00000000)
*HwTLS: e19f7d3a
Stack : 807e0000 807dde98 80867cd8 80791814 00000001 80687974 807dde18 00000000
807f5ed0 807f628c 807f628c 8080fdd4 80870000 00000001 807e0000 00000044
0000005c 807dde00 00000000 00000001 80810000 8075a660 80870001 00000000
00000100 00000000 807e0000 00000000 00000000 00000001 80860000 808492b4
87d75608 800e3028 00000100 00000000 00000001 80058c08 80870000 80870000
...
Call Trace:
[<800167b8>] clear_page+0x18/0x128
[<800e2860>] get_page_from_freelist+0xa94/0xdd4
[<800e3028>] __alloc_pages_nodemask+0xf4/0xbb8
[<800e3b08>] __get_free_pages+0x1c/0x58
[<80013430>] setup_zero_pages+0x3c/0xe4
[<80826eac>] mem_init+0x40/0x50
[<808219c0>] start_kernel+0x250/0x510
Code: cc9e0040 cc9e0060 cc9e0080 <ac800000> ac800004 ac800008 ac80000c 24840020 ac80fff0

I think this is just the early setup of the pages.

> >
> > Anyway I'd really appreciate any guidance that anyone could provide
> > on
> > this. Even if it's just "go look at this SoC".
> >
> > Thanks,
> > Chris
> >
> >
>
> --
> Jiaxun Yang