Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes

[Vanshidhar Konda]

On Thu, Oct 29, 2020 at 01:37:10PM +0000, Catalin Marinas wrote:
> On Wed, Oct 21, 2020 at 11:29:41PM +0100, Valentin Schneider wrote:
> > On 21/10/20 12:02, Jonathan Cameron wrote:
> > > On Wed, 21 Oct 2020 09:43:21 +0530
> > > Anshuman Khandual <anshuman.khandual@xxxxxxx> wrote:
> > >> Agreed. Do we really need to match X86 right now ? Do we really have
> > >> systems that has 64 nodes ? We should not increase the default node
> > >> value and then try to solve some new problems, when there might not
> > >> be any system which could even use that. I would suggest increase
> > >> NODES_SHIFT value upto as required by a real and available system.
> > >
> > > I'm not going to give precise numbers on near future systems but it is public
> > > that we ship 8 NUMA node ARM64 systems today.  Things will get more
> > > interesting as CXL and CCIX enter the market on ARM systems,
> > > given chances are every CXL device will look like another NUMA
> > > node (CXL spec says they should be presented as such) and you
> > > may be able to rack up lots of them.
> > >
> > > So I'd argue minimum that makes sense today is 16 nodes, but looking forward
> > > even a little and 64 is not a great stretch.
> > > I'd make the jump to 64 so we can forget about this again for a year or two.
> > > People will want to run today's distros on these new machines and we'd
> > > rather not have to go around all the distros asking them to carry a patch
> > > increasing this count (I assume they are already carrying such a patch
> > > due to those 8 node systems)
> >
> > I agree that 4 nodes is somewhat anemic; I've had to bump that just to
> > run some scheduler tests under QEMU. However I still believe we should
> > exercise caution before cranking it too high, especially when seeing things
> > like:
> >
> >   ee38d94a0ad8 ("page flags: prioritize kasan bits over last-cpuid")
> >
> > To give some numbers, a defconfig build gives me:
> >
> >   SECTIONS_WIDTH=0 ZONES_WIDTH=2 NODES_SHIFT=2 LAST_CPUPID_SHIFT=(8+8) KASAN_TAG_WIDTH=0
> >   BITS_PER_LONG=64 NR_PAGEFLAGS=24
> >
> > IOW, we need 18 + NODES_SHIFT <= 40 -> NODES_SHIFT <= 22. That looks to be
> > plenty, however this can get cramped fairly easily with any combination of:
> >
> >   CONFIG_SPARSEMEM_VMEMMAP=n (-18)
> >   CONFIG_IDLE_PAGE_TRACKING=y (-2)
> >   CONFIG_KASAN=y + CONFIG_KASAN_SW_TAGS (-8)
> >
> > Taking Arnd's above example, a randconfig build picking !VMEMMAP already
> > limits the NODES_SHIFT to 4 *if* we want to keep the CPUPID thing within
> > the flags (it gets a dedicated field at the tail of struct page
> > otherwise). If that is something we don't care too much about, then
> > consider my concerns taken care of.
>
> I don't think there's any value in allowing SPARSEMEM_VMEMMAP to be
> disabled but the option is in the core mm/Kconfig file. We could make
> NODES_SHIFT depend on SPARSEMEM_VMEMMAP (there's DISCONTIGMEM as well
> but hopefully that's going away soon).
>
> > One more thing though: NR_CPUS can be cranked up to 4096 but we've only set
> > it to 256 IIRC to support the TX2. From that PoV, I'm agreeing with
> > Anshuman in that we should set it to match the max encountered on platforms
> > that are in use right now.
>
> I agree. Let's bump NODES_SHIFT to 4 now to cover existing platforms. If
> distros have a 10-year view, they can always ship a kernel configured to
> 64 nodes, no need to change Kconfig (distros never ship with defconfig).
>
> It may have an impact on more memory constrained platforms but that's
> not what defconfig is about. It should allow existing hardware to run
> Linux but not necessarily run it in the most efficient way possible.

From the discussion it looks like 4 is an acceptable number to support
current hardware. I'll send a patch with NODES_SHIFT set to 4. Is it 
still possible to add this change to the 5.10 kernel?

Vanshi

> --
> Catalin