Re: 2.6.32-rc3: low mem - only 378MB on x86_32 with 64GB. Why?

From: Linus Torvalds
Date: Sat Oct 10 2009 - 14:39:13 EST




On Sat, 10 Oct 2009, Valdis.Kletnieks@xxxxxx wrote:
>
> When the x86 went 64-bit, the register pressure relief from the
> additional registers usually more then outweighs the additional memory
> bandwidth (basically, if you're spending twice as much time on each
> load/store, but only doing it 40% as often, you come out ahead...)

That's mainly stack traffic, and x86 has always been good at it. More
registers makes for simpler (and fewer) instructions due to less reloads,
but for kernel loads, it's not the biggest advantage.

If you have 8GB of RAM or more, the biggest advantage _by_far_ for the
kernel is that you don't spend 25% of your system time playing with
k[un]map() and the TLB flushing that goes along with it. You also have
much more freedom to allocate (and thus cache) inodes, dentries and
various other fundamental kernel data structures.

Also, the reason MIPS and Sparc had a slowdown for 64-bit code was only
partially the bigger cache footprint (and that depends a lot on the app
anyway: many applications aren't that pointer-intensive. The kernel is
_very_ pointer-intensive, but even for something like that, most data
structures tend to blow up by 50%, not 100%).

The other reason for slowdown is that generating those pointers (for
function calls in particular) is more complex, and x86-64 is better at
that than MIPS and Sparc. That complex instruction encoding with
variable-size instructions means that you don't have to try to fit all
constants in the instruction stream either in the fixed-sized instruction,
or by doing indirect data access to memory through a GP register.

So x86-64 not only had the register expansion advantage, it had less of a
code generation downside to 64-bit mode to begin with. Want to have large
constants in the code? No problem. Sure, it makes your code bigger, but
you can still have them predecoded in the instruction stream rather than
have to load them from memory. Much nicer for everybody.

And for the kernel, the bigger virtual address space really is a _huge_
deal. HIGHMEM accesses really are very slow. You don't see that in user
space, but I really have seen 25% performance differences between
non-highmem builds and CONFIG_HIGHMEM4G enabled for things that try to put
a lot of data in highmem (and the 64G one is even more expensive). And
that was just with 2GB of RAM.

And when it makes the difference between doing IO or not doign IO (ie
caching or not caching - when the dentry cache can't grow any more because
it _needs_ to be in lowmem), you can literally see an order-of-magnitude
difference.

With 8GB+ of ram, I guarantee you that the kernel spent tons of time on
just mappign high pages, _and_ it couldn't grow inodes and dentry caches
nearly as big as it would have wanted to. Going to x86-64 makes all those
issues just go away entirely.

So it's not "you can save a few instructions by not spilling to stack as
much". It's a much bigger deal than that. There's a reason I personally
refuse to even care about >2GB 32-bit machines. There's just no excuse
these days to do that.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/