ASLR, stack location, and BSS nonsense

[Joshua Hudson]

I have this program with a 16 GB BSS segment that stopped loading.
Apparently there's been a phase change and some systems won't
opportunistically load a binary with too big of BSS anymore. The job
size is multi-GB in size so saying use a smaller memory size doesn't
really cut it. To be fair I too would rather allocate dynamically and
display a reasonable error message on running out altogether.

The root algorithm is a single stretchy array and can't be easily
adopted for a fragmented memory pool. So I ponder my other options
(all of which include growing stretchy array with mmap() rather than a
giant BSS):

1) Try locating stack self-relative (it's a PIE executable).

1A) PT_GNU_STACK always ignores its address

1B) Supposing I could enumerate my address space and free the original
stack and set my new stack where I want it; the original stack might
eventually be allocated again, and treated as arguments by /proc.
Oops.

2) relaunch self with personality set to no ASLR. Oops; ever try to
relaunch yourself in a chroot jail that deliberately does not have
/proc mounted? It's hard.

3) Determine what the absolute limits are of ASLR and see if this is a
problem. Oops, insufficient documentation

4) Determine what the absolute limits are of ASLR, sed my load
address, and see if this is a problem. Oops, insufficient
documentation (still).

5) Determine stack top and bottom from value of rsp and which of three
memory regions to start my stretchy array. I think I can force my
initial size via PT_GNU_STACK which means I can estimate from rsp.

So the regions would be
5a) 0x10000 to load address (easy to get from rsp)
5b) load address + size of program to top of stack
5c) bottom of stack + small constant to top of address space

Problem: what if the environment is too big
Problem: what is the value of (small constant)

Anyway, all potential solutions have problems. Personally I'd love it
if setting a base address into PT_GNU_STACK actually worked but it's
probably not the best solution (see, what if the environment is too
big (to be fair I don't need any part of the environment but there
doesn't seem to be a good way to say that)). This doesn't defeat ASLR
because it's a PIE binary so the base is self relative.

Note that "insufficient documentation" isn't _quite_ resolved by read
kernel source. It's not "what does it to today". It's "what can I
depend on not breaking under me". I just got burned and don't want to
get burned again.

The source code is published on github. You don't want to deal with
it. https://github.com/joshudson/emergency64/blob/master/ren.asm It's
kind of an oddball in that project in that it has other uses besides
just rescue.