AArch64 executables allocate huge stack

[compor]

AArch64 executables as statically linked by our toolchain using musl libc seem to allocate a huge stack.

This causes gdb to hang during coredump generation since it is forced to write to the coredump file a huge VMA (in the order of hundreds of GBs) for the stack allocation.

Output of cat /proc/[pid]/maps:

004f0000-004f1000 r--p 00000000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00500000-00507000 r-xp 00100000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00600000-00602000 r--p 00200000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00700000-00702000 rw-p 00300000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00800000-00802000 rw-p 00000000 00:00 0
00900000-00901000 r--p 00400000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00a00000-00a01000 r--p 00500000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00a01000-00a03000 rw-p 00501000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
7ffffeffd000-7fffffffd000 rw-p 00000000 00:00 0
7fffffffd000-7ffffffff000 r--p 00000000 00:00 0                          [vvar]
7ffffffff000-800000000000 r-xp 00000000 00:00 0                          [vdso]
800000000000-fffffffdf000 ---p 00000000 00:00 0   <------------ stack allocation!

Output of cat /proc/[pid]/smaps:

800000000000-fffffffdf000 ---p 00000000 00:00 0
Size:           137438953340 kB   <--------------------------------- this is several gigabytes!
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    0
VmFlags: mr mw me

In contrast, the corresponding X86 executable seems to have a normal stack size that is also properly identified as such:

7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
Size:                132 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    1
ProtectionKey:         0
VmFlags: rd wr mr mw me gd ac

[blackgeorge-boom]

It would be also useful if @abarbala would take a look at this. Any insight would be helpful! I think Ceasar had mentioned he had a similar issue.

[compor]

So, with the help of @blackgeorge-boom I've had another look at the _start_c code and it turns out that the bracket tags for non-file backed memory segments (e.g., [vvar], [vdso]`) might not necessarily make sense as they are produced by the kernel's procfs subsystem and are not in sync with our placement of regions in musl libc.

Hence, in the aforementioned /proc/[pid]/maps the last segment is the padding required

004f0000-004f1000 r--p 00000000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00500000-00507000 r-xp 00100000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00600000-00602000 r--p 00200000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00700000-00702000 rw-p 00300000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00800000-00802000 rw-p 00000000 00:00 0
00900000-00901000 r--p 00400000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00a00000-00a01000 r--p 00500000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
00a01000-00a03000 rw-p 00501000 08:12 2901681                            /home/vasil/migration/call_leaf_aarch64_aligned.out
7ffffeffd000-7fffffffd000 rw-p 00000000 00:00 0                          <------ actual stack
7fffffffd000-7ffffffff000 r--p 00000000 00:00 0                          [vvar]
7ffffffff000-800000000000 r-xp 00000000 00:00 0                          [vdso]
800000000000-fffffffdf000 ---p 00000000 00:00 0

[compor]

Commenting out the __syscall(SYS_mmap) call in the STACK_RELOC_PROTECT-guarded part in crt1.c seems not to change the address ranges for the rest of the segments (as shown above).

Thus, the current question is if we do need the stack relocation protection.

[blackgeorge-boom]

Hello @abarbala.

Could you remind me what is the purpose of this line?

https://github.com/systems-nuts/musl-stack-reloc/blob/unasl/crt/crt1.c#L326

Because, as Chris mentioned, it seems to be the one causing the trouble.

[abarbala]

because ARM has a larger virtual address space and we would like to enforce a smaller virtual address space what we do here is to enforce the same virtual address space as x86.
We do that by creating a mapping that is with PROT_NONE: pages cannot be accessed. So that no code may think to allocate anything in that area that doesn't exist in x86.

[blackgeorge-boom]

@abarbala Thank you for your reply!

There are still two points I don't understand:

1. Where is this PROT_NONE happening in the code? Because I can't seem to find it.
2. "So that no code may think to allocate anything in that area that doesn't exist in x86." Since we already assigned the stack elsewhere, how could the code allocate something in the padding area?

[abarbala]

0 == PROT_NONE
__syscall(SYS_mmap, STACK_END_ADDR, arch_vaddr_max() - STACK_END_ADDR, 0, <<<<<<<<
(MAP_PRIVATE|MAP_ANON|MAP_FIXED), -1, 0);

[abarbala]

If you do mmap without specifying an address, the OS can return a memory area anywhere in the address space. Hence, in order to make sure no one will allocate or use that memory, you need to mprotect.

[blackgeorge-boom]

If you do mmap without specifying an address, the OS can return a memory area anywhere in the address space. Hence, in order to make sure no one will allocate or use that memory, you need to mprotect.

Ok, that makes sense, thank you!

[compor]

Small (belated) update:

This also causes an error for criu:

Warn  (criu/image.c:134): Failed to open parent directory
Error (criu/pagemap-cache.c:54): pagemap-cache: pmc_init: Can't allocate 274877906664 bytes
Error (criu/pagemap-cache.c:85): pagemap-cache: Failed to init pagemap for 1177254
Error (criu/mem.c:546): Can't dump page with parasite
Error (criu/cr-dump.c:1732): Dumping FAILED.

An idea would be to modify the memory dumping code to exclude VMAs that are:

1. over a specific size, or
2. at those specific start/end addresses