1) RLIMIT_DATA rlim_cur soft limit of 2 pages is set. ld maps this program on
a page boundary and gives us one page (out of the two) to work with so we brk
at the start_brk + 1 page. We are allowed to read/write memory up to the brk
set by brk() without mallocing. Subsequent mallocs grab memory further up the
address space, but not directly following curbrk.
2) We attempt to read/write to start_brk + 1 page + 4 bytes which exceeds cur
brk and creates a segmentation fault.
3) Setting RLIMIT_AS rlim_cur soft limit to RLIMIT_DATA rlim_cur soft limit
limits the address space memory that can be consumed by the program.
Effectively this stops the mallocs from succeeding after we brk up to the
RLIMIT_DATA soft limit. Malloc returns NULL.
1.) mmap isn't governed by RLIMIT_DATA and malloc may resort to mmap when
sbrk() fails. Setting the RLIMIT_AS soft-limit to RLIMIT_DATA soft-limit will
prevent malloc from exceeding the RLIMIT_DATA soft-limit. I haven't tried
setting RLIMIT_AS soft-limit below RLIMIT_DATA soft-limit but I
1a.) Setting RLIMIT_AS below RLIMIT_DATA doesn't seem to prevent malloc from
using all the memory in the data segment up the RLIMIT_DATA because malloc
probably uses sbrk() until it returns 0.
2.) The kernel allocates memory on page boundaries so all soft-limits are only
enforced at the upward page boundary.
3.) I never experienced a seg fault with malloc() when mallocing data beyond
RLIMIT_DATA rlim_cur soft-limit.
After further analysis I still think this is working as designed.