mapping shared libraries at fixed address...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

mapping shared libraries at fixed address...

Francois-Rene Rideau
So as to be able to mmap BIG data files on i386, I would like to get
the shared libraries out of the way. With no userspace specification,
ld.so and libc.so are mmap()ed by the kernel at one third of the
address space (starting at 0x55555000), which means that I can't mmap
contiguously a file larger than 1.6GB approximately (less than I
need), at opposed to a maximum of 2.9GB or so if I could get the
libraries loaded near the bottom of the memory.

There are three problems:
(1) ld.so itself is defined as a shared library with no load address
specification, and the kernel loads it at TASK_UNMAPPED_BASE. I don't
understand how I need to edit the ld script so that it would do
otherwise -- my attempts so far get me segfaults and/or link errors.
(2) ld.so itself must load things at a lower address. I could somehow
try to define __elf_preferred_address for i386 or to intercept
__mmap() with code that tries to load from some address in low memory.
(3) the libc itself might use mmap(0,...) at which point I could have
it use a custom __mmap() defined with LD_PRELOAD. This would not be
required if I get a chance to mmap() my own stuff soon enough for the
address space to not be clobbered yet.

I'm at loss how to solve problem (1), and that's where I'm requesting
help. Between what the kernel wants, how to achieve it with ld, and
how to fulfill the runtime expectations of ld.so while doing it, I
don't know what to do.

For (2) I think I could implement __elf_preferred_address if I tried,
but I'm not sure it would spare me the need to intercept mmap, and if
not, then it's redundant anyway.

For (2) and/or (3), I think I understand how to intercept mmap(); I'm
concerned however about how to do it right. I must maintain a
userspace mirror of enough of the kernel vma structures so as to
emulate mmap with a smaller TASK_UNMAPPED_BASE. (a) this raises the
question of ensuring that all calls to mmap are properly intercepted
by the same copy of the code. (b) this also raises the question of
what libc calls I may use to initialize the map: typically, I suppose
I could try to read from /proc/self/maps though this requires at least
strtol to be present if not sscanf, and/or I could suppose that the
only thing mapped into memory initially is ld.so, at its known address
and size.

Ideally, I should be able to backport those changes to older version
of ld.so (2.2.5), but I could live with shipping around the set of a
binary and all its dependent dynamic libraries.

Workarounds:
* One "easy" workaround would be to recompile the kernel with a small
TASK_UNMAPPED_BASE, but I can't change the kernel on all the targetted
machines.
* Another workaround would be to compile my binaries statically, but
the whole environment crucially depends on dynamic libraries at the
time being, and it would be quite some engineering to make it not so.

PS: I suppose that such features could be useful to other people than
me, especially if no mmap() interception is required in ld.so -- at
which time it could usefully be made a runtime option controlled by
some LD_PREFERRED_ADDRESS environment variable or some such.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
When everything seems to be going against you, remember that the airplane takes
off against the wind, not with it. -- Henry Ford
Reply | Threaded
Open this post in threaded view
|

Re: mapping shared libraries at fixed address...

H.J. Lu-27
On Wed, Jan 18, 2006 at 12:21:47AM -0500, Far? wrote:
> So as to be able to mmap BIG data files on i386, I would like to get
> the shared libraries out of the way. With no userspace specification,
> ld.so and libc.so are mmap()ed by the kernel at one third of the
> address space (starting at 0x55555000), which means that I can't mmap
> contiguously a file larger than 1.6GB approximately (less than I

That is what x86-64 is for.


H.J.
Reply | Threaded
Open this post in threaded view
|

Re: mapping shared libraries at fixed address...

Francois-Rene Rideau
On 18/01/06, H. J. Lu <[hidden email]> wrote:
> On Wed, Jan 18, 2006 at 12:21:47AM -0500, Far? wrote:
> > So as to be able to mmap BIG data files on i386, I would like to get
> > the shared libraries out of the way. With no userspace specification,
> > ld.so and libc.so are mmap()ed by the kernel at one third of the
> > address space (starting at 0x55555000), which means that I can't mmap
> > contiguously a file larger than 1.6GB approximately (less than I
>
> That is what x86-64 is for.
>
Yes, but I'd like to buy our farm of thousands of i386 servers many a
month of life extension. The price difference between buying thousands
of amd64 machines now or then can be quite substantial.

It seems that loading libraries at a fixed address ought to be very
doable, and I can do the C coding part of it just fine. However, I
admit I'm baffled by the ld script hacking part of it. And if
possible, I'd like to do thing Right(tm) so as to be able to propose a
patch for official inclusion.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Guns & bullets don't kill people -- blood loss and organ damage kills people.
Reply | Threaded
Open this post in threaded view
|

Re: mapping shared libraries at fixed address...

Alexandre Julliard
In reply to this post by Francois-Rene Rideau
Faré <[hidden email]> writes:

> So as to be able to mmap BIG data files on i386, I would like to get
> the shared libraries out of the way. With no userspace specification,
> ld.so and libc.so are mmap()ed by the kernel at one third of the
> address space (starting at 0x55555000), which means that I can't mmap
> contiguously a file larger than 1.6GB approximately (less than I
> need), at opposed to a maximum of 2.9GB or so if I could get the
> libraries loaded near the bottom of the memory.

We had a similar problem in Wine, since we need to load Windows
binaries at fixed addresses. What we ended up doing is a preloader
that allows you to reserve a range of addresses before ld.so and libc
are mapped. You can find the code here:

  http://source.winehq.org/git/?p=wine.git;a=blob;hb=HEAD;f=loader/preloader.c

--
Alexandre Julliard
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: mapping shared libraries at fixed address...

Jakub Jelinek
In reply to this post by Francois-Rene Rideau
On Wed, Jan 18, 2006 at 12:21:47AM -0500, Far? wrote:
> So as to be able to mmap BIG data files on i386, I would like to get
> the shared libraries out of the way. With no userspace specification,
> ld.so and libc.so are mmap()ed by the kernel at one third of the
> address space (starting at 0x55555000), which means that I can't mmap
> contiguously a file larger than 1.6GB approximately (less than I
> need), at opposed to a maximum of 2.9GB or so if I could get the
> libraries loaded near the bottom of the memory.

With the flexible mmap memory layout in 2.6.9 and later this shouldn't be
an issue.

> There are three problems:
> (1) ld.so itself is defined as a shared library with no load address
> specification, and the kernel loads it at TASK_UNMAPPED_BASE. I don't
> understand how I need to edit the ld script so that it would do
> otherwise -- my attempts so far get me segfaults and/or link errors.

If for whatever reason 2.6.9+ doesn't DTRT for you, you can:
1) use prelink and tell it where to mmap the libraries (with --exec-shield
   option it will by default put libraries below binaries if possible)
2) if you don't want to have libraries prelinked, you can just use
   prelink -r 0x12345000 /lib/ld-2.3.*.so that will just relocate
   ld.so (and similarly for other libraries you need)

        Jakub
Reply | Threaded
Open this post in threaded view
|

Re: mapping shared libraries at fixed address...

Francois-Rene Rideau
Dear Jakub,

> With the flexible mmap memory layout in 2.6.9 and later this shouldn't be
> an issue.
It looks like an interesting approach, if I can convince the
administrators to upgrade kernel for all/most of my server farm.
However, half an hour of Googling around and peering at the source
doesn't find me some reliable documentation on how this feature is
meant to be used. I understand that I must somehow set the kernel
personality and exec, but I'm not quite sure how to do it. Can you
hint me with a clue stick?

> If for whatever reason 2.6.9+ doesn't DTRT for you, you can:
> 1) use prelink and tell it where to mmap the libraries (with --exec-shield
>    option it will by default put libraries below binaries if possible)
> 2) if you don't want to have libraries prelinked, you can just use
>    prelink -r 0x12345000 /lib/ld-2.3.*.so that will just relocate
>    ld.so (and similarly for other libraries you need)
Will try that too! But will that convince the libc to mmap its locale
data out of the way? Otherwise, I might still have to preload an
interceptor for mmap...

Before you told me about prelink, I managed to cheat the linker into
producing a ld.so that specifies a fixed load address (first, changing
the ld script to specify such an address, then changing ET_DYN==3 into
ET_EXEC==2 into offset 16 of the binary) -- that's evil but it's all I
had found to convince the kernel to load it at fixed address. That (or
preferrably prelink if it works) plus an interception of mmap (patch
to ld.so if no prelink on old kernels, LD_PRELOAD to use unchanged
libc) will do the trick if I can't DTRT with recent kernels most
everywhere. (Many thanks to Alexandre Julliard for the example of a
working preloader.)

Thanks a lot for your support.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Pacifism is a shifty doctrine under which a man accepts the benefits of the
social group without being willing to pay - and claims a halo for his
dishonesty. -- Robert Heinlein