Fwd: Preventing preemption of 'protected' symbols in GNU ld 2.26

classic Classic list List threaded Threaded
63 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Fwd: Preventing preemption of 'protected' symbols in GNU ld 2.26

Joe Groff
On Mar 29, 2016, at 8:44 AM, H.J. Lu <[hidden email]> wrote:

>
> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <[hidden email]> wrote:
>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>> new version of the library?
>>>>>
>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>> are simply allocated in bss in the executable.
>>>>
>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>
>>> No, references from within the .so would have always used the GOT.
>>> Non-protected global symbols in a shared library are still
>>> pre-emptible, so they are always indirect, and there's always a
>>> dynamic relocation for the GOT entry. Whether the prevailing
>>> definition winds up in the executable or the shared library, the
>>> dynamic loader still has to bind the symbol and apply the relocation.
>>
>> HJ's changes to protected visibility meant compiler changes so that
>> protected visibility in shared libraries is no longer seen as local.
>> So yes, protected visibility symbols in shared libraries now go
>> through the GOT.  Prior to his changes, they were optimized to a
>> pc-relative access.  Joe is correct in pointing out that shared
>> libraries needed a change.  Bad luck if you're using an older
>> compiler.  Also bad luck if you want to use protected visibility to
>> optimize your shared library.
>>
>> HJ also made glibc ld.so changes to ensure the semantics of protected
>> visibility symbols remain unchanged when multiple shared libraries
>> define the same protected visibility symbol.
>>
>> Apparently most people in the gcc and glibc communities saw these
>> toolchain modifications as fiendishly clever.
>>
>
> As I said before, copy relocation and protected symbol are fundamentally
> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
> GCC, glibc and ld to make protected symbol to work with copy relocation.
> That is protected symbol may be external, but won't be preempted.  The
> price I paid is that protected symbol won't be accessed via PC-relative
> relocation within the shared object.  To access protected symbol via
> PC-relative relocation within the shared object, we need to disable copy
> relocation in executable, which is a psABI change.  That is why I proposed
> to mark the object as such so that we won't get surprise at run-time.

I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.

-Joe

Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

H.J. Lu-30
On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <[hidden email]> wrote:

> On Mar 29, 2016, at 8:44 AM, H.J. Lu <[hidden email]> wrote:
>>
>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <[hidden email]> wrote:
>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>> new version of the library?
>>>>>>
>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>> are simply allocated in bss in the executable.
>>>>>
>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>
>>>> No, references from within the .so would have always used the GOT.
>>>> Non-protected global symbols in a shared library are still
>>>> pre-emptible, so they are always indirect, and there's always a
>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>> definition winds up in the executable or the shared library, the
>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>
>>> HJ's changes to protected visibility meant compiler changes so that
>>> protected visibility in shared libraries is no longer seen as local.
>>> So yes, protected visibility symbols in shared libraries now go
>>> through the GOT.  Prior to his changes, they were optimized to a
>>> pc-relative access.  Joe is correct in pointing out that shared
>>> libraries needed a change.  Bad luck if you're using an older
>>> compiler.  Also bad luck if you want to use protected visibility to
>>> optimize your shared library.
>>>
>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>> visibility symbols remain unchanged when multiple shared libraries
>>> define the same protected visibility symbol.
>>>
>>> Apparently most people in the gcc and glibc communities saw these
>>> toolchain modifications as fiendishly clever.
>>>
>>
>> As I said before, copy relocation and protected symbol are fundamentally
>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>> That is protected symbol may be external, but won't be preempted.  The
>> price I paid is that protected symbol won't be accessed via PC-relative
>> relocation within the shared object.  To access protected symbol via
>> PC-relative relocation within the shared object, we need to disable copy
>> relocation in executable, which is a psABI change.  That is why I proposed
>> to mark the object as such so that we won't get surprise at run-time.
>
> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.

That is what I meant by "That is why I proposed to mark the object as such so
that we won't get surprise at run-time."


--
H.J.
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Joe Groff

> On Mar 29, 2016, at 12:33 PM, H.J. Lu <[hidden email]> wrote:
>
> On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <[hidden email]> wrote:
>> On Mar 29, 2016, at 8:44 AM, H.J. Lu <[hidden email]> wrote:
>>>
>>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <[hidden email]> wrote:
>>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>>> new version of the library?
>>>>>>>
>>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>>> are simply allocated in bss in the executable.
>>>>>>
>>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>>
>>>>> No, references from within the .so would have always used the GOT.
>>>>> Non-protected global symbols in a shared library are still
>>>>> pre-emptible, so they are always indirect, and there's always a
>>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>>> definition winds up in the executable or the shared library, the
>>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>>
>>>> HJ's changes to protected visibility meant compiler changes so that
>>>> protected visibility in shared libraries is no longer seen as local.
>>>> So yes, protected visibility symbols in shared libraries now go
>>>> through the GOT.  Prior to his changes, they were optimized to a
>>>> pc-relative access.  Joe is correct in pointing out that shared
>>>> libraries needed a change.  Bad luck if you're using an older
>>>> compiler.  Also bad luck if you want to use protected visibility to
>>>> optimize your shared library.
>>>>
>>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>>> visibility symbols remain unchanged when multiple shared libraries
>>>> define the same protected visibility symbol.
>>>>
>>>> Apparently most people in the gcc and glibc communities saw these
>>>> toolchain modifications as fiendishly clever.
>>>>
>>>
>>> As I said before, copy relocation and protected symbol are fundamentally
>>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>>> That is protected symbol may be external, but won't be preempted.  The
>>> price I paid is that protected symbol won't be accessed via PC-relative
>>> relocation within the shared object.  To access protected symbol via
>>> PC-relative relocation within the shared object, we need to disable copy
>>> relocation in executable, which is a psABI change.  That is why I proposed
>>> to mark the object as such so that we won't get surprise at run-time.
>>
>> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.
>
> That is what I meant by "That is why I proposed to mark the object as such so
> that we won't get surprise at run-time."

I don't see why it would be a runtime failure. ld could refuse to resolve the relocation at static link time, couldn't it?

-Joe
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

H.J. Lu-30
On Tue, Mar 29, 2016 at 12:36 PM, Joe Groff <[hidden email]> wrote:

>
>> On Mar 29, 2016, at 12:33 PM, H.J. Lu <[hidden email]> wrote:
>>
>> On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <[hidden email]> wrote:
>>> On Mar 29, 2016, at 8:44 AM, H.J. Lu <[hidden email]> wrote:
>>>>
>>>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <[hidden email]> wrote:
>>>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>>>> new version of the library?
>>>>>>>>
>>>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>>>> are simply allocated in bss in the executable.
>>>>>>>
>>>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>>>
>>>>>> No, references from within the .so would have always used the GOT.
>>>>>> Non-protected global symbols in a shared library are still
>>>>>> pre-emptible, so they are always indirect, and there's always a
>>>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>>>> definition winds up in the executable or the shared library, the
>>>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>>>
>>>>> HJ's changes to protected visibility meant compiler changes so that
>>>>> protected visibility in shared libraries is no longer seen as local.
>>>>> So yes, protected visibility symbols in shared libraries now go
>>>>> through the GOT.  Prior to his changes, they were optimized to a
>>>>> pc-relative access.  Joe is correct in pointing out that shared
>>>>> libraries needed a change.  Bad luck if you're using an older
>>>>> compiler.  Also bad luck if you want to use protected visibility to
>>>>> optimize your shared library.
>>>>>
>>>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>>>> visibility symbols remain unchanged when multiple shared libraries
>>>>> define the same protected visibility symbol.
>>>>>
>>>>> Apparently most people in the gcc and glibc communities saw these
>>>>> toolchain modifications as fiendishly clever.
>>>>>
>>>>
>>>> As I said before, copy relocation and protected symbol are fundamentally
>>>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>>>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>>>> That is protected symbol may be external, but won't be preempted.  The
>>>> price I paid is that protected symbol won't be accessed via PC-relative
>>>> relocation within the shared object.  To access protected symbol via
>>>> PC-relative relocation within the shared object, we need to disable copy
>>>> relocation in executable, which is a psABI change.  That is why I proposed
>>>> to mark the object as such so that we won't get surprise at run-time.
>>>
>>> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.
>>
>> That is what I meant by "That is why I proposed to mark the object as such so
>> that we won't get surprise at run-time."
>
> I don't see why it would be a runtime failure. ld could refuse to resolve the relocation at static link time, couldn't it?
>

The link-time shared library may be very different from the run-time
shared library.  Since copy relocation works correctly against protected
symbol with the current gcc, binutils and glibc, we can't simply disallow
copy relocation against protected symbol at link-time.  It should be done
with a link command-line as well as markers on the input files.


--
H.J.
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Joe Groff

> On Mar 29, 2016, at 12:43 PM, H.J. Lu <[hidden email]> wrote:
>
> The link-time shared library may be very different from the run-time
> shared library.  Since copy relocation works correctly against protected
> symbol with the current gcc, binutils and glibc, we can't simply disallow
> copy relocation against protected symbol at link-time.

However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.

-Joe

Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

H.J. Lu-30
On Tue, Mar 29, 2016 at 12:51 PM, Joe Groff <[hidden email]> wrote:

>
>> On Mar 29, 2016, at 12:43 PM, H.J. Lu <[hidden email]> wrote:
>>
>> The link-time shared library may be very different from the run-time
>> shared library.  Since copy relocation works correctly against protected
>> symbol with the current gcc, binutils and glibc, we can't simply disallow
>> copy relocation against protected symbol at link-time.
>
> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>

Protected symbol never worked correctly on x86 before.  My
change closed a few long-standing bugs.  There is no going-back.

--
H.J.
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

H.J. Lu-30
On Tue, Mar 29, 2016 at 12:54 PM, H.J. Lu <[hidden email]> wrote:

> On Tue, Mar 29, 2016 at 12:51 PM, Joe Groff <[hidden email]> wrote:
>>
>>> On Mar 29, 2016, at 12:43 PM, H.J. Lu <[hidden email]> wrote:
>>>
>>> The link-time shared library may be very different from the run-time
>>> shared library.  Since copy relocation works correctly against protected
>>> symbol with the current gcc, binutils and glibc, we can't simply disallow
>>> copy relocation against protected symbol at link-time.
>>
>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>
>
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

If you want to avoid copy relocation, you can do

1. Pass -z noextern-protected-data to build the shared object.
2. Use GOT to access external data in executable.


H.J.
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Alan Modra-3
In reply to this post by H.J. Lu-30
On Tue, Mar 29, 2016 at 12:54:36PM -0700, H.J. Lu wrote:
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

The swift bug report and this thread should be telling you that
protected visibility symbols don't work correctly *now* on x86.

They did in fact work correctly before your changes, but had many
annoying restrictions.  Those restrictions, all due to non-PIC, can be
overcome in a number of ways without breaking shared libraries.
eg. ppc32 is non-PIC by default just like x64, but see bfd/elf32-ppc.c
          /* Convert lis;addi or lis;load/store accessing a protected
             variable defined in a shared library to PIC.  */
Yes, this makes ppc32 executables slower.  I'd rather that than making
shared libraries slower.

FWIW, I agree with all the points Cary makes in
https://sourceware.org/ml/binutils/2016-03/msg00331.html.

--
Alan Modra
Australia Development Lab, IBM
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Cary Coutant-3
In reply to this post by H.J. Lu-30
>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>
>
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

You keep countering my arguments with assertions like, "it was a bug
and I fixed it," but you present no arguments of your own to support
your position. I'm not sure what long-standing bugs you're referring
to -- the only one I can find, PR target/65248 [1], was filed by you
yourself, so you can't really use that as support. In fact, PR
ld/15228 [2], was filed against ld for *not* refusing to make a COPY
relocation to a protected symbol, and Alan fixed that. Gold has the
same bug, and I intend to fix it there, too.

You have effectively made protected symbols "work" by disabling the
very thing they were designed to do. I don't know how to point out the
absurdity any more clearly.

<reductio ad absurdum>
By your logic, you should make GCC generate PIC code by default and
turn -fno-pic into a no-op, because non-PIC code doesn't "work" in a
shared library. (Ironically, that would actually make protected
symbols work again without your changes.) Over the years, I've heard
many complaints from developers that they couldn't put code into a
shared library without compiling it with a special option. When you
first heard that complaint, why wasn't your response to change the
compiler? Isn't that the only way to make shared libraries "work"?
</reductio ad absurdum>

-cary

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65248

[2] https://sourceware.org/bugzilla/show_bug.cgi?id=15228
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Jeff Law
On 03/29/2016 07:46 PM, Cary Coutant wrote:

>>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>>
>>
>> Protected symbol never worked correctly on x86 before.  My
>> change closed a few long-standing bugs.  There is no going-back.
>
> You keep countering my arguments with assertions like, "it was a bug
> and I fixed it," but you present no arguments of your own to support
> your position. I'm not sure what long-standing bugs you're referring
> to -- the only one I can find, PR target/65248 [1], was filed by you
> yourself, so you can't really use that as support. In fact, PR
> ld/15228 [2], was filed against ld for *not* refusing to make a COPY
> relocation to a protected symbol, and Alan fixed that. Gold has the
> same bug, and I intend to fix it there, too.
And FWIW, there are some folks on the GCC side of things that think that
HJ's change for 65248 is broken and needs to be reverted before gcc-6
releases.

I'm not familiar enough with all the issues, but I am familiar enough
with the work of HJ, Alan and yourself that if you & Alan say HJ's GCC
change is wrong, then, well, it's wrong and needs to be reverted.

It would help me immensely on the GCC side if things if you and Alan
could easily summarize correct behavior and the impact if we were to
just revert HJ's change.  A testcase would be amazingly helpful too.

I'm sure I could extract the relevant info out of the thread, but I'm
just buried right now.

jeff
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Cary Coutant-3
> And FWIW, there are some folks on the GCC side of things that think that
> HJ's change for 65248 is broken and needs to be reverted before gcc-6
> releases.

Thanks; I was thinking the next step was to move this discussion over
to the GCC side to see if anyone there was in agreement.

> It would help me immensely on the GCC side if things if you and Alan could
> easily summarize correct behavior and the impact if we were to just revert
> HJ's change.  A testcase would be amazingly helpful too.

On the compiler side, the main thing to check is that any reference to
a protected data symbol, whether extern, common, weak, or initialized,
is accessed with a pc-relative or GOT-relative addressing mode (i.e.,
not indirect through a GOT entry). This is the fundamental reason to
declare a data symbol protected.

The testcases in gcc.target/i386/pr65248-[1234].c look suitable to me,
if we simply switch "scan-assembler-not" with "scan-assembler"
throughout. These basically take a source like this:

__attribute__((visibility("protected")))
int xxx;

int
foo ()
{
  return xxx;
}

and compile with -fpic. Prior to GCC 5, we got

   movl    xxx(%rip), %eax

but with GCC 5, we now get

   movq    xxx@GOTPCREL(%rip), %rax
   movl    (%rax), %eax

There was another change to GCC at r218397, in Dec. 2014, to make data
access in PIE mode work more like non-PIC mode. This required linker
changes to enable the use of COPY relocations when generating PIE
output. This patch was originally developed by Sriraman Tallam on the
Google branch, but was checked on trunk in by HJ after considerable
discussion. As far as I'm concerned (and I advised Sri during the
development of this patch), this change was fine -- it simply
recognizes that a position-independent executable does not need to use
PIC-style references to data when a pc-relative or got-relative access
is possible. As in non-PIC mode, these accesses work fine for data
defined in the main program, but result in occasional COPY relocations
if the data happens to be defined in a shared library. Unfortunately,
I think this was the patch that set the rest in motion -- when
developing test cases, HJ noted that COPY relocations to protected
symbols were not allowed, and set out to fix that.

Ultimately, I'd like to see GCC support the use of declspec(dllimport)
or __attribute__((dllimport)) to declare a global variable known to be
defined externally in a shared library. It should then generate a
PIC-style reference even in non-PIC mode. This could be used to
eliminate all COPY relocations.

On the linker side, the linkers need to check that a reference from a
main program to a protected symbol will result in neither a text
relocation nor a COPY relocation, but instead issue an error message
explaining fairly clearly that a protected symbol in a shared library
cannot be referenced from non-PIC code. See PR ld/15228 and PR
gold/19823 (the latter being the bug report that started this current
thread).

-cary
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Cary Coutant-3
> See PR ld/15228 and PR
> gold/19823 (the latter being the bug report that started this current
> thread).

Sorry, it was actually Joe's note about Swift that started this
thread, but that came in right after the gold PR, so I was already
primed.

-cary
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Alan Modra-3
In reply to this post by Jeff Law
OK, I'll try to list the technical issues re. pr65248 "Copy relocation
against protected symbol doesn't work".

Given a shared library that defines a variable, and a non-PIC
executable that references that variable, the linker makes a duplicate
of the variable in the executable .dynbss section and arranges to have
the copy initialized by the dynamic loader with a copy relocation.
.dynbss is a linker created section that becomes part of the
executable bss segment.  The idea is that at run-time both the
executable and the shared library will use the executable copy of the
variable.  It's a rather ancient linker hack to avoid dynamic text
relocations, invented well before symbol visibility.

The problem with protected visibility variables in shared libraries is
that the shared library continues to access its own copy.  So we have
two copies of a variable where the source only contained one variable
definition.  (It's not really the copy relocation as such that causes
a problem, but the fact that we have a duplicate in .dynbss.)

Solutions are:

1) Have the linker emit an error, suggesting that the executable needs
to be PIC to access a protected visibility variable in a shared
library.  Pass the pain on to users.

2) Emit dynamic text relocations for non-PIC executable access to
protected visibility variables defined in shared libraries.  Makes
executable pages with text relocations non-shared between processes.
(But can be shared again after relocation if two processes have the
same layout.)

3) Teach the linker to edit non-PIC to PIC.  I've done this for ppc32,
but it's not very easy for x86 due to the fact that you can't find the
start of an instruction from a relocated field.  It also results in
slow code, since PIC sequences are usually larger than non-PIC,
requiring a branch into a patch area and a branch back.

4) Have the compiler always emit PIC for external variable access.
The .dynbss hack could then die a natural death.  Coupled with linker
editing of PIC to non-PIC, you'd get correct and reasonably fast code
with perhaps a few nops.  Again, x86 has difficulty due to variable
length insns, but this could be solved with marker relocs.

5) HJ's solution.  Make the compiler emit the same code for protected
visibility variables in shared libraries as it does for default
visibility variables.  ie. shared libraries access their protected
visibility variables via the GOT, which allows the .dynbss hack to
work.  Also, modify the dynamic loader to ensure protected visibility
semantics are enforced.  (Consider the case of shared lib A that
defines protected visibility var V, and shared lib B that also defines
protected visibility var V, and an executable that references V.
Assuming A is before B in breadth-first order, then the executable and
A should use the .dynbss copy of V, but B should use its own V!)  I've
got to hand it to HJ, I think he has all this working properly to
satisfy the semantics of protected visibility.

However, shared library access of protected visibility variables is of
course no quicker than default visibility variables.  Which defeats
the main reason to use protected visibility variables in shared
libraries!  Note also that it unnecessarily slows down shared library
code using protected visibility variables that are not accessed by the
executable (but perhaps accessed from some other shared library).

Also, I don't believe there is anything to prevent a newly compiled
program from running with an old glibc, which would mean protected
visibility semantics are broken on such a system.  To be fair, they
were probably broken on such a system before the linker complained
about non-PIC protected visibility access, anyway.

--
Alan Modra
Australia Development Lab, IBM
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Cary Coutant-3
In reply to this post by Jeff Law
> It would help me immensely on the GCC side if things if you and Alan could
> easily summarize correct behavior and the impact if we were to just revert
> HJ's change.  A testcase would be amazingly helpful too.

It looks like it's not just the one change. There's this patch:

   https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html

which took the idea that protected can still be pre-empted by a COPY
relocation and extended it to three more targets that use COPY
relocations.

I wonder how many other patches have been based on the same misunderstanding?

-cary
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Cary Coutant-3
In reply to this post by Alan Modra-3
> 2) Emit dynamic text relocations for non-PIC executable access to
> protected visibility variables defined in shared libraries.  Makes
> executable pages with text relocations non-shared between processes.
> (But can be shared again after relocation if two processes have the
> same layout.)

Note that this is not always possible, depending on the addressing
range of the code emitted. For example, a pc-relative access may have
sufficient range to reach a variable copied into the executable's bss,
but not sufficient range to reach the original definition in a shared
library.

-cary
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Jeff Law
In reply to this post by Cary Coutant-3
On 03/30/2016 06:40 PM, Cary Coutant wrote:

>> It would help me immensely on the GCC side if things if you and Alan could
>> easily summarize correct behavior and the impact if we were to just revert
>> HJ's change.  A testcase would be amazingly helpful too.
>
> It looks like it's not just the one change. There's this patch:
>
>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>
> which took the idea that protected can still be pre-empted by a COPY
> relocation and extended it to three more targets that use COPY
> relocations.
>
> I wonder how many other patches have been based on the same misunderstanding?
I don't think it was many -- I certainly recall the arm/aarch64 variant.
  There may have been one other varasm.c change in this space or I might
be conflating it with the arm/aarch64 change.  Tracking those down is
naturally part of this work.

jeff
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Ramana Radhakrishnan-7
On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <[hidden email]> wrote:

> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>
>>> It would help me immensely on the GCC side if things if you and Alan
>>> could
>>> easily summarize correct behavior and the impact if we were to just
>>> revert
>>> HJ's change.  A testcase would be amazingly helpful too.
>>
>>
>> It looks like it's not just the one change. There's this patch:
>>
>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>
>> which took the idea that protected can still be pre-empted by a COPY
>> relocation and extended it to three more targets that use COPY
>> relocations.
>>
>> I wonder how many other patches have been based on the same
>> misunderstanding?
>
> I don't think it was many -- I certainly recall the arm/aarch64 variant.
> There may have been one other varasm.c change in this space or I might be
> conflating it with the arm/aarch64 change.  Tracking those down is naturally
> part of this work.

The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
the same time.  IIUC, the reason the patch above went in were to fix
failures on arm / aarch64 with those tests. I haven't yet worked out
whether all this is the same issue.

CC'ing Szabolcs.

Thanks,
Ramana

>
> jeff
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

H.J. Lu-30
On Thu, Mar 31, 2016 at 6:26 AM, Ramana Radhakrishnan
<[hidden email]> wrote:

> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <[hidden email]> wrote:
>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>
>>>> It would help me immensely on the GCC side if things if you and Alan
>>>> could
>>>> easily summarize correct behavior and the impact if we were to just
>>>> revert
>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>
>>>
>>> It looks like it's not just the one change. There's this patch:
>>>
>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>
>>> which took the idea that protected can still be pre-empted by a COPY
>>> relocation and extended it to three more targets that use COPY
>>> relocations.
>>>
>>> I wonder how many other patches have been based on the same
>>> misunderstanding?
>>
>> I don't think it was many -- I certainly recall the arm/aarch64 variant.
>> There may have been one other varasm.c change in this space or I might be
>> conflating it with the arm/aarch64 change.  Tracking those down is naturally
>> part of this work.
>
> The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
> the same time.  IIUC, the reason the patch above went in were to fix
> failures on arm / aarch64 with those tests. I haven't yet worked out
> whether all this is the same issue.
>

These testcases show the problem between copy relocation and
protected data symbol.  I updated my program property extension
to GNU ABI to prevent copy relocation against protected data symbol:

https://sourceware.org/ml/gnu-gabi/2016-q1/msg00041.html

--
H.J.
Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Jeff Law
In reply to this post by Cary Coutant-3
On 03/30/2016 06:40 PM, Cary Coutant wrote:

>> It would help me immensely on the GCC side if things if you and Alan could
>> easily summarize correct behavior and the impact if we were to just revert
>> HJ's change.  A testcase would be amazingly helpful too.
>
> It looks like it's not just the one change. There's this patch:
>
>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>
> which took the idea that protected can still be pre-empted by a COPY
> relocation and extended it to three more targets that use COPY
> relocations.
>
> I wonder how many other patches have been based on the same misunderstanding?
The patches around BZ65780 may be tangled in this mess as well.    And
it bled into the s390 & darwin ports as well.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65780

Alan, could you take a peek at 65780 -- you're better versed than I in
this stuff.  Essentially the question that needs to be answered is if we
roll back 65280, do we need to also roll back 65780?

jeff


Reply | Threaded
Open this post in threaded view
|

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

Alan Modra-3
On Fri, Apr 01, 2016 at 01:51:33PM -0600, Jeff Law wrote:
> Essentially the question that needs to be answered is if we roll
> back 65280, do we need to also roll back 65780?

No, I don't think so.  As I wrote in the bugzilla (paraphrased)

What's going on here is that HJ has made changes to the linker for x86
to always copy variables defined in a shared library and referenced by
the executable, into the executable's .dynbss.  This is necessary to
support non-PIC without text relocations, but is optional for PIEs.

If you do this for PIEs and gain some performance without affecting
shared library performance, and that is the case for x86_64, then the
optimisation is quite reasonable.  I think powerpc64 would gain too,
but I haven't done anything about it yet (and for powerpc64 we should
be able to implement it entirely in the linker, no gcc changes
needed).

I believe common variables can be treated like any other variable so
far as this optimisation goes.

--
Alan Modra
Australia Development Lab, IBM
1234