SHF_LINK_ORDER's original semantics make upgrade difficult

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

SHF_LINK_ORDER's original semantics make upgrade difficult

Sourceware - binutils list mailing list
For your convenience, the agreed semantics of SHF_LINK_ORDER is attached:

https://groups.google.com/d/msg/generic-abi/_CbBM6T6WeM/eGF9A0AnAAAJ

     SHF_LINK_ORDER

### Original semantics
        This flag adds special ordering requirements for link editors. The
        requirements apply to the referenced section identified by the sh_link
        field of this section's header.  If this section is combined with other
        sections in the output file, the section must appear in the same
        relative order with respect to those sections, as the referenced
        section appears with respect to sections the referenced section is
        combined with.

        A typical use of this flag is to build a table that references
        text or data sections in address order.

### Extended semantics for metadata sections
        In addition to adding ordering requirements, SHF_LINK_ORDER indicates
        that the section contains metadata describing the referenced section.
        When performing unused section elimination, the link editor should
        ensure that both the section and the referenced section are retained
        or discarded together. Furthermore, relocations from this section
        into the referenced section should not be taken as evidence that the
        referenced section should be retained.

Say we have a monolithic metadata section (.foo) associated to a text section.
The metadata section is only referenced by the associated text section.  If we
want to "upgrade" the metadata section to SHF_LINK_ORDER in the future (to get
--gc-sections functionality with fragmented .foo sections), unfortunately the
original semantics make the design space narrow.

For compatibility reasons, we still have some object files with
non-SHF_LINK_ORDER monolithic .foo .  New .foo have the SHF_LINK_ORDER flag.
However, mixed SHF_LINK_ORDER and non-SHF_LINK_ORDER sections have an issue with
the linked-to order requirement (Original semantics). In practice, linkers
forbid a mix.

GNU ld: .gcc_except_table has both ordered [.foo' in a.o] and unordered [.foo' in b.o] sections
LLD is more relaxed, but it does not allow non-contiguous SHF_LINK_ORDER sections: a.o:(.foo): SHF_LINK_ORDER sections in .foo are not contiguous
   (https://reviews.llvm.org/D77007 )

In many cases, .foo doesn't really need original semantics and .foo can be
combined in an arbitrary order. What can we do to make such "upgrade" feasible?
The current design simply requires such metadata section to be designed with
SHF_LINK_ORDER in mind in the very beginning.
Reply | Threaded
Open this post in threaded view
|

Fwd: SHF_LINK_ORDER's original semantics make upgrade difficult

Sourceware - binutils list mailing list
Cross-posting as my original attempt only went to the gabi list.

---------- Forwarded message ---------
From: James Henderson <[hidden email]>
Date: Thu, 16 Jul 2020 at 08:27
Subject: Re: SHF_LINK_ORDER's original semantics make upgrade difficult
To: Generic System V Application Binary Interface <
[hidden email]>


I've run into this same problem whilst doing some prototyping work on
fragmenting DWARF sections into smaller pieces. These pieces are a mixture
of common pieces and function/variable-related pieces, with the latter
being associated with the corresponding text/data sections via
SHF_LINK_ORDER. However, in some cases, the section is laid out such that
there are multiple common pieces intermixed with the
function/variable-specific pieces (e.g. it might look like "common,
function, function, common, function, variable, common"). I actually don't
want this section to be ordered at all, since the original ordering should
be preserved.I've been using a hacked LLD for this, which removes the
ordering for SHF_LINK_ORDER completely, so that the sections remain in the
"natural" order, whilst the associated-with semantics for --gc-sections are
still preserved.

Whilst doing this, I couldn't help but feel that the associated-with and
ordering semantics are somewhat orthoganol. Clearly if a section is
ordered, it needs the associated-with semantics, but it seems like section
association is a different thing. There are multiple sections now where the
ordering is irrelevant, but which still wants association in some way.
Examples include LLVM's stack sizes section and debug data.The original
SHF_LINK_ORDER extension discussion actually started out with discussing a
SHF_ASSOCIATED flag. Maybe we should revisit that idea in some form? Thus,
SHF_ASSOCIATED implies the section should be discarded when it's linked
section is discarded, and SHF_LINK_ORDER requires ordering. I think both
for backwards-compatibility's sake and to avoid redundancy, we could say
SHF_LINK_ORDER implies SHF_ASSOCIATED.

--
You received this message because you are subscribed to the Google Groups
"Generic System V Application Binary Interface" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email].
To view this discussion on the web visit
https://groups.google.com/d/msgid/generic-abi/14c7f411-c727-4f92-a5b5-ca3adc543c22n%40googlegroups.com
<https://groups.google.com/d/msgid/generic-abi/14c7f411-c727-4f92-a5b5-ca3adc543c22n%40googlegroups.com?utm_medium=email&utm_source=footer>
.
Reply | Threaded
Open this post in threaded view
|

Re: SHF_LINK_ORDER's original semantics make upgrade difficult

Sourceware - binutils list mailing list
In reply to this post by Sourceware - binutils list mailing list
> GNU ld: .gcc_except_table has both ordered [.foo' in a.o] and unordered [.foo' in b.o] sections

Correction: .foo has both ordered [`.foo' in a.o] and unordered [`.foo' in b.o] sections

(I was experimenting with SHF_LINK_ORDER .gcc_except_table . For this
discussion I just used a metasyntactic section name .foo but forgot to
replace all occurrences of .gcc_except_table)

On 2020-07-16, James Henderson wrote:

> I've run into this same problem whilst doing some prototyping work on
>fragmenting DWARF sections into smaller pieces. These pieces are a mixture
>of common pieces and function/variable-related pieces, with the latter
>being associated with the corresponding text/data sections via
>SHF_LINK_ORDER. However, in some cases, the section is laid out such that
>there are multiple common pieces intermixed with the
>function/variable-specific pieces (e.g. it might look like "common,
>function, function, common, function, variable, common"). I actually don't
>want this section to be ordered at all, since the original ordering should
>be preserved.I've been using a hacked LLD for this, which removes the
>ordering for SHF_LINK_ORDER completely, so that the sections remain in the
>"natural" order, whilst the associated-with semantics for --gc-sections are
>still preserved.
>
>Whilst doing this, I couldn't help but feel that the associated-with and
>ordering semantics are somewhat orthoganol. Clearly if a section is
>ordered, it needs the associated-with semantics, but it seems like section
>association is a different thing. There are multiple sections now where the
>ordering is irrelevant, but which still wants association in some way.
>Examples include LLVM's stack sizes section and debug data.The original
>SHF_LINK_ORDER extension discussion actually started out with discussing a
>SHF_ASSOCIATED flag. Maybe we should revisit that idea in some form? Thus,
>SHF_ASSOCIATED implies the section should be discarded when it's linked
>section is discarded, and SHF_LINK_ORDER requires ordering. I think both
>for backwards-compatibility's sake and to avoid redundancy, we could say
>SHF_LINK_ORDER implies SHF_ASSOCIATED.

I am in favor of a new section flag SHF_ASSOCIATED as well. We can use
the next bit:

#define SHF_ASSOCIATED 0x1000

In the assembler, assign it a .section flag 'm', e.g.

     .section __patchable_function_entries,"am",@progbits,foo
     # Create a __patchable_function_entries section with the SHF_ASSOCIATED flag
     # and sh_link referencing the section defining 'foo'

Unfortunately we will have a binutils release (2.35) with SHF_LINK_ORDER
syntax (.section flag 'o')
https://sourceware.org/bugzilla/show_bug.cgi?id=24526
Unless we could be really quick and fix it, we might have to let SHF_LINK_ORDER
imply SHF_ASSOCIATED permanently.
(On the LLVM side, I think we could keep LLD's SHF_LINK_ORDER implying SHF_ASSOCIATED
for one or two releases and then drop that)


SHF_ASSOCIATED

     SHF_ASSOCIATED indicates that the section contains metadata describing
     the referenced section.  When performing unused section elimination, the
     link editor should ensure that both the section and the referenced
     section are retained or discarded together. Furthermore, relocations
     from this section into the referenced section should not be taken as
     evidence that the referenced section should be retained.
Reply | Threaded
Open this post in threaded view
|

Re: SHF_LINK_ORDER's original semantics make upgrade difficult

Sourceware - binutils list mailing list
In reply to this post by Sourceware - binutils list mailing list
On 2020-07-16, Ali Bahrami wrote:

>On 7/16/20 1:27 AM, James Henderson wrote:
>>I've run into this same problem whilst doing some prototyping work on fragmenting DWARF sections into smaller pieces. These pieces are a mixture of common pieces and function/variable-related pieces, with the latter being associated with the corresponding text/data sections via SHF_LINK_ORDER. However, in some cases, the section is laid out such that there are multiple common pieces intermixed with the function/variable-specific pieces (e.g. it might look like "common, function, function, common, function, variable, common"). I actually don't want this section to be ordered at all, since the original ordering should be preserved.I've been using a hacked LLD for this, which removes the ordering for SHF_LINK_ORDER completely, so that the sections remain in the "natural" order, whilst the associated-with semantics for --gc-sections are still preserved.
>>
>>Whilst doing this, I couldn't help but feel that the associated-with and ordering semantics are somewhat orthoganol. Clearly if a section is ordered, it needs the associated-with semantics, but it seems like section association is a different thing. There are multiple sections now where the ordering is irrelevant, but which still wants association in some way. Examples include LLVM's stack sizes section and debug data.The original SHF_LINK_ORDER extension discussion actually started out with discussing a SHF_ASSOCIATED flag. Maybe we should revisit that idea in some form? Thus, SHF_ASSOCIATED implies the section should be discarded when it's linked section is discarded, and SHF_LINK_ORDER requires ordering. I think both for backwards-compatibility's sake and to avoid redundancy, we could say SHF_LINK_ORDER implies SHF_ASSOCIATED.
>>
>
>
>
>   I guess I don't understand why SHT_GROUP sections aren't the general
>solution to this scenario, and any others where sections need to be
>associated.
>
>    https://docs.oracle.com/cd/E37838_01/html/E36783/chapter7-26.html
>
>In your scenario, I think you'd set the flags field to 0, rather
>than GRP_COMDAT, and then you'd have an associated set of sections
>that are kept, or discarded, as a unit. And, GROUP conveniently go
>away as part of a "final" link that produces an executable or shared
>object, as you propose above.

If I understand correctly, linkers support GRP_COMDAT but not other values. So
you are right that we can make use of other values if we want to do some fancy
things.

In practice, I guess one reason people want to mess with a section flag is that
adding a .group has non-trivial cost (sizeof(Elf64_Shdr)=64) if an object file
may have many of them (especially when -ffunction-sections is enabled).

For example, I added -fpatchable-function-entry=N[,M] in clang for the Linux
kernel. Every .text section has an accompanying __patchable_function_entries
with SHF_LINK_ORDER set. If I want to express this with a section group, I need
to add a .group (SHT_GROUP) for each .text, the cost will go up from double to
triple.

>>On 7/16/20 1:27 AM, James Henderson wrote:
>> Whilst doing this, I couldn't help but feel that the associated-with and ordering semantics are somewhat orthoganol.

Yeah, I am a big fan of non-orthogonality.. However, a gABI section flag seems a
no-go now.

>If the GNU linkers were modified to allow both ordered and
>non-ordered sections, then I think you could use GROUP for
>associating, and SHF_LINK_ORDER for actual ordering, where
>that is desired. We probably don't need a third mechanism,
>do we?

This solution is acceptable. I will do this for LLD
(https://reviews.llvm.org/D72904 ). We need an assembler syntax for
(SHF_LINK_ORDER & sh_link=0).

I created  https://sourceware.org/bugzilla/show_bug.cgi?id=26253 for a GNU as feature request.
Peter Collingbourne proposed

   .section .meta,"ao",@progbits,0

I will expect that GNU as and LLVM integrated assembler match
(probably https://reviews.llvm.org/D72899#2157020 )
Reply | Threaded
Open this post in threaded view
|

Re: SHF_LINK_ORDER's original semantics make upgrade difficult

Sourceware - binutils list mailing list
On 2020-07-16, Fangrui Song wrote:

>If I understand correctly, linkers support GRP_COMDAT but not other values. So
>you are right that we can make use of other values if we want to do some fancy
>things.
>
>In practice, I guess one reason people want to mess with a section flag is that
>adding a .group has non-trivial cost (sizeof(Elf64_Shdr)=64) if an object file
>may have many of them (especially when -ffunction-sections is enabled).
>
>For example, I added -fpatchable-function-entry=N[,M] in clang for the Linux
>kernel. Every .text section has an accompanying __patchable_function_entries
>with SHF_LINK_ORDER set. If I want to express this with a section group, I need
>to add a .group (SHT_GROUP) for each .text, the cost will go up from double to
>triple.
>
>>>On 7/16/20 1:27 AM, James Henderson wrote:
>>>Whilst doing this, I couldn't help but feel that the
>>>associated-with and ordering semantics are somewhat orthoganol.
>
>Yeah, I am a big fan of non-orthogonality.. However, a gABI section flag seems a
>no-go now.
>
>>If the GNU linkers were modified to allow both ordered and
>>non-ordered sections, then I think you could use GROUP for
>>associating, and SHF_LINK_ORDER for actual ordering, where
>>that is desired. We probably don't need a third mechanism,
>>do we?
>
>This solution is acceptable. I will do this for LLD
>(https://reviews.llvm.org/D72904 ). We need an assembler syntax for
>(SHF_LINK_ORDER & sh_link=0).
>
>I created  https://sourceware.org/bugzilla/show_bug.cgi?id=26253 for a GNU as feature request.
>Peter Collingbourne proposed
>
>  .section .meta,"ao",@progbits,0
>
>I will expect that GNU as and LLVM integrated assembler match
>(probably https://reviews.llvm.org/D72899#2157020 )

Forgot to mention that there are two feature requests for ld and one for as.

* ld: A SHF_LINK_ORDER section with sh_link=0 is treated like a non-SHF_LINK_ORDER section.
   (sorry to send llvm links here: https://reviews.llvm.org/D72904
   but I hope the interested can get a bit more context from the links)

* ld: Arbitrary mix of SHF_LINK_ORDER and non-SHF_LINK_ORDER components
   within an output section is allowed.
   (https://reviews.llvm.org/D84001 )

   I just saw https://sourceware.org/bugzilla/show_bug.cgi?id=16833 after
   I had filed https://sourceware.org/bugzilla/show_bug.cgi?id=26256

   There is actually an interesting question about how we perform sorting
   with input section descriptions. My feeling is that sorting should be
   scoped within an input section description. See the links for more information.

* as: Support SHF_LINK_ORDER with sh_link=0
   https://sourceware.org/bugzilla/show_bug.cgi?id=26253
   (https://reviews.llvm.org/D72899 )

   There is actually an LLVM IR design issue requiring sh_link=0.
   Supporting this will help GNU ld and gold (if gold implements
   SHF_LINK_ORDER) with clang -flto (which requires LLVMgold.so )
Reply | Threaded
Open this post in threaded view
|

Re: SHF_LINK_ORDER's original semantics make upgrade difficult

Sourceware - binutils list mailing list
In reply to this post by Sourceware - binutils list mailing list
On Fri, 17 Jul 2020 at 00:32, Fangrui Song <[hidden email]> wrote:

> On 2020-07-16, Ali Bahrami wrote:
> >On 7/16/20 1:27 AM, James Henderson wrote:
> >>I've run into this same problem whilst doing some prototyping work on
> fragmenting DWARF sections into smaller pieces. These pieces are a mixture
> of common pieces and function/variable-related pieces, with the latter
> being associated with the corresponding text/data sections via
> SHF_LINK_ORDER. However, in some cases, the section is laid out such that
> there are multiple common pieces intermixed with the
> function/variable-specific pieces (e.g. it might look like "common,
> function, function, common, function, variable, common"). I actually don't
> want this section to be ordered at all, since the original ordering should
> be preserved.I've been using a hacked LLD for this, which removes the
> ordering for SHF_LINK_ORDER completely, so that the sections remain in the
> "natural" order, whilst the associated-with semantics for --gc-sections are
> still preserved.
> >>
> >>Whilst doing this, I couldn't help but feel that the associated-with and
> ordering semantics are somewhat orthoganol. Clearly if a section is
> ordered, it needs the associated-with semantics, but it seems like section
> association is a different thing. There are multiple sections now where the
> ordering is irrelevant, but which still wants association in some way.
> Examples include LLVM's stack sizes section and debug data.The original
> SHF_LINK_ORDER extension discussion actually started out with discussing a
> SHF_ASSOCIATED flag. Maybe we should revisit that idea in some form? Thus,
> SHF_ASSOCIATED implies the section should be discarded when it's linked
> section is discarded, and SHF_LINK_ORDER requires ordering. I think both
> for backwards-compatibility's sake and to avoid redundancy, we could say
> SHF_LINK_ORDER implies SHF_ASSOCIATED.
> >>
> >
> >
> >
> >   I guess I don't understand why SHT_GROUP sections aren't the general
> >solution to this scenario, and any others where sections need to be
> >associated.
> >
> >    https://docs.oracle.com/cd/E37838_01/html/E36783/chapter7-26.html
> >
> >In your scenario, I think you'd set the flags field to 0, rather
> >than GRP_COMDAT, and then you'd have an associated set of sections
> >that are kept, or discarded, as a unit. And, GROUP conveniently go
> >away as part of a "final" link that produces an executable or shared
> >object, as you propose above.
>
> If I understand correctly, linkers support GRP_COMDAT but not other
> values. So
> you are right that we can make use of other values if we want to do some
> fancy
> things.
>
> In practice, I guess one reason people want to mess with a section flag is
> that
> adding a .group has non-trivial cost (sizeof(Elf64_Shdr)=64) if an object
> file
> may have many of them (especially when -ffunction-sections is enabled).
>
> For example, I added -fpatchable-function-entry=N[,M] in clang for the
> Linux
> kernel. Every .text section has an accompanying
> __patchable_function_entries
> with SHF_LINK_ORDER set. If I want to express this with a section group, I
> need
> to add a .group (SHT_GROUP) for each .text, the cost will go up from
> double to
> triple.
>
Fangrui more or less covered it, but to add, I have in the past
experimented with non-comdat groups, but since they weren't supported in
LLVM (no assembler syntax, no LLD support), it was a non-trivial amount of
work to get them to work nicely at the time. Plus, as mentioned, they are a
significantly higher file space (and likely link-time) overhead than just
using the sh_link value. Given that one of the main concerns with the debug
data fragmentation concept was file I/O overhead costs, due to there being
potentially many more section headers, adding another set per
function/variable (due to -ffunction-sections/-fdata-sections which is
where this approach would be used) makes the matter even worse.

Of course using the SHF_LINK_ORDER/a proposed SHF_ASSOCIATED flag does not
solve the question of what happens if sh_link is used for something else,
but that hasn't been relevant in my situation so far, since the sections
have all been SHT_PROGBITS.


>
> >>On 7/16/20 1:27 AM, James Henderson wrote:
> >> Whilst doing this, I couldn't help but feel that the associated-with
> and ordering semantics are somewhat orthoganol.
>
> Yeah, I am a big fan of non-orthogonality.. However, a gABI section flag
> seems a
> no-go now.
>
> >If the GNU linkers were modified to allow both ordered and
> >non-ordered sections, then I think you could use GROUP for
> >associating, and SHF_LINK_ORDER for actual ordering, where
> >that is desired. We probably don't need a third mechanism,
> >do we?
>
> This solution is acceptable. I will do this for LLD
> (https://reviews.llvm.org/D72904 ). We need an assembler syntax for
> (SHF_LINK_ORDER & sh_link=0).
>
> I created  https://sourceware.org/bugzilla/show_bug.cgi?id=26253 for a
> GNU as feature request.
> Peter Collingbourne proposed
>
>    .section .meta,"ao",@progbits,0
>
> I will expect that GNU as and LLVM integrated assembler match
> (probably https://reviews.llvm.org/D72899#2157020 )
>