RFH: Annotating ELF binaries

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

RFH: Annotating ELF binaries

Nick Clifton
Hello Everyone,

  We (the tools team at Red Hat) are working on a project to add
  annotations to ELF binaries, so that we can answer various questions
  about them.  We have set up a wiki page about the project here:

https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects

  We would very much like this to be something more than just an
  internal project, and so we are reaching out to you for your opinions,
  suggestions and advice.  If you are interested in being able answer
  questions such as 'how was this function compiled ?' or 'is this
  library compatible with this application ?' then please take a minute
  to have a look at the proposal.

  Thanks very much.

Cheers
  Nick Clifton
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Joel Sherrill <joel@OARcorp.com>
Hi

Great idea! I recall Ada toolsets annotating objects
long ago so there was a traceable record of how they
were compiled, what source file version, etc. That
all required closed source tools though.

Chris Johns from RTEMS has been investigating
add ELF annotations to RTEMS and Newlib to indicate which
standards a particular method adheres to. We think this
would assist in evaluating the conformance of a program
to various standards. It hasn't reached the point of
making a public proposal but the underlying idea of
leveraging ELF annotation is similar.

I have cc'ed him so he is aware of this. Perhaps there
is some synergy.

--joel

On 11/4/2016 10:02 AM, Nick Clifton wrote:

> Hello Everyone,
>
>   We (the tools team at Red Hat) are working on a project to add
>   annotations to ELF binaries, so that we can answer various questions
>   about them.  We have set up a wiki page about the project here:
>
> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>
>   We would very much like this to be something more than just an
>   internal project, and so we are reaching out to you for your opinions,
>   suggestions and advice.  If you are interested in being able answer
>   questions such as 'how was this function compiled ?' or 'is this
>   library compatible with this application ?' then please take a minute
>   to have a look at the proposal.
>
>   Thanks very much.
>
> Cheers
>   Nick Clifton
>

--
Joel Sherrill, Ph.D.             Director of Research & Development
[hidden email]        On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35806
Support Available                (256) 722-9985
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Tristan Gingold-2
In reply to this post by Nick Clifton

> On 04 Nov 2016, at 16:02, Nick Clifton <[hidden email]> wrote:
>
> Hello Everyone,
>
>  We (the tools team at Red Hat) are working on a project to add
>  annotations to ELF binaries, so that we can answer various questions
>  about them.  We have set up a wiki page about the project here:
>
> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>
>  We would very much like this to be something more than just an
>  internal project, and so we are reaching out to you for your opinions,
>  suggestions and advice.  If you are interested in being able answer
>  questions such as 'how was this function compiled ?' or 'is this
>  library compatible with this application ?' then please take a minute
>  to have a look at the proposal.

This will generalise attributes used by some architectures (ppc, arm), won't it ?

Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Nick Clifton
Hi Tristan,

>> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects

> This will generalise attributes used by some architectures (ppc, arm), won't it ?

Yes.  Or at least it would if implemented as currently proposed.  Maybe a better
solution would be to only record attributes where they are not already covered by
some target specific solution.

Personally I would prefer a nice, generalised solution, but the current target
specific attributes are mandated by the particular ABIs and so presumably are not
going to go away.

Cheers
  Nick
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Tristan Gingold-2

> On 04 Nov 2016, at 16:22, Nick Clifton <[hidden email]> wrote:
>
> Hi Tristan,
>
>>> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>
>> This will generalise attributes used by some architectures (ppc, arm), won't it ?
>
> Yes.  Or at least it would if implemented as currently proposed.  Maybe a better
> solution would be to only record attributes where they are not already covered by
> some target specific solution.
>
> Personally I would prefer a nice, generalised solution, but the current target
> specific attributes are mandated by the particular ABIs and so presumably are not
> going to go away.

It would be nice to have self-describing properties: a property is compatible only with the same value or with values less or equal, or (even better) within a range.  So that a user can add properties without changing tools.  That would work only for some properties however.

Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Richard W.M. Jones-2
In reply to this post by Nick Clifton
On Fri, Nov 04, 2016 at 03:02:22PM +0000, Nick Clifton wrote:

> Hello Everyone,
>
>   We (the tools team at Red Hat) are working on a project to add
>   annotations to ELF binaries, so that we can answer various questions
>   about them.  We have set up a wiki page about the project here:
>
> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>
>   We would very much like this to be something more than just an
>   internal project, and so we are reaching out to you for your opinions,
>   suggestions and advice.  If you are interested in being able answer
>   questions such as 'how was this function compiled ?' or 'is this
>   library compatible with this application ?' then please take a minute
>   to have a look at the proposal.

My only comment would be - don't assume that GCC (and LLVM) are
the only things which generate objects.  Amongst others there are:

 - ocamlopt
 - GHC
 - golang (or is it using LLVM?)
 - raw assembler files
 - scripts that use objcopy and other techniques to turn data into object files

I don't mean to say it's your responsibility to fix all these things,
but good, clear documentation about exactly what to change is needed.
This was missing at first when the execstack stuff was added years ago.

Also the hardening stuff often doesn't apply in safe languages, so the
tools you build around this shouldn't automatically assume
no hardening == bad; or that 'long double' or 'wchar_t' are meaningful.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Ian Lance Taylor-3
In reply to this post by Nick Clifton
On Fri, Nov 4, 2016 at 8:02 AM, Nick Clifton <[hidden email]> wrote:

>
>   We (the tools team at Red Hat) are working on a project to add
>   annotations to ELF binaries, so that we can answer various questions
>   about them.  We have set up a wiki page about the project here:
>
> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>
>   We would very much like this to be something more than just an
>   internal project, and so we are reaching out to you for your opinions,
>   suggestions and advice.  If you are interested in being able answer
>   questions such as 'how was this function compiled ?' or 'is this
>   library compatible with this application ?' then please take a minute
>   to have a look at the proposal.

Wearing my Go hat, I observe that you are mixing together general
properties ("Which (architecture specific) ABI variant is in use in
object X?") with language-specific properties ("agree upon the format
of long double?").  I encourage a clear separation of those two
different kinds of notes.

Ian
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Florian Weimer-5
On 11/05/2016 12:57 AM, Ian Lance Taylor wrote:

> On Fri, Nov 4, 2016 at 8:02 AM, Nick Clifton <[hidden email]> wrote:
>>
>>   We (the tools team at Red Hat) are working on a project to add
>>   annotations to ELF binaries, so that we can answer various questions
>>   about them.  We have set up a wiki page about the project here:
>>
>> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>>
>>   We would very much like this to be something more than just an
>>   internal project, and so we are reaching out to you for your opinions,
>>   suggestions and advice.  If you are interested in being able answer
>>   questions such as 'how was this function compiled ?' or 'is this
>>   library compatible with this application ?' then please take a minute
>>   to have a look at the proposal.
>
> Wearing my Go hat, I observe that you are mixing together general
> properties ("Which (architecture specific) ABI variant is in use in
> object X?") with language-specific properties ("agree upon the format
> of long double?").  I encourage a clear separation of those two
> different kinds of notes.

I'm not sure how to draw the line.  I consider the size and format of
“long double” very much an ABI matter.

The psABI uses C types throughout, although it is used for much more
than just C .  There is no language-independent name I know of for the
floating point type larger than the type which resembles IEEE double
precision floats.  But I would expect that Ada's definition of
Long_Long_Float would have to be adjusted to align with ABI changes in
this area.

Thanks,
Florian

Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Ian Lance Taylor-3
On Sun, Nov 6, 2016 at 8:30 AM, Florian Weimer <[hidden email]> wrote:

> On 11/05/2016 12:57 AM, Ian Lance Taylor wrote:
>>
>> On Fri, Nov 4, 2016 at 8:02 AM, Nick Clifton <[hidden email]> wrote:
>>>
>>>
>>>   We (the tools team at Red Hat) are working on a project to add
>>>   annotations to ELF binaries, so that we can answer various questions
>>>   about them.  We have set up a wiki page about the project here:
>>>
>>> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>>>
>>>   We would very much like this to be something more than just an
>>>   internal project, and so we are reaching out to you for your opinions,
>>>   suggestions and advice.  If you are interested in being able answer
>>>   questions such as 'how was this function compiled ?' or 'is this
>>>   library compatible with this application ?' then please take a minute
>>>   to have a look at the proposal.
>>
>>
>> Wearing my Go hat, I observe that you are mixing together general
>> properties ("Which (architecture specific) ABI variant is in use in
>> object X?") with language-specific properties ("agree upon the format
>> of long double?").  I encourage a clear separation of those two
>> different kinds of notes.
>
>
> I'm not sure how to draw the line.  I consider the size and format of “long
> double” very much an ABI matter.

Yes, it's an ABI matter, but it's specific to C/C++ programs.

> The psABI uses C types throughout, although it is used for much more than
> just C .  There is no language-independent name I know of for the floating
> point type larger than the type which resembles IEEE double precision
> floats.  But I would expect that Ada's definition of Long_Long_Float would
> have to be adjusted to align with ABI changes in this area.

The only sensible way to write the psABI is for it to say "this is how
to pass an 80 or 128 bit floating point value."  The psABI can't say
"this is the size of the type 'long double'", because that is a matter
for the language and perhaps the compiler.  The best the ABI can do is
say "In C/C++, the type 'long double' is 80 (or 128) bits," which is
separate from the discussion of such a type should be passed between
functions.  Then each language can decide how to implement the
appropriate type--whether it should be the same as the C/C++ type
'long double' or not.

Ian
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Florian Weimer-5
In reply to this post by Richard W.M. Jones-2
On 11/04/2016 07:34 PM, Richard W.M. Jones wrote:

> Also the hardening stuff often doesn't apply in safe languages, so the
> tools you build around this shouldn't automatically assume
> no hardening == bad; or that 'long double' or 'wchar_t' are meaningful.

Sorry, this isn't true.  As long as you don't have a bytecode
interpreter (which is a very attractive target for code injection
attacks, to the degree that additional hardening may not matter at all),
even supposedly memory-safe languages have type system trapdoors, or
perhaps they do not enforce memory safety in the presence of data races.
  This means that many of the hardening settings still matter.

Thanks,
Florian
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Richard W.M. Jones-2
On Mon, Nov 07, 2016 at 03:28:12PM +0100, Florian Weimer wrote:

> On 11/04/2016 07:34 PM, Richard W.M. Jones wrote:
>
> >Also the hardening stuff often doesn't apply in safe languages, so the
> >tools you build around this shouldn't automatically assume
> >no hardening == bad; or that 'long double' or 'wchar_t' are meaningful.
>
> Sorry, this isn't true.  As long as you don't have a bytecode
> interpreter (which is a very attractive target for code injection
> attacks, to the degree that additional hardening may not matter at
> all), even supposedly memory-safe languages have type system
> trapdoors, or perhaps they do not enforce memory safety in the
> presence of data races.  This means that many of the hardening
> settings still matter.

I did say "often".

If we take the OCaml case, then stack hardening is fairly useless
since only simple values and pointers are stored there; and racing
threads isn't possible.  (Stronger enforcement of object boundaries on
the minor heap could be a useful thing to have -- in 2015 a compiler
bug caused a CVE which allowed you to overwrite object headers on the
heap if you could cause an allocation of a certain very large size --
but by default it's not possible to over- or under-run an object
allocation.)

What you really want to know for OCaml code is whether it was compiled
with the -unsafe flag which turns off bounds checking and/or any
unsafe_* functions were called (same) or any Obj.* functions (subverts
type safety).  These would be very useful flags to have but are really
nothing to do with C hardening settings -- they are OCaml-specific and
they are "un-"hardening.  They are also source code red flags which we
already check for.

My point anyway is that OCaml code is hardened by default and you have
to take explicit steps or (in the CVE case) exploit a compiler bug to
remove that hardening.  Very different from C where even a careful and
conscientous programmer can introduce security bugs by accident even
when the compiler is working perfectly and so we have to do all this
work to add second line defenses to our C compilers.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Joseph Myers
In reply to this post by Ian Lance Taylor-3
On Sun, 6 Nov 2016, Ian Lance Taylor wrote:

> The only sensible way to write the psABI is for it to say "this is how
> to pass an 80 or 128 bit floating point value."  The psABI can't say

I would advise using IEEE 754-2008 names in psABIs where available (e.g.
binary128).  (Of course you need to deal with the non-IEEE types such as
x86 extended precision, but IEEE names are standard and unambiguous up to
endianness for binary interchange formats, and unambiguous up to
endianness and BID / DPD choice for decimal interchange formats.)

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Florian Weimer-5
On 11/07/2016 07:00 PM, Joseph Myers wrote:

> On Sun, 6 Nov 2016, Ian Lance Taylor wrote:
>
>> The only sensible way to write the psABI is for it to say "this is how
>> to pass an 80 or 128 bit floating point value."  The psABI can't say
>
> I would advise using IEEE 754-2008 names in psABIs where available (e.g.
> binary128).  (Of course you need to deal with the non-IEEE types such as
> x86 extended precision, but IEEE names are standard and unambiguous up to
> endianness for binary interchange formats, and unambiguous up to
> endianness and BID / DPD choice for decimal interchange formats.)

Most people in the community don't have access to IEEE 754-2008 and
therefore will not know what these terms mean (and considering how much
stuff has been made up about IEEE 754 due to its secretive nature, I
wouldn't want to rely on a paraphrase).

Florian

Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

H.J. Lu-30
In reply to this post by Nick Clifton
On Fri, Nov 4, 2016 at 8:22 AM, Nick Clifton <[hidden email]> wrote:

> Hi Tristan,
>
>>> https://fedoraproject.org/wiki/Toolchain/Watermark#Markup_for_ELF_objects
>
>> This will generalise attributes used by some architectures (ppc, arm), won't it ?
>
> Yes.  Or at least it would if implemented as currently proposed.  Maybe a better
> solution would be to only record attributes where they are not already covered by
> some target specific solution.
>
> Personally I would prefer a nice, generalised solution, but the current target
> specific attributes are mandated by the particular ABIs and so presumably are not
> going to go away.
>
> Cheers
>   Nick

We have 2 different proposals for program properties.  Mine:

https://sourceware.org/ml/gnu-gabi/2016-q4/msg00025.html

has a much smaller scope.  New features on upcoming Intel platforms,
like 5-level paging, need this extension for loader decision at run-time.
How should we move forward with program property extensions?

--
H.J.
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Nick Clifton
Hi H.J.

> We have 2 different proposals for program properties.  Mine:
>
> https://sourceware.org/ml/gnu-gabi/2016-q4/msg00025.html
>
> has a much smaller scope.  New features on upcoming Intel platforms,
> like 5-level paging, need this extension for loader decision at run-time.
> How should we move forward with program property extensions?

I would like to combine the two approaches.  Ie use your notes for
properties that need to be examined at run-time (specifically the
loader, although I imagine that the application itself might be
interested in reading its own notes).  Plus use the note scheme I
am proposing for static analysis tools.

I am currently using a gcc plugin to generate the notes, and I think
that this should be extendable to generate your notes as well.  (Using
a plugin is advantageous in that it is not tied to the latest gcc release.
It can be built to run with older gcc's, and it can be updated
independently of the gcc release cycle).

What do you think ?

Cheers
  Nick


Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Carlos O'Donell-6
On 01/16/2017 09:37 AM, Nick Clifton wrote:

> Hi H.J.
>
>> We have 2 different proposals for program properties.  Mine:
>>
>> https://sourceware.org/ml/gnu-gabi/2016-q4/msg00025.html
>>
>> has a much smaller scope.  New features on upcoming Intel platforms,
>> like 5-level paging, need this extension for loader decision at run-time.
>> How should we move forward with program property extensions?
>
> I would like to combine the two approaches.  Ie use your notes for
> properties that need to be examined at run-time (specifically the
> loader, although I imagine that the application itself might be
> interested in reading its own notes).  Plus use the note scheme I
> am proposing for static analysis tools.
>
> I am currently using a gcc plugin to generate the notes, and I think
> that this should be extendable to generate your notes as well.  (Using
> a plugin is advantageous in that it is not tied to the latest gcc release.
> It can be built to run with older gcc's, and it can be updated
> independently of the gcc release cycle).
>
> What do you think ?

I've added 2 questions to the Toolchain/Watermark wiki but will post them
here for posterity:

(1) What happened to SHT_GNU_ATTRIBUTES and how does it relate to what
    you are proposing?
 
(2) What is being done to ensure the attributes are space and time
    efficient for dynamic link comparison in the dynamic linker?
    Speed of checking 10,000 DSOs (scalability) for ABI compatibility is
    going to be a very important requirement.

Comments:

(a) Ian requests clear separation between language and psABI notes.

Notes are notes. The attribute system should be sufficiently flexible to
handle both, and the "clear sepration" is just a documentation aspect,
and still important, but just about writing down what the notes mean in
clear detail.

(b) Loadable notes and space/time efficiency vs. non-loadable notes and
    static analysis tools.

Run-time checking of properties is radically different from offline
checking of properties and we absolutely need two different designs to
meet these needs. However, if we could weld the two together in a compatible
way, that would be great. For example if the dynamic loader could map from
a 'run-time property' to a 'link-time property' to increase the verbosity
of the error in a failure scenario, then that might be beneficial. If we
could translate 'link-time notes' into 'a collection of run-time properties' in
a semi-automatic fashion given strict rules about the notes application,
then that would also be awesome.

(c) The case against SHT_GNU_ATTRIBUTES (Question 2).

Not used. -- Then we should just use them for x86.

IFUNC complication. -- Any new framework must be able to tolerate that
a given interface may have a "range" of ABIs it works with, and those
represent the set of ABIs it can change to. Attributes that don't allow
sets of values are going to be problematic in the face of IFUNCs.

No loadable segment. -- Correct. They were designed for link-time support only.

Most attributes don't apply to dynamic loading. -- Correct. Space inefficient.

Layout not optimal for loading.  -- Correct. Time/Space inefficient.

In summary SHT_GNU_ATTRIBUTES might not work for run-time properties, but
what about link-time properties? Why not resuse this framework?

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

H.J. Lu-30
On Mon, Jan 16, 2017 at 9:48 AM, Carlos O'Donell <[hidden email]> wrote:

> On 01/16/2017 09:37 AM, Nick Clifton wrote:
>> Hi H.J.
>>
>>> We have 2 different proposals for program properties.  Mine:
>>>
>>> https://sourceware.org/ml/gnu-gabi/2016-q4/msg00025.html
>>>
>>> has a much smaller scope.  New features on upcoming Intel platforms,
>>> like 5-level paging, need this extension for loader decision at run-time.
>>> How should we move forward with program property extensions?
>>
>> I would like to combine the two approaches.  Ie use your notes for
>> properties that need to be examined at run-time (specifically the
>> loader, although I imagine that the application itself might be
>> interested in reading its own notes).  Plus use the note scheme I
>> am proposing for static analysis tools.
>>
>> I am currently using a gcc plugin to generate the notes, and I think
>> that this should be extendable to generate your notes as well.  (Using
>> a plugin is advantageous in that it is not tied to the latest gcc release.
>> It can be built to run with older gcc's, and it can be updated
>> independently of the gcc release cycle).
>>
>> What do you think ?
>
> I've added 2 questions to the Toolchain/Watermark wiki but will post them
> here for posterity:
>
> (1) What happened to SHT_GNU_ATTRIBUTES and how does it relate to what
>     you are proposing?
>
> (2) What is being done to ensure the attributes are space and time
>     efficient for dynamic link comparison in the dynamic linker?
>     Speed of checking 10,000 DSOs (scalability) for ABI compatibility is
>     going to be a very important requirement.
>
> Comments:
>
> (a) Ian requests clear separation between language and psABI notes.
>
> Notes are notes. The attribute system should be sufficiently flexible to
> handle both, and the "clear sepration" is just a documentation aspect,
> and still important, but just about writing down what the notes mean in
> clear detail.
>
> (b) Loadable notes and space/time efficiency vs. non-loadable notes and
>     static analysis tools.
>
> Run-time checking of properties is radically different from offline
> checking of properties and we absolutely need two different designs to
> meet these needs. However, if we could weld the two together in a compatible
> way, that would be great. For example if the dynamic loader could map from
> a 'run-time property' to a 'link-time property' to increase the verbosity
> of the error in a failure scenario, then that might be beneficial. If we
> could translate 'link-time notes' into 'a collection of run-time properties' in
> a semi-automatic fashion given strict rules about the notes application,
> then that would also be awesome.
>
> (c) The case against SHT_GNU_ATTRIBUTES (Question 2).
>
> Not used. -- Then we should just use them for x86.
>
> IFUNC complication. -- Any new framework must be able to tolerate that
> a given interface may have a "range" of ABIs it works with, and those
> represent the set of ABIs it can change to. Attributes that don't allow
> sets of values are going to be problematic in the face of IFUNCs.
>
> No loadable segment. -- Correct. They were designed for link-time support only.
>
> Most attributes don't apply to dynamic loading. -- Correct. Space inefficient.
>
> Layout not optimal for loading.  -- Correct. Time/Space inefficient.
>
> In summary SHT_GNU_ATTRIBUTES might not work for run-time properties, but
> what about link-time properties? Why not resuse this framework?
>

My property proposal is an extension of existing .note.ABI-tag and
.note.gnu.build-id notes, which are loaded sections and don't require
SHT_GNU_ATTRIBUTES.  However, it only supports object scope
property, not symbol scope property.  It is sufficient for run-time loader,
but inadequate for static tools.   Should we evaluate 2 approaches as 2
orthogonal ones?  Use loaded note to cover object scope properties and
non-loaded note for symbol scope properties.

--
H.J.
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Nick Clifton
In reply to this post by Carlos O'Donell-6
Hi Carlos,

> I've added 2 questions to the Toolchain/Watermark wiki but will post them
> here for posterity:

Thanks - I'll try answering them here first, and if my answers make sense
then I will update the wiki.

> (1) What happened to SHT_GNU_ATTRIBUTES and how does it relate to what
>     you are proposing?

Good question, and unfortunately I do not know the answer.  The problem
is, I have been unable to locate any documentation that describes
SHT_GNU_ATTRIBUTES and how it is supposed to be used.

I think that the two schemes are quite similar, although this new proposal
is intended to be able to cope with attributes that only apply to part of
an executable and not necessarily the executable as a whole.  (Also, IMHO,
my proposal has better documentation...)


> (2) What is being done to ensure the attributes are space and time
>     efficient for dynamic link comparison in the dynamic linker?
>     Speed of checking 10,000 DSOs (scalability) for ABI compatibility is
>     going to be a very important requirement.

I believe that H.J's design for the dynamic link notes does take efficiency
into consideration, but I will leave that for him to comment on further.

One thing that I have already done for the static notes is to implement a
new option for objcopy called "--merge-notes" which eliminates redundancies.
Theoretically this option could be extended to work with the dynamic notes
too, helping to make them as space efficient as possible.

Another possibility is that the linker could be extended so that when it
creates a dynamic executable it also inserts a "master" dynamic linker note,
which contains all of the information that the dynamic linker will need,
without it having to search through all of the shared libraries used by the
application.  (This does assume that the shared libraries examined at static
link time are the same ones that are loaded/used at dynamic link time).



> (b) Loadable notes and space/time efficiency vs. non-loadable notes and
>     static analysis tools.
>
> Run-time checking of properties is radically different from offline
> checking of properties and we absolutely need two different designs to
> meet these needs. However, if we could weld the two together in a compatible
> way, that would be great. For example if the dynamic loader could map from
> a 'run-time property' to a 'link-time property' to increase the verbosity
> of the error in a failure scenario, then that might be beneficial.

I think that this might not be easy to do in a way that both imposes a low
code-increase cost on the dynamic linker and a keep-the-dynamic-link-notes-small
space requirement.  There is no harm in investigating though.

> If we
> could translate 'link-time notes' into 'a collection of run-time properties' in
> a semi-automatic fashion given strict rules about the notes application,
> then that would also be awesome.

Now this might well be feasible.  I am thinking of another new option to objcopy
here that examines the static notes and generates dynamic notes from them.  This
should be quite straightforward, provided that the static notes have captures
the right information.

Cheers
  Nick

Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Joseph Myers
On Wed, 18 Jan 2017, Nick Clifton wrote:

> Good question, and unfortunately I do not know the answer.  The problem
> is, I have been unable to locate any documentation that describes
> SHT_GNU_ATTRIBUTES and how it is supposed to be used.

See "Object Attributes" in as.texinfo.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: RFH: Annotating ELF binaries

Carlos O'Donell-6
In reply to this post by Nick Clifton
On 01/18/2017 12:02 PM, Nick Clifton wrote:

> Hi Carlos,
>
>> I've added 2 questions to the Toolchain/Watermark wiki but will post them
>> here for posterity:
>
> Thanks - I'll try answering them here first, and if my answers make sense
> then I will update the wiki.
>
>> (1) What happened to SHT_GNU_ATTRIBUTES and how does it relate to what
>>     you are proposing?
>
> Good question, and unfortunately I do not know the answer.  The problem
> is, I have been unable to locate any documentation that describes
> SHT_GNU_ATTRIBUTES and how it is supposed to be used.
>
> I think that the two schemes are quite similar, although this new proposal
> is intended to be able to cope with attributes that only apply to part of
> an executable and not necessarily the executable as a whole.  (Also, IMHO,
> my proposal has better documentation...)
 
I'm including Joseph Myers in this discussion.

It's very very well documented by ARM.

The point of SHT_GNU_ATTRIBUTES is to mark a section as containing
GNU-based Attributes for the object in question. Rather than have a generic
SHT_NOTE. The GNU-based attributes can be applied to the file, section, or
symbol (and extended further).

Attributes can be anything you want them to be. Today most of the GNU-attributes
are ABI related and the static linker uses them at link time to issue errors
when objects of mixed incompatible ABIs are attempted to be used together.
That is to say they are Tag_File-based attributes (see binutils source for this).

The generic section which holds attributes is SHT_GNU_ATTRIBUTES, but machines
can override this e.g. SHT_ARM_ATTRIBUTES for ARM-specific attributes.

The ELF attributes section design was originally based on the need at the time
from ARM for ARM EABI Attributes.

See 4.3.6 Build Attributes:

http://infocenter.arm.com/help/topic/com.arm.doc.ihi0044f/IHI0044F_aaelf.pdf

.gnu.attributes (section)

        * <format version>, currently 'A' (0x41)
        * <section length> (4 byte unsigned int in ELF endian order of all subsequent data)
                * "vendor name" ("GNU") for generic + NULL pointer.
                * <file-tag> <szie> (all things which are file scope) <attribute>
                * <section-tag> (all things which are section scope) <size> <section #> <attribute>
                * <symbol-tag> (all things which are symbol scope) <size> <symbol #> <attribute>

The ARM documentation goes into a lot of detail about what attributes it
supports.

Given that we have a GNU-generic version of this... might we not extend things
to a new Tag_Range and keep all the existing machinery we already have in binutils
for processing all of these attributes?

Again, the SHT_GNU_ATTRIBUTES section was never designed to be loaded by the
dynamic linker, and it's the reason I recommend you look at the design and think
about how we can avoid another attributes format that is incompatible with the
one we already have in binutils.

Harmonize the designs please :-)

>> (2) What is being done to ensure the attributes are space and time
>>     efficient for dynamic link comparison in the dynamic linker?
>>     Speed of checking 10,000 DSOs (scalability) for ABI compatibility is
>>     going to be a very important requirement.
>
> I believe that H.J's design for the dynamic link notes does take efficiency
> into consideration, but I will leave that for him to comment on further.

It does, mostly be using bits, and AND/OR/EQ handling.

I have yet to see exactly how it works.

I might suggest we version the section like .gnu.attributes so we can
increment the version if we have a flag day changing the layout.

> One thing that I have already done for the static notes is to implement a
> new option for objcopy called "--merge-notes" which eliminates redundancies.
> Theoretically this option could be extended to work with the dynamic notes
> too, helping to make them as space efficient as possible.

OK.

> Another possibility is that the linker could be extended so that when it
> creates a dynamic executable it also inserts a "master" dynamic linker note,
> which contains all of the information that the dynamic linker will need,
> without it having to search through all of the shared libraries used by the
> application.  (This does assume that the shared libraries examined at static
> link time are the same ones that are loaded/used at dynamic link time).
 
We can't assume that unfortunately.
 

>> (b) Loadable notes and space/time efficiency vs. non-loadable notes and
>>     static analysis tools.
>>
>> Run-time checking of properties is radically different from offline
>> checking of properties and we absolutely need two different designs to
>> meet these needs. However, if we could weld the two together in a compatible
>> way, that would be great. For example if the dynamic loader could map from
>> a 'run-time property' to a 'link-time property' to increase the verbosity
>> of the error in a failure scenario, then that might be beneficial.
>
> I think that this might not be easy to do in a way that both imposes a low
> code-increase cost on the dynamic linker and a keep-the-dynamic-link-notes-small
> space requirement.  There is no harm in investigating though.

What I would like to see is parity between the notes the dynamic loader
is using and the notes the static linker is using to generate warnings or outright
link failures.

>> If we
>> could translate 'link-time notes' into 'a collection of run-time properties' in
>> a semi-automatic fashion given strict rules about the notes application,
>> then that would also be awesome.
>
> Now this might well be feasible.  I am thinking of another new option to objcopy
> here that examines the static notes and generates dynamic notes from them.  This
> should be quite straightforward, provided that the static notes have captures
> the right information.

Agreed.

--
Cheers,
Carlos.
12