[Bug threads/22960] New: brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Bug threads/22960] New: brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

            Bug ID: 22960
           Summary: brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os
                    high sierra 10.13.3
           Product: gdb
           Version: 8.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: threads
          Assignee: unassigned at sourceware dot org
          Reporter: xdavidliu at gmail dot com
  Target Milestone: ---

I filed this bug at the homebrew page, so the relevant info can be found there.
https://github.com/Homebrew/homebrew-core/issues/25172

If someone wants me to copy the info into a post here, for further convenience,
I would be happy to.

This may actually not be the same as bug # 20266 here, which was from two years
ago, since the issue occurs only on gdb 8.1 in homebrew in mac os, *not* gdb
8.0.1 homebrew. In 8.0.1, gdb actually works correctly (assuming codesigning is
done correctly, which is unrelated but has caused many users trouble), only
with a dyld version warning.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

xdavidliu at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|threads                     |gdb

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

thor.lilei at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |thor.lilei at gmail dot com

--- Comment #1 from thor.lilei at gmail dot com ---
Same problem with me.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

Ray Seyfarth <ray.seyfarth at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ray.seyfarth at gmail dot com

--- Comment #2 from Ray Seyfarth <ray.seyfarth at gmail dot com> ---
Same problem with me.

I have to wonder how Apple gets lldb to work with no problems.  It appears to
not be codesigned nor setuid/gid or anything special.  There is an option which
works which Apple should share with the open source community.  They have
wasted a lot of my time.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

Pedro Alves <palves at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |palves at redhat dot com

--- Comment #3 from Pedro Alves <palves at redhat dot com> ---
If gdb 8.0 works, but gdb 8.1 doesn't, then that suggests doing a git bisect to
find the exact change in gdb that caused the problem.  Any takers?

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #5 from Saagar Jha <saagar at saagarjha dot com> ---
(In reply to Ray Seyfarth from comment #2)
> Same problem with me.
>
> I have to wonder how Apple gets lldb to work with no problems.  It appears
> to not be codesigned nor setuid/gid or anything special.  There is an option
> which works which Apple should share with the open source community.  They
> have wasted a lot of my time.

LLDB is signed with Apple's certificate:

$ codesign -dvv `xcrun -find lldb`
Executable=/Applications/Xcode-beta.app/Contents/Developer/usr/bin/lldb
Identifier=com.apple.lldb
Format=Mach-O thin (x86_64)
CodeDirectory v=20200 size=622 flags=0x0(none) hashes=15+2 location=embedded
Signature size=4535
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Info.plist entries=6
TeamIdentifier=59GAB85EFG
Sealed Resources=none
Internal requirements count=1 size=64

--- Comment #6 from Pedro Alves <palves at redhat dot com> ---
(In reply to Saagar Jha from comment #4)

> I think I've found the culprit:
>
> $ git bisect run ../gdb-bisect.sh
> running /Users/saagarjha/Git/bisect-test.sh
>
> [Snip]
>
> f6ac5f3d63e03a81c4ff3749aba234961cc9090e is the first bad commit
> commit f6ac5f3d63e03a81c4ff3749aba234961cc9090e
> Author: Pedro Alves <[hidden email]>
> Date:   Thu May 3 00:37:22 2018 +0100
>
>    Convert struct target_ops to C++
>
> [Snip]
>
> bisect run success

That commit can't be the culprit for the issue reported in this bug,
because that commit is recent, it is in master only, not in 8.1.
It if caused some breakage, it's something else.  A separate bug report
would have been better.

>
> Could someone confirm this for me? Commits before this one can successfully
> follow the debuggee to completion without incident, but the ones after and
> including this one crash with a null pointer dereference in
> gdb`push_target(struct target_ops *) at target.c:653. From a cursory glance,
> it seems a little fishy that darwin-nat.c doesn't have any sort of
> add_target call in it,

The add_target call is in i386-darwin-nat.c:_initialize_i386_darwin_nat

  add_inf_child_target (&darwin_target);

> but I can't understand the code in the C/C++
> frankenstein state it's in right now,

Yeah.  Anything in particular you'd like to point out?

> so I wasn't able to come up with a
> fix. (I did find a bunch of undefined behavior being hit, though, which I
> *do* have patches for. Let me know if you're curious in seeing them.)

Yes please.  If you could contribute fixes, it'd be awesome:

 https://sourceware.org/gdb/wiki/ContributionChecklist

In case it isn't obvious, the macOS port is in real need of someone motivated
to maintain it.  I'm afraid that none of the day-to-day maintainers uses macOS,
AFAIK.  You can see it as an opportunity.

>
> <rant>
> Just as a FYI, confirming this particular commit took well over two days and
> testing over two hundred revisions, which is something that I find as an
> outside observer to be truly horrible.

Wow.  Sorry about that.  Two hundred revisions sounds way too many for a git
bisect?  How could that have happened?

> Does GDB have *no* automated testing
> or continuous integration whatsoever?

It does, see <https://sourceware.org/gdb/wiki/BuildBot>.  The problem is nobody
ever contributed a macOS buildslave.

> Putting aside the fact that any such
> infrastructure would catch simple bugs like this one, which are easy to
> reproduce, it would have also made my life bisecting a lot easier. Many
> intermediate commits are broken, as in they *literally don't build on
> macOS*, because someone forgot a header file or messed up a Makefile. Others
> dereference null pointers or overflow ints during startup, which really
> threw off my bisect script with false positives: I had to restart the bisect
> from the beginning at least half a dozen times because it homed in on the
> wrong bug.

:-(  Sound like maybe "git bisect skip" would have helped?

> I'm aghast that it's possible for such clearly broken patches to
> land in the master branch. I do apologize for the vitriolic tone here, but
> I'm extremely frustrated at the amount of time I had to spend finding this
> when it should have been a rather trivial task. I do hope none of you take
> it personally–but if you're looking for things to improve, this is one thing
> I think you should focus on.

Nope, sorry.  The thing to improve is _getting someone that actually cares
about the port to step up and help maintain it_.  That could be you.
Otherwise, I fear that at some point, the port will just end up deprecated and
removed.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

Saagar Jha <saagar at saagarjha dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |saagar at saagarjha dot com

--- Comment #4 from Saagar Jha <saagar at saagarjha dot com> ---
I think I've found the culprit:

        $ git bisect run ../gdb-bisect.sh
        running /Users/saagarjha/Git/bisect-test.sh

        [Snip]

        f6ac5f3d63e03a81c4ff3749aba234961cc9090e is the first bad commit
        commit f6ac5f3d63e03a81c4ff3749aba234961cc9090e
        Author: Pedro Alves <[hidden email]>
        Date:   Thu May 3 00:37:22 2018 +0100

            Convert struct target_ops to C++

        [Snip]

        bisect run success

Could someone confirm this for me? Commits before this one can successfully
follow the debuggee to completion without incident, but the ones after and
including this one crash with a null pointer dereference in
gdb`push_target(struct target_ops *) at target.c:653. From a cursory glance, it
seems a little fishy that darwin-nat.c doesn't have any sort of add_target call
in it, but I can't understand the code in the C/C++ frankenstein state it's in
right now, so I wasn't able to come up with a fix. (I did find a bunch of
undefined behavior being hit, though, which I *do* have patches for. Let me
know if you're curious in seeing them.)

<rant>
Just as a FYI, confirming this particular commit took well over two days and
testing over two hundred revisions, which is something that I find as an
outside observer to be truly horrible. Does GDB have *no* automated testing or
continuous integration whatsoever? Putting aside the fact that any such
infrastructure would catch simple bugs like this one, which are easy to
reproduce, it would have also made my life bisecting a lot easier. Many
intermediate commits are broken, as in they *literally don't build on macOS*,
because someone forgot a header file or messed up a Makefile. Others
dereference null pointers or overflow ints during startup, which really threw
off my bisect script with false positives: I had to restart the bisect from the
beginning at least half a dozen times because it homed in on the wrong bug. I'm
aghast that it's possible for such clearly broken patches to land in the master
branch. I do apologize for the vitriolic tone here, but I'm extremely
frustrated at the amount of time I had to spend finding this when it should
have been a rather trivial task. I do hope none of you take it personally–but
if you're looking for things to improve, this is one thing I think you should
focus on.
</rant>

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #7 from Pedro Alves <palves at redhat dot com> ---
> including this one crash with a null pointer dereference in
> gdb`push_target(struct target_ops *) at target.c:653.

I think I see what is going on here.  I'll send a patch.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

grassfedcode at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |grassfedcode at gmail dot com

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #8 from Saagar Jha <saagar at saagarjha dot com> ---
Sorry, I took a break from because I couldn't figure it out: my bisect kept
ending up on 4bbd4ef219c5b4c7d437618ba8937af86dd1032e, with a one character
diff. My guess is that this commit changes what methods get called, so it might
be able to discover what this changes if I could log every method call, but I
don't know how to do that in gdb.

> Yeah.  Anything in particular you'd like to point out?

The darwin-nat/i386-darwin-nat thing was kind of confusing to me, since I
thought darwin-nat was for x86_64 and i386-darwin-nat was for, well, i386. Plus
this one didn't really follow the example set by other platforms so I didn't
have much to go off of. Just my thoughts.

> Yes please.  If you could contribute fixes, it'd be awesome

I have a couple of clumsy patches for issues I found up here (as well as
yours), if you find them useful: https://github.com/saagarjha/binutils-gdb. If
they're useful I could try to format them to follow the guidelines.

> Two hundred revisions sounds way too many for a git bisect?  How could that have happened?

Well, each bisect ideally should have been around a dozen commits to test, but
I kept needing to run bisect again because my bisect script, having no real way
of testing whether the current commit was good, ended up doing something along
the lines of checking "echo r | gdb -return-child-result a.out". But a lot of
commits had issues such as not building (which meant that my script, which "git
bisect skip"ed any commit that didn't build, ended up mired in a 90-odd commit
block where none of the commits built, jumping around randomly to find one that
compiled), or many that had some sort of undefined behavior that was recognized
much later, which means I had to manually backport fixes to rule out the false
positives it discovered.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #9 from Pedro Alves <palves at redhat dot com> ---
(In reply to Saagar Jha from comment #8)

> > Yes please.  If you could contribute fixes, it'd be awesome
>
> I have a couple of clumsy patches for issues I found up here (as well as
> yours), if you find them useful: https://github.com/saagarjha/binutils-gdb.
> If they're useful I could try to format them to follow the guidelines.

They do seem to point at real issues that should be fixed somehow.  If you send
the fixes to the list, they can be discussed there.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #10 from Pedro Alves <palves at redhat dot com> ---
>
> > Yeah.  Anything in particular you'd like to point out?
>
> The darwin-nat/i386-darwin-nat thing was kind of confusing to me, since I
> thought darwin-nat was for x86_64 and i386-darwin-nat was for, well, i386.
> Plus this one didn't really follow the example set by other platforms so I
> didn't have much to go off of. Just my thoughts.

OK.  There are exceptions for single-arch ports, but $OS-nat.c is usually _not_
architecture-specific.  E.g., linux-nat.c is for all Linux architectures, and
then we have i386-linux-nat.c/amd64-linux-nat.c.  Same with fbsd-nat.c, etc.
Maybe we should rename i386-darwin-nat to x86-darwin-nat though, as that's the
convention we follow most everywhere (i386=>32-bit, amd64=>64-bit, x86=>both).
Please do feel free to pop in to #gdb on freenode, where several maintainers
hang.  I'd be happy to help you get around the codebase a bit more, if you're
interested.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #11 from Pedro Alves <palves at redhat dot com> ---
Back to the original topic:

(In reply to Saagar Jha from comment #8)
> Sorry, I took a break from because I couldn't figure it out: my bisect kept
> ending up on 4bbd4ef219c5b4c7d437618ba8937af86dd1032e, with a one character
> diff. My guess is that this commit changes what methods get called, so it
> might be able to discover what this changes if I could log every method
> call, but I don't know how to do that in gdb.

Hmm, at least that is indeed changing something in the darwin-related code.
The original patch was submitted here, but it didn't come with any sort of
detail: <https://sourceware.org/ml/gdb-patches/2017-07/msg00447.html>.

Did you try reverting that commit on top of current master, see if it makes a
difference?

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #12 from Saagar Jha <saagar at saagarjha dot com> ---
> They do seem to point at real issues that should be fixed somehow.  If you send the fixes to the list, they can be discussed there.

Sure, I'll make sure to stop by after we get this figured out so I can make a
clean set of patches.

> Did you try reverting that commit on top of current master, see if it makes a difference?

Yup, the issue (mostly) goes away if I do that. There are other latent issues
hiding out somewhere that we can get to later, but at least I can get GDB to
execute by program to successful termination as it did in 8.0.1.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2018-06-27
                 CC|                            |tromey at sourceware dot org
     Ever confirmed|0                           |1

--- Comment #13 from Tom Tromey <tromey at sourceware dot org> ---
I built gdb 8.0 from git and that did not work for me on macOS 10.13.5.
Neither did git master.  I also tried the 8.0.1 from brew.

They both fail in the same way, with "Unknown signal".

I tend to think this is a dup of 20266.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #14 from Pedro Alves <palves at redhat dot com> ---
Tom, does reverting the offending commit work for you?

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #15 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Pedro Alves from comment #14)
> Tom, does reverting the offending commit work for you?

No.

What happens for me is that darwin_decode_message gets
a MACH_NOTIFY_DEAD_NAME (the "== 0x48") case.  Then the
subsequent wait4() call returns with wstatus=5.

wstatus=5 is a strange response.  It is not WIFEXITED,
but neither is it WIFSIGNALED.  So far I haven't found
any documentation about what it might be.

One wild guess is that maybe this mach message actually
does carry the name of the new port and it could be
extracted via darwin_find_new_inferior.  But that seems
like a longshot.

I looked at the lldb patch that Jason Molenda posted
(see https://sourceware.org/bugzilla/show_bug.cgi?id=20266#c6),
but lldb seems to work in a completely different way here,
I guess hooking into some low-level mach thing somehow?  Like,
those functions aren't obviously called from anywhere.
So, I do wonder whether the answer is a bigger rewrite of
darwin-nat.c, to use mach stuff everywhere and not ptrace
or wait.  However, this experience has shown me that even
minor revisions of macOS can come with big changes, so
modifying this code seems somewhat tricky.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #16 from Pedro Alves <palves at redhat dot com> ---
Are you testing with "set startup-with-shell off", perhaps?  Or maybe Saagar
was?  I could see that impacting whether affecting whether you see the SIGTRAP,
since this all seems to be exec-event related.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #17 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Pedro Alves from comment #16)
> Are you testing with "set startup-with-shell off", perhaps?  Or maybe Saagar
> was?  I could see that impacting whether affecting whether you see the
> SIGTRAP, since this all seems to be exec-event related.

I have tried it both ways to no avail.

--
You are receiving this mail because:
You are on the CC list for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug gdb/22960] brew gdb 8.1 (but not 8.0.1) breakpoint trap in mac os high sierra 10.13.3

agentzh at gmail dot com
In reply to this post by agentzh at gmail dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=22960

--- Comment #18 from Tom Tromey <tromey at sourceware dot org> ---
In my case the first problem was that I was trying "gdb /bin/ls" --
but that is subject to System Integrity Protection.
Using my own test executable gives a different problem.

I'll file a separate bug about detecting SIP.
gdb could at least tell the user what is going on.

--
You are receiving this mail because:
You are on the CC list for the bug.
12