Unable to break into a PAM module

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to break into a PAM module

Full Name
   I am developing a PAM module to be used by the OpenSSH daemon in CentOS 7.  I would like to be able to get gdb to break into my module - but I I have yet to succeed.  This is how I am launching gdb:

# gdb <sshd-path>/sshd
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from <sshd-path>/sshd...done.
(gdb) b pam_sm_authenticate
Function "pam_sm_authenticate" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (pam_sm_authenticate) pending.

(gdb) run -d -D -p 324

The breakpoints that the b pam_sm_authenticate command sets are the following:

 (gdb) i br
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   <MULTIPLE>        
        breakpoint already hit 3 times
1.1                         y     0x00007ffff179ce50 in pam_sm_authenticate
                                                   at pam_lastlog.c:572
1.2                         y     0x00007ffff39cd080 in pam_sm_authenticate
                                                   at pam_selinux.c:766
1.3                         y     0x00007ffff41e4840 in pam_sm_authenticate
                                                   at pam_permit.c:36
1.4                         y     0x00007ffff43e6a20 in pam_sm_authenticate
                                                   at pam_localuser.c:61
1.5                         y     0x00007ffff45e9d00 in pam_sm_authenticate
                                                   at pam_nologin.c:141
1.6                         y     0x00007ffff47ec700 in pam_sm_authenticate
                                                   at pam_deny.c:33
1.7                         y     0x00007ffff49ef860 in pam_sm_authenticate
                                                   at pam_succeed_if.c:406
1.8                         y     0x00007ffff4e0f390 in pam_sm_authenticate
                                                   at pam_unix_auth.c:106
1.9                         y     0x00007ffff5026b70 in pam_sm_authenticate
                                                   at pam_faildelay.c:158
1.10                      y     0x00007ffff522b2c0 in pam_sm_authenticate
                                                   at pam_env.c:763
1.11                      y     0x00007ffff56342d7 in pam_sm_authenticate
                                                   at src/pam_my_module.c:1087
1.12                      y     0x00007ffff5ee6760 in pam_sm_authenticate
                                                   at pam_sepermit.c:369

The breakpoint I am interested in is 1.11.

When I run gdb and make an SSH connection, out of all the breakpoints above gdb only breaks at 1.7.  However, I know for a fact that pam_sm_authenticate() in src/pam_my_module.c is being executed,  for it has debugging statements unique to this function that send output to syslog, and when I run the code I do indeed see such output in syslog. Even setting a breakpoint explicitly at the relevant file and line does not make gdb to break there. gdb should break at 1.11, but it is not.

Any thoughts on what is going on here? Other relevant information:

#  gdb -v
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)

My PAM module is built as follows:

cc -g -Wall -fPIC -D_GNU_SOURCE -c src/pam_rmy_module.c -o pam_my_module.o
cc -shared pam_my_module.o -lpam -o pam_my_module.so
Reply | Threaded
Open this post in threaded view
|

Re: Unable to break into a PAM module

Jan Kratochvil-2
On Thu, 16 Jan 2020 22:57:25 +0100, Full Name wrote:
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7

As a quick check you could try first newer GDB on your CentOS-7:
        https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/


Jan

Reply | Threaded
Open this post in threaded view
|

Re: Unable to break into a PAM module

Full Name
In reply to this post by Full Name
     Well, I got it to work in the end. I would like to share what I did, in case it could be of use to someone.

     Just launching the SSH daemon from the debugger and setting breakpoints in the relevant PAM source code file won't usually work. This is because the SSH forks children processes, and gdb seems to unable to follow them properly. And, yes, I tried with the different 'set follow-*' commands in gdb.

  The following is what worked for me:

  1) Launch an instance of the SSH daemon, if necessary at some non-standard port:

    # <full-pathname>/sshd -p 8221

  2) Attempt a password-authenticated SSH connection to this daemon.

  3) When prompted for a password at the client, the SSH daemon will be blocking at the pam_sm_authenticate() function of the PAM module in charge
of the authentication.

  4) At the CLI in some terminal emulator issue the following command:

  $ ps awux | egrep sshd | egrep pam

The output from this should be similar to the following:

  root     20017  0.0  0.0 108312  1396 ?        S    14:03   0:00 sshd: <username> [pam]

where <username> is the username received from the client.

  5) Launch gdb:

  # gdb <full-pathname>/sshd

  6) At the gdb prompt issue the following command:

  (gdb) attach <process-id>

where <process-id> is the ID of the process that is doing the PAM stuff. In the example above, this would be 20017.

  7) The command above will load symbols for all relevant shared libraries. At this point, we can set a breakpoint somewhere at the target PAM module's
pam_sm_authenticate() function - e.g.:

  (gdb) b pam_my_module.c: 1121

  8) At the gdb prompt press execute 'continue'.

  9) At the client side, type in the password. This will cause gdb to break at the line specified.

  Bear in mind that, for this to work, the breakpoint has to be set in the PAM source code's pam_sm_authenticate() function  _after_ it prompts the
client for a password - otherwise, gdb will of course not be able to break where told. If one wants to step through the pam_sm_authenticate() code
from the top, one should insert a line like

  sleep(60) ;

at the very start of this function. This will afford us 60 seconds to accomplish the nine steps above, which should be enough to set a breakpoint anywhere after the sleep() invocation.

  Finally, stepping through the PAM code will be far easier if the target PAM module has been compiled without optimizations.
Reply | Threaded
Open this post in threaded view
|

Re: Unable to break into a PAM module

Full Name
In reply to this post by Full Name
    Thanks for your suggestion. I'll have a go when I get some spare time to look into this.

-----Original Message-----
From: "Jan Kratochvil" [[hidden email]]
Date: 01/17/2020 05:02 PM
To: "Full Name" <[hidden email]>
CC: [hidden email]
Subject: Re: Unable to break into a PAM module

On Thu, 16 Jan 2020 22:57:25 +0100, Full Name wrote:
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7

As a quick check you could try first newer GDB on your CentOS-7:
        https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/


Jan



Reply | Threaded
Open this post in threaded view
|

Re: Unable to break into a PAM module

Pedro Alves-7
In reply to this post by Full Name
On 1/17/20 10:03 PM, Full Name wrote:
>      Well, I got it to work in the end. I would like to share what I did, in case it could be of use to someone.
>
>      Just launching the SSH daemon from the debugger and setting breakpoints in the relevant PAM source code file won't usually work. This is because the SSH forks children processes, and gdb seems to unable to follow them properly. And, yes, I tried with the different 'set follow-*' commands in gdb.

For the next time, I suggest also trying "set detach-on-fork off",
and "set schedule-multiple on".

Thanks,
Pedro Alves

Reply | Threaded
Open this post in threaded view
|

Re: Unable to break into a PAM module

Full Name
In reply to this post by Full Name
   Thanks, that works too. Mind you, the SSH session is not properly started - I don't know why - but one can break into the relevant PAM module consistently.


-----Original Message-----
From: "Pedro Alves" [[hidden email]]
Date: 01/22/2020 08:09 AM
To: "Full Name" <[hidden email]>, [hidden email]
Subject: Re: Unable to break into a PAM module

On 1/17/20 10:03 PM, Full Name wrote:
>      Well, I got it to work in the end. I would like to share what I did, in case it could be of use to someone.
>
>      Just launching the SSH daemon from the debugger and setting breakpoints in the relevant PAM source code file won't usually work. This is because the SSH forks children processes, and gdb seems to unable to follow them properly. And, yes, I tried with the different 'set follow-*' commands in gdb.

For the next time, I suggest also trying "set detach-on-fork off",
and "set schedule-multiple on".

Thanks,
Pedro Alves