[Bug translator/24720] New: stap Can't insert module and exit with error "Invalid module format"

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] New: stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

            Bug ID: 24720
           Summary: stap Can't insert module and exit with error "Invalid
                    module format"
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: translator
          Assignee: systemtap at sourceware dot org
          Reporter: gromero at linux dot vnet.ibm.com
  Target Milestone: ---

stap can't install module in Pass 5 and fails with "Invalid module format":

gromero@2q14gto1:~/systemtap/bin$ sudo ./stap -g -k -v -r
/home/gromero/git/linux tm.d
Pass 1: parsed user script and 496 library scripts using
101632virt/89920res/7936shr/81472data kb, in 520usr/10sys/529real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using
148544virt/139200res/8960shr/128384data kb, in 880usr/10sys/881real ms.
Pass 3: translated to C into "/tmp/stap1fSq2k/stap_25649_src.c" using
148544virt/139456res/9216shr/128384data kb, in 0usr/10sys/11real ms.
Pass 4: compiled C into "stap_25649.ko" in 2690usr/310sys/3134real ms.
Pass 5: starting run.
ERROR: Couldn't insert module '/tmp/stap1fSq2k/stap_25649.ko': Invalid module
format
WARNING: /home/gromero/systemtap/bin/staprun exited with status: 1
Pass 5: run completed in 10usr/0sys/3real ms.
Pass 5: run failed.  [man error::pass5]
Keeping temporary directory "/tmp/stap1fSq2k"

If Guru mode is disabled it installs correctly:

gromero@2q14gto1:~/systemtap/bin$ sudo ./stap -k -v -r /home/gromero/git/linux
tm.d
Pass 1: parsed user script and 496 library scripts using
101568virt/89856res/7936shr/81408data kb, in 510usr/10sys/525real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using
148416virt/139136res/8960shr/128256data kb, in 860usr/10sys/873real ms.
Pass 3: translated to C into "/tmp/stap2dpqpG/stap_22990_src.c" using
148416virt/139392res/9216shr/128256data kb, in 10usr/0sys/11real ms.
Pass 4: compiled C into "stap_22990.ko" in 14460usr/900sys/15826real ms.
Pass 5: starting run.
tm_unavailable() -> apache2
tm_unavailable() -> apache2
tm_unavailable() -> apache2
^CPass 5: run completed in 0usr/20sys/10304real ms.
Keeping temporary directory "/tmp/stap2dpqpG"

Also, passing flags "-B CONFIG_MODVERSIONS=y" with Guru mode work-around the
issue:

gromero@2q14gto1:~/systemtap/bin$ sudo ./stap -B CONFIG_MODVERSIONS=y -g -k -v
-r /home/gromero/git/linux tm.d
Pass 1: parsed user script and 496 library scripts using
101568virt/89920res/7936shr/81408data kb, in 520usr/10sys/528real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using
148480virt/139136res/8960shr/128320data kb, in 860usr/10sys/882real ms.
Pass 3: translated to C into "/tmp/stapXWnZ4f/stap_24910_src.c" using
148480virt/139392res/9216shr/128320data kb, in 10usr/10sys/11real ms.
Pass 4: compiled C into "stap_24910.ko" in 14420usr/900sys/15758real ms.
Pass 5: starting run.
tm_unavailable() -> apache2
tm_unavailable() -> apache2
tm_unavailable() -> apache2
tm_unavailable() -> apache2

Probe in use is trivial but will only work on recent PPC64 kernels:

gromero@2q14gto1:~/systemtap/bin$ cat tm.d
probe kernel.function("tm_unavailable").inline {
  printf("tm_unavailable() -> %s\n", execname());
}

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

Gustavo Romero <gromero at linux dot vnet.ibm.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gromero at linux dot vnet.ibm.com

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #1 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Additional information on kernel, gcc, and stap version used:

gromero@2q14gto1:~/systemtap/bin$ ./stap --version
Systemtap translator/driver (version 4.2/0.168, commit
release-4.1-35-g82354b99d037)
Copyright (C) 2005-2019 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 5.1-rc2
enabled features: AVAHI BPF LIBSQLITE3 NLS NSS
gromero@2q14gto1:~/systemtap/bin$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/powerpc64le-linux-gnu/6/lto-wrapper
Target: powerpc64le-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1'
--with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs
--enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-6 --program-prefix=powerpc64le-linux-gnu- --enable-shared
--enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/
--enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes
--with-default-libstdcxx-abi=new --enable-gnu-unique-object
--disable-libquadmath --enable-plugin --enable-default-pie --with-system-zlib
--disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-ppc64el/jre --enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-ppc64el
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-ppc64el
--with-arch-directory=ppc64le --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
--enable-objc-gc=auto --enable-secureplt --with-cpu=power8
--enable-targets=powerpcle-linux --disable-multilib --enable-multiarch
--with-long-double-128 --enable-checking=release --build=powerpc64le-linux-gnu
--host=powerpc64le-linux-gnu --target=powerpc64le-linux-gnu
Thread model: posix
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
gromero@2q14gto1:~/systemtap/bin$ uname -a
Linux 2q14gto1 5.2.0-rc5-debuginfo-00239-g241e39004581 #4 SMP Fri Jun 21
12:06:45 CDT 2019 ppc64le GNU/Linux

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #2 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Created attachment 11856
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11856&action=edit
kernel config

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #3 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Created attachment 11857
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11857&action=edit
stap-report wo/ root

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #4 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Created attachment 11858
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11858&action=edit
stap-report w/ root

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #5 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Git source is also in sync with booted kernel:

gromero@2q14gto1:~/git/linux$ git describe
v5.2-rc5-239-g241e39004581
gromero@2q14gto1:~/git/linux$ uname -a
Linux 2q14gto1 5.2.0-rc5-debuginfo-00239-g241e39004581 #4 SMP Fri Jun 21
12:06:45 CDT 2019 ppc64le GNU/Linux

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

Serhei Makarov <me at serhei dot io> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2019-06-21
                 CC|                            |me at serhei dot io
     Ever confirmed|0                           |1

--- Comment #6 from Serhei Makarov <me at serhei dot io> ---
Thanks for providing the detailed info. This problem was caused by the prior
commit [1] attempting to fix a much rarer problem (a guru-mode module compiled
for one kernel can sometimes run on a different kernel version if there is no
difference in the ABIs checked by modsymversioning).

[1]:
https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=c8084763b75fe4c8c61a0ddfa69e2d13f04de18e;hp=3a29a2eef30f191afbc1bc117b6e5a2812dfe91b

For now I reverted the commit in SystemTap git master since it clearly had
undesirable results. I'll use your info to reproduce the problem for myself and
see if there's a better way to fix it that works across all the different
configs. Keeping the PR open to track that work.

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

fweimer at redhat dot com
In reply to this post by fweimer at redhat dot com
https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #7 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
(In reply to Serhei Makarov from comment #6)

> Thanks for providing the detailed info. This problem was caused by the prior
> commit [1] attempting to fix a much rarer problem (a guru-mode module
> compiled for one kernel can sometimes run on a different kernel version if
> there is no difference in the ABIs checked by modsymversioning).
>
> [1]:
> https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=c8084763b75fe4c8c61a0ddfa69e2d13f04de18e;
> hp=3a29a2eef30f191afbc1bc117b6e5a2812dfe91b
>
> For now I reverted the commit in SystemTap git master since it clearly had
> undesirable results. I'll use your info to reproduce the problem for myself
> and see if there's a better way to fix it that works across all the
> different configs. Keeping the PR open to track that work.

Thanks for the analysis and explanations, Serhei!

If you need any help on reproducing/testing it please let me know.

--
You are receiving this mail because:
You are the assignee for the bug.