guru mode: invalid module format

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

guru mode: invalid module format

Aubrey Li
Hi,

Here is mine systemtap version

$ stap -V
Systemtap translator/driver (version 4.1/0.169, commit
release-4.0-191-g8bb72aa68b70 + changes)
Copyright (C) 2005-2019 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 5.1-rc2
enabled features: BPF PYTHON2 PYTHON3 NLS

Normal mode works properly

$ sudo stap -v helloworld.stp
Pass 1: parsed user script and 476 library scripts using
113524virt/86020res/5588shr/80776data kb, in 160usr/70sys/222real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
using 115108virt/87728res/5724shr/82360data kb, in 0usr/0sys/7real ms.
Pass 3: translated to C into
"/tmp/stap0IY8bu/stap_987097eb77b9beb03d8e647034de5773_1001_src.c"
using 115108virt/87728res/5724shr/82360data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into
"stap_987097eb77b9beb03d8e647034de5773_1001.ko" in
1990usr/2520sys/4301real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 10usr/90sys/556real ms.

But guru mode does not work for me

$ sudo rm -rf ~/.systemtap/cache/
$ sudo stap -v helloworld.stp -g
Pass 1: parsed user script and 476 library scripts using
113528virt/86164res/5732shr/80780data kb, in 140usr/80sys/221real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
using 115112virt/87932res/5932shr/82364data kb, in 10usr/10sys/7real
ms.
Pass 3: translated to C into
"/tmp/stap9WYfyJ/stap_987097eb77b9beb03d8e647034de5773_1001_src.c"
using 115112virt/87932res/5932shr/82364data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into
"stap_987097eb77b9beb03d8e647034de5773_1001.ko" in
10650usr/6000sys/16469real ms.
Pass 5: starting run.
ERROR: Couldn't insert module
'/tmp/stap9WYfyJ/stap_987097eb77b9beb03d8e647034de5773_1001.ko':
Invalid module format
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run completed in 0usr/10sys/8real ms.
Pass 5: run failed.  [man error::pass5]

Is this "invalid module format" a known error?

Thanks,
-Aubrey
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Aubrey Li
On Tue, May 7, 2019 at 3:11 PM Aubrey Li <[hidden email]> wrote:

>
> Hi,
>
> Here is mine systemtap version
>
> $ stap -V
> Systemtap translator/driver (version 4.1/0.169, commit
> release-4.0-191-g8bb72aa68b70 + changes)
> Copyright (C) 2005-2019 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> tested kernel versions: 2.6.18 ... 5.1-rc2
> enabled features: BPF PYTHON2 PYTHON3 NLS
>
> Normal mode works properly
>
> $ sudo stap -v helloworld.stp
> Pass 1: parsed user script and 476 library scripts using
> 113524virt/86020res/5588shr/80776data kb, in 160usr/70sys/222real ms.
> Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
> using 115108virt/87728res/5724shr/82360data kb, in 0usr/0sys/7real ms.
> Pass 3: translated to C into
> "/tmp/stap0IY8bu/stap_987097eb77b9beb03d8e647034de5773_1001_src.c"
> using 115108virt/87728res/5724shr/82360data kb, in 0usr/0sys/0real ms.
> Pass 4: compiled C into
> "stap_987097eb77b9beb03d8e647034de5773_1001.ko" in
> 1990usr/2520sys/4301real ms.
> Pass 5: starting run.
> hello world
> Pass 5: run completed in 10usr/90sys/556real ms.
>
> But guru mode does not work for me
>
> $ sudo rm -rf ~/.systemtap/cache/
> $ sudo stap -v helloworld.stp -g
> Pass 1: parsed user script and 476 library scripts using
> 113528virt/86164res/5732shr/80780data kb, in 140usr/80sys/221real ms.
> Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
> using 115112virt/87932res/5932shr/82364data kb, in 10usr/10sys/7real
> ms.
> Pass 3: translated to C into
> "/tmp/stap9WYfyJ/stap_987097eb77b9beb03d8e647034de5773_1001_src.c"
> using 115112virt/87932res/5932shr/82364data kb, in 0usr/0sys/0real ms.
> Pass 4: compiled C into
> "stap_987097eb77b9beb03d8e647034de5773_1001.ko" in
> 10650usr/6000sys/16469real ms.
> Pass 5: starting run.
> ERROR: Couldn't insert module
> '/tmp/stap9WYfyJ/stap_987097eb77b9beb03d8e647034de5773_1001.ko':
> Invalid module format
> WARNING: /usr/local/bin/staprun exited with status: 1
> Pass 5: run completed in 0usr/10sys/8real ms.
> Pass 5: run failed.  [man error::pass5]
>
> Is this "invalid module format" a known error?
>

The problem remains when I upgrade systemtap source and elfutils

$ sudo stap -V
Systemtap translator/driver (version 4.1/0.176, commit
release-4.0-220-g1e96111d0d89 + changes)
Copyright (C) 2005-2019 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 5.1-rc2
enabled features: BPF PYTHON2 PYTHON3 NLS

$ sudo stap -v helloworld.stp -g
Pass 1: parsed user script and 476 library scripts using
113588virt/86248res/5816shr/80776data kb, in 140usr/90sys/226real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
using 115172virt/87956res/5952shr/82360data kb, in 0usr/0sys/7real ms.
Pass 3: translated to C into
"/tmp/stapVVb5wu/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001_src.c"
using 115172virt/87956res/5952shr/82360data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into
"stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko" in
10860usr/5820sys/16486real ms.
Pass 5: starting run.
ERROR: Couldn't insert module
'/tmp/stapVVb5wu/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko':
Invalid module format
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run completed in 20usr/0sys/9real ms.
Pass 5: run failed.  [man error::pass5]
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Frank Ch. Eigler
Aubrey Li <[hidden email]> writes:

>> Normal mode works properly
>> $ sudo stap -v helloworld.stp
>> But guru mode does not work for me
>> $ sudo rm -rf ~/.systemtap/cache/

(This should not be necessary!)

>> $ sudo stap -v helloworld.stp -g
>> Invalid module format
>> WARNING: /usr/local/bin/staprun exited with status: 1
>> Pass 5: run completed in 0usr/10sys/8real ms.
>> Pass 5: run failed.  [man error::pass5]

It would help to know the kernel version, configuration, anything left
in the dmesg w.r.t. modules, security, etc.


- FChE
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Aubrey Li
On Wed, May 8, 2019 at 3:58 AM Frank Ch. Eigler <[hidden email]> wrote:
>
> Aubrey Li <[hidden email]> writes:
>
> >> Normal mode works properly
> >> $ sudo stap -v helloworld.stp
> >> But guru mode does not work for me
> >> $ sudo rm -rf ~/.systemtap/cache/
>
> (This should not be necessary!)

-g run successfully without this.

$ sudo stap helloworld.stp
hello world
$ sudo stap helloworld.stp -g
hello world
$ sudo rm -rf .systemtap/cache/
$ sudo stap helloworld.stp -g
ERROR: Couldn't insert module
'/tmp/stapg2kVhy/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko':
Invalid module format
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run failed.  [man error::pass5]

> >> $ sudo stap -v helloworld.stp -g
> >> Invalid module format
> >> WARNING: /usr/local/bin/staprun exited with status: 1
> >> Pass 5: run completed in 0usr/10sys/8real ms.
> >> Pass 5: run failed.  [man error::pass5]
>
> It would help to know the kernel version, configuration, anything left
> in the dmesg w.r.t. modules, security, etc.
>
Kernel version
============
$ uname -a
Linux aubrey-skl 5.1.0-rc5+ #96 SMP Tue May 7 06:34:44 CST 2019 x86_64
x86_64 x86_64 GNU/Linux

configuration is attached.

I didn't see anything in dmesg related to this. That is, I "tail -f
/var/log/kern.log"
before I run the following steps, and nothing new is shown on the screen.

$ sudo stap -p4 helloworld.stp -g
/home/aubrey/.systemtap/cache/04/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko

$ sudo insmod .systemtap/cache/04/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko
insmod: ERROR: could not insert module
.systemtap/cache/04/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko:
Invalid module format

$ modinfo .systemtap/cache/04/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko
filename:
/home/aubrey/.systemtap/cache/04/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko
license:        GPL
description:    systemtap-generated probe
license:        GPL
srcversion:     75A43EE3D05DA23E3130841
depends:
retpoline:      Y
name:           stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001
vermagic:       5.1.0-rc5+ SMP mod_unload modversions
parm:           _stp_bufsize:buffer size (int)

Please let me know if I should provide more information.

Thanks,
-Aubrey

kernel_config (220K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Frank Ch. Eigler
Hi -


> $ sudo stap helloworld.stp
> hello world
> $ sudo stap helloworld.stp -g
> hello world
> $ sudo rm -rf .systemtap/cache/
> $ sudo stap helloworld.stp -g
> ERROR: Couldn't insert module
> '/tmp/stapg2kVhy/stap_04e0f4dfbaee29e9464955e58a2f9dbf_1001.ko':
> Invalid module format

Thanks for your patience.  I have not been able to reproduce this here
on Fedora, so didn't look deeper until today.  I believe the problem
may be buildrun.cxx:124, which for guru mode adds a
CONFIG_MODEVERSIONS= override into the module kbuild makefile.
Non-fedora kernels may interpret this differently.

commit c8084763b75fe4c8c61a0ddfa69e2d13f04de18e
Author: Serhei Makarov <[hidden email]>
Date:   Wed Jan 16 13:56:23 2019 -0500

    PR10280 initial fix: force vermagic for guru-mode scripts


- FChE
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Serhei Makarov-3
On Thu, Jun 13, 2019, at 7:48 PM, Frank Ch. Eigler wrote:
> Thanks for your patience.  I have not been able to reproduce this here
> on Fedora, so didn't look deeper until today.  I believe the problem
> may be buildrun.cxx:124, which for guru mode adds a
> CONFIG_MODEVERSIONS= override into the module kbuild makefile.
> Non-fedora kernels may interpret this differently.
I haven't been able to reproduce on Ubuntu either. Trying to think if there's any other distribution/kernel where this option is likely to work differently.

The reproduction steps point to another potential problem -- the regular and guru mode scripts are being compiled differently, but they appear to be treated equivalently by stap's compilation cache. Therefore deleting the cache changes the outcome of $ stap helloworld.stp -g

--
All the best,
    Serhei
    https://serhei.io
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Serhei Makarov-3
On Fri, Jun 14, 2019, at 10:55 AM, Serhei Makarov wrote:

> On Thu, Jun 13, 2019, at 7:48 PM, Frank Ch. Eigler wrote:
> > Thanks for your patience.  I have not been able to reproduce this here
> > on Fedora, so didn't look deeper until today.  I believe the problem
> > may be buildrun.cxx:124, which for guru mode adds a
> > CONFIG_MODEVERSIONS= override into the module kbuild makefile.
> > Non-fedora kernels may interpret this differently.
> I haven't been able to reproduce on Ubuntu either. Trying to think if
> there's any other distribution/kernel where this option is likely to
> work differently.
>
> The reproduction steps point to another potential problem -- the
> regular and guru mode scripts are being compiled differently, but they
> appear to be treated equivalently by stap's compilation cache.
> Therefore deleting the cache changes the outcome of $ stap
> helloworld.stp -g

I pushed a commit to include guru-mode status in the hashing data to avoid this problem.

For overriding SystemTap's current behaviour wrt modversions it may be worth trying
$ stap -B CONFIG_MODVERSIONS=y -g hello.stp

Later I'll test on a kernel config where CONFIG_MODVERSIONS= disabled modules are rejected.
If the problem is too complex, I can revert the change to PR10280, at the cost of more risk when transferring guru-mode modules to a different kernel version.

All the best,
      Serhei
Reply | Threaded
Open this post in threaded view
|

Re: guru mode: invalid module format

Aubrey Li
On Tue, Jun 18, 2019 at 4:13 AM Serhei Makarov <[hidden email]> wrote:

>
> On Fri, Jun 14, 2019, at 10:55 AM, Serhei Makarov wrote:
> > On Thu, Jun 13, 2019, at 7:48 PM, Frank Ch. Eigler wrote:
> > > Thanks for your patience.  I have not been able to reproduce this here
> > > on Fedora, so didn't look deeper until today.  I believe the problem
> > > may be buildrun.cxx:124, which for guru mode adds a
> > > CONFIG_MODEVERSIONS= override into the module kbuild makefile.
> > > Non-fedora kernels may interpret this differently.
> > I haven't been able to reproduce on Ubuntu either. Trying to think if
> > there's any other distribution/kernel where this option is likely to
> > work differently.
> >
> > The reproduction steps point to another potential problem -- the
> > regular and guru mode scripts are being compiled differently, but they
> > appear to be treated equivalently by stap's compilation cache.
> > Therefore deleting the cache changes the outcome of $ stap
> > helloworld.stp -g
>
> I pushed a commit to include guru-mode status in the hashing data to avoid this problem.
>
> For overriding SystemTap's current behaviour wrt modversions it may be worth trying
> $ stap -B CONFIG_MODVERSIONS=y -g hello.stp

This works for me. Thanks a lot!

>
> Later I'll test on a kernel config where CONFIG_MODVERSIONS= disabled modules are rejected.
> If the problem is too complex, I can revert the change to PR10280, at the cost of more risk when transferring guru-mode modules to a different kernel version.
>
> All the best,
>       Serhei