user-space tracing on Debian... or..?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

user-space tracing on Debian... or..?

corpaul
Hi,

So for the last days I have been trying to get user-space tracing working on Debian.
The first problem was that utrace is not supported, so we tried to compile a custom kernel with uprobes support.
This worked, however, process probes are not supported.


To my understanding these are a fundamental part of user-space tracing... (if not, please explain how I can avoid using them :) )

So it seems that user-space tracing is not supported on Debian.. what do you advise?
We tried the latest Fedora core but  as this uses uprobes as well, the same problem arises.

Any help is appreciated.

Thanks, CP
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Frank Ch. Eigler
Hi -

c.bezemer wrote:

> [...] we tried to compile a custom kernel with uprobes support.
> This worked, however, process probes are not supported.

Please follow the instructions in "man 7stap error::reporting",
or http://sourceware.org/systemtap/wiki/HowToReportBugs to supply
us with enough information to help you.

- FChE
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Timo Juhani Lindfors
In reply to this post by corpaul
corpaul <[hidden email]> writes:
> So for the last days I have been trying to get user-space tracing working on
> Debian.
> The first problem was that utrace is not supported, so we tried to compile a
> custom kernel with uprobes support.
> This worked, however, process probes are not supported.

How exactly did you build your custom kernel? Which kernel version did
you use? I tested this with squeeze two years ago and I think it
worked. The kernel I built is still at

http://lindi.iki.fi/lindi/systemtap/linux/

and includes the source (the *.dsc file) that describes exactly the
patches that were applied.

Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

corpaul
I tried with a custom 3.8.x. kernel.
Also I tried with the latest Fedora which uses a 3.6.x kernel.
Both use Systemtap 2.1.

For both kernels I get an error when I try to run the probe:
probe process("./test").function("main").return
{
        printf("Main function exited\n");
}

Error:
Pass 1: parsed user script and 97 library script(s) using 216852virt/35972res/2972shr/33668data kb, in 230usr/50sys/507real ms.
semantic error: while resolving probe point: identifier 'process' at test.stp:2:7
        source: probe process("./test").function("main").return
                      ^

semantic error: process return probes not available [man error::inode-uprobes]
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 217908virt/37272res/3116shr/34724data kb, in 10usr/10sys/27real ms.
Pass 2: analysis failed.  [man error::pass2]

From what I understand is that it has something to do with the fact that Systemtap uses Uprobes instead of Utrace in newer kernels. Is there any way I can run probes of this type? Or is there a workaround to be able to run a probe with the same effect?

Thanks! CP
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Lukas Berk
Hey,

[...]

> Error:
> Pass 1: parsed user script and 97 library script(s) using
> 216852virt/35972res/2972shr/33668data kb, in 230usr/50sys/507real ms.
> semantic error: while resolving probe point: identifier 'process' at
> test.stp:2:7
>         source: probe process("./test").function("main").return
>                       ^
>
> semantic error: process return probes not available [man
> error::inode-uprobes]
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s)
> using 217908virt/37272res/3116shr/34724data kb, in 10usr/10sys/27real ms.
> Pass 2: analysis failed.  [man error::pass2]
>
> From what I understand is that it has something to do with the fact that
> Systemtap uses Uprobes instead of Utrace in newer kernels. Is there any way
> I can run probes of this type? Or is there a workaround to be able to run a
> probe with the same effect?
Taken from the error::inode-uprobes man page;
function.return probes
  process.function().return probes require "return-probes" or "uretprobes"
functionality, that has not been implemented yet in the kernel.  Until  it
is re-added, consider using process.statement() probes placed on source
line numbers at the function's return statements.

It may also be possible to use the --runtime=dyninst option to use
function.return style probes, however I'm not sure on the state of
dyninst on debian.  So for script portability it might be worth using the
process.statement approach.

Please let us know if any of the error:: man page entries are unclear
and we'll do our best to make them more precise and/or readable.

Cheers,

Lukas

attachment0 (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Timo Juhani Lindfors
Lukas Berk <[hidden email]> writes:
> It may also be possible to use the --runtime=dyninst option to use
> function.return style probes, however I'm not sure on the state of
> dyninst on debian.  So for script portability it might be worth using the
> process.statement approach.

systemtap in Debian has not been built with dyninst support since
dyninst is not yet part of Debian. I can consider packaging it if it
offers real advantages.

Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

corpaul
In reply to this post by Lukas Berk
Thanks, this has gotten me a little bit further.. however, setting probes at line numbers is very inconvenient for large source files (unless I misunderstood something :) ). Also, this does not seem to be very evolvable to me - I should change the line numbers everytime the source file changes..

So is there another way to set probes on all function returns, rather than creating a new probe explicitly for every return using the line number?
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Frank Ch. Eigler

c.bezemer wrote:

> [...]  So is there another way to set probes on all function
> returns, rather than creating a new probe explicitly for every
> return using the line number?  [...]

With a basic facility missing in the kernel, systemtap users are
forced to wait for "uretprobes" to be finished / merged.

An alternative is the stap --runtime=dyninst backend, which would have
to be enabled in your systemtap build/distro, and is still relatively
young code.

(Another alternative is to use an older Fedora or any RHEL[56] distro,
where the kernel utrace patches were applied, and thus systemtap users
have full function-return-probing powers.)

- FChE
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

corpaul
Unfortunately, using a RHEL distr is not possible for me as the application I want to probe does not work well on RHEL distros.

I have tried to build Systemtap 2.1.1 with dyninst 8.0.0 (the newest dyninst gives compilation problems) the following way:
./configure  --with-dyninst=/home/corpaul/dyninst/ --prefix=/home/corpaul/systemtap-2.1 --with-elfutils=/home/corpaul/Play/systemtap2.1/systemtap-2.1/elfutils/elfutils-0.153

make results in the following errors (last part of output):
touch stamp-elfutils
make  all-recursive
make[1]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
Making all in .
make[2]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
  CXX    stap-main.o
  CXX    stap-session.o
  CXX    stap-parse.o
  CXX    stap-staptree.o
  CXX    stap-elaborate.o
  CXX    stap-translate.o
  CXX    stap-tapsets.o
  CXX    stap-buildrun.o
  CC     stap-loc2c.o
  CXX    stap-hash.o
  CXX    stap-cache.o
  CXX    stap-util.o
  CXX    stap-coveragedb.o
  CXX    stap-dwarf_wrappers.o
  CXX    stap-tapset-been.o
  CXX    stap-tapset-procfs.o
  CXX    stap-tapset-timers.o
  CXX    stap-tapset-netfilter.o
  CXX    stap-tapset-perfmon.o
  CXX    stap-tapset-mark.o
  CXX    stap-tapset-itrace.o
  CXX    stap-tapset-utrace.o
  CXX    stap-task_finder.o
  CXX    stap-dwflpp.o
  CXX    stap-rpm_finder.o
  CXX    stap-setupdwfl.o
  CXX    stap-remote.o
  CXX    stap-tapset-dynprobe.o
  CXX    stap-stapregex.o
  CXX    stap-csclient.o
  CXXLD  stap
  CC     loc2c_test-loc2c-test.o
  CC     loc2c_test-loc2c.o
  CCLD   loc2c-test
  CXX    stap_sign_module-util.o
  CXXLD  stap-sign-module
  CXX    stap_authorize_cert-util.o
  CXXLD  stap-authorize-cert
  CXX    stap_serverd-util.o
  CXXLD  stap-serverd
  CXX    stap_gen_cert-util.o
  CXXLD  stap-gen-cert
make[2]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
Making all in stapdyn
make[2]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
make -C .. git_version.stamp
make[3]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
/bin/bash ./git_version.sh -k -s . -o git_version.h
make[3]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
make  all-am
make[3]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
  CXX    stapdyn-stapdyn.o
  CXX    stapdyn-util.o
  CXXLD  stapdyn
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_get_fde_info_for_cfa_reg3'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_errmsg'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dieoffset'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lowpc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_augmentation_data'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_next_cu_header'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_whatform'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formref'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_loclist_from_expr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_linesrc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formsdata'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formudata'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formaddr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_hasattr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_loclist_from_expr_a'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineaddr'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_get_fde_info_for_reg3'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_siblingof'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_child'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_elf_init'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_cie_info'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formexprloc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_die_CU_offset'
/usr/lib/libstackwalk.so.8.0: undefined reference to `dwarf_errno'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_get_fde_at_pc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_diename'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_n'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_srclines'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_form_class'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_tag'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_set_frame_cfa_value'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineno'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_fde_cie_list_dealloc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_finish'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_range'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_get_fde_list'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_offdie'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_loclist_n'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_cie_augmentation_data'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_list_eh'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formstring'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_srcfiles'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_cie_of_fde'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dealloc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineendsequence'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_attr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_highpc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_global_formref'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_next_cu_header_c'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineoff'
collect2: error: ld returned 1 exit status
make[3]: *** [stapdyn] Error 1
make[3]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
make: *** [all] Error 2



Any suggestions on this?
I'm about to give up on user-space tracing :')

Thanks.

--CP
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Frank Ch. Eigler
corpaul <[hidden email]> writes:

> Unfortunately, using a RHEL distr is not possible for me as the application I
> want to probe does not work well on RHEL distros.
> [...]
> /usr/lib/libdynDwarf.so.8.0: undefined reference to
> `dwarf_get_fde_info_for_cfa_reg3'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_errmsg'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dieoffset'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lowpc'
> [...]

That suggests to me that dyninst was not linked quite right, it's
missing -ldwarf.  See "ldd /usr/lib/libsymtabAPI.so.8.0" - it should
list some libdwarf.so file.

To fix that, could try rebuilding dyninst more correctly, or adding
-ldwarf to the systemtap stapdyn/Makefile stapdyn_LDADD line, or
perhaps using
% env LD_PRELOAD=/usr/lib/libdwarf.so.something stap


- FChE
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Josh Stone-5
In reply to this post by corpaul
On 03/28/2013 03:26 AM, corpaul wrote:
> Unfortunately, using a RHEL distr is not possible for me as the application I
> want to probe does not work well on RHEL distros.
>
> I have tried to build Systemtap 2.1.1 with dyninst 8.0.0 (the newest dyninst
> gives compilation problems) the following way:
[...]
>   CXXLD  stapdyn
> /usr/lib/libdynDwarf.so.8.0: undefined reference to
> `dwarf_get_fde_info_for_cfa_reg3'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_errmsg'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dieoffset'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lowpc'
[...]

Is your libdwarf static?  There is a thread discussing issues with this
on the dyninst-api list:
https://lists.cs.wisc.edu/archive/dyninst-api/2013/msg00026.shtml
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Josh Stone-5
On 03/28/2013 09:25 AM, Josh Stone wrote:
> Is your libdwarf static?  There is a thread discussing issues with this
> on the dyninst-api list:
> https://lists.cs.wisc.edu/archive/dyninst-api/2013/msg00026.shtml

Now I feel I should suggest some actual solution too. :)

We could probably add a link test during configure, trying a small test
first with just -ldyninstAPI, and if that fails we can try again with
the whole-archive -ldwarf options added.

Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

corpaul
I was finally able to get user-space probes working in Debian.

What I did was (thanks to Andrew Bernat from Dyninst for providing excellent support):

1 - Install libdwarf as shared library
Download libdwarf
cd dwarf-2013-02-07/libdwarf
./configure --enable-shared
make
mkdir -p /where/you/want/dwarf/include/files
mkdir -p /where/you/want/dwarf/lib/files
cp libdwarf.h dwarf.h /where/you/want/dwarf/include/files
cp libdwarf.so /where/you/want/dwarf/lib/files
Add /where/you/want/dwarf/lib/files to LD_LIBRARY_PATH
add dwarf.conf to /etc/ld.so.conf.d/ with /where/you/want/dwarf/lib/files as contents

(not sure all these steps were mandatory but at least they worked for me)

2 - Build Dyninst 8.1.1
Configure Dyninst with --with-libdwarf-incdir=/where/you/want/dwarf/include/files and --with-libdwarf-libdir=/where/you/want/dwarf/lib/files --prefix=/where/you/want/dyninst

3 - Build Systemtap 2.2
Configure with: ./configure --with-dyninst=/where/you/want/dyninst --prefix=/where/you/want/systemtap --with-elfutils=elfutils/elfutils-0.153  
and make the source code

and the probe ./stap -e 'probe process.function("main") { print(pn()); }' --runtime=dyninst -c './stap -V'
should work now.



Now... next step is getting python user-space probing to work :')
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Timo Juhani Lindfors
corpaul <[hidden email]> writes:
> I was finally able to get user-space probes working in Debian.

Thanks for the info. So this was on amd64 and in debian unstable?

Do you have any interest in Debian packaging? I could consider adding
dyninst to Debian but I really lack the time to extensively test it.
Reply | Threaded
Open this post in threaded view
|

RE: user-space tracing on Debian... or..?

corpaul
This was on amd64 and on Debian squeeze.

Somewhere in the following months I will have to package something to make installation on a test environment easier (e.g. VMs). I will keep you posted on the progress we make.

--CP



________________________________________
From: Timo Juhani Lindfors [[hidden email]]
Sent: Friday, April 05, 2013 3:58 PM
To: Cor-paul Bezemer - EWI
Cc: [hidden email]
Subject: Re: user-space tracing on Debian... or..?

corpaul <[hidden email]> writes:
> I was finally able to get user-space probes working in Debian.

Thanks for the info. So this was on amd64 and in debian unstable?

Do you have any interest in Debian packaging? I could consider adding
dyninst to Debian but I really lack the time to extensively test it.
Reply | Threaded
Open this post in threaded view
|

Re: user-space tracing on Debian... or..?

Timo Juhani Lindfors
Hi,

Cor-paul Bezemer - EWI <[hidden email]> writes:
> Somewhere in the following months I will have to package something to
> make installation on a test environment easier (e.g. VMs). I will keep
> you posted on the progress we make.

ok, do you have previous packaging experience?

-Timo
Reply | Threaded
Open this post in threaded view
|

RE: user-space tracing on Debian... or..?

corpaul
No, but a colleague of mine who is managing the test environment has.

--CP
________________________________________
From: Timo Juhani Lindfors [[hidden email]]
Sent: Friday, April 05, 2013 4:06 PM
To: Cor-paul Bezemer - EWI
Cc: [hidden email]
Subject: Re: user-space tracing on Debian... or..?

Hi,

Cor-paul Bezemer - EWI <[hidden email]> writes:
> Somewhere in the following months I will have to package something to
> make installation on a test environment easier (e.g. VMs). I will keep
> you posted on the progress we make.

ok, do you have previous packaging experience?

-Timo