python tracing

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

python tracing

corpaul
Was anyone here successful on python tracing?

I patched Python 2.7 using the patches on http://bugs.python.org/issue13405
Now when I try to run http://sourceware.org/systemtap/wiki/PythonMarkers?action=AttachFile&do=view&target=systemtap-example.stp , I get the error:

semantic error: while resolving probe point: identifier 'python' at stap-python.stp:11:7
        source: probe python.function.entry
                      ^

semantic error: probe point mismatch  (alternatives: begin begin(number) end end(number) error error(number) kernel kprobe module(string) netfilter never perf process process(number) process(string) procfs procfs(string) stap staprun timer): identifier 'python' at :11:7
        source: probe python.function.entry
                      ^

semantic error: while resolving probe point: identifier 'python' at :16:7
        source: probe python.function.return
                      ^

semantic error: probe point mismatch  (alternatives: begin begin(number) end end(number) error error(number) kernel kprobe module(string) netfilter never perf process process(number) process(string) procfs procfs(string) stap staprun timer): identifier 'python' at :16:7
        source: probe python.function.return
                      ^

Pass 2: analysis failed.  [man error::pass2]

Does this mean that Python was not patched correctly? or do i need to change something in Systemtap?

Thanks!

--CP
Reply | Threaded
Open this post in threaded view
|

Re: python tracing

David Smith-19
On 04/08/2013 08:11 AM, corpaul wrote:

> Was anyone here successful on python tracing?
>
> I patched Python 2.7 using the patches on http://bugs.python.org/issue13405
> Now when I try to run
> http://sourceware.org/systemtap/wiki/PythonMarkers?action=AttachFile&do=view&target=systemtap-example.stp
> , I get the error:
>
> semantic error: while resolving probe point: identifier 'python' at
> stap-python.stp:11:7
>         source: probe python.function.entry

I haven't done traced python with systemtap, but from the error it looks
like there is supposed to be a systemtap python tapset file (probably
named something like 'python.stp') that is supposed to be added to
systemtap.

Did that exist somewhere in those patches?

--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

RE: python tracing

corpaul
You are right, the tapset was missing.
I used this one: http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/root/usr/share/systemtap/tapset/libpython2.7-64.stp

but am getting the error
 ./stap test_python.stp --runtime=dyninst -c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
semantic error: while resolving probe point: identifier 'process' at /home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
        source: probe python.function.return = process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__return")
                                               ^

semantic error: no match
semantic error: while resolving probe point: identifier 'python' at test_python.stp:1:7
        source: probe python.function.return
                      ^

Pass 2: analysis failed.  [man error::pass2]

--CP
________________________________________
From: David Smith [[hidden email]]
Sent: Monday, April 08, 2013 4:14 PM
To: Cor-paul Bezemer - EWI
Cc: [hidden email]
Subject: Re: python tracing

On 04/08/2013 08:11 AM, corpaul wrote:

> Was anyone here successful on python tracing?
>
> I patched Python 2.7 using the patches on http://bugs.python.org/issue13405
> Now when I try to run
> http://sourceware.org/systemtap/wiki/PythonMarkers?action=AttachFile&do=view&target=systemtap-example.stp
> , I get the error:
>
> semantic error: while resolving probe point: identifier 'python' at
> stap-python.stp:11:7
>         source: probe python.function.entry

I haven't done traced python with systemtap, but from the error it looks
like there is supposed to be a systemtap python tapset file (probably
named something like 'python.stp') that is supposed to be added to
systemtap.

Did that exist somewhere in those patches?

--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

Re: python tracing

David Smith-19
On 04/08/2013 09:41 AM, Cor-paul Bezemer - EWI wrote:

> You are right, the tapset was missing.
> I used this one: http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/root/usr/share/systemtap/tapset/libpython2.7-64.stp
>
> but am getting the error
>  ./stap test_python.stp --runtime=dyninst -c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
> semantic error: while resolving probe point: identifier 'process' at /home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
>         source: probe python.function.return = process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__return")
>                                                ^
>
> semantic error: no match
> semantic error: while resolving probe point: identifier 'python' at test_python.stp:1:7
>         source: probe python.function.return
>                       ^
>
> Pass 2: analysis failed.  [man error::pass2]

OK, we're getting a bit further now.

Can you try the following command? This should show us what tracepoints
stap thinks are present in your python executable.

# stap -L
'process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("*")'

--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

RE: python tracing

corpaul
I tried and noticed that the library path is incorrect (nonexistent). How can I found it which library python is using?

Should I have a python process running when executing this command?
Or is it trying to hook to the 'default' python executable? (my patched Python is in a different directory)

--CP


________________________________________
From: David Smith [[hidden email]]
Sent: Monday, April 08, 2013 4:59 PM
To: Cor-paul Bezemer - EWI
Cc: [hidden email]
Subject: Re: python tracing

On 04/08/2013 09:41 AM, Cor-paul Bezemer - EWI wrote:

> You are right, the tapset was missing.
> I used this one: http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/root/usr/share/systemtap/tapset/libpython2.7-64.stp
>
> but am getting the error
>  ./stap test_python.stp --runtime=dyninst -c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
> semantic error: while resolving probe point: identifier 'process' at /home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
>         source: probe python.function.return = process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__return")
>                                                ^
>
> semantic error: no match
> semantic error: while resolving probe point: identifier 'python' at test_python.stp:1:7
>         source: probe python.function.return
>                       ^
>
> Pass 2: analysis failed.  [man error::pass2]

OK, we're getting a bit further now.

Can you try the following command? This should show us what tracepoints
stap thinks are present in your python executable.

# stap -L
'process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("*")'

--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

Re: python tracing

David Smith-19
On 04/09/2013 03:05 AM, Cor-paul Bezemer - EWI wrote:
> I tried and noticed that the library path is incorrect (nonexistent). How can I found it which library python is using?
>
> Should I have a python process running when executing this command?
> Or is it trying to hook to the 'default' python executable? (my patched Python is in a different directory

Ah. The command I gave you (and the tapset itself) are trying to probe
the system python, not your private python.

You'll need to modify the tapset to point to your private version of
python (instead of the system python). You don't have to have a python
process running when executing the 'stap -L' command.

So, modify that stap command to look something like the following, where
'PRIV_PYTHON' is the path to your private python executable (probably
somewhere down in ~/Play/python2.7-2.7.4-rc1 if I'm reading your email
correctly) and 'PRIV_PYTHON_LIB' is the path to the main python shared
library.

# stap -L 'process("PRIV_PYTHON").library("PRIV_PYTHON_LIB").mark("*")'


>
> --CP
>
>
> ________________________________________
> From: David Smith [[hidden email]]
> Sent: Monday, April 08, 2013 4:59 PM
> To: Cor-paul Bezemer - EWI
> Cc: [hidden email]
> Subject: Re: python tracing
>
> On 04/08/2013 09:41 AM, Cor-paul Bezemer - EWI wrote:
>> You are right, the tapset was missing.
>> I used this one: http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/root/usr/share/systemtap/tapset/libpython2.7-64.stp
>>
>> but am getting the error
>>  ./stap test_python.stp --runtime=dyninst -c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
>> semantic error: while resolving probe point: identifier 'process' at /home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
>>         source: probe python.function.return = process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__return")
>>                                                ^
>>
>> semantic error: no match
>> semantic error: while resolving probe point: identifier 'python' at test_python.stp:1:7
>>         source: probe python.function.return
>>                       ^
>>
>> Pass 2: analysis failed.  [man error::pass2]
>
> OK, we're getting a bit further now.
>
> Can you try the following command? This should show us what tracepoints
> stap thinks are present in your python executable.
>
> # stap -L
> 'process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("*")'
>
> --
> David Smith
> [hidden email]
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)
>


--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

RE: python tracing

corpaul
OK, I have tried this (eventually I updated my path). Still, the list returned by the command is empty :(

--CP
________________________________________
From: David Smith [[hidden email]]
Sent: Tuesday, April 09, 2013 3:24 PM
To: Cor-paul Bezemer - EWI
Cc: [hidden email]
Subject: Re: python tracing

On 04/09/2013 03:05 AM, Cor-paul Bezemer - EWI wrote:
> I tried and noticed that the library path is incorrect (nonexistent). How can I found it which library python is using?
>
> Should I have a python process running when executing this command?
> Or is it trying to hook to the 'default' python executable? (my patched Python is in a different directory

Ah. The command I gave you (and the tapset itself) are trying to probe
the system python, not your private python.

You'll need to modify the tapset to point to your private version of
python (instead of the system python). You don't have to have a python
process running when executing the 'stap -L' command.

So, modify that stap command to look something like the following, where
'PRIV_PYTHON' is the path to your private python executable (probably
somewhere down in ~/Play/python2.7-2.7.4-rc1 if I'm reading your email
correctly) and 'PRIV_PYTHON_LIB' is the path to the main python shared
library.

# stap -L 'process("PRIV_PYTHON").library("PRIV_PYTHON_LIB").mark("*")'


>
> --CP
>
>
> ________________________________________
> From: David Smith [[hidden email]]
> Sent: Monday, April 08, 2013 4:59 PM
> To: Cor-paul Bezemer - EWI
> Cc: [hidden email]
> Subject: Re: python tracing
>
> On 04/08/2013 09:41 AM, Cor-paul Bezemer - EWI wrote:
>> You are right, the tapset was missing.
>> I used this one: http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/root/usr/share/systemtap/tapset/libpython2.7-64.stp
>>
>> but am getting the error
>>  ./stap test_python.stp --runtime=dyninst -c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
>> semantic error: while resolving probe point: identifier 'process' at /home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
>>         source: probe python.function.return = process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__return")
>>                                                ^
>>
>> semantic error: no match
>> semantic error: while resolving probe point: identifier 'python' at test_python.stp:1:7
>>         source: probe python.function.return
>>                       ^
>>
>> Pass 2: analysis failed.  [man error::pass2]
>
> OK, we're getting a bit further now.
>
> Can you try the following command? This should show us what tracepoints
> stap thinks are present in your python executable.
>
> # stap -L
> 'process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("*")'
>
> --
> David Smith
> [hidden email]
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)
>


--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

Re: python tracing

David Smith-19
Let's try a couple of more things.

First, let's look for the special section that using markers create. Try
this, substituting PRIV_PYTHON_LIB as before:

# eu-readelf -S PRIV_PYTHON_LIB | grep stapsdt

On libc on my system, that reports:

# eu-readelf -S /usr/lib64/libc-2.17.so | fgrep stapsdt
[16] .stapsdt.base        PROGBITS     00000030f5780be0 00180be0
00000001  0 A      0   0  1
[35] .note.stapsdt        NOTE         0000000000000000 001ba8f0
00000294  0        0   0  4

You should see similar sections in your private python library. If you
see those sections, try the following, substituting PRIV_PYTHON_LIB as
before:

# stap -L 'process("PRIV_PYTHON_LIB").mark("*")'

On 04/09/2013 08:32 AM, Cor-paul Bezemer - EWI wrote:

> OK, I have tried this (eventually I updated my path). Still, the list returned by the command is empty :(
>
> --CP
> ________________________________________
> From: David Smith [[hidden email]]
> Sent: Tuesday, April 09, 2013 3:24 PM
> To: Cor-paul Bezemer - EWI
> Cc: [hidden email]
> Subject: Re: python tracing
>
> On 04/09/2013 03:05 AM, Cor-paul Bezemer - EWI wrote:
>> I tried and noticed that the library path is incorrect (nonexistent). How can I found it which library python is using?
>>
>> Should I have a python process running when executing this command?
>> Or is it trying to hook to the 'default' python executable? (my patched Python is in a different directory
>
> Ah. The command I gave you (and the tapset itself) are trying to probe
> the system python, not your private python.
>
> You'll need to modify the tapset to point to your private version of
> python (instead of the system python). You don't have to have a python
> process running when executing the 'stap -L' command.
>
> So, modify that stap command to look something like the following, where
> 'PRIV_PYTHON' is the path to your private python executable (probably
> somewhere down in ~/Play/python2.7-2.7.4-rc1 if I'm reading your email
> correctly) and 'PRIV_PYTHON_LIB' is the path to the main python shared
> library.
>
> # stap -L 'process("PRIV_PYTHON").library("PRIV_PYTHON_LIB").mark("*")'
>
>
>>
>> --CP
>>
>>
>> ________________________________________
>> From: David Smith [[hidden email]]
>> Sent: Monday, April 08, 2013 4:59 PM
>> To: Cor-paul Bezemer - EWI
>> Cc: [hidden email]
>> Subject: Re: python tracing
>>
>> On 04/08/2013 09:41 AM, Cor-paul Bezemer - EWI wrote:
>>> You are right, the tapset was missing.
>>> I used this one: http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/root/usr/share/systemtap/tapset/libpython2.7-64.stp
>>>
>>> but am getting the error
>>>  ./stap test_python.stp --runtime=dyninst -c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
>>> semantic error: while resolving probe point: identifier 'process' at /home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
>>>         source: probe python.function.return = process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__return")
>>>                                                ^
>>>
>>> semantic error: no match
>>> semantic error: while resolving probe point: identifier 'python' at test_python.stp:1:7
>>>         source: probe python.function.return
>>>                       ^
>>>
>>> Pass 2: analysis failed.  [man error::pass2]
>>
>> OK, we're getting a bit further now.
>>
>> Can you try the following command? This should show us what tracepoints
>> stap thinks are present in your python executable.
>>
>> # stap -L
>> 'process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("*")'
>>
>> --
>> David Smith
>> [hidden email]
>> Red Hat
>> http://www.redhat.com
>> 256.217.0141 (direct)
>> 256.837.0057 (fax)
>>
>
>
> --
> David Smith
> [hidden email]
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)
>


--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

RE: python tracing

corpaul
Also empty :/
Looks like the patch didn't work?



--CP

-----Original Message-----
From: David Smith [mailto:[hidden email]]
Sent: dinsdag 9 april 2013 17:16
To: Cor-paul Bezemer - EWI
Cc: [hidden email]
Subject: Re: python tracing

Let's try a couple of more things.

First, let's look for the special section that using markers create. Try
this, substituting PRIV_PYTHON_LIB as before:

# eu-readelf -S PRIV_PYTHON_LIB | grep stapsdt

On libc on my system, that reports:

# eu-readelf -S /usr/lib64/libc-2.17.so | fgrep stapsdt
[16] .stapsdt.base        PROGBITS     00000030f5780be0 00180be0
00000001  0 A      0   0  1
[35] .note.stapsdt        NOTE         0000000000000000 001ba8f0
00000294  0        0   0  4

You should see similar sections in your private python library. If you see
those sections, try the following, substituting PRIV_PYTHON_LIB as
before:

# stap -L 'process("PRIV_PYTHON_LIB").mark("*")'

On 04/09/2013 08:32 AM, Cor-paul Bezemer - EWI wrote:

> OK, I have tried this (eventually I updated my path). Still, the list
> returned by the command is empty :(
>
> --CP
> ________________________________________
> From: David Smith [[hidden email]]
> Sent: Tuesday, April 09, 2013 3:24 PM
> To: Cor-paul Bezemer - EWI
> Cc: [hidden email]
> Subject: Re: python tracing
>
> On 04/09/2013 03:05 AM, Cor-paul Bezemer - EWI wrote:
>> I tried and noticed that the library path is incorrect (nonexistent). How
can I found it which library python is using?

>>
>> Should I have a python process running when executing this command?
>> Or is it trying to hook to the 'default' python executable? (my
>> patched Python is in a different directory
>
> Ah. The command I gave you (and the tapset itself) are trying to probe
> the system python, not your private python.
>
> You'll need to modify the tapset to point to your private version of
> python (instead of the system python). You don't have to have a python
> process running when executing the 'stap -L' command.
>
> So, modify that stap command to look something like the following,
> where 'PRIV_PYTHON' is the path to your private python executable
> (probably somewhere down in ~/Play/python2.7-2.7.4-rc1 if I'm reading
> your email
> correctly) and 'PRIV_PYTHON_LIB' is the path to the main python shared
> library.
>
> # stap -L 'process("PRIV_PYTHON").library("PRIV_PYTHON_LIB").mark("*")'
>
>
>>
>> --CP
>>
>>
>> ________________________________________
>> From: David Smith [[hidden email]]
>> Sent: Monday, April 08, 2013 4:59 PM
>> To: Cor-paul Bezemer - EWI
>> Cc: [hidden email]
>> Subject: Re: python tracing
>>
>> On 04/08/2013 09:41 AM, Cor-paul Bezemer - EWI wrote:
>>> You are right, the tapset was missing.
>>> I used this one:
>>> http://packaging-farm.dachary.org/packaging-farm/fedora/x86_64/f14/r
>>> oot/usr/share/systemtap/tapset/libpython2.7-64.stp
>>>
>>> but am getting the error
>>>  ./stap test_python.stp --runtime=dyninst
-c'~/Play/python/python2.7-2.7.4~rc1/build-shared/python test.py'
>>> semantic error: while resolving probe point: identifier 'process' at
/home/corpaul/stap/share/systemtap/tapset/python.stp:12:32
>>>         source: probe python.function.return =
process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("function__
return")
>>>                                                ^
>>>
>>> semantic error: no match
>>> semantic error: while resolving probe point: identifier 'python' at
test_python.stp:1:7

>>>         source: probe python.function.return
>>>                       ^
>>>
>>> Pass 2: analysis failed.  [man error::pass2]
>>
>> OK, we're getting a bit further now.
>>
>> Can you try the following command? This should show us what
>> tracepoints stap thinks are present in your python executable.
>>
>> # stap -L
>> 'process("python").library("/usr/lib64/libpython2.7.so.1.0").mark("*")'
>>
>> --
>> David Smith
>> [hidden email]
>> Red Hat
>> http://www.redhat.com
>> 256.217.0141 (direct)
>> 256.837.0057 (fax)
>>
>
>
> --
> David Smith
> [hidden email]
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)
>


--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Reply | Threaded
Open this post in threaded view
|

Re: python tracing

David Smith-19
On 04/09/2013 10:51 AM, Cor-Paul Bezemer wrote:
> Also empty :/
> Looks like the patch didn't work?

It appears so. I'll ask a basic question. After applying the patches to
python, did you use the '--with-dtrace' configure argument?

--
David Smith
[hidden email]
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
Reply | Threaded
Open this post in threaded view
|

Re: python tracing

Stan Cox
In reply to this post by corpaul
We test a variation of this in the upstream systemtap testsuite
(testsuite/systemtap.apps/python.exp)  What the test does is: fetch
upstream python sources, apply either one of python2.patch or
python3.patch, build python, run test using a scenario similar to the
following:

stap celsius-var.stp -c "python -sS celsius.py 30" '*' python.log
stap celsius-var.stp -c "python3 -sS celsius.py 30" '*' python.log
Reply | Threaded
Open this post in threaded view
|

Re: python tracing

corpaul
I was finally able to get this working by using the following patched version:

hg clone http://hg.jcea.es/cpython-2011
cd cpython-2011
hg checkout dtrace-issue13405_2.7
./configure --with-dtrace
cp Modules/Setup.dist Modules/Setup
make -j_Number_of_cores_in_the_machine_

and the right library path can be found by running on the built python:
ldd python

We're working on building a .deb file for the python version as it now misses some linking with libraries.
Figured I'd post this here in case it might be of use to anyone.