Re: Compilation issue with pthreads-win32

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

Ross Johnson-2
Hi Claude,

Glad you found the library useful and thankyou for the feedback.

You are right that all Unixes (that I know of) define pthread_t as a
scalar type and that pthread_win32 deviates from this convention by
using a struct. The projects FAQ offers a history and explanation in
Question 11, which I've quoted at the end.

IIRC Solaris does not use a pointer but uses an int type that sequences
to provide a unique id for each new thread in a process. I have no idea
how Solaris maps this counting value to thread storage when needed nor
how it does it efficiently when the set of living threads becomes
sparse, but they do it. Linux and BSD use pointers which are not unique
if the thread exits and its memory has been allocated to a new
pthread_t, or any other type for that matter. I know from comments in
their code that the BSD developers have thought about this issue and the
possibility of changing away from a pointer type at some point.

Having done so ourselves we no longer get questions about the many
complex problems that arise when using pointers. We have had a few
questions like yours to do with porting, all three of which have been
fairly easily solved AFAIK. I take the blame for the decision but I
thought it was better to provide application reliability, predictability
etc. for everyone and accept the occasional but fixable compiler breakage.

But the crux of it is this: the POSIX (and now the SUSv3) standard
allows pthread_t to be scalar or non-scalar and in making it non-scalar
in pthreads-win we are not just taking advantage of a loophole in the
standard; the standard has deliberately not defined pthread_t so that
implementations can define it how they see fit. The notes within the
standard actually suggest defining pthread_t as a struct exactly as it
is defined here, to allow inclusion of a 'sequence' counter to render
the handle unique over time.

But back to your code:
What would happen if you did not set SSD->id = 0, i.e. just leave it
unitialised? I'm curious because your code appears not to attempt a
comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
you would see a compiler error attempting to compare a struct.

If you can't avoid initialising a pthread_t variable, I would suggest
doing it by declaring a special pthread_t constant with the value you
want (0 in this case), e.g.:

typedef union {
     pthread_t t;
     int filler[sizeof(pthread_t)/sizeof(int)];
} init_t;

const init_t u_init = {.filler = {0}}; # Relies on having a C99
compliant compiler

Then you can do this to initialise:

SSD->id = u_init.t;

Note that the initialisation of the array u_init.filler only sets the
first element to 0 explicitly and the remaining elements, if any, are
set to 0 by default. Since you don't really know how many elements there
are you should probably avoid trying to initialise more than one
element, i.e. don't do " = {.filler = {0 , 0}};".

This method means you don't need to break the opacity of the pthread_t
and it should also be portable.

Also for portability, you should only ever use the pthread_equal()
function to compare pthread_t types, e.g.:

if (pthread_equal(SSD->id, u_init.t) {
     ...
}

And one more thing that I can mention. In pthreads-win32 you can call
pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
a living thread. It will return ESRCH if invalid. However this is not
portable and therefore not safe but can sometimes be better than
nothing. It probably also works for Solaris and works here because we
can guarantee that threadID is a unique value within the process scope
and we can determine all of the previous values of living and dead
threadIDs. (This is not absolutely strictly true of course but is true
within the practical lifetimes of processes.)

FAQ
===
Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
like it is for other POSIX threads implementations?
----

Originally pthread_t was defined as a pointer (to the opaque pthread_t_
struct) and later it was changed to a struct containing the original
pointer plus a sequence counter. This is allowed under both the original
POSIX Threads Standard and the current Single Unix Specification.

When pthread_t is a simple pointer to a struct some very difficult to
debug problems arise from the process of freeing and later allocing
thread structs because new pthread_t handles can acquire the identity of
previously detached threads. The change to a struct was made, along with
some changes to their internal managment, in order to guarantee (for
practical applications) that the pthread_t handle will be unique over the
life of the running process.

Where application code attempts to compare one pthread_t against another
directly, a compiler error will be emitted because structs can't be
compared at that level. This should signal a potentially serious problem
in the code design, which would go undetected if pthread_t was a scalar.

The POSIX Threading API provides a function named pthread_equal() to
compare pthread_t thread handles.

Other pthreads implementations, such as Sun's, use an int as the handle
but do guarantee uniqueness within the process scope. Win32 scalar typed
thread handles also guarantee uniqueness in system scope. It wasn't clear
how well the internal management of these handles would scale as the
number of threads and the fragmentation of the sequence numbering
increased for applications where thousands or millions of threads are
created and detached over time. The current management of threads within
pthreads-win32 using structs for pthread_t, and reusing without ever
freeing them, reduces the management time overheads to a constant, which
could be important given that pthreads-win32 threads are built on top of
Win32 threads and will therefore include that management overhead on top
of their own. The cost is that the memory resources used for thread
handles will remain at the peak level until the process exits.

While it may be inconvenient for developers to be forced away from making
assumptions about the internals of pthread_t, the advantage for the
future development of pthread-win32, as well as those applications that
use it and other pthread implementations, is that the library is free to
change pthread_t internals and management as better methods arise.


On 5/02/2011 1:35 AM, Claude LALYRE wrote:
 > Hi Ross,
 >
 > I would like to thank a lot your pthreads-win32 team for the great
work they achieved.
 > This week, I was in a situation of migrating UNIX source code to
Windows environment.
 > And helpfully with your project phreads-win32 that task was easily
possible.
 >
 > But I encountered some compilation issues. And I have had to declare
some missing
 > typedef and macros in my code, picked from cygwin header files. And
surprisingly
 > it was enough for my code being able to compile.
 >
 > So as it was just a small issue, I thought I should give you my point
of view and
 > the little declarations I made. I think that it should be easily
integrated in your
 > source code. Just have a look at the posix.h attached file.
 >
 > Another point is concerning the declaration of your type "pthread_t".
In all UNIX
 > platforms this is formerly a pointer, but in your Windows
implementation this
 > is a struct object. The issue is that I was given a source file
containing a SSD
 > object containing a field "id" of type pthread_t.
 >
 > struct SSD {
 >    pthread_t id;
 >    int dummy;
 > }
 >
 > And somewhere else in the code they gave me, I have this
 > SSD->id = 0;
 >
 > And that line of code was not accepted by cl.exe (Windows) compiler !
 > So I face the situation by adding this ugly fix
 >
 > #ifdef WIN32
 > SSD->id.p = 0
 > #else /* WIN32 */
 > SSD->id = 0;
 > #endif /* WIN32 */
 >
 > So I am sorry to tell you this about the most basic type of your
pthreads-win32 library,
 > but it would have been great to keep the pthread_t type as a pointer
rather than a struct
 > object. However, as I managed to fix this situation its a tiny issue, an
 > enhancement suggestion rather than a bug...
 >
 > Thank a lot for all youy great job !
 >
 > Claude.
 >
 >
 >
 >

Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

Alexander Terekhov-2
Ross Johnson <[hidden email]> wrote:
[...]
> You are right that all Unixes (that I know of) define
> pthread_t as a scalar type and that pthread_win32
> deviates from this convention by using a struct. The
> projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.

Mainframe UNIX

http://en.wikipedia.org/wiki/Single_UNIX_Specification#z.2FOS

also uses a struct.

Unfortunately pthreads do not provide standard

pthread_null/compare/hash

interfaces, see

http://tinyurl.com/buj9

but you might want to add _np() functions...

regards,
alexander.


Ross Johnson <[hidden email]>@sourceware.org on 05.02.2011
09:51:35

Sent by:    [hidden email]


To:    [hidden email]
cc:
Subject:    Re: Compilation issue with pthreads-win32


Hi Claude,

Glad you found the library useful and thankyou for the feedback.

You are right that all Unixes (that I know of) define pthread_t as a
scalar type and that pthread_win32 deviates from this convention by
using a struct. The projects FAQ offers a history and explanation in
Question 11, which I've quoted at the end.

IIRC Solaris does not use a pointer but uses an int type that sequences
to provide a unique id for each new thread in a process. I have no idea
how Solaris maps this counting value to thread storage when needed nor
how it does it efficiently when the set of living threads becomes
sparse, but they do it. Linux and BSD use pointers which are not unique
if the thread exits and its memory has been allocated to a new
pthread_t, or any other type for that matter. I know from comments in
their code that the BSD developers have thought about this issue and the
possibility of changing away from a pointer type at some point.

Having done so ourselves we no longer get questions about the many
complex problems that arise when using pointers. We have had a few
questions like yours to do with porting, all three of which have been
fairly easily solved AFAIK. I take the blame for the decision but I
thought it was better to provide application reliability, predictability
etc. for everyone and accept the occasional but fixable compiler breakage.

But the crux of it is this: the POSIX (and now the SUSv3) standard
allows pthread_t to be scalar or non-scalar and in making it non-scalar
in pthreads-win we are not just taking advantage of a loophole in the
standard; the standard has deliberately not defined pthread_t so that
implementations can define it how they see fit. The notes within the
standard actually suggest defining pthread_t as a struct exactly as it
is defined here, to allow inclusion of a 'sequence' counter to render
the handle unique over time.

But back to your code:
What would happen if you did not set SSD->id = 0, i.e. just leave it
unitialised? I'm curious because your code appears not to attempt a
comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
you would see a compiler error attempting to compare a struct.

If you can't avoid initialising a pthread_t variable, I would suggest
doing it by declaring a special pthread_t constant with the value you
want (0 in this case), e.g.:

typedef union {
     pthread_t t;
     int filler[sizeof(pthread_t)/sizeof(int)];
} init_t;

const init_t u_init = {.filler = {0}}; # Relies on having a C99
compliant compiler

Then you can do this to initialise:

SSD->id = u_init.t;

Note that the initialisation of the array u_init.filler only sets the
first element to 0 explicitly and the remaining elements, if any, are
set to 0 by default. Since you don't really know how many elements there
are you should probably avoid trying to initialise more than one
element, i.e. don't do " = {.filler = {0 , 0}};".

This method means you don't need to break the opacity of the pthread_t
and it should also be portable.

Also for portability, you should only ever use the pthread_equal()
function to compare pthread_t types, e.g.:

if (pthread_equal(SSD->id, u_init.t) {
     ...
}

And one more thing that I can mention. In pthreads-win32 you can call
pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
a living thread. It will return ESRCH if invalid. However this is not
portable and therefore not safe but can sometimes be better than
nothing. It probably also works for Solaris and works here because we
can guarantee that threadID is a unique value within the process scope
and we can determine all of the previous values of living and dead
threadIDs. (This is not absolutely strictly true of course but is true
within the practical lifetimes of processes.)

FAQ
===
Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
like it is for other POSIX threads implementations?
----

Originally pthread_t was defined as a pointer (to the opaque pthread_t_
struct) and later it was changed to a struct containing the original
pointer plus a sequence counter. This is allowed under both the original
POSIX Threads Standard and the current Single Unix Specification.

When pthread_t is a simple pointer to a struct some very difficult to
debug problems arise from the process of freeing and later allocing
thread structs because new pthread_t handles can acquire the identity of
previously detached threads. The change to a struct was made, along with
some changes to their internal managment, in order to guarantee (for
practical applications) that the pthread_t handle will be unique over the
life of the running process.

Where application code attempts to compare one pthread_t against another
directly, a compiler error will be emitted because structs can't be
compared at that level. This should signal a potentially serious problem
in the code design, which would go undetected if pthread_t was a scalar.

The POSIX Threading API provides a function named pthread_equal() to
compare pthread_t thread handles.

Other pthreads implementations, such as Sun's, use an int as the handle
but do guarantee uniqueness within the process scope. Win32 scalar typed
thread handles also guarantee uniqueness in system scope. It wasn't clear
how well the internal management of these handles would scale as the
number of threads and the fragmentation of the sequence numbering
increased for applications where thousands or millions of threads are
created and detached over time. The current management of threads within
pthreads-win32 using structs for pthread_t, and reusing without ever
freeing them, reduces the management time overheads to a constant, which
could be important given that pthreads-win32 threads are built on top of
Win32 threads and will therefore include that management overhead on top
of their own. The cost is that the memory resources used for thread
handles will remain at the peak level until the process exits.

While it may be inconvenient for developers to be forced away from making
assumptions about the internals of pthread_t, the advantage for the
future development of pthread-win32, as well as those applications that
use it and other pthread implementations, is that the library is free to
change pthread_t internals and management as better methods arise.


On 5/02/2011 1:35 AM, Claude LALYRE wrote:
 > Hi Ross,
 >
 > I would like to thank a lot your pthreads-win32 team for the great
work they achieved.
 > This week, I was in a situation of migrating UNIX source code to
Windows environment.
 > And helpfully with your project phreads-win32 that task was easily
possible.
 >
 > But I encountered some compilation issues. And I have had to declare
some missing
 > typedef and macros in my code, picked from cygwin header files. And
surprisingly
 > it was enough for my code being able to compile.
 >
 > So as it was just a small issue, I thought I should give you my point
of view and
 > the little declarations I made. I think that it should be easily
integrated in your
 > source code. Just have a look at the posix.h attached file.
 >
 > Another point is concerning the declaration of your type "pthread_t".
In all UNIX
 > platforms this is formerly a pointer, but in your Windows
implementation this
 > is a struct object. The issue is that I was given a source file
containing a SSD
 > object containing a field "id" of type pthread_t.
 >
 > struct SSD {
 >    pthread_t id;
 >    int dummy;
 > }
 >
 > And somewhere else in the code they gave me, I have this
 > SSD->id = 0;
 >
 > And that line of code was not accepted by cl.exe (Windows) compiler !
 > So I face the situation by adding this ugly fix
 >
 > #ifdef WIN32
 > SSD->id.p = 0
 > #else /* WIN32 */
 > SSD->id = 0;
 > #endif /* WIN32 */
 >
 > So I am sorry to tell you this about the most basic type of your
pthreads-win32 library,
 > but it would have been great to keep the pthread_t type as a pointer
rather than a struct
 > object. However, as I managed to fix this situation its a tiny issue, an
 > enhancement suggestion rather than a bug...
 >
 > Thank a lot for all youy great job !
 >
 > Claude.
 >
 >
 >
 >



Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

Lubashev, Igor
In reply to this post by Ross Johnson-2
Why is this an issue?

If some code wants to have treads comperable/hashable, they can roll their own class/struct that wraps pthread_t and provides whatever functionality it wants.

In fact, for the simple requirment of isNull, compare, and hash, just use a (pthread_t *) for a thread handle!


- Igor Lubashev

-----Original Message-----
From: Alexander Terekhov [[hidden email]]
Received: Saturday, 05 Feb 2011, 10:18am
To: Ross Johnson [[hidden email]]
CC: [hidden email] [[hidden email]]
Subject: Re: Compilation issue with pthreads-win32

Ross Johnson <[hidden email]> wrote:
[...]
> You are right that all Unixes (that I know of) define
> pthread_t as a scalar type and that pthread_win32
> deviates from this convention by using a struct. The
> projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.

Mainframe UNIX

http://en.wikipedia.org/wiki/Single_UNIX_Specification#z.2FOS

also uses a struct.

Unfortunately pthreads do not provide standard

pthread_null/compare/hash

interfaces, see

http://tinyurl.com/buj9

but you might want to add _np() functions...

regards,
alexander.


Ross Johnson <[hidden email]>@sourceware.org on 05.02.2011
09:51:35

Sent by:    [hidden email]


To:    [hidden email]
cc:
Subject:    Re: Compilation issue with pthreads-win32


Hi Claude,

Glad you found the library useful and thankyou for the feedback.

You are right that all Unixes (that I know of) define pthread_t as a
scalar type and that pthread_win32 deviates from this convention by
using a struct. The projects FAQ offers a history and explanation in
Question 11, which I've quoted at the end.

IIRC Solaris does not use a pointer but uses an int type that sequences
to provide a unique id for each new thread in a process. I have no idea
how Solaris maps this counting value to thread storage when needed nor
how it does it efficiently when the set of living threads becomes
sparse, but they do it. Linux and BSD use pointers which are not unique
if the thread exits and its memory has been allocated to a new
pthread_t, or any other type for that matter. I know from comments in
their code that the BSD developers have thought about this issue and the
possibility of changing away from a pointer type at some point.

Having done so ourselves we no longer get questions about the many
complex problems that arise when using pointers. We have had a few
questions like yours to do with porting, all three of which have been
fairly easily solved AFAIK. I take the blame for the decision but I
thought it was better to provide application reliability, predictability
etc. for everyone and accept the occasional but fixable compiler breakage.

But the crux of it is this: the POSIX (and now the SUSv3) standard
allows pthread_t to be scalar or non-scalar and in making it non-scalar
in pthreads-win we are not just taking advantage of a loophole in the
standard; the standard has deliberately not defined pthread_t so that
implementations can define it how they see fit. The notes within the
standard actually suggest defining pthread_t as a struct exactly as it
is defined here, to allow inclusion of a 'sequence' counter to render
the handle unique over time.

But back to your code:
What would happen if you did not set SSD->id = 0, i.e. just leave it
unitialised? I'm curious because your code appears not to attempt a
comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
you would see a compiler error attempting to compare a struct.

If you can't avoid initialising a pthread_t variable, I would suggest
doing it by declaring a special pthread_t constant with the value you
want (0 in this case), e.g.:

typedef union {
     pthread_t t;
     int filler[sizeof(pthread_t)/sizeof(int)];
} init_t;

const init_t u_init = {.filler = {0}}; # Relies on having a C99
compliant compiler

Then you can do this to initialise:

SSD->id = u_init.t;

Note that the initialisation of the array u_init.filler only sets the
first element to 0 explicitly and the remaining elements, if any, are
set to 0 by default. Since you don't really know how many elements there
are you should probably avoid trying to initialise more than one
element, i.e. don't do " = {.filler = {0 , 0}};".

This method means you don't need to break the opacity of the pthread_t
and it should also be portable.

Also for portability, you should only ever use the pthread_equal()
function to compare pthread_t types, e.g.:

if (pthread_equal(SSD->id, u_init.t) {
     ...
}

And one more thing that I can mention. In pthreads-win32 you can call
pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
a living thread. It will return ESRCH if invalid. However this is not
portable and therefore not safe but can sometimes be better than
nothing. It probably also works for Solaris and works here because we
can guarantee that threadID is a unique value within the process scope
and we can determine all of the previous values of living and dead
threadIDs. (This is not absolutely strictly true of course but is true
within the practical lifetimes of processes.)

FAQ
===
Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
like it is for other POSIX threads implementations?
----

Originally pthread_t was defined as a pointer (to the opaque pthread_t_
struct) and later it was changed to a struct containing the original
pointer plus a sequence counter. This is allowed under both the original
POSIX Threads Standard and the current Single Unix Specification.

When pthread_t is a simple pointer to a struct some very difficult to
debug problems arise from the process of freeing and later allocing
thread structs because new pthread_t handles can acquire the identity of
previously detached threads. The change to a struct was made, along with
some changes to their internal managment, in order to guarantee (for
practical applications) that the pthread_t handle will be unique over the
life of the running process.

Where application code attempts to compare one pthread_t against another
directly, a compiler error will be emitted because structs can't be
compared at that level. This should signal a potentially serious problem
in the code design, which would go undetected if pthread_t was a scalar.

The POSIX Threading API provides a function named pthread_equal() to
compare pthread_t thread handles.

Other pthreads implementations, such as Sun's, use an int as the handle
but do guarantee uniqueness within the process scope. Win32 scalar typed
thread handles also guarantee uniqueness in system scope. It wasn't clear
how well the internal management of these handles would scale as the
number of threads and the fragmentation of the sequence numbering
increased for applications where thousands or millions of threads are
created and detached over time. The current management of threads within
pthreads-win32 using structs for pthread_t, and reusing without ever
freeing them, reduces the management time overheads to a constant, which
could be important given that pthreads-win32 threads are built on top of
Win32 threads and will therefore include that management overhead on top
of their own. The cost is that the memory resources used for thread
handles will remain at the peak level until the process exits.

While it may be inconvenient for developers to be forced away from making
assumptions about the internals of pthread_t, the advantage for the
future development of pthread-win32, as well as those applications that
use it and other pthread implementations, is that the library is free to
change pthread_t internals and management as better methods arise.


On 5/02/2011 1:35 AM, Claude LALYRE wrote:
 > Hi Ross,
 >
 > I would like to thank a lot your pthreads-win32 team for the great
work they achieved.
 > This week, I was in a situation of migrating UNIX source code to
Windows environment.
 > And helpfully with your project phreads-win32 that task was easily
possible.
 >
 > But I encountered some compilation issues. And I have had to declare
some missing
 > typedef and macros in my code, picked from cygwin header files. And
surprisingly
 > it was enough for my code being able to compile.
 >
 > So as it was just a small issue, I thought I should give you my point
of view and
 > the little declarations I made. I think that it should be easily
integrated in your
 > source code. Just have a look at the posix.h attached file.
 >
 > Another point is concerning the declaration of your type "pthread_t".
In all UNIX
 > platforms this is formerly a pointer, but in your Windows
implementation this
 > is a struct object. The issue is that I was given a source file
containing a SSD
 > object containing a field "id" of type pthread_t.
 >
 > struct SSD {
 >    pthread_t id;
 >    int dummy;
 > }
 >
 > And somewhere else in the code they gave me, I have this
 > SSD->id = 0;
 >
 > And that line of code was not accepted by cl.exe (Windows) compiler !
 > So I face the situation by adding this ugly fix
 >
 > #ifdef WIN32
 > SSD->id.p = 0
 > #else /* WIN32 */
 > SSD->id = 0;
 > #endif /* WIN32 */
 >
 > So I am sorry to tell you this about the most basic type of your
pthreads-win32 library,
 > but it would have been great to keep the pthread_t type as a pointer
rather than a struct
 > object. However, as I managed to fix this situation its a tiny issue, an
 > enhancement suggestion rather than a bug...
 >
 > Thank a lot for all youy great job !
 >
 > Claude.
 >
 >
 >
 >




Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

Ross Johnson-2
In reply to this post by Ross Johnson-2
Yes, you can role your own. There's an example of a usable compare
function here:

http://sourceware.org/ml/pthreads-win32/2009/msg00023.html

But if you are comparing/sorting/hashing handles then, whatever you use
to represent a thread, it will be much less problematic if it represents
that thread and only that thread for the life of the process, i.e. even
after the thread exits.

On 6/02/2011 4:17 AM, Lubashev, Igor wrote:

> Why is this an issue?
>
> If some code wants to have treads comperable/hashable, they can roll
> their own class/struct that wraps pthread_t and provides whatever
> functionality it wants.
>
> In fact, for the simple requirment of isNull, compare, and hash, just
> use a (pthread_t *) for a thread handle!
>
> - Igor
>
> - Igor Lubashev
>
> -----Original Message-----
> *From:* Alexander Terekhov [[hidden email]]
> *Received:* Saturday, 05 Feb 2011, 10:18am
> *To:* Ross Johnson [[hidden email]]
> *CC:* [hidden email] [[hidden email]]
> *Subject:* Re: Compilation issue with pthreads-win32
>
> Ross Johnson <[hidden email]> wrote:
> [...]
> > You are right that all Unixes (that I know of) define
> > pthread_t as a scalar type and that pthread_win32
> > deviates from this convention by using a struct. The
> > projects FAQ offers a history and explanation in
> > Question 11, which I've quoted at the end.
>
> Mainframe UNIX
>
> http://en.wikipedia.org/wiki/Single_UNIX_Specification#z.2FOS
>
> also uses a struct.
>
> Unfortunately pthreads do not provide standard
>
> pthread_null/compare/hash
>
> interfaces, see
>
> http://tinyurl.com/buj9
>
> but you might want to add _np() functions...
>
> regards,
> alexander.
>
>
> Ross Johnson <[hidden email]>@sourceware.org on 05.02.2011
> 09:51:35
>
> Sent by: [hidden email]
>
>
> To: [hidden email]
> cc:
> Subject:    Re: Compilation issue with pthreads-win32
>
>
> Hi Claude,
>
> Glad you found the library useful and thankyou for the feedback.
>
> You are right that all Unixes (that I know of) define pthread_t as a
> scalar type and that pthread_win32 deviates from this convention by
> using a struct. The projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.
>
> IIRC Solaris does not use a pointer but uses an int type that sequences
> to provide a unique id for each new thread in a process. I have no idea
> how Solaris maps this counting value to thread storage when needed nor
> how it does it efficiently when the set of living threads becomes
> sparse, but they do it. Linux and BSD use pointers which are not unique
> if the thread exits and its memory has been allocated to a new
> pthread_t, or any other type for that matter. I know from comments in
> their code that the BSD developers have thought about this issue and the
> possibility of changing away from a pointer type at some point.
>
> Having done so ourselves we no longer get questions about the many
> complex problems that arise when using pointers. We have had a few
> questions like yours to do with porting, all three of which have been
> fairly easily solved AFAIK. I take the blame for the decision but I
> thought it was better to provide application reliability, predictability
> etc. for everyone and accept the occasional but fixable compiler breakage.
>
> But the crux of it is this: the POSIX (and now the SUSv3) standard
> allows pthread_t to be scalar or non-scalar and in making it non-scalar
> in pthreads-win we are not just taking advantage of a loophole in the
> standard; the standard has deliberately not defined pthread_t so that
> implementations can define it how they see fit. The notes within the
> standard actually suggest defining pthread_t as a struct exactly as it
> is defined here, to allow inclusion of a 'sequence' counter to render
> the handle unique over time.
>
> But back to your code:
> What would happen if you did not set SSD->id = 0, i.e. just leave it
> unitialised? I'm curious because your code appears not to attempt a
> comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
> you would see a compiler error attempting to compare a struct.
>
> If you can't avoid initialising a pthread_t variable, I would suggest
> doing it by declaring a special pthread_t constant with the value you
> want (0 in this case), e.g.:
>
> typedef union {
>      pthread_t t;
>      int filler[sizeof(pthread_t)/sizeof(int)];
> } init_t;
>
> const init_t u_init = {.filler = {0}}; # Relies on having a C99
> compliant compiler
>
> Then you can do this to initialise:
>
> SSD->id = u_init.t;
>
> Note that the initialisation of the array u_init.filler only sets the
> first element to 0 explicitly and the remaining elements, if any, are
> set to 0 by default. Since you don't really know how many elements there
> are you should probably avoid trying to initialise more than one
> element, i.e. don't do " = {.filler = {0 , 0}};".
>
> This method means you don't need to break the opacity of the pthread_t
> and it should also be portable.
>
> Also for portability, you should only ever use the pthread_equal()
> function to compare pthread_t types, e.g.:
>
> if (pthread_equal(SSD->id, u_init.t) {
>      ...
> }
>
> And one more thing that I can mention. In pthreads-win32 you can call
> pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
> a living thread. It will return ESRCH if invalid. However this is not
> portable and therefore not safe but can sometimes be better than
> nothing. It probably also works for Solaris and works here because we
> can guarantee that threadID is a unique value within the process scope
> and we can determine all of the previous values of living and dead
> threadIDs. (This is not absolutely strictly true of course but is true
> within the practical lifetimes of processes.)
>
> FAQ
> ===
> Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
> like it is for other POSIX threads implementations?
> ----
>
> Originally pthread_t was defined as a pointer (to the opaque pthread_t_
> struct) and later it was changed to a struct containing the original
> pointer plus a sequence counter. This is allowed under both the original
> POSIX Threads Standard and the current Single Unix Specification.
>
> When pthread_t is a simple pointer to a struct some very difficult to
> debug problems arise from the process of freeing and later allocing
> thread structs because new pthread_t handles can acquire the identity of
> previously detached threads. The change to a struct was made, along with
> some changes to their internal managment, in order to guarantee (for
> practical applications) that the pthread_t handle will be unique over the
> life of the running process.
>
> Where application code attempts to compare one pthread_t against another
> directly, a compiler error will be emitted because structs can't be
> compared at that level. This should signal a potentially serious problem
> in the code design, which would go undetected if pthread_t was a scalar.
>
> The POSIX Threading API provides a function named pthread_equal() to
> compare pthread_t thread handles.
>
> Other pthreads implementations, such as Sun's, use an int as the handle
> but do guarantee uniqueness within the process scope. Win32 scalar typed
> thread handles also guarantee uniqueness in system scope. It wasn't clear
> how well the internal management of these handles would scale as the
> number of threads and the fragmentation of the sequence numbering
> increased for applications where thousands or millions of threads are
> created and detached over time. The current management of threads within
> pthreads-win32 using structs for pthread_t, and reusing without ever
> freeing them, reduces the management time overheads to a constant, which
> could be important given that pthreads-win32 threads are built on top of
> Win32 threads and will therefore include that management overhead on top
> of their own. The cost is that the memory resources used for thread
> handles will remain at the peak level until the process exits.
>
> While it may be inconvenient for developers to be forced away from making
> assumptions about the internals of pthread_t, the advantage for the
> future development of pthread-win32, as well as those applications that
> use it and other pthread implementations, is that the library is free to
> change pthread_t internals and management as better methods arise.
>
>
> On 5/02/2011 1:35 AM, Claude LALYRE wrote:
> > Hi Ross,
> >
> > I would like to thank a lot your pthreads-win32 team for the great
> work they achieved.
> > This week, I was in a situation of migrating UNIX source code to
> Windows environment.
> > And helpfully with your project phreads-win32 that task was easily
> possible.
> >
> > But I encountered some compilation issues. And I have had to declare
> some missing
> > typedef and macros in my code, picked from cygwin header files. And
> surprisingly
> > it was enough for my code being able to compile.
> >
> > So as it was just a small issue, I thought I should give you my point
> of view and
> > the little declarations I made. I think that it should be easily
> integrated in your
> > source code. Just have a look at the posix.h attached file.
> >
> > Another point is concerning the declaration of your type "pthread_t".
> In all UNIX
> > platforms this is formerly a pointer, but in your Windows
> implementation this
> > is a struct object. The issue is that I was given a source file
> containing a SSD
> > object containing a field "id" of type pthread_t.
> >
> > struct SSD {
> >    pthread_t id;
> >    int dummy;
> > }
> >
> > And somewhere else in the code they gave me, I have this
> > SSD->id = 0;
> >
> > And that line of code was not accepted by cl.exe (Windows) compiler !
> > So I face the situation by adding this ugly fix
> >
> > #ifdef WIN32
> > SSD->id.p = 0
> > #else /* WIN32 */
> > SSD->id = 0;
> > #endif /* WIN32 */
> >
> > So I am sorry to tell you this about the most basic type of your
> pthreads-win32 library,
> > but it would have been great to keep the pthread_t type as a pointer
> rather than a struct
> > object. However, as I managed to fix this situation its a tiny issue, an
> > enhancement suggestion rather than a bug...
> >
> > Thank a lot for all youy great job !
> >
> > Claude.
> >
> >
> >
> >
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

John Bossom
In reply to this post by Ross Johnson-2

HPUX's DCE Threads (which was an implementation of the draft
version of the standard) used a struct. I'll check out Hpux 11.x when  
I get into work to see is the final version followed suit.

Regardless, the standard specifically allows
the flexibility to implement the library either way.
That is why it is pass-by-ref on the API and the
reason for methods such as pthread_equal

John E. Bossom


Quoting Ross Johnson <[hidden email]>:

> Hi Claude,
>
> Glad you found the library useful and thankyou for the feedback.
>
> You are right that all Unixes (that I know of) define pthread_t as a
> scalar type and that pthread_win32 deviates from this convention by
> using a struct. The projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.
>
> IIRC Solaris does not use a pointer but uses an int type that sequences
> to provide a unique id for each new thread in a process. I have no idea
> how Solaris maps this counting value to thread storage when needed nor
> how it does it efficiently when the set of living threads becomes
> sparse, but they do it. Linux and BSD use pointers which are not unique
> if the thread exits and its memory has been allocated to a new
> pthread_t, or any other type for that matter. I know from comments in
> their code that the BSD developers have thought about this issue and
> the possibility of changing away from a pointer type at some point.
>
> Having done so ourselves we no longer get questions about the many
> complex problems that arise when using pointers. We have had a few
> questions like yours to do with porting, all three of which have been
> fairly easily solved AFAIK. I take the blame for the decision but I
> thought it was better to provide application reliability,
> predictability etc. for everyone and accept the occasional but fixable
> compiler breakage.
>
> But the crux of it is this: the POSIX (and now the SUSv3) standard
> allows pthread_t to be scalar or non-scalar and in making it non-scalar
> in pthreads-win we are not just taking advantage of a loophole in the
> standard; the standard has deliberately not defined pthread_t so that
> implementations can define it how they see fit. The notes within the
> standard actually suggest defining pthread_t as a struct exactly as it
> is defined here, to allow inclusion of a 'sequence' counter to render
> the handle unique over time.
>
> But back to your code:
> What would happen if you did not set SSD->id = 0, i.e. just leave it
> unitialised? I'm curious because your code appears not to attempt a
> comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
> you would see a compiler error attempting to compare a struct.
>
> If you can't avoid initialising a pthread_t variable, I would suggest
> doing it by declaring a special pthread_t constant with the value you
> want (0 in this case), e.g.:
>
> typedef union {
>     pthread_t t;
>     int filler[sizeof(pthread_t)/sizeof(int)];
> } init_t;
>
> const init_t u_init = {.filler = {0}}; # Relies on having a C99
> compliant compiler
>
> Then you can do this to initialise:
>
> SSD->id = u_init.t;
>
> Note that the initialisation of the array u_init.filler only sets the
> first element to 0 explicitly and the remaining elements, if any, are
> set to 0 by default. Since you don't really know how many elements
> there are you should probably avoid trying to initialise more than one
> element, i.e. don't do " = {.filler = {0 , 0}};".
>
> This method means you don't need to break the opacity of the pthread_t
> and it should also be portable.
>
> Also for portability, you should only ever use the pthread_equal()
> function to compare pthread_t types, e.g.:
>
> if (pthread_equal(SSD->id, u_init.t) {
>     ...
> }
>
> And one more thing that I can mention. In pthreads-win32 you can call
> pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
> a living thread. It will return ESRCH if invalid. However this is not
> portable and therefore not safe but can sometimes be better than
> nothing. It probably also works for Solaris and works here because we
> can guarantee that threadID is a unique value within the process scope
> and we can determine all of the previous values of living and dead
> threadIDs. (This is not absolutely strictly true of course but is true
> within the practical lifetimes of processes.)
>
> FAQ
> ===
> Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
> like it is for other POSIX threads implementations?
> ----
>
> Originally pthread_t was defined as a pointer (to the opaque pthread_t_
> struct) and later it was changed to a struct containing the original
> pointer plus a sequence counter. This is allowed under both the original
> POSIX Threads Standard and the current Single Unix Specification.
>
> When pthread_t is a simple pointer to a struct some very difficult to
> debug problems arise from the process of freeing and later allocing
> thread structs because new pthread_t handles can acquire the identity of
> previously detached threads. The change to a struct was made, along with
> some changes to their internal managment, in order to guarantee (for
> practical applications) that the pthread_t handle will be unique over the
> life of the running process.
>
> Where application code attempts to compare one pthread_t against another
> directly, a compiler error will be emitted because structs can't be
> compared at that level. This should signal a potentially serious problem
> in the code design, which would go undetected if pthread_t was a scalar.
>
> The POSIX Threading API provides a function named pthread_equal() to
> compare pthread_t thread handles.
>
> Other pthreads implementations, such as Sun's, use an int as the handle
> but do guarantee uniqueness within the process scope. Win32 scalar typed
> thread handles also guarantee uniqueness in system scope. It wasn't clear
> how well the internal management of these handles would scale as the
> number of threads and the fragmentation of the sequence numbering
> increased for applications where thousands or millions of threads are
> created and detached over time. The current management of threads within
> pthreads-win32 using structs for pthread_t, and reusing without ever
> freeing them, reduces the management time overheads to a constant, which
> could be important given that pthreads-win32 threads are built on top of
> Win32 threads and will therefore include that management overhead on top
> of their own. The cost is that the memory resources used for thread
> handles will remain at the peak level until the process exits.
>
> While it may be inconvenient for developers to be forced away from making
> assumptions about the internals of pthread_t, the advantage for the
> future development of pthread-win32, as well as those applications that
> use it and other pthread implementations, is that the library is free to
> change pthread_t internals and management as better methods arise.
>
>
> On 5/02/2011 1:35 AM, Claude LALYRE wrote:
>> Hi Ross,
>>
>> I would like to thank a lot your pthreads-win32 team for the great
> work they achieved.
>> This week, I was in a situation of migrating UNIX source code to
> Windows environment.
>> And helpfully with your project phreads-win32 that task was easily possible.
>>
>> But I encountered some compilation issues. And I have had to declare
> some missing
>> typedef and macros in my code, picked from cygwin header files. And
> surprisingly
>> it was enough for my code being able to compile.
>>
>> So as it was just a small issue, I thought I should give you my point
> of view and
>> the little declarations I made. I think that it should be easily
> integrated in your
>> source code. Just have a look at the posix.h attached file.
>>
>> Another point is concerning the declaration of your type "pthread_t".
> In all UNIX
>> platforms this is formerly a pointer, but in your Windows
> implementation this
>> is a struct object. The issue is that I was given a source file
> containing a SSD
>> object containing a field "id" of type pthread_t.
>>
>> struct SSD {
>>    pthread_t id;
>>    int dummy;
>> }
>>
>> And somewhere else in the code they gave me, I have this
>> SSD->id = 0;
>>
>> And that line of code was not accepted by cl.exe (Windows) compiler !
>> So I face the situation by adding this ugly fix
>>
>> #ifdef WIN32
>> SSD->id.p = 0
>> #else /* WIN32 */
>> SSD->id = 0;
>> #endif /* WIN32 */
>>
>> So I am sorry to tell you this about the most basic type of your
> pthreads-win32 library,
>> but it would have been great to keep the pthread_t type as a pointer
> rather than a struct
>> object. However, as I managed to fix this situation its a tiny issue, an
>> enhancement suggestion rather than a bug...
>>
>> Thank a lot for all youy great job !
>>
>> Claude.
>>
>>
>>
>>


Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

Alexander Terekhov-2
In reply to this post by Lubashev, Igor
"Lubashev, Igor" <[hidden email]> wrote:
>
> Why is this an issue?
>
> If some code wants to have treads comperable/hashable, they can roll
> their own class/struct that wraps pthread_t and provides whatever
> functionality it wants.

You probably mean something along the lines of

http://terekhov.de/mythread.c

which is turning pthread_t into a pointer. Well, that's way to
invasive to begin with.

>
> In fact, for the simple requirment of isNull, compare, and hash,
> just use a (pthread_t *) for a thread handle!

It is not that simple. OTOH it is really simple to provide native
null, compare, and hash interfaces for pthread_t.

regards,
alexander.


"Lubashev, Igor" <[hidden email]>@sourceware.org on 05.02.2011
18:30:03

Sent by:    [hidden email]


To:
cc:    "[hidden email]" <[hidden email]>
Subject:    Re: Compilation issue with pthreads-win32


Why is this an issue?

If some code wants to have treads comperable/hashable, they can roll their
own class/struct that wraps pthread_t and provides whatever functionality
it wants.

In fact, for the simple requirment of isNull, compare, and hash, just use a
(pthread_t *) for a thread handle!


- Igor Lubashev

-----Original Message-----
From: Alexander Terekhov [[hidden email]]
Received: Saturday, 05 Feb 2011, 10:18am
To: Ross Johnson [[hidden email]]
CC: [hidden email] [[hidden email]]
Subject: Re: Compilation issue with pthreads-win32

Ross Johnson <[hidden email]> wrote:
[...]
> You are right that all Unixes (that I know of) define
> pthread_t as a scalar type and that pthread_win32
> deviates from this convention by using a struct. The
> projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.

Mainframe UNIX

http://en.wikipedia.org/wiki/Single_UNIX_Specification#z.2FOS

also uses a struct.

Unfortunately pthreads do not provide standard

pthread_null/compare/hash

interfaces, see

http://tinyurl.com/buj9

but you might want to add _np() functions...

regards,
alexander.


Ross Johnson <[hidden email]>@sourceware.org on 05.02.2011
09:51:35

Sent by:    [hidden email]


To:    [hidden email]
cc:
Subject:    Re: Compilation issue with pthreads-win32


Hi Claude,

Glad you found the library useful and thankyou for the feedback.

You are right that all Unixes (that I know of) define pthread_t as a
scalar type and that pthread_win32 deviates from this convention by
using a struct. The projects FAQ offers a history and explanation in
Question 11, which I've quoted at the end.

IIRC Solaris does not use a pointer but uses an int type that sequences
to provide a unique id for each new thread in a process. I have no idea
how Solaris maps this counting value to thread storage when needed nor
how it does it efficiently when the set of living threads becomes
sparse, but they do it. Linux and BSD use pointers which are not unique
if the thread exits and its memory has been allocated to a new
pthread_t, or any other type for that matter. I know from comments in
their code that the BSD developers have thought about this issue and the
possibility of changing away from a pointer type at some point.

Having done so ourselves we no longer get questions about the many
complex problems that arise when using pointers. We have had a few
questions like yours to do with porting, all three of which have been
fairly easily solved AFAIK. I take the blame for the decision but I
thought it was better to provide application reliability, predictability
etc. for everyone and accept the occasional but fixable compiler breakage.

But the crux of it is this: the POSIX (and now the SUSv3) standard
allows pthread_t to be scalar or non-scalar and in making it non-scalar
in pthreads-win we are not just taking advantage of a loophole in the
standard; the standard has deliberately not defined pthread_t so that
implementations can define it how they see fit. The notes within the
standard actually suggest defining pthread_t as a struct exactly as it
is defined here, to allow inclusion of a 'sequence' counter to render
the handle unique over time.

But back to your code:
What would happen if you did not set SSD->id = 0, i.e. just leave it
unitialised? I'm curious because your code appears not to attempt a
comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
you would see a compiler error attempting to compare a struct.

If you can't avoid initialising a pthread_t variable, I would suggest
doing it by declaring a special pthread_t constant with the value you
want (0 in this case), e.g.:

typedef union {
     pthread_t t;
     int filler[sizeof(pthread_t)/sizeof(int)];
} init_t;

const init_t u_init = {.filler = {0}}; # Relies on having a C99
compliant compiler

Then you can do this to initialise:

SSD->id = u_init.t;

Note that the initialisation of the array u_init.filler only sets the
first element to 0 explicitly and the remaining elements, if any, are
set to 0 by default. Since you don't really know how many elements there
are you should probably avoid trying to initialise more than one
element, i.e. don't do " = {.filler = {0 , 0}};".

This method means you don't need to break the opacity of the pthread_t
and it should also be portable.

Also for portability, you should only ever use the pthread_equal()
function to compare pthread_t types, e.g.:

if (pthread_equal(SSD->id, u_init.t) {
     ...
}

And one more thing that I can mention. In pthreads-win32 you can call
pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
a living thread. It will return ESRCH if invalid. However this is not
portable and therefore not safe but can sometimes be better than
nothing. It probably also works for Solaris and works here because we
can guarantee that threadID is a unique value within the process scope
and we can determine all of the previous values of living and dead
threadIDs. (This is not absolutely strictly true of course but is true
within the practical lifetimes of processes.)

FAQ
===
Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
like it is for other POSIX threads implementations?
----

Originally pthread_t was defined as a pointer (to the opaque pthread_t_
struct) and later it was changed to a struct containing the original
pointer plus a sequence counter. This is allowed under both the original
POSIX Threads Standard and the current Single Unix Specification.

When pthread_t is a simple pointer to a struct some very difficult to
debug problems arise from the process of freeing and later allocing
thread structs because new pthread_t handles can acquire the identity of
previously detached threads. The change to a struct was made, along with
some changes to their internal managment, in order to guarantee (for
practical applications) that the pthread_t handle will be unique over the
life of the running process.

Where application code attempts to compare one pthread_t against another
directly, a compiler error will be emitted because structs can't be
compared at that level. This should signal a potentially serious problem
in the code design, which would go undetected if pthread_t was a scalar.

The POSIX Threading API provides a function named pthread_equal() to
compare pthread_t thread handles.

Other pthreads implementations, such as Sun's, use an int as the handle
but do guarantee uniqueness within the process scope. Win32 scalar typed
thread handles also guarantee uniqueness in system scope. It wasn't clear
how well the internal management of these handles would scale as the
number of threads and the fragmentation of the sequence numbering
increased for applications where thousands or millions of threads are
created and detached over time. The current management of threads within
pthreads-win32 using structs for pthread_t, and reusing without ever
freeing them, reduces the management time overheads to a constant, which
could be important given that pthreads-win32 threads are built on top of
Win32 threads and will therefore include that management overhead on top
of their own. The cost is that the memory resources used for thread
handles will remain at the peak level until the process exits.

While it may be inconvenient for developers to be forced away from making
assumptions about the internals of pthread_t, the advantage for the
future development of pthread-win32, as well as those applications that
use it and other pthread implementations, is that the library is free to
change pthread_t internals and management as better methods arise.


On 5/02/2011 1:35 AM, Claude LALYRE wrote:
 > Hi Ross,
 >
 > I would like to thank a lot your pthreads-win32 team for the great
work they achieved.
 > This week, I was in a situation of migrating UNIX source code to
Windows environment.
 > And helpfully with your project phreads-win32 that task was easily
possible.
 >
 > But I encountered some compilation issues. And I have had to declare
some missing
 > typedef and macros in my code, picked from cygwin header files. And
surprisingly
 > it was enough for my code being able to compile.
 >
 > So as it was just a small issue, I thought I should give you my point
of view and
 > the little declarations I made. I think that it should be easily
integrated in your
 > source code. Just have a look at the posix.h attached file.
 >
 > Another point is concerning the declaration of your type "pthread_t".
In all UNIX
 > platforms this is formerly a pointer, but in your Windows
implementation this
 > is a struct object. The issue is that I was given a source file
containing a SSD
 > object containing a field "id" of type pthread_t.
 >
 > struct SSD {
 >    pthread_t id;
 >    int dummy;
 > }
 >
 > And somewhere else in the code they gave me, I have this
 > SSD->id = 0;
 >
 > And that line of code was not accepted by cl.exe (Windows) compiler !
 > So I face the situation by adding this ugly fix
 >
 > #ifdef WIN32
 > SSD->id.p = 0
 > #else /* WIN32 */
 > SSD->id = 0;
 > #endif /* WIN32 */
 >
 > So I am sorry to tell you this about the most basic type of your
pthreads-win32 library,
 > but it would have been great to keep the pthread_t type as a pointer
rather than a struct
 > object. However, as I managed to fix this situation its a tiny issue, an
 > enhancement suggestion rather than a bug...
 >
 > Thank a lot for all youy great job !
 >
 > Claude.
 >
 >
 >
 >






Reply | Threaded
Open this post in threaded view
|

Re: Compilation issue with pthreads-win32

Alexander Terekhov-2
In reply to this post by Ross Johnson-2
Ross Johnson <[hidden email]> wrote:
>
> Yes, you can role your own. There's an example of a usable compare
> function here:
>
> http://sourceware.org/ml/pthreads-win32/2009/msg00023.html

Raw bits of the pthread_t representation are not helpful because
the same pthread_t value could have more than one representation
(different values in raw bits).

regards,
alexander.


Ross Johnson <[hidden email]>@sourceware.org on 06.02.2011
10:44:34

Sent by:    [hidden email]


To:    "[hidden email]" <[hidden email]>
cc:
Subject:    Re: Compilation issue with pthreads-win32


Yes, you can role your own. There's an example of a usable compare
function here:

http://sourceware.org/ml/pthreads-win32/2009/msg00023.html

But if you are comparing/sorting/hashing handles then, whatever you use
to represent a thread, it will be much less problematic if it represents
that thread and only that thread for the life of the process, i.e. even
after the thread exits.

On 6/02/2011 4:17 AM, Lubashev, Igor wrote:

> Why is this an issue?
>
> If some code wants to have treads comperable/hashable, they can roll
> their own class/struct that wraps pthread_t and provides whatever
> functionality it wants.
>
> In fact, for the simple requirment of isNull, compare, and hash, just
> use a (pthread_t *) for a thread handle!
>
> - Igor
>
> - Igor Lubashev
>
> -----Original Message-----
> *From:* Alexander Terekhov [[hidden email]]
> *Received:* Saturday, 05 Feb 2011, 10:18am
> *To:* Ross Johnson [[hidden email]]
> *CC:* [hidden email] [[hidden email]]
> *Subject:* Re: Compilation issue with pthreads-win32
>
> Ross Johnson <[hidden email]> wrote:
> [...]
> > You are right that all Unixes (that I know of) define
> > pthread_t as a scalar type and that pthread_win32
> > deviates from this convention by using a struct. The
> > projects FAQ offers a history and explanation in
> > Question 11, which I've quoted at the end.
>
> Mainframe UNIX
>
> http://en.wikipedia.org/wiki/Single_UNIX_Specification#z.2FOS
>
> also uses a struct.
>
> Unfortunately pthreads do not provide standard
>
> pthread_null/compare/hash
>
> interfaces, see
>
> http://tinyurl.com/buj9
>
> but you might want to add _np() functions...
>
> regards,
> alexander.
>
>
> Ross Johnson <[hidden email]>@sourceware.org on 05.02.2011
> 09:51:35
>
> Sent by: [hidden email]
>
>
> To: [hidden email]
> cc:
> Subject:    Re: Compilation issue with pthreads-win32
>
>
> Hi Claude,
>
> Glad you found the library useful and thankyou for the feedback.
>
> You are right that all Unixes (that I know of) define pthread_t as a
> scalar type and that pthread_win32 deviates from this convention by
> using a struct. The projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.
>
> IIRC Solaris does not use a pointer but uses an int type that sequences
> to provide a unique id for each new thread in a process. I have no idea
> how Solaris maps this counting value to thread storage when needed nor
> how it does it efficiently when the set of living threads becomes
> sparse, but they do it. Linux and BSD use pointers which are not unique
> if the thread exits and its memory has been allocated to a new
> pthread_t, or any other type for that matter. I know from comments in
> their code that the BSD developers have thought about this issue and the
> possibility of changing away from a pointer type at some point.
>
> Having done so ourselves we no longer get questions about the many
> complex problems that arise when using pointers. We have had a few
> questions like yours to do with porting, all three of which have been
> fairly easily solved AFAIK. I take the blame for the decision but I
> thought it was better to provide application reliability, predictability
> etc. for everyone and accept the occasional but fixable compiler
breakage.

>
> But the crux of it is this: the POSIX (and now the SUSv3) standard
> allows pthread_t to be scalar or non-scalar and in making it non-scalar
> in pthreads-win we are not just taking advantage of a loophole in the
> standard; the standard has deliberately not defined pthread_t so that
> implementations can define it how they see fit. The notes within the
> standard actually suggest defining pthread_t as a struct exactly as it
> is defined here, to allow inclusion of a 'sequence' counter to render
> the handle unique over time.
>
> But back to your code:
> What would happen if you did not set SSD->id = 0, i.e. just leave it
> unitialised? I'm curious because your code appears not to attempt a
> comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
> you would see a compiler error attempting to compare a struct.
>
> If you can't avoid initialising a pthread_t variable, I would suggest
> doing it by declaring a special pthread_t constant with the value you
> want (0 in this case), e.g.:
>
> typedef union {
>      pthread_t t;
>      int filler[sizeof(pthread_t)/sizeof(int)];
> } init_t;
>
> const init_t u_init = {.filler = {0}}; # Relies on having a C99
> compliant compiler
>
> Then you can do this to initialise:
>
> SSD->id = u_init.t;
>
> Note that the initialisation of the array u_init.filler only sets the
> first element to 0 explicitly and the remaining elements, if any, are
> set to 0 by default. Since you don't really know how many elements there
> are you should probably avoid trying to initialise more than one
> element, i.e. don't do " = {.filler = {0 , 0}};".
>
> This method means you don't need to break the opacity of the pthread_t
> and it should also be portable.
>
> Also for portability, you should only ever use the pthread_equal()
> function to compare pthread_t types, e.g.:
>
> if (pthread_equal(SSD->id, u_init.t) {
>      ...
> }
>
> And one more thing that I can mention. In pthreads-win32 you can call
> pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
> a living thread. It will return ESRCH if invalid. However this is not
> portable and therefore not safe but can sometimes be better than
> nothing. It probably also works for Solaris and works here because we
> can guarantee that threadID is a unique value within the process scope
> and we can determine all of the previous values of living and dead
> threadIDs. (This is not absolutely strictly true of course but is true
> within the practical lifetimes of processes.)
>
> FAQ
> ===
> Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
> like it is for other POSIX threads implementations?
> ----
>
> Originally pthread_t was defined as a pointer (to the opaque pthread_t_
> struct) and later it was changed to a struct containing the original
> pointer plus a sequence counter. This is allowed under both the original
> POSIX Threads Standard and the current Single Unix Specification.
>
> When pthread_t is a simple pointer to a struct some very difficult to
> debug problems arise from the process of freeing and later allocing
> thread structs because new pthread_t handles can acquire the identity of
> previously detached threads. The change to a struct was made, along with
> some changes to their internal managment, in order to guarantee (for
> practical applications) that the pthread_t handle will be unique over the
> life of the running process.
>
> Where application code attempts to compare one pthread_t against another
> directly, a compiler error will be emitted because structs can't be
> compared at that level. This should signal a potentially serious problem
> in the code design, which would go undetected if pthread_t was a scalar.
>
> The POSIX Threading API provides a function named pthread_equal() to
> compare pthread_t thread handles.
>
> Other pthreads implementations, such as Sun's, use an int as the handle
> but do guarantee uniqueness within the process scope. Win32 scalar typed
> thread handles also guarantee uniqueness in system scope. It wasn't clear
> how well the internal management of these handles would scale as the
> number of threads and the fragmentation of the sequence numbering
> increased for applications where thousands or millions of threads are
> created and detached over time. The current management of threads within
> pthreads-win32 using structs for pthread_t, and reusing without ever
> freeing them, reduces the management time overheads to a constant, which
> could be important given that pthreads-win32 threads are built on top of
> Win32 threads and will therefore include that management overhead on top
> of their own. The cost is that the memory resources used for thread
> handles will remain at the peak level until the process exits.
>
> While it may be inconvenient for developers to be forced away from making
> assumptions about the internals of pthread_t, the advantage for the
> future development of pthread-win32, as well as those applications that
> use it and other pthread implementations, is that the library is free to
> change pthread_t internals and management as better methods arise.
>
>
> On 5/02/2011 1:35 AM, Claude LALYRE wrote:
> > Hi Ross,
> >
> > I would like to thank a lot your pthreads-win32 team for the great
> work they achieved.
> > This week, I was in a situation of migrating UNIX source code to
> Windows environment.
> > And helpfully with your project phreads-win32 that task was easily
> possible.
> >
> > But I encountered some compilation issues. And I have had to declare
> some missing
> > typedef and macros in my code, picked from cygwin header files. And
> surprisingly
> > it was enough for my code being able to compile.
> >
> > So as it was just a small issue, I thought I should give you my point
> of view and
> > the little declarations I made. I think that it should be easily
> integrated in your
> > source code. Just have a look at the posix.h attached file.
> >
> > Another point is concerning the declaration of your type "pthread_t".
> In all UNIX
> > platforms this is formerly a pointer, but in your Windows
> implementation this
> > is a struct object. The issue is that I was given a source file
> containing a SSD
> > object containing a field "id" of type pthread_t.
> >
> > struct SSD {
> >    pthread_t id;
> >    int dummy;
> > }
> >
> > And somewhere else in the code they gave me, I have this
> > SSD->id = 0;
> >
> > And that line of code was not accepted by cl.exe (Windows) compiler !
> > So I face the situation by adding this ugly fix
> >
> > #ifdef WIN32
> > SSD->id.p = 0
> > #else /* WIN32 */
> > SSD->id = 0;
> > #endif /* WIN32 */
> >
> > So I am sorry to tell you this about the most basic type of your
> pthreads-win32 library,
> > but it would have been great to keep the pthread_t type as a pointer
> rather than a struct
> > object. However, as I managed to fix this situation its a tiny issue,
an

> > enhancement suggestion rather than a bug...
> >
> > Thank a lot for all youy great job !
> >
> > Claude.
> >
> >
> >
> >
>
>
>




Reply | Threaded
Open this post in threaded view
|

RE: Compilation issue with pthreads-win32

Lubashev, Igor
In reply to this post by Alexander Terekhov-2
> You probably mean something along the lines of
>
> http://terekhov.de/mythread.c

This is more complicated than necessary for the simple requirements of "nullable, hashable, and comparable".  But this is a good threading abstraction.

What I had in mind is "typedef pthread_t *my_thread_t;" and the code that stores thread_t in a global data structure and uses pointers to the thread_t fields in that structure.

But you are correct that a native library can provide nullable, hashable, and thread handles easily.


- Igor


P.S.

My big pet peeve with pthreads-win32 is that is breaks otherwise valid client code that merely includes <pthread.h>, because <pthread.h> defines broken strtok_r and rand_r macros!  I've been bitten by this already.  Ideally <pthread.h> would not drop anything into the global namespace that is not thread-related.




-----Original Message-----
From: Alexander Terekhov [mailto:[hidden email]]
Sent: Monday, February 07, 2011 5:41 AM
To: Lubashev, Igor
Cc: [hidden email]
Subject: Re: Compilation issue with pthreads-win32

"Lubashev, Igor" <[hidden email]> wrote:
>
> Why is this an issue?
>
> If some code wants to have treads comperable/hashable, they can roll
> their own class/struct that wraps pthread_t and provides whatever
> functionality it wants.

You probably mean something along the lines of

http://terekhov.de/mythread.c

which is turning pthread_t into a pointer. Well, that's way to
invasive to begin with.

>
> In fact, for the simple requirment of isNull, compare, and hash,
> just use a (pthread_t *) for a thread handle!

It is not that simple. OTOH it is really simple to provide native
null, compare, and hash interfaces for pthread_t.

regards,
alexander.


"Lubashev, Igor" <[hidden email]>@sourceware.org on 05.02.2011
18:30:03

Sent by:    [hidden email]


To:
cc:    "[hidden email]" <[hidden email]>
Subject:    Re: Compilation issue with pthreads-win32


Why is this an issue?

If some code wants to have treads comperable/hashable, they can roll their
own class/struct that wraps pthread_t and provides whatever functionality
it wants.

In fact, for the simple requirment of isNull, compare, and hash, just use a
(pthread_t *) for a thread handle!


- Igor Lubashev

-----Original Message-----
From: Alexander Terekhov [[hidden email]]
Received: Saturday, 05 Feb 2011, 10:18am
To: Ross Johnson [[hidden email]]
CC: [hidden email] [[hidden email]]
Subject: Re: Compilation issue with pthreads-win32

Ross Johnson <[hidden email]> wrote:
[...]
> You are right that all Unixes (that I know of) define
> pthread_t as a scalar type and that pthread_win32
> deviates from this convention by using a struct. The
> projects FAQ offers a history and explanation in
> Question 11, which I've quoted at the end.

Mainframe UNIX

http://en.wikipedia.org/wiki/Single_UNIX_Specification#z.2FOS

also uses a struct.

Unfortunately pthreads do not provide standard

pthread_null/compare/hash

interfaces, see

http://tinyurl.com/buj9

but you might want to add _np() functions...

regards,
alexander.


Ross Johnson <[hidden email]>@sourceware.org on 05.02.2011
09:51:35

Sent by:    [hidden email]


To:    [hidden email]
cc:
Subject:    Re: Compilation issue with pthreads-win32


Hi Claude,

Glad you found the library useful and thankyou for the feedback.

You are right that all Unixes (that I know of) define pthread_t as a
scalar type and that pthread_win32 deviates from this convention by
using a struct. The projects FAQ offers a history and explanation in
Question 11, which I've quoted at the end.

IIRC Solaris does not use a pointer but uses an int type that sequences
to provide a unique id for each new thread in a process. I have no idea
how Solaris maps this counting value to thread storage when needed nor
how it does it efficiently when the set of living threads becomes
sparse, but they do it. Linux and BSD use pointers which are not unique
if the thread exits and its memory has been allocated to a new
pthread_t, or any other type for that matter. I know from comments in
their code that the BSD developers have thought about this issue and the
possibility of changing away from a pointer type at some point.

Having done so ourselves we no longer get questions about the many
complex problems that arise when using pointers. We have had a few
questions like yours to do with porting, all three of which have been
fairly easily solved AFAIK. I take the blame for the decision but I
thought it was better to provide application reliability, predictability
etc. for everyone and accept the occasional but fixable compiler breakage.

But the crux of it is this: the POSIX (and now the SUSv3) standard
allows pthread_t to be scalar or non-scalar and in making it non-scalar
in pthreads-win we are not just taking advantage of a loophole in the
standard; the standard has deliberately not defined pthread_t so that
implementations can define it how they see fit. The notes within the
standard actually suggest defining pthread_t as a struct exactly as it
is defined here, to allow inclusion of a 'sequence' counter to render
the handle unique over time.

But back to your code:
What would happen if you did not set SSD->id = 0, i.e. just leave it
unitialised? I'm curious because your code appears not to attempt a
comparison of the value with 0, e.g. "if (SSD->id == 0) ..." otherwise
you would see a compiler error attempting to compare a struct.

If you can't avoid initialising a pthread_t variable, I would suggest
doing it by declaring a special pthread_t constant with the value you
want (0 in this case), e.g.:

typedef union {
     pthread_t t;
     int filler[sizeof(pthread_t)/sizeof(int)];
} init_t;

const init_t u_init = {.filler = {0}}; # Relies on having a C99
compliant compiler

Then you can do this to initialise:

SSD->id = u_init.t;

Note that the initialisation of the array u_init.filler only sets the
first element to 0 explicitly and the remaining elements, if any, are
set to 0 by default. Since you don't really know how many elements there
are you should probably avoid trying to initialise more than one
element, i.e. don't do " = {.filler = {0 , 0}};".

This method means you don't need to break the opacity of the pthread_t
and it should also be portable.

Also for portability, you should only ever use the pthread_equal()
function to compare pthread_t types, e.g.:

if (pthread_equal(SSD->id, u_init.t) {
     ...
}

And one more thing that I can mention. In pthreads-win32 you can call
pthread_kill(threadID, 0) to check if threadID is valid, i.e. refers to
a living thread. It will return ESRCH if invalid. However this is not
portable and therefore not safe but can sometimes be better than
nothing. It probably also works for Solaris and works here because we
can guarantee that threadID is a unique value within the process scope
and we can determine all of the previous values of living and dead
threadIDs. (This is not absolutely strictly true of course but is true
within the practical lifetimes of processes.)

FAQ
===
Q 11 Why isn't pthread_t defined as a scalar (e.g. pointer or int)
like it is for other POSIX threads implementations?
----

Originally pthread_t was defined as a pointer (to the opaque pthread_t_
struct) and later it was changed to a struct containing the original
pointer plus a sequence counter. This is allowed under both the original
POSIX Threads Standard and the current Single Unix Specification.

When pthread_t is a simple pointer to a struct some very difficult to
debug problems arise from the process of freeing and later allocing
thread structs because new pthread_t handles can acquire the identity of
previously detached threads. The change to a struct was made, along with
some changes to their internal managment, in order to guarantee (for
practical applications) that the pthread_t handle will be unique over the
life of the running process.

Where application code attempts to compare one pthread_t against another
directly, a compiler error will be emitted because structs can't be
compared at that level. This should signal a potentially serious problem
in the code design, which would go undetected if pthread_t was a scalar.

The POSIX Threading API provides a function named pthread_equal() to
compare pthread_t thread handles.

Other pthreads implementations, such as Sun's, use an int as the handle
but do guarantee uniqueness within the process scope. Win32 scalar typed
thread handles also guarantee uniqueness in system scope. It wasn't clear
how well the internal management of these handles would scale as the
number of threads and the fragmentation of the sequence numbering
increased for applications where thousands or millions of threads are
created and detached over time. The current management of threads within
pthreads-win32 using structs for pthread_t, and reusing without ever
freeing them, reduces the management time overheads to a constant, which
could be important given that pthreads-win32 threads are built on top of
Win32 threads and will therefore include that management overhead on top
of their own. The cost is that the memory resources used for thread
handles will remain at the peak level until the process exits.

While it may be inconvenient for developers to be forced away from making
assumptions about the internals of pthread_t, the advantage for the
future development of pthread-win32, as well as those applications that
use it and other pthread implementations, is that the library is free to
change pthread_t internals and management as better methods arise.


On 5/02/2011 1:35 AM, Claude LALYRE wrote:
 > Hi Ross,
 >
 > I would like to thank a lot your pthreads-win32 team for the great
work they achieved.
 > This week, I was in a situation of migrating UNIX source code to
Windows environment.
 > And helpfully with your project phreads-win32 that task was easily
possible.
 >
 > But I encountered some compilation issues. And I have had to declare
some missing
 > typedef and macros in my code, picked from cygwin header files. And
surprisingly
 > it was enough for my code being able to compile.
 >
 > So as it was just a small issue, I thought I should give you my point
of view and
 > the little declarations I made. I think that it should be easily
integrated in your
 > source code. Just have a look at the posix.h attached file.
 >
 > Another point is concerning the declaration of your type "pthread_t".
In all UNIX
 > platforms this is formerly a pointer, but in your Windows
implementation this
 > is a struct object. The issue is that I was given a source file
containing a SSD
 > object containing a field "id" of type pthread_t.
 >
 > struct SSD {
 >    pthread_t id;
 >    int dummy;
 > }
 >
 > And somewhere else in the code they gave me, I have this
 > SSD->id = 0;
 >
 > And that line of code was not accepted by cl.exe (Windows) compiler !
 > So I face the situation by adding this ugly fix
 >
 > #ifdef WIN32
 > SSD->id.p = 0
 > #else /* WIN32 */
 > SSD->id = 0;
 > #endif /* WIN32 */
 >
 > So I am sorry to tell you this about the most basic type of your
pthreads-win32 library,
 > but it would have been great to keep the pthread_t type as a pointer
rather than a struct
 > object. However, as I managed to fix this situation its a tiny issue, an
 > enhancement suggestion rather than a bug...
 >
 > Thank a lot for all youy great job !
 >
 > Claude.
 >
 >
 >
 >