PSA: glibc buildbot slave up for aarch64

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
Hello,

This is to let everyone know that I have added a buildbot slave for
aarch64 running on a mustang box I have lying around.  As of now there
are two failures (both being backtrace symbol resolution issues) that
are preventing the slave from running full green, so it would be great
if someone has a look at that.  I have logged it in bz in case anyone
wants to take a look:

https://sourceware.org/bugzilla/show_bug.cgi?id=21428

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Szabolcs Nagy-2
On 26/04/17 05:36, Siddhesh Poyarekar wrote:
> This is to let everyone know that I have added a buildbot slave for
> aarch64 running on a mustang box I have lying around.  As of now there

thanks

> are two failures (both being backtrace symbol resolution issues) that
> are preventing the slave from running full green, so it would be great
> if someone has a look at that.  I have logged it in bz in case anyone
> wants to take a look:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=21428

this is a known issue, the cancellation cleanup will fix it.

Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Carlos O'Donell-6
On 04/27/2017 04:27 AM, Szabolcs Nagy wrote:

> On 26/04/17 05:36, Siddhesh Poyarekar wrote:
>> This is to let everyone know that I have added a buildbot slave for
>> aarch64 running on a mustang box I have lying around.  As of now there
>
> thanks
>
>> are two failures (both being backtrace symbol resolution issues) that
>> are preventing the slave from running full green, so it would be great
>> if someone has a look at that.  I have logged it in bz in case anyone
>> wants to take a look:
>>
>> https://sourceware.org/bugzilla/show_bug.cgi?id=21428
>
> this is a known issue, the cancellation cleanup will fix it.
>

Can we XFAIL this for AArch64 to get a clean build?

One think I'd like to see is more clean builds regardless of failures.

The buildbots _must_ be all green for them to mean anything.

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
On Thursday 27 April 2017 07:36 PM, Carlos O'Donell wrote:
> Can we XFAIL this for AArch64 to get a clean build?
>
> One think I'd like to see is more clean builds regardless of failures.
>
> The buildbots _must_ be all green for them to mean anything.

I can't convince myself of this - marking testsuite failures as XFAIL
seems like putting things under the rug.  We as a community are not that
great in pulling things back out from under the rug and cleaning them up
for real because of the kind of resources we have, so what goes under
the rug essentially stays under the rug.

I would prefer if for now red buildbot statuses become release blockers
for 2.26.  Once 2.26 is released, any failure should either be
immediately fixed or the offending patch reverted.

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Joseph Myers
On Fri, 28 Apr 2017, Siddhesh Poyarekar wrote:

> On Thursday 27 April 2017 07:36 PM, Carlos O'Donell wrote:
> > Can we XFAIL this for AArch64 to get a clean build?
> >
> > One think I'd like to see is more clean builds regardless of failures.
> >
> > The buildbots _must_ be all green for them to mean anything.
>
> I can't convince myself of this - marking testsuite failures as XFAIL
> seems like putting things under the rug.  We as a community are not that
> great in pulling things back out from under the rug and cleaning them up
> for real because of the kind of resources we have, so what goes under
> the rug essentially stays under the rug.

Bugzilla is how we track known bugs.  Just because a particular bug causes
a test failure, or a failure on a system with a buildbot, doesn't make it
more important than others.  It's true we have too many open bugs needing
fixing, and that simply saying "this test fails for me" (or, worse, "this
omnibus set of possibly unrelated tests all fail for me") is not a good
bug report without sufficient analysis to show it's a bug in glibc and
identify any relevant conditions for it to fail.

It's a lot friendlier for users and distributors if our collective
knowledge about expected failures is reflected in XFAILs (with appropriate
comments referencing the bugs in Bugzilla) than if people need to refer to
lists on the wiki to identify whether particular failures are known.

> I would prefer if for now red buildbot statuses become release blockers
> for 2.26.  Once 2.26 is released, any failure should either be
> immediately fixed or the offending patch reverted.

That logic would say the hppa build failures should have resulted in the
reversion of the NPTL changes causing them, if someone had an hppa
buildbot.  We need to judge case by case how important a given regression
is (and for non-regression failures - in which category I include failures
from a newly added test that only works on some platforms - I don't think
they're generally going to be release blockers).

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Carlos O'Donell-6
In reply to this post by Siddhesh Poyarekar-8
On 04/28/2017 12:53 AM, Siddhesh Poyarekar wrote:

> On Thursday 27 April 2017 07:36 PM, Carlos O'Donell wrote:
>> Can we XFAIL this for AArch64 to get a clean build?
>>
>> One think I'd like to see is more clean builds regardless of failures.
>>
>> The buildbots _must_ be all green for them to mean anything.
>
> I can't convince myself of this - marking testsuite failures as XFAIL
> seems like putting things under the rug.  We as a community are not that
> great in pulling things back out from under the rug and cleaning them up
> for real because of the kind of resources we have, so what goes under
> the rug essentially stays under the rug.

It is better in the long run to mark these XFAIL, associate them with a bug,
and have a green board. The green board makes it immediately easy to identify
breakage.

The XFAIL is the encoding of expert knowledge. We _will_ get to fixing issues,
but we will do so based on priority order of the things we have in the bugzilla
and things identified by the community.

> I would prefer if for now red buildbot statuses become release blockers
> for 2.26.  Once 2.26 is released, any failure should either be
> immediately fixed or the offending patch reverted.

Why? They might not be important things. Priority is what matters.

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
In reply to this post by Joseph Myers
On Friday 28 April 2017 08:21 PM, Joseph Myers wrote:

> Bugzilla is how we track known bugs.  Just because a particular bug causes
> a test failure, or a failure on a system with a buildbot, doesn't make it
> more important than others.  It's true we have too many open bugs needing
> fixing, and that simply saying "this test fails for me" (or, worse, "this
> omnibus set of possibly unrelated tests all fail for me") is not a good
> bug report without sufficient analysis to show it's a bug in glibc and
> identify any relevant conditions for it to fail.
>
> It's a lot friendlier for users and distributors if our collective
> knowledge about expected failures is reflected in XFAILs (with appropriate
> comments referencing the bugs in Bugzilla) than if people need to refer to
> lists on the wiki to identify whether particular failures are known.

For that the XFAILs should have an additional reference point than just
bugzilla reports.  The red status on buildbot is an eye-sore and I am
hoping that the eyesore will drive at least some of us to try and fix it.

>> I would prefer if for now red buildbot statuses become release blockers
>> for 2.26.  Once 2.26 is released, any failure should either be
>> immediately fixed or the offending patch reverted.
>
> That logic would say the hppa build failures should have resulted in the
> reversion of the NPTL changes causing them, if someone had an hppa
> buildbot.  We need to judge case by case how important a given regression
> is (and for non-regression failures - in which category I include failures
> from a newly added test that only works on some platforms - I don't think
> they're generally going to be release blockers).

The key point here is that there *is* no hppa buildbot and the reason is
quite clear IMO - there is nobody bothered enough to take the effort to
host one and that is what takes the responsibility away from the NPTL
patch submitter.

In any case I agree that this should be decided on a case to case basis
and it shouldn't be a rule set in stone.  What I want to convey is that
the XFAIL mechanism doesn't mean anything useful currently other than
using it to make buildbot statuses green and that is only going to
result in more bugs being pushed under the rug.

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
In reply to this post by Carlos O'Donell-6
On Friday 28 April 2017 08:30 PM, Carlos O'Donell wrote:
> It is better in the long run to mark these XFAIL, associate them with a bug,
> and have a green board. The green board makes it immediately easy to identify
> breakage.
>
> The XFAIL is the encoding of expert knowledge. We _will_ get to fixing issues,
> but we will do so based on priority order of the things we have in the bugzilla
> and things identified by the community.

My concern is that we won't get to fixing things urgently enough given
that technical debt repayment has not been a priority for us
traditionally.  That is not to say that nobody does it - we have fixed a
huge pile of issues over the years.  It is just that we don't invest the
kind of resources that we should and as a result a number of issues that
are of medium/low priority stay languishing in there forever.  The red
buildbot is hopefully enough of an eyesore to get to fixing things
sooner rather than later.

>> I would prefer if for now red buildbot statuses become release blockers
>> for 2.26.  Once 2.26 is released, any failure should either be
>> immediately fixed or the offending patch reverted.
>
> Why? They might not be important things. Priority is what matters.

Sure, I can settle for that, marking low priority bugs that nobody cares
about and are probably not easy to fix anyway as XFAILs.  In that
context I concede that the backtrace bug fits that description.  Also in
this context, I'd rather keep the buildbot red for aarch64 to track the
backtrace fix because the patch is in review and the red status is again
hopefully an incentive to expedite the review.

In general though we need to be careful with what we mark as XFAIL with
the view of making the buildbot statuses green.

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Adhemerval Zanella-2


> On 28 Apr 2017, at 13:03, Siddhesh Poyarekar <[hidden email]> wrote:
>
>> On Friday 28 April 2017 08:30 PM, Carlos O'Donell wrote:
>> It is better in the long run to mark these XFAIL, associate them with a bug,
>> and have a green board. The green board makes it immediately easy to identify
>> breakage.
>>
>> The XFAIL is the encoding of expert knowledge. We _will_ get to fixing issues,
>> but we will do so based on priority order of the things we have in the bugzilla
>> and things identified by the community.
>
> My concern is that we won't get to fixing things urgently enough given
> that technical debt repayment has not been a priority for us
> traditionally.  That is not to say that nobody does it - we have fixed a
> huge pile of issues over the years.  It is just that we don't invest the
> kind of resources that we should and as a result a number of issues that
> are of medium/low priority stay languishing in there forever.  The red
> buildbot is hopefully enough of an eyesore to get to fixing things
> sooner rather than later.
>
>>> I would prefer if for now red buildbot statuses become release blockers
>>> for 2.26.  Once 2.26 is released, any failure should either be
>>> immediately fixed or the offending patch reverted.
>>
>> Why? They might not be important things. Priority is what matters.
>
> Sure, I can settle for that, marking low priority bugs that nobody cares
> about and are probably not easy to fix anyway as XFAILs.  In that
> context I concede that the backtrace bug fits that description.  Also in
> this context, I'd rather keep the buildbot red for aarch64 to track the
> backtrace fix because the patch is in review and the red status is again
> hopefully an incentive to expedite the review.
>
> In general though we need to be careful with what we mark as XFAIL with
> the view of making the buildbot statuses green.
>
> Siddhesh

For the tst-backtrace{5,6} on aarch64 I think we can use the same strategy for PowerPC, where GLIBC saves the vDSO symbol address and use it to get the correct stackstrace on signal handlers. Also, since frame pointer save is required for aarch64 ABI, we can avoid using x86_64 code which uses libgcc and walk the frames directly. I have a local implementation that fixes all current testcases and I will send it for 2.26.
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Joseph Myers
In reply to this post by Siddhesh Poyarekar-8
On Fri, 28 Apr 2017, Siddhesh Poyarekar wrote:

> > It's a lot friendlier for users and distributors if our collective
> > knowledge about expected failures is reflected in XFAILs (with appropriate
> > comments referencing the bugs in Bugzilla) than if people need to refer to
> > lists on the wiki to identify whether particular failures are known.
>
> For that the XFAILs should have an additional reference point than just
> bugzilla reports.  The red status on buildbot is an eye-sore and I am
> hoping that the eyesore will drive at least some of us to try and fix it.

Well, you could have an orthogonal measure of port status distinct from
"are there regressions yet to be analysed", without hiding new regressions
because of an always-present failure.

> and it shouldn't be a rule set in stone.  What I want to convey is that
> the XFAIL mechanism doesn't mean anything useful currently other than
> using it to make buildbot statuses green and that is only going to
> result in more bugs being pushed under the rug.

It achieves that everyone testing on a platform doesn't need to rediscover
for themselves what is or is not a known failure.  It achieves that
distributors don't all need to replicate the same list of known failures
locally.  It achieves that when a regression occurs on a platform it
actually becomes visible in the results of buildbots rather than being
hidden by an old known failure.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
On Friday 28 April 2017 09:52 PM, Joseph Myers wrote:
> Well, you could have an orthogonal measure of port status distinct from
> "are there regressions yet to be analysed", without hiding new regressions
> because of an always-present failure.

Hmm, we could do that by marking regressions that we XFAIL with a
keyword in bz, say testsuite-xfail.  If it sounds like a good idea then
can someone with bz access add a keyword? I can then come up with a
script to get a report from bz and have it emailed to lic-alpha on a
weekly basis.

> It achieves that everyone testing on a platform doesn't need to rediscover
> for themselves what is or is not a known failure.  It achieves that
> distributors don't all need to replicate the same list of known failures
> locally.  It achieves that when a regression occurs on a platform it
> actually becomes visible in the results of buildbots rather than being
> hidden by an old known failure.

I don't disagree, just that I think we need something more than XFAIL.
An xfail weekly spam report ought to bridge that gap.

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Joseph Myers
On Fri, 28 Apr 2017, Siddhesh Poyarekar wrote:

> On Friday 28 April 2017 09:52 PM, Joseph Myers wrote:
> > Well, you could have an orthogonal measure of port status distinct from
> > "are there regressions yet to be analysed", without hiding new regressions
> > because of an always-present failure.
>
> Hmm, we could do that by marking regressions that we XFAIL with a
> keyword in bz, say testsuite-xfail.  If it sounds like a good idea then
> can someone with bz access add a keyword? I can then come up with a
> script to get a report from bz and have it emailed to lic-alpha on a
> weekly basis.

I'm not sure it's even regressions being XFAILed; it's generally tests
that never worked, or never worked on a particular platform (and where
fixing the issue may well require global changes across other toolchain
components and the Linux kernel, e.g. the MIPS XFAIL of check-execstack).

Bug reports to libc-alpha may well be useful, but I don't think "is there
a test in the testsuite for this bug" is a particularly useful criterion
for them.  Numbers of open bugs in total and in each component, with how
that changed over the past week / month / year, or a list of recently
opened / reopened bugs, seem like the sort of thing that might be more
helpful for keeping on top of known bug state.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
On Friday 28 April 2017 10:58 PM, Joseph Myers wrote:
> I'm not sure it's even regressions being XFAILed; it's generally tests
> that never worked, or never worked on a particular platform (and where
> fixing the issue may well require global changes across other toolchain
> components and the Linux kernel, e.g. the MIPS XFAIL of check-execstack).

Right, I should not have said regressions, just bugs for which there is
a test case in the testsuite and it is currently failing.  Also, this is
kinda the reason why I am uncomfortable with using the XFAIL mechanism
indiscriminately - they conflate these kinds of convoluted issues with
bugs than could have been fixed in a specified time frame but weren't
because they weren't thought to be important enough at that time.

> Bug reports to libc-alpha may well be useful, but I don't think "is there
> a test in the testsuite for this bug" is a particularly useful criterion
> for them.  Numbers of open bugs in total and in each component, with how
> that changed over the past week / month / year, or a list of recently
> opened / reopened bugs, seem like the sort of thing that might be more
> helpful for keeping on top of known bug state.

OK, I'll write a script that sends in such an email, although I still
maintain that identifying bugs that have failing test cases in the
testsuite are good candidates to identify separately, especially as a
metric of how we have been using the xfail mechanism and also as a guide
for those looking for a problem to solve in glibc.  A slightly
complicated issue that isn't critical is a nice place to get started
with contributing to glibc and a test case to reproduce the problem and
step through code helps.

I also added a comment on the TODO Master wiki page about bug fixes,
with a special mention of XFAIL bugs.

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Joseph Myers
On Fri, 28 Apr 2017, Siddhesh Poyarekar wrote:

> Right, I should not have said regressions, just bugs for which there is
> a test case in the testsuite and it is currently failing.  Also, this is
> kinda the reason why I am uncomfortable with using the XFAIL mechanism
> indiscriminately - they conflate these kinds of convoluted issues with
> bugs than could have been fixed in a specified time frame but weren't
> because they weren't thought to be important enough at that time.

The GDB testsuite distinguishes XFAIL and KFAIL, but I've never been
convinced that distinction is useful in practice (and e.g. GCC does not
use KFAIL).

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Carlos O'Donell-6
In reply to this post by Siddhesh Poyarekar-8
On 04/28/2017 12:03 PM, Siddhesh Poyarekar wrote:

> On Friday 28 April 2017 08:30 PM, Carlos O'Donell wrote:
>> It is better in the long run to mark these XFAIL, associate them with a bug,
>> and have a green board. The green board makes it immediately easy to identify
>> breakage.
>>
>> The XFAIL is the encoding of expert knowledge. We _will_ get to fixing issues,
>> but we will do so based on priority order of the things we have in the bugzilla
>> and things identified by the community.
>
> My concern is that we won't get to fixing things urgently enough given
> that technical debt repayment has not been a priority for us
> traditionally.  That is not to say that nobody does it - we have fixed a
> huge pile of issues over the years.  It is just that we don't invest the
> kind of resources that we should and as a result a number of issues that
> are of medium/low priority stay languishing in there forever.  The red
> buildbot is hopefully enough of an eyesore to get to fixing things
> sooner rather than later.

The concern about technical debt repayment will not be fixed by XFAIL
management, it will be fixed by each of the senior community members lobbying
and slowly building back a large community of engaged and interested developers.
We can't do that if our failure list is large and undocumented (no XFAIL data,
and no bugs filed).

I don't think any red buildbot will solve this problem, it will only make it
harder to recruit junior people who don't understand why it's red.

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
On Monday 01 May 2017 06:54 PM, Carlos O'Donell wrote:
> The concern about technical debt repayment will not be fixed by XFAIL
> management, it will be fixed by each of the senior community members lobbying
> and slowly building back a large community of engaged and interested developers.
> We can't do that if our failure list is large and undocumented (no XFAIL data,
> and no bugs filed).
>
> I don't think any red buildbot will solve this problem, it will only make it
> harder to recruit junior people who don't understand why it's red.

Like I said elsewhere in the thread, I don't disagree with it being a
partial hack to try and get people interested and I've settled for
sending out weekly spam to the list instead.

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
On Tuesday 02 May 2017 09:13 AM, Siddhesh Poyarekar wrote:
> Like I said elsewhere in the thread, I don't disagree with it being a
> partial hack to try and get people interested and I've settled for
> sending out weekly spam to the list instead.

Actually on that note, can we create a 'testsuite-fail' keyword to track
these separately?

Siddhesh
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Joseph Myers
On Tue, 2 May 2017, Siddhesh Poyarekar wrote:

> On Tuesday 02 May 2017 09:13 AM, Siddhesh Poyarekar wrote:
> > Like I said elsewhere in the thread, I don't disagree with it being a
> > partial hack to try and get people interested and I've settled for
> > sending out weekly spam to the list instead.
>
> Actually on that note, can we create a 'testsuite-fail' keyword to track
> these separately?

Would such keywords also cover cases marked as expected to fail within the
tests themselves (so the overall status of a test ends up as PASS)?  E.g.
bugs 14362, 16437, 16919, 17786, 18228, 18231, 18232, 21260, 21278, 21279
have such local xfail markings within the conform/ tests so they don't
hide failures for other test assertions within the same (standard, header)
pairs.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PSA: glibc buildbot slave up for aarch64

Siddhesh Poyarekar-8
On Tuesday 02 May 2017 08:51 PM, Joseph Myers wrote:
> Would such keywords also cover cases marked as expected to fail within the
> tests themselves (so the overall status of a test ends up as PASS)?  E.g.
> bugs 14362, 16437, 16919, 17786, 18228, 18231, 18232, 21260, 21278, 21279
> have such local xfail markings within the conform/ tests so they don't
> hide failures for other test assertions within the same (standard, header)
> pairs.

Given that they are reproducible via the testsuite (despite the overall
PASS), yes they should have a testsuite-fail keyword.

Siddhesh