Sending test results from build robot?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Sending test results from build robot?

Jan-Benedict Glaw
Hi!

I've written a small shell script to regularly build
cross-{binutils,gcc,uclibc} and then with that native binutils+gcc.
That's for vax-dec-linux-uclibc (and I'm working on
vax-dec-linux-gnu).

Would it be of some help (at least to document that at least some VAX
targets are still alive) to regluarly submit testsuite results? At a
maximum, it could bulid one complete run in about 30min, but I guess
you don't want to have a report ~50 times per day :->

Apart from the fact that I monitor changes, how often should I send
reports?

Thanks, JBG

--
Jan-Benedict Glaw       [hidden email]    . +49-172-7608481             _ O _
"Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg  _ _ O
 für einen Freien Staat voll Freier Bürger"  | im Internet! |   im Irak!   O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Sending test results from build robot?

Hans-Peter Nilsson-2
On Mon, 19 Dec 2005, Jan-Benedict Glaw wrote:
> Would it be of some help (at least to document that at least some VAX
> targets are still alive) to regluarly submit testsuite results? At a
> maximum, it could bulid one complete run in about 30min, but I guess
> you don't want to have a report ~50 times per day :->

If you trig your script on commits, this will happen a few times
a day.  You don't have to poll the CVS server, you can do this
for emails to the binutils-cvs list.  I do this for my
autotester and it works fine, but I should mention that I have a
delay for 15m until the actual cvs update (actually re-checkout
over existing checked-out tree; never update in src) begins,
which works fine to avoid getting an unfinished set of commits.

(It doesn't auto-report as I still want to inspect the results
before reporting and I wouldn't want an external
semi-catastrophic failure to cause it to mailbomb the list.  I
might be over-cautious.  It's been running fine for a while, but
as I don't have control over the machines where it runs, a
mistaken change in installed packages can cause it to spuriously
fail.)

> Apart from the fact that I monitor changes, how often should I send
> reports?

Most useful would IMHO be to only send reports when there's a
failure.

Dejagnu has the nice property (though limited: you can't tweak
e.g. the number or identity of unexpected errors for exit != 0;
you have to edit the test-cases) that you get a nonzero return
code from "make check" only when there are errors or FAILs
present, so this makes report-on-errors easy to implement.

The binutils testsuite had no known FAILs until H.J.:s recent
commit.

brgds, H-P
Reply | Threaded
Open this post in threaded view
|

Re: Sending test results from build robot?

Christian Joensson-2
On 12/19/05, Hans-Peter Nilsson <[hidden email]> wrote:

What do you guys think of something similar to gcc-testresults list,
i.e., a binutils-testresults list?

And... as you've seen, I guess, what about looking into gcc's script
for generating test results, it's kinda neat, don't you think?

--
Cheers,

/ChJ