[PATCH buildbot] keep build directory after failure

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH buildbot] keep build directory after failure

Szabolcs Nagy-2
aarch64 buildbot failed with

FAIL: malloc/tst-malloc-tcache-leak

4 times so far even though there were no related commits,
the failure happens about 1 out of 15 builds and only if
the build was done in an unclean directory, but i could
not manually reproduce it yet.

the buildbot first builds glibc in the previous build
directory, if that fails then that immediately gets
deleted and glibc is rebuilt starting from an empty
build directory, so it's impossible to tell what went
wrong in case of such non-reproducible failures.

i suggest keeping the last failed build directory for
analysis.

the build directory rename is not optimal since that
breaks absolute paths pointing to it (e.g. paths in
testrun.sh), but it allows looking at test outputs
and failed binaries.

keep.diff (946 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH buildbot] keep build directory after failure

Szabolcs Nagy-2
On 02/11/18 11:33, Szabolcs Nagy wrote:
> aarch64 buildbot failed with
>
> FAIL: malloc/tst-malloc-tcache-leak
>
> 4 times so far even though there were no related commits,
> the failure happens about 1 out of 15 builds and only if
> the build was done in an unclean directory, but i could
> not manually reproduce it yet.

this is most likely caused by a cgroup task limit.

is there a known safe task limit setting for make check -j1 ?

the current setting is
$ cat /sys/fs/cgroup/pids/user.slice/user-1002.slice/pids.max
12288

it also caused other failures

env GCONV_PATH=.... /home/szabolcs/tx1-ubuntu-aarch64/glibc-aarch64-linux/build/build/nptl/tst-eintr1  >
/home/szabolcs/tx1-ubuntu-aarch64/glibc-aarch64-linux/build/build/nptl/tst-eintr1.out; \
../scripts/evaluate-test.sh nptl/tst-eintr1 $? false false >
/home/szabolcs/tx1-ubuntu-aarch64/glibc-aarch64-linux/build/build/nptl/tst-eintr1.test-result
/bin/sh: 2: Cannot fork
make[2]: *** [/home/szabolcs/tx1-ubuntu-aarch64/glibc-aarch64-linux/build/build/nptl/tst-eintr1.out] Error 2


>
> the buildbot first builds glibc in the previous build
> directory, if that fails then that immediately gets
> deleted and glibc is rebuilt starting from an empty
> build directory, so it's impossible to tell what went
> wrong in case of such non-reproducible failures.
>
> i suggest keeping the last failed build directory for
> analysis.
>
> the build directory rename is not optimal since that
> breaks absolute paths pointing to it (e.g. paths in
> testrun.sh), but it allows looking at test outputs
> and failed binaries.
>