How to handle long running tests in nptl.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to handle long running tests in nptl.

Stefan Liebler-2
Hi,

the nptl tests are currently running in sequence. Some of them are
running very long:
-tst-cond16: 20s
-tst-cond17: 20s
-tst-cond18: 20s
-tst-rwlock-tryrdlock-stall: 20s
-tst-mutex10: 16s
-tst-rwlock20: 10s
-tst-rwlock-trywrlock-stall: 10s
-tst-rwlock-pwn: 10s

The listed tests are responsible for over two minutes of runtime of
"make check". They all are running a test in a loop for a large amount
of iterations or seconds in order to trigger e.g. a race condition.

How to handle those long running tests with respect of "make check" runtime?
- reduce runtime by reducing number of iterations or seconds
- move those tests to "make xcheck"
- reduce runtime while running "make check" and rerun with unchanged
runtime in "make xcheck"
- change nothing
- other ideas?

Bye,
Stefan

Reply | Threaded
Open this post in threaded view
|

Re: How to handle long running tests in nptl.

Carlos O'Donell-6
On 9/12/19 9:21 AM, Stefan Liebler wrote:

> Hi,
>
> the nptl tests are currently running in sequence. Some of them are running very long:
> -tst-cond16: 20s
> -tst-cond17: 20s
> -tst-cond18: 20s
> -tst-rwlock-tryrdlock-stall: 20s
> -tst-mutex10: 16s
> -tst-rwlock20: 10s
> -tst-rwlock-trywrlock-stall: 10s
> -tst-rwlock-pwn: 10s
>
> The listed tests are responsible for over two minutes of runtime of "make check". They all are running a test in a loop for a large amount of iterations or seconds in order to trigger e.g. a race condition.
>
> How to handle those long running tests with respect of "make check" runtime?
> - reduce runtime by reducing number of iterations or seconds
> - move those tests to "make xcheck"
> - reduce runtime while running "make check" and rerun with unchanged runtime in "make xcheck"
> - change nothing
> - other ideas?

The use of 'make xcheck' is for tests that need specific persmissions
to run, like root.

This issue has come up already with regards to the test-in-container
tests because it takes time to do installed-tree testing because you
have to update the installed tree.

I would like to see a discussion around:

- What do developers want from day-to-day 'make check?'
  - Do you want 'make check' to take a constant amount of time?
  - Do you want 'make check' to give you good coverage?

- What do developers want to run before checkin?
  - Is "before checkin" different from day-to-day testing?

This issue is a topic for GNU Cauldron 2019:
https://sourceware.org/glibc/wiki/Cauldron2019
~~~
Update the glibc build infrastructure

    Discuss getting accurate dependency information into the build system so we can do incremental build and test in as fast a time as possible.
    Breaking up tests? Fast. Short. Fixed time.
~~~

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: How to handle long running tests in nptl.

Cyril Hrubis
In reply to this post by Stefan Liebler-2
Hi!
> How to handle those long running tests with respect of "make check" runtime?
> - reduce runtime by reducing number of iterations or seconds
> - move those tests to "make xcheck"
> - reduce runtime while running "make check" and rerun with unchanged
> runtime in "make xcheck"
> - change nothing
> - other ideas?

We do have a nice fuzzy sync library in LTP that is designed especially
to trigger race conditions, which was developed to speed up LTP CVE
related tests. With this library runtime for race test can usually be
reduced from minutes to couple of seconds.

See:
https://github.com/linux-test-project/ltp/blob/master/include/tst_fuzzy_sync.h

We also plan to prepare a talk about this library for next FOSDEM in
order to spare people from reinventing the wheel again and again.

--
Cyril Hrubis
[hidden email]