"make check" times

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

"make check" times

DJ Delorie-2

While digging for some low-hanging fruit in "make" times, I did this:

diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
index 919bbae253..7088ef6996 100755
--- a/scripts/merge-test-results.sh
+++ b/scripts/merge-test-results.sh
@@ -35,7 +35,11 @@ case $type in
     subdir=${subdir:+$subdir/}
     for t in "$@"; do
       if [ -s "$objpfx$t.test-result" ]; then
- head -n1 "$objpfx$t.test-result"
+  #head -n1 "$objpfx$t.test-result"
+  exec 6<"$objpfx$t.test-result"
+  read line <&6
+  echo $line
+  exec 6<&-
       else
  echo "UNRESOLVED: $subdir$t"
       fi

That one instance of "head" is called over 6000 times per "make
check", and as it's the only non-builtin in that script, it adds about
11 seconds of overhead compared to just reading that one line with
builtins.

My question here is: how much of a time savings is worth the
complexity of said savings?
Reply | Threaded
Open this post in threaded view
|

Re: "make check" times

Carlos O'Donell-6
On 7/19/19 6:11 PM, DJ Delorie wrote:

> While digging for some low-hanging fruit in "make" times, I did this:
>
> diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
> index 919bbae253..7088ef6996 100755
> --- a/scripts/merge-test-results.sh
> +++ b/scripts/merge-test-results.sh
> @@ -35,7 +35,11 @@ case $type in
>       subdir=${subdir:+$subdir/}
>       for t in "$@"; do
>         if [ -s "$objpfx$t.test-result" ]; then
> - head -n1 "$objpfx$t.test-result"
> +  #head -n1 "$objpfx$t.test-result"
> +  exec 6<"$objpfx$t.test-result"
> +  read line <&6
> +  echo $line
> +  exec 6<&-
>         else
>   echo "UNRESOLVED: $subdir$t"
>         fi
>
> That one instance of "head" is called over 6000 times per "make
> check", and as it's the only non-builtin in that script, it adds about
> 11 seconds of overhead compared to just reading that one line with
> builtins.
>
> My question here is: how much of a time savings is worth the
> complexity of said savings?

One needs to tackle the problem the other way around.

* First determine those things we cannot change and remove them
   form the accounting.
   - We cannot avoid compiling each file.

* Whatever is left over after removing the things we can't
   avoid doing is our "overhead".

* Sort the "overhead" by the amount of time spent, and find ways
   to reduce it.

If the "overhead" is small, then there isn't much value in adding
complexity.

If the "overhead" is large, we may want to rewrite the whole build
system to get that time back.

Does that make sense?

I can't answer your question without knowing how much is 11 seconds
out of the total time taken.

If the build took 20 seconds to complete, then yes this is a valuable
change ;-)

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: "make check" times

DJ Delorie-2
"Carlos O'Donell" <[hidden email]> writes:
> I can't answer your question without knowing how much is 11 seconds
> out of the total time taken.

In this case, it's about a third of "make check" if you don't need to
rebuild/recheck anything (reduces 30 seconds to 19 seconds), but
compared to a full test run (lots of minutes) it's in the noise.

I've been looking at it from a "why does running one test take so long"
viewpoint - when a test takes under a second, but "make check" takes 30
seconds, there's a lot of overhead.

>   - We cannot avoid compiling each file.

There are ways of optimizing this, too, though... like better
parallelism.
Reply | Threaded
Open this post in threaded view
|

Re: "make check" times

Carlos O'Donell-6
On 7/19/19 9:31 PM, DJ Delorie wrote:
> "Carlos O'Donell" <[hidden email]> writes:
>> I can't answer your question without knowing how much is 11 seconds
>> out of the total time taken.
>
> In this case, it's about a third of "make check" if you don't need to
> rebuild/recheck anything (reduces 30 seconds to 19 seconds), but
> compared to a full test run (lots of minutes) it's in the noise.

Then I think this is a worthwhile fix.

In the old days I often would remove *one* output file to trigger the
test to be re-run and then just hit 'make check'.

This fix would, IIUC, accelerate that use case, which is probably a
common "newcomer" behaviour e.g. remove the test target result, and
rerun 'make check' to re-run that test.
 
> I've been looking at it from a "why does running one test take so long"
> viewpoint - when a test takes under a second, but "make check" takes 30
> seconds, there's a lot of overhead.

Agreed. It should be way way less.
 
>>    - We cannot avoid compiling each file.
>
> There are ways of optimizing this, too, though... like better
> parallelism.

Agreed completely. To do that we need to start understanding our
dependency graph, and I know you know that ;-)

--
Cheers,
Carlos.