nss_db: protect against empty mappings

classic Classic list List threaded Threaded
44 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: CI/CD in glibc

Zack Weinberg-2
On Tue, Jul 16, 2019 at 8:00 AM Florian Weimer <[hidden email]> wrote:

>
> One experiment I'd like to do (and maybe someone wants to help with
> that): Instrument CC and CXX invocations with something that captures
> the current directory and the command lines during a regular build.
> Afterwards, replay all the compiler invocations, in parallel.  (This may
> need some manual tweaks for adding barriers, but perhaps not if we run
> this test on an already-built tree.)  This should gives a number: How
> much time does it take, at minimum, to build glibc, without make
> overhead or artificial serialization.  It will tell us how inefficient
> the current build system really is.  Is the make overhead for a
> from-scratch build just those 12 seconds I mentioned above, or is it
> much larger?
>
> This should give us some guidance whether we need to focus on
> from-scratch performance, or on making incremental builds accurate.
This isn't the data point you were asking for, but it's a
complementary one: I wrote a simple program (attached; it's in Python,
so we've got interpreter overhead in here, but I expect the dominant
factor is OS overhead) that calls lstat() on every file in a set of
directory trees, except not descending into any subdirectory named
'.git', and records the result in a data structure, and then reports
how long it took to do that.  This should approximate the minimum
possible time for an ideal build tool to determine that an incremental
build has nothing to do.

On the computer I'm typing this on (Xeon E3-1245v3, Linux 4.19,
speculative execution mitigations active, SSD), I ran this test 50
times on a glibc source and build tree and got a median time of 0.181
seconds, with interquartile range of 0.00163 seconds.

zw

stat-perf.py (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CI/CD in glibc

Florian Weimer-5
* Zack Weinberg:

> This isn't the data point you were asking for, but it's a
> complementary one: I wrote a simple program (attached; it's in Python,
> so we've got interpreter overhead in here, but I expect the dominant
> factor is OS overhead) that calls lstat() on every file in a set of
> directory trees, except not descending into any subdirectory named
> '.git', and records the result in a data structure, and then reports
> how long it took to do that.  This should approximate the minimum
> possible time for an ideal build tool to determine that an incremental
> build has nothing to do.

An ideal build tool could do this multi-threaded.  That's not so
outrageous: Even make manages to distribute some of its overhead across
several processes running in parallel.

> On the computer I'm typing this on (Xeon E3-1245v3, Linux 4.19,
> speculative execution mitigations active, SSD), I ran this test 50
> times on a glibc source and build tree and got a median time of 0.181
> seconds, with interquartile range of 0.00163 seconds.

It might be instructive to compare this with “git status” (which is
multi-threaded these days), on a fake tree with both the sources and the
build tree.

Thanks,
Florian
Reply | Threaded
Open this post in threaded view
|

Re: [PATCHv8] nss_db: protect against empty mappings

Carlos O'Donell-6
In reply to this post by DJ Delorie-2
On 7/12/19 11:13 PM, DJ Delorie wrote:
> 2. Any text other than the PASS/FAIL/etc lines should be assumed ignored
>     by any script that parses the output, or by developers who
>     auto-filter-out everything else because "it's never relevent".

Then mark them FAIL?

I'm trying to get to a point where a full `make` followed by `make check`
results in a functioning and correct regression test environment.

This whole point is moot though if we agree to a `make fastcheck` which
doesn't so the install, and a `make check` that always does the install.

--
Cheers,
Carlos.
Reply | Threaded
Open this post in threaded view
|

Re: [PATCHv8] nss_db: protect against empty mappings

DJ Delorie-2
"Carlos O'Donell" <[hidden email]> writes:
> Then mark them FAIL?

Mark them UNSUPPORTED if they weren't run at all.
123