Age | Commit message (Collapse) | Author | Files | Lines |
|
This eliminates some unnecessary (presumably cached) I/O.
|
|
http://hydra.nixos.org/build/32085949
|
|
|
|
|
|
|
|
Also, move a few free-standing functions into StoreAPI and Derivation.
Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
|
|
|
|
Previously files in the Nix store were owned by root or by nixbld,
depending on whether they were created by a substituter or by a
builder. This doesn't matter much, but causes spurious diffoscope
differences. So use root everywhere.
|
|
Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent
ambiguity.
|
|
And make exportPath() less spammy by default.
|
|
|
|
|
|
|
|
|
|
This broke NixOS VM tests.
Mostly reverts 27b7b94923d2f207781b438bb7a57669bddf7d2b,
5ce50cd99e740d0d0f18c30327ae687be9356553,
afa433e58c3fe6029660a43fdc2073c9d15b4210.
|
|
|
|
|
|
|
|
I.e., not readable to the nixbld group. This improves purity a bit for
non-chroot builds, because it prevents a builder from enumerating
store paths (i.e. it can only access paths it knows about).
|
|
See NixOS/nixpkgs@9245516
|
|
|
|
The pid field can be -1 if forking the substituter process failed.
|
|
|
|
Fixes #269.
|
|
|
|
|
|
|
|
This causes nix-copy-closure to show what it's doing before rather
than after.
|
|
C++11 lambdas ftw.
|
|
|
|
This automatically creates /nix/var/nix/profiles/per-user and sets the
permissions/ownership on /nix/store to 1775 and root:nixbld.
|
|
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
|
|
Not bind-mounting the /dev from the host also solves the problem with
/dev/shm being a symlink to something not in the chroot.
|
|
This will allow Hydra to detect that a build should not be marked as
"permanently failed", allowing it to be retried later.
|
|
Namely:
nix-store: derivations.cc:242: nix::Hash nix::hashDerivationModulo(nix::StoreAPI&, nix::Derivation): Assertion `store.isValidPath(i->first)' failed.
This happened because of the derivation output correctness check being
applied before the references of a derivation are valid.
|
|
Previously we would say "error: setting synchronous mode: unable to
open database file" which isn't very helpful.
|
|
This should fix building on Illumos.
|
|
AFAIK, nobody uses it, it's not maintained, and it has no tests.
|
|
To deal with SQLITE_PROTOCOL, we also need to retry read-only
operations.
|
|
Registering the path as failed can fail if another process does the
same thing after the call to hasPathFailed(). This is extremely
unlikely though.
|
|
|
|
There is no risk of getting an inconsistent result here: if the ID
returned by queryValidPathId() is deleted from the database
concurrently, subsequent queries involving that ID will simply fail
(since IDs are never reused).
|
|
|
|
In the Hydra build farm we fairly regularly get SQLITE_PROTOCOL errors
(e.g., "querying path in database: locking protocol"). The docs for
this error code say that it "is returned if some other process is
messing with file locks and has violated the file locking protocol
that SQLite uses on its rollback journal files." However, the SQLite
source code reveals that this error can also occur under high load:
if( cnt>5 ){
int nDelay = 1; /* Pause time in microseconds */
if( cnt>100 ){
VVA_ONLY( pWal->lockError = 1; )
return SQLITE_PROTOCOL;
}
if( cnt>=10 ) nDelay = (cnt-9)*238; /* Max delay 21ms. Total delay 996ms */
sqlite3OsSleep(pWal->pVfs, nDelay);
}
i.e. if certain locks cannot be not acquired, SQLite will retry a
number of times before giving up and returing SQLITE_PROTOCOL. The
comments say:
Circumstances that cause a RETRY should only last for the briefest
instances of time. No I/O or other system calls are done while the
locks are held, so the locks should not be held for very long. But
if we are unlucky, another process that is holding a lock might get
paged out or take a page-fault that is time-consuming to resolve,
during the few nanoseconds that it is holding the lock. In that case,
it might take longer than normal for the lock to free.
...
The total delay time before giving up is less than 1 second.
On a heavily loaded machine like lucifer (the main Hydra server),
which often has dozens of processes waiting for I/O, it seems to me
that a page fault could easily take more than a second to resolve.
So, let's treat SQLITE_PROTOCOL as SQLITE_BUSY and retry the
transaction.
Issue NixOS/hydra#14.
|
|
|
|
On a system with multiple CPUs, running Nix operations through the
daemon is significantly slower than "direct" mode:
$ NIX_REMOTE= nix-instantiate '<nixos>' -A system
real 0m0.974s
user 0m0.875s
sys 0m0.088s
$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real 0m2.118s
user 0m1.463s
sys 0m0.218s
The main reason seems to be that the client and the worker get moved
to a different CPU after every call to the worker. This patch adds a
hack to lock them to the same CPU. With this, the overhead of going
through the daemon is very small:
$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real 0m1.074s
user 0m0.809s
sys 0m0.098s
|
|
Common operations like instantiating a NixOS system config no longer
fitted in 8192 pages, leading to more fsyncs. So increase this limit.
|
|
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests. This also speeds things up because we don't send queries
to those substituters.
|
|
|
|
Otherwise subsequent invocations of "--repair" will keep rebuilding
the path. This only happens if the path content differs between
builds (e.g. due to timestamps).
|