about summary refs log tree commit diff
path: root/src/libstore
AgeCommit message (Collapse)AuthorFilesLines
2016-11-16 seccomp: Forge return codes for POSIX ACL syscallsaszlig1-0/+4
Commands such as "cp -p" also use fsetxattr() in addition to fchown(), so we need to make sure these syscalls always return successful as well in order to avoid nasty "Invalid value" errors. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 libstore/build: Forge chown() to return successaszlig1-0/+41
What we basically want is a seccomp mode 2 BPF program like this but for every architecture: BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0), BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW), BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO) However, on 32 bit architectures we do have chown32, lchown32 and fchown32, so we'd need to add all the architecture blurb which libseccomp handles for us. So we only need to make sure that we add the 32bit seccomp arch while we're on x86_64 and otherwise we just stay at the native architecture which was set during seccomp_init(), which more or less replicates setting 32bit personality during runChild(). The FORCE_SUCCESS() macro here could be a bit less ugly but I think repeating the seccomp_rule_add() all over the place is way uglier. Another way would have been to create a vector of syscalls to iterate over, but that would make error messages uglier because we can either only print the (libseccomp-internal) syscall number or use seccomp_syscall_resolve_num_arch() to get the name or even make the vector a pair number/name, essentially duplicating everything again. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 Add build dependency for libseccompaszlig1-0/+4
We're going to use libseccomp instead of creating the raw BPF program, because we have different syscall numbers on different architectures. Although our initial seccomp rules will be quite small it really doesn't make sense to generate the raw BPF program because we need to duplicate it and/or make branches on every single architecture we want to suuport. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 Run builds as root in user namespace againaszlig1-29/+16
This reverts commit ff0c0b645cc1448959126185bb2fafe41cf0bddf. We're going to use seccomp to allow "cp -p" and force chown-related syscalls to always return 0. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-14 Don't rely on %mEelco Dolstra1-1/+2
2016-11-14 daemon: Do not error out when deduplication fails due to ENOSPC.Ludovic Courtès1-5/+19
This solves a problem whereby if /gnu/store/.links had enough entries, ext4's directory index would be full, leading to link(2) returning ENOSPC. * nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon ENOSPC from link(2), print a message and return instead of throwing a 'SysError'.
2016-11-10 Store::computeFSClosure(): Support a set of pathsEelco Dolstra3-7/+18
This way, callers can exploits the parallelism of computeFSClosure() when they have multiple paths that they need the (combined) closure of.
2016-11-09 copyClosure() / copyStorePath(): Expose dontCheckSigsEelco Dolstra2-6/+6
Needed by Hydra.
2016-11-09 Merge branch 'ssh-store' of https://github.com/shlevy/nixEelco Dolstra10-86/+355
2016-11-09 Implement backwards-compatible RemoteStore::addToStore()Eelco Dolstra1-1/+22
The SSHStore PR adds this functionality to the daemon, but we have to handle the case where the Nix daemon is 1.11. Also, don't require signatures for trusted users. This restores 1.11 behaviour. Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-08 TypoEelco Dolstra1-1/+1
2016-10-31 Support optional sandbox pathsEelco Dolstra1-9/+26
For example, you can now set build-sandbox-paths = /dev/nvidiactl? to specify that /dev/nvidiactl should only be mounted in the sandbox if it exists in the host filesystem. This is useful e.g. for EC2 images that should support both CUDA and non-CUDA instances.
2016-10-27 Add nix.conf options for -k and -KShea Levy1-0/+2
Fixes #1084
2016-10-26 Fix build log output in nix-store --serveEelco Dolstra1-1/+1
2016-10-26 Restore the CachedFailure status codeEelco Dolstra1-0/+4
The removal of CachedFailure caused the value of TimedOut to change, which broke timed-out handling in Hydra (so timed-out builds would show up as "aborted" and would be retried, e.g. at http://hydra.nixos.org/build/42537427).
2016-10-21 Remove addPathToAccessorEelco Dolstra8-108/+96
2016-10-21 BinaryCacheStore: Optionally write a NAR listingEelco Dolstra10-20/+81
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-19 Merge branch 'priorityqueue' of https://github.com/groxxda/nixEelco Dolstra1-12/+17
2016-10-19 Shut up clang warningEelco Dolstra1-1/+1
2016-10-19 Fix uninitialised variableEelco Dolstra1-1/+1
2016-10-18 downloader: use priority_queueAlexander Ried1-12/+17
2016-10-14 Fix /dev/ptmx in sandboxesEelco Dolstra1-1/+6
This was broken since ff0c0b645cc1448959126185bb2fafe41cf0bddf. Since I can't figure out how to mount a devpts instance in the sandbox, let's just bind-mount the host devpts.
2016-10-14 CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUMEelco Dolstra1-4/+3
http://hydra.nixos.org/build/42025230
2016-10-13 Fix HTTP/2 supportEelco Dolstra1-3/+3
Commit 86e8c67efc33cf756500a1dec7fd6313658f2664 broke it, because CURL_* are not actually #defines.
2016-10-13 SSL_CERT_FILE -> NIX_SSL_CERT_FILEEelco Dolstra1-1/+2
This prevents collisions with the "native" OpenSSL, in particular on OS X. Fixes #921.
2016-10-12 Shut up some warningsEelco Dolstra1-1/+1
2016-10-07 querySubstitutablePaths(): Don't query paths for which we already have a ↵Eelco Dolstra1-3/+16
substituter
2016-10-07 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths()Eelco Dolstra1-7/+3
2016-10-07 Store::queryValidPaths(): Use async queryPathInfo()Eelco Dolstra1-4/+39
This allows the binary cache substituter to pipeline requests.
2016-10-07 Implement generic Store::queryValidPaths()Eelco Dolstra3-4/+12
2016-10-07 Add copyClosure utility function for HydraEelco Dolstra2-0/+29
2016-10-07 importPaths(): Fix accessor support for HydraEelco Dolstra4-7/+27
2016-10-06 Fix getS3Stats()Eelco Dolstra2-2/+2
2016-10-05 Use std::random_deviceEelco Dolstra1-0/+1
2016-10-04 Fix build with older versions of libcurlEelco Dolstra1-0/+6
2016-09-22 Handle the case where signed-binary-caches consists of whitespaceEelco Dolstra1-1/+1
2016-09-21 Add sandbox-dev-shm-size optionEelco Dolstra1-1/+2
Fixes #1069.
2016-09-21 printMsg(lvlError, ...) -> printError(...) etc.Eelco Dolstra11-80/+80
2016-09-21 Some notational convenience for formatting stringsEelco Dolstra2-4/+4
We can now write throw Error("file '%s' not found", path); instead of throw Error(format("file '%s' not found") % path); and similarly printError("file '%s' not found", path); instead of printMsg(lvlError, format("file '%s' not found") % path);
2016-09-20 TweakEelco Dolstra1-5/+1
2016-09-20 Improve robustnessEelco Dolstra1-2/+6
2016-09-20 Fix "Promise already satisfied" errorEelco Dolstra1-1/+3
2016-09-16 armv5tel can be built by armv6l and armv7l (#1063)Eric Litak1-0/+1
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra14-202/+335
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-14 Revive binary-caches-parallel-connectionsEelco Dolstra1-2/+8
It's a slight misnomer now because it actually limits *all* downloads, not just binary cache lookups. Also add a "enable-http2" option to allow disabling use of HTTP/2 (enabled by default).
2016-09-14 Enable HTTP/2 supportEelco Dolstra4-199/+422
The binary cache store can now use HTTP/2 to do lookups. This is much more efficient than HTTP/1.1 due to multiplexing: we can issue many requests in parallel over a single TCP connection. Thus it's no longer necessary to use a bunch of concurrent TCP connections (25 by default). For example, downloading 802 .narinfo files from https://cache.nixos.org/, using a single TCP connection, takes 11.8s with HTTP/1.1, but only 0.61s with HTTP/2. This did require a fairly substantial rewrite of the Downloader class to use the curl multi interface, because otherwise curl wouldn't be able to do multiplexing for us. As a bonus, we get connection reuse even with HTTP/1.1. All downloads are now handled by a single worker thread. Clients call Downloader::enqueueDownload() to tell the worker thread to start the download, getting a std::future to the result.
2016-09-14 Move some .drv parsing functions out of utilEelco Dolstra1-0/+46
2016-09-12 ssh-store: Start master on-demandShea Levy1-8/+13
2016-09-12 Inline ssh-store.hh into ssh-store.ccShea Levy2-41/+34
2016-09-08 Don't run builds as root in the user namespaceEelco Dolstra1-16/+31
This largely reverts c68e5913c71badc89ff346d1c6948517ba720c93. Running builds as root breaks "cp -p", since when running as root, "cp -p" assumes that it can succesfully chown() files. But that's not actually the case since the user namespace doesn't provide a complete uid mapping. So it barfs with a fatal error message ("cp: failed to preserve ownership for 'foo': Invalid argument").
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-23T13·02+0000