about summary refs log tree commit diff
path: root/src/libstore/local-store.cc
AgeCommit message (Collapse)AuthorFilesLines
2018-05-03 LocalStore::addValidPath(): Throw error when the CA assertion is wrongEelco Dolstra1-1/+2
Closes #2133.
2018-04-13 Fix #1921Eelco Dolstra1-2/+10
2018-03-16 Reduce substitution memory consumptionEelco Dolstra1-12/+22
copyStorePath() now pipes the output of srcStore->narFromPath() directly into dstStore->addToStore(). The sink used by the former is converted into a source usable by the latter using boost::coroutine2. This is based on [1]. This reduces the maximum resident size of $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs from 418592 KiB to 53416 KiB. (The previous commit also reduced the runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will be to download files into a Sink. [1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18 Issue #1969.
2018-03-07 Fix #1957Eelco Dolstra1-1/+1
2018-02-12 Fix hang in build-remoteEelco Dolstra1-1/+0
2018-02-12 Fix 'deadlock: trying to re-acquire self-held lock'Eelco Dolstra1-2/+2
This was caused by derivations with 'allowSubstitutes = false'. Such derivations will be built locally. However, if there is another SubstitionGoal that has the output of the first derivation in its closure, then the path will be simultaneously built and substituted. There was a check to catch this situation (via pathIsLockedByMe()), but it no longer worked reliably because substitutions are now done in another thread. (Thus the comment 'It can't happen between here and the lockPaths() call below because we're not allowing multi-threading' was no longer valid.) The fix is to handle the path already being locked in both SubstitutionGoal and DerivationGoal.
2017-09-05 Add automatic garbage collectionEelco Dolstra1-0/+18
Nix can now automatically run the garbage collector during builds or while adding paths to the store. The option "min-free = <bytes>" specifies that Nix should run the garbage collector whenever free space in the Nix store drops below <bytes>. It will then delete garbage until "max-free" bytes are available. Garbage collection during builds is asynchronous; running builds are not paused and new builds are not blocked. However, there also is a synchronous GC run prior to the first build/substitution. Currently, no old GC roots are deleted (as in "nix-collect-garbage -d").
2017-09-05 GC: Don't delete own temproots fileEelco Dolstra1-3/+5
Since file locks are per-process rather than per-file-descriptor, the garbage collector would always acquire a lock on its own temproots file and conclude that it's stale.
2017-07-30 Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim1-43/+43
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-04 Support base-64 hashesEelco Dolstra1-18/+8
Also simplify the Hash API. Fixes #1437.
2017-07-03 Replace a few bool flags with enumsEelco Dolstra1-8/+8
Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-20 Call SetDllDirectory("") after sqlite3 init on cygwinDavid McFarland1-0/+14
Cygwin sqlite3 is patched to call SetDllDirectory("/usr/bin") on init, which affects the current process and is inherited by child processes. It causes DLLs to be loaded from /usr/bin/ before $PATH, which breaks all sorts of things. A typical failures would be header/lib version mismatches (e.g. openssl when running checkPhase on openssh). We'll just set it back to the default value. Note that this is a problem with the cygwin version of sqlite3 (currently 3.18.0). nixpkgs doesn't have the problematic patch.
2017-06-19 macOS: Remove flagsEelco Dolstra1-0/+10
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the path to be garbage-collected or optimised. See https://github.com/NixOS/nixpkgs/issues/25819. + the file from being garbage-collected.
2017-06-14 canonicalisePathMetaData(): Ignore security.selinux attributeEelco Dolstra1-2/+6
Untested, hopefully fixes #1406.
2017-05-31 Remove listxattr assertionEelco Dolstra1-2/+0
It appears that sometimes, listxattr() returns a different value for the query case (i.e. when the buffer size is 0).
2017-05-30 Darwin sandbox: Use sandbox-defaults.sbEelco Dolstra1-1/+1
Issue #759. Also, remove nix.conf from the sandbox since I don't really see a legitimate reason for builders to access the Nix configuration.
2017-05-30 canonicalisePathMetaData(): Remove extended attributes / ACLsEelco Dolstra1-0/+22
EAs/ACLs are not part of the NAR canonicalisation. Worse, setting an ACL allows a builder to create writable files in the Nix store. So get rid of them. Closes #185.
2017-05-29 Fix typoEelco Dolstra1-1/+1
2017-05-11 Change the meaning of info.ultimateEelco Dolstra1-2/+0
It now means "paths that were built locally". It no longer includes paths that were added locally. For those we don't need info.ultimate, since we have the content-addressability assertion (info.ca).
2017-05-11 LocalStore::addToStore(): Check info.narSizeEelco Dolstra1-2/+6
It allowed the client to specify bogus narSize values. In particular, Downloader::downloadCached wasn't setting narSize at all.
2017-05-01 build-remote: Don't require signaturesEelco Dolstra1-0/+2
This restores the old behaviour.
2017-04-13 Convert Settings to the new config systemEelco Dolstra1-2/+2
This makes all config options self-documenting. Unknown or unparseable config settings and --option flags now cause a warning.
2017-04-13 Merge branch 'rework-options' of https://github.com/copumpkin/nixEelco Dolstra1-1/+1
2017-04-13 Add a Config class to simplify adding configuration settingsEelco Dolstra1-2/+3
The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-03-21 Require signatures by defaultEelco Dolstra1-1/+1
This corresponds to the NixOS default.
2017-03-16 copyPaths(): Use queryValidPaths() to reduce SSH latencyEelco Dolstra1-1/+1
2017-03-01 Tweak messageEelco Dolstra1-1/+1
2017-02-24 Verify content-addressability assertions at registration timeEelco Dolstra1-0/+2
2017-02-22 Explicitly model all settings and fail on unrecognized onesDan Peebles1-2/+2
Previously, the Settings class allowed other code to query for string properties, which led to a proliferation of code all over the place making up new options without any sort of central registry of valid options. This commit pulls all those options back into the central Settings class and removes the public get() methods, to discourage future abuses like that. Furthermore, because we know the full set of options ahead of time, we now fail loudly if someone enters an unrecognized option, thus preventing subtle typos. With some template fun, we could probably also dump the full set of options (with documentation, defaults, etc.) to the command line, but I'm not doing that yet here.
2017-02-22 RemoteStore::addToStore(): Pass content-addressability assertionEelco Dolstra1-1/+1
... and use this in Downloader::downloadCached(). This fixes $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-08 Include config.h implicitly with '-include config.h' in CFLAGSTuomas Tynkkynen1-1/+0
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not every compilation unit includes config.h, we currently compile half of Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS unset. This causes major havoc with the Settings class on e.g. 32-bit ARM, where different compilation units disagree with the struct layout. E.g.: diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc @@ -166,6 +166,8 @@ void Settings::update() _get(useSubstitutes, "build-use-substitutes"); + fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes); _get(buildUsersGroup, "build-users-group"); diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc +++ b/src/libstore/remote-store.cc @@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn) void RemoteStore::setOptions(Connection & conn) { + fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes); conn.to << wopSetOptions Gave me: at Settings::update(): &useSubstitutes = 0xb6e5c5cb at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7 That was not a fun one to debug!
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-39/+39
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-39/+39
2016-11-09 Merge branch 'ssh-store' of https://github.com/shlevy/nixEelco Dolstra1-1/+2
2016-10-21 Remove addPathToAccessorEelco Dolstra1-1/+1
2016-10-21 BinaryCacheStore: Optionally write a NAR listingEelco Dolstra1-3/+3
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-12 Shut up some warningsEelco Dolstra1-1/+1
2016-10-07 querySubstitutablePaths(): Don't query paths for which we already have a ↵Eelco Dolstra1-3/+16
substituter
2016-10-07 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths()Eelco Dolstra1-7/+3
2016-09-22 Handle the case where signed-binary-caches consists of whitespaceEelco Dolstra1-1/+1
2016-09-21 printMsg(lvlError, ...) -> printError(...) etc.Eelco Dolstra1-16/+16
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-27/+32
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-02 Factor out the unix domain socket-specific code from RemoteStoreShea Levy1-1/+2
2016-08-24 Fix queryPathFromHashPart()Eelco Dolstra1-1/+1
The inner lambda was returning a SQLite-internal char * rather than a std::string, leading to Hydra errors liks Caught exception in Hydra::Controller::Root->narinfo "path ‘ø˜£â€™ is not in the Nix store at /nix/store/6mvvyb8fgwj23miyal5mdr8ik4ixk15w-hydra-0.1.1234.abcdef/libexec/hydra/lib/Hydra/Controller/Root.pm line 352."
2016-08-10 SQLite:: Add some convenienceEelco Dolstra1-17/+8
2016-08-10 Mark content-addressed paths in the Nix database and in .narinfoEelco Dolstra1-7/+22
This allows such paths to be imported without signatures.
2016-08-10 Add a "root" parameter to local storesEelco Dolstra1-1/+1
This makes it easier to create a diverted store, i.e. NIX_REMOTE="local?root=/tmp/root" instead of NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 Remove $NIX_DB_DIREelco Dolstra1-1/+1
This variable has no reason to exist, given $NIX_STATE_DIR.
2016-07-26 makeFixedOutputPath(): Drop superfluous HashType argumentEelco Dolstra1-4/+4
2016-07-11 Modernize AutoCloseFDShea Levy1-11/+11
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-07T15·52+0000