about summary refs log tree commit diff
path: root/src/libstore/misc.cc
AgeCommit message (Collapse)AuthorFilesLines
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-40/+64
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-07-21 Store::queryMissing(): Use a thread poolEelco Dolstra1-85/+104
For one particular NixOS configuration, this cut the runtime of "nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in the size calculation that was causing certain paths to be counted twice, e.g. before: these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked): and after: these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21 Store::computeFSClosure(): Use thread poolEelco Dolstra1-27/+48
This speeds up queries against the binary cache.
2016-06-03 Consistent quotesEelco Dolstra1-1/+1
2016-06-03 Show both cycle endsNikolay Amiantov1-5/+5
2016-06-02 Make derivationFromPath work on diverted storesEelco Dolstra1-8/+0
2016-04-19 Move path info caching from BinaryCacheStore to StoreEelco Dolstra1-8/+9
Caching path info is generally useful. For instance, it speeds up "nix path-info -rS /run/current-system" (i.e. showing the closure sizes of all paths in the closure of the current system) from 5.6s to 0.15s. This also eliminates some APIs like Store::queryDeriver() and Store::queryReferences().
2016-02-04 StoreAPI -> StoreEelco Dolstra1-4/+4
Calling a class an API is a bit redundant...
2016-02-04 Eliminate the "store" global variableEelco Dolstra1-58/+50
Also, move a few free-standing functions into StoreAPI and Derivation. Also, introduce a non-nullable smart pointer, ref<T>, which is just a wrapper around std::shared_ptr ensuring that the pointer is never null. (For reference-counted values, this is better than passing a "T&", because the latter doesn't maintain the refcount. Usually, the caller will have a shared_ptr keeping the value alive, but that's not always the case, e.g., when passing a reference to a std::thread via std::bind.)
2015-07-17 OCD: foreach -> C++11 ranged forEelco Dolstra1-43/+43
2015-06-04 Allow substitutes for builds that have preferLocalBuild setEelco Dolstra1-2/+2
Not substituting builds with "preferLocalBuild = true" was a bad idea, because it didn't take the cost of dependencies into account. For instance, if we can't substitute a fetchgit call, then we have to download/build git and all its dependencies. Partially reverts 5558652709f27e8a887580b77b93c705659d7a4b and adds a new derivation attribute "allowSubstitutes" to specify whether a derivation may be substituted.
2014-08-20 Use proper quotes everywhereEelco Dolstra1-2/+2
2014-04-08 If a .drv cannot be parsed, show its pathEelco Dolstra1-1/+1
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2013-06-20 Don't substitute derivations that have preferLocalBuild setEelco Dolstra1-2/+3
In particular this means that "trivial" derivations such as writeText are not substituted, reducing the number of GET requests to the binary cache by about 200 on a typical NixOS configuration.
2012-12-20 nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settingsEelco Dolstra1-17/+37
So if a path is not garbage solely because it's reachable from a root due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store -q --roots’ now shows that root.
2012-11-26 queryMissing(): Handle partially valid derivationsEelco Dolstra1-5/+6
2012-11-26 Only substitute wanted outputs of a derivationEelco Dolstra1-10/+17
If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 Make "nix-build -A <derivation>.<output>" do the right thingEelco Dolstra1-4/+7
For example, given a derivation with outputs "out", "man" and "bin": $ nix-build -A pkg produces ./result pointing to the "out" output; $ nix-build -A pkg.man produces ./result-man pointing to the "man" output; $ nix-build -A pkg.all produces ./result, ./result-man and ./result-bin; $ nix-build -A pkg.all -A pkg2 produces ./result, ./result-man, ./result-bin and ./result-2.
2012-07-30 Refactor settings processingEelco Dolstra1-13/+11
Put all Nix configuration flags in a Settings object.
2012-07-06 download-from-binary-cache: parallelise fetching of NAR info filesEelco Dolstra1-27/+79
Getting substitute information using the binary cache substituter has non-trivial latency overhead. A package or NixOS system configuration can have hundreds of dependencies, and in the worst case (when the local info cache is empty) we have to do a separate HTTP request for each of these. If the ping time to the server is t, getting N info files will take tN seconds; e.g., with a ping time of 0.1s to nixos.org, sequentially downloading 1000 info files (a typical NixOS config) will take at least 100 seconds. To fix this problem, the binary cache substituter can now perform requests in parallel. This required changing the substituter interface to support a function querySubstitutablePathInfos() that queries multiple paths at the same time, and rewriting queryMissing() to take advantage of parallelism. (Due to local caching, parallelising queryMissing() is sufficient for most use cases, since it's almost always called before building a derivation and thus fills the local info cache.) For example, parallelism speeds up querying all 1056 paths in a particular NixOS system configuration from 116s to 2.6s. It works so well because the eccentricity of the top-level derivation in the dependency graph is only 9. So we only need 10 round-trips (when using an unlimited number of parallel connections) to get everything. Currently we do a maximum of 150 parallel connections to the server. Thus it's important that the binary cache server (e.g. nixos.org) has a high connection limit. Alternatively we could use HTTP pipelining, but WWW::Curl doesn't support it and libcurl has a hard-coded limit of 5 requests per pipeline.
2012-04-30 * Add an option ‘build-use-substitutes’, which can be set to ‘false’Eelco Dolstra1-1/+3
to disable use of substitutes; i.e., force building from source. Fixes Nix/221.
2011-12-30 * Oops.Eelco Dolstra1-1/+1
2011-12-30 * Move topoSortPaths() out of gc.cc.Eelco Dolstra1-0/+36
2011-08-31 * Eliminate all uses of the global variable ‘store’ from libstore.Eelco Dolstra1-15/+15
This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2010-11-17 * Before a build, show the disk space that the downloaded store pathsEelco Dolstra1-2/+3
will approximately require.
2010-05-12 * Sync with the trunk.Eelco Dolstra1-5/+1
2010-04-19 * Don't use the ATerm library for parsing/printing .drv files.Eelco Dolstra1-5/+1
2010-02-22 * Get derivation outputs from the database instead of the .drv file,Eelco Dolstra1-4/+4
which requires more I/O.
2010-01-25 * Made `nix-store -qR --include-outputs' much faster if there areEelco Dolstra1-2/+9
multiple paths specified on the command line (from O(n * m) to O(n + m), where n is the number of arguments and m is the size of the closure).
2009-04-21 * Use foreach in a lot of places.Eelco Dolstra1-10/+5
2008-08-04 * nix-env --dry-run: show the total size of the substituterEelco Dolstra1-1/+5
downloads.
2008-08-04 * querySubstitutablePathInfo: work properly when run via the daemon.Eelco Dolstra1-6/+8
* --dry-run: print the paths that we don't know how to build/substitute.
2008-08-02 * Make nix-env --dry-run print the paths to be substituted correctlyEelco Dolstra1-3/+7
again. (After the previous substituter mechanism refactoring I didn't update the code that obtains the references of substitutable paths.) This required some refactoring: the substituter programs are now kept running and receive/respond to info requests via stdin/stdout.
2008-06-09 * Merged the no-bdb branch (-r10900:HEADEelco Dolstra1-1/+0
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2007-08-12 * Get rid of the substitutes database table (NIX-47). Instead, if weEelco Dolstra1-4/+3
need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2006-11-30 * More remote operations.Eelco Dolstra1-2/+2
* Added new operation hasSubstitutes(), which is more efficient than querySubstitutes().size() > 0.
2006-11-30 * Put building in the store API.Eelco Dolstra1-2/+1
2006-11-30 * Refactoring. There is now an abstract interface class StoreAPIEelco Dolstra1-9/+9
containing functions that operate on the Nix store. One implementation is LocalStore, which operates on the Nix store directly. The next step, to enable secure multi-user Nix, is to create a different implementation RemoteStore that talks to a privileged daemon process that uses LocalStore to perform the actual operations.
2006-09-04 * Remove unnecessary inclusions of aterm2.h.Eelco Dolstra1-0/+2
2006-09-04 * Use a proper namespace.Eelco Dolstra1-0/+9
* Optimise header file usage a bit. * Compile the parser as C++.
2006-03-06 * `nix-env (-i|-u) --dry-run' now shows exactly which missing pathsEelco Dolstra1-1/+46
will be built or substituted.
2005-12-13 * Change `referer' to `referrer' throughout. In particular, theEelco Dolstra1-1/+1
nix-store query options `--referer' and `--referer-closure' have been changed to `--referrer' and `--referrer-closure' (but the old ones are still accepted for compatibility).
2005-02-14 * It is now possible to add store derivations or paths directly to aEelco Dolstra1-0/+9
user environment, e.g., $ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv or $ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1 This is useful because it allows Nix expressions to be bypassed entirely. For instance, if only a nix-pull manifest is provided, plus the top-level path of some component, it can be installed without having to supply the Nix expression (e.g., for obfuscation, or to be independent of Nix expression language changes or context dependencies).
2005-02-08 * Updated `nix-store --verify' to the new schema.Eelco Dolstra1-2/+2
2005-01-25 * Maintain the references/referers relation also for derivations.Eelco Dolstra1-41/+0
This simplifies garbage collection and `nix-store --query --requisites' since we no longer need to treat derivations specially. * Better maintaining of the invariants, e.g., setReferences() can only be called on a valid/substitutable path.
2005-01-25 * In nix-store: added query `--referers-closure' that returns theEelco Dolstra1-3/+6
closure of the referers relation rather than the references relation, i.e., the set of all paths that directly or indirectly refer to the given path. Note that contrary to the references closure this set is not fixed; it can change as paths are added to or removed from the store.
2005-01-20 * Another change to low-level derivations. The last one this year, IEelco Dolstra1-2/+5
promise :-) This allows derivations to specify on *what* output paths of input derivations they are dependent. This helps to prevent unnecessary downloads. For instance, a build might be dependent on the `devel' and `lib' outputs of some library component, but not the `docs' output.
2005-01-19 * Renamed `normalise.cc' -> `build.cc', `storeexprs.cc' ->Eelco Dolstra1-1/+1
`derivations.cc', etc. * Store the SHA-256 content hash of store paths in the database after they have been built/added. This is so that we can check whether the store has been messed with (a la `rpm --verify'). * When registering path validity, verify that the closure property holds.
2005-01-19 * Change extension `.store' to `.drv'.Eelco Dolstra1-51/+25
* Re-enable `nix-store --query --requisites'.
2005-01-19 * Started removing closure store expressions, i.e., the explicitEelco Dolstra1-6/+23
representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.