about summary refs log tree commit diff
path: root/src/libstore/misc.cc
AgeCommit message (Collapse)AuthorFilesLines
2015-07-17 OCD: foreach -> C++11 ranged forEelco Dolstra1-43/+43
2015-06-04 Allow substitutes for builds that have preferLocalBuild setEelco Dolstra1-2/+2
Not substituting builds with "preferLocalBuild = true" was a bad idea, because it didn't take the cost of dependencies into account. For instance, if we can't substitute a fetchgit call, then we have to download/build git and all its dependencies. Partially reverts 5558652709f27e8a887580b77b93c705659d7a4b and adds a new derivation attribute "allowSubstitutes" to specify whether a derivation may be substituted.
2014-08-20 Use proper quotes everywhereEelco Dolstra1-2/+2
2014-04-08 If a .drv cannot be parsed, show its pathEelco Dolstra1-1/+1
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2013-06-20 Don't substitute derivations that have preferLocalBuild setEelco Dolstra1-2/+3
In particular this means that "trivial" derivations such as writeText are not substituted, reducing the number of GET requests to the binary cache by about 200 on a typical NixOS configuration.
2012-12-20 nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settingsEelco Dolstra1-17/+37
So if a path is not garbage solely because it's reachable from a root due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store -q --roots’ now shows that root.
2012-11-26 queryMissing(): Handle partially valid derivationsEelco Dolstra1-5/+6
2012-11-26 Only substitute wanted outputs of a derivationEelco Dolstra1-10/+17
If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26 Make "nix-build -A <derivation>.<output>" do the right thingEelco Dolstra1-4/+7
For example, given a derivation with outputs "out", "man" and "bin": $ nix-build -A pkg produces ./result pointing to the "out" output; $ nix-build -A pkg.man produces ./result-man pointing to the "man" output; $ nix-build -A pkg.all produces ./result, ./result-man and ./result-bin; $ nix-build -A pkg.all -A pkg2 produces ./result, ./result-man, ./result-bin and ./result-2.
2012-07-30 Refactor settings processingEelco Dolstra1-13/+11
Put all Nix configuration flags in a Settings object.
2012-07-06 download-from-binary-cache: parallelise fetching of NAR info filesEelco Dolstra1-27/+79
Getting substitute information using the binary cache substituter has non-trivial latency overhead. A package or NixOS system configuration can have hundreds of dependencies, and in the worst case (when the local info cache is empty) we have to do a separate HTTP request for each of these. If the ping time to the server is t, getting N info files will take tN seconds; e.g., with a ping time of 0.1s to nixos.org, sequentially downloading 1000 info files (a typical NixOS config) will take at least 100 seconds. To fix this problem, the binary cache substituter can now perform requests in parallel. This required changing the substituter interface to support a function querySubstitutablePathInfos() that queries multiple paths at the same time, and rewriting queryMissing() to take advantage of parallelism. (Due to local caching, parallelising queryMissing() is sufficient for most use cases, since it's almost always called before building a derivation and thus fills the local info cache.) For example, parallelism speeds up querying all 1056 paths in a particular NixOS system configuration from 116s to 2.6s. It works so well because the eccentricity of the top-level derivation in the dependency graph is only 9. So we only need 10 round-trips (when using an unlimited number of parallel connections) to get everything. Currently we do a maximum of 150 parallel connections to the server. Thus it's important that the binary cache server (e.g. nixos.org) has a high connection limit. Alternatively we could use HTTP pipelining, but WWW::Curl doesn't support it and libcurl has a hard-coded limit of 5 requests per pipeline.
2012-04-30 * Add an option ‘build-use-substitutes’, which can be set to ‘false’Eelco Dolstra1-1/+3
to disable use of substitutes; i.e., force building from source. Fixes Nix/221.
2011-12-30 * Oops.Eelco Dolstra1-1/+1
2011-12-30 * Move topoSortPaths() out of gc.cc.Eelco Dolstra1-0/+36
2011-08-31 * Eliminate all uses of the global variable ‘store’ from libstore.Eelco Dolstra1-15/+15
This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2010-11-17 * Before a build, show the disk space that the downloaded store pathsEelco Dolstra1-2/+3
will approximately require.
2010-05-12 * Sync with the trunk.Eelco Dolstra1-5/+1
2010-04-19 * Don't use the ATerm library for parsing/printing .drv files.Eelco Dolstra1-5/+1
2010-02-22 * Get derivation outputs from the database instead of the .drv file,Eelco Dolstra1-4/+4
which requires more I/O.
2010-01-25 * Made `nix-store -qR --include-outputs' much faster if there areEelco Dolstra1-2/+9
multiple paths specified on the command line (from O(n * m) to O(n + m), where n is the number of arguments and m is the size of the closure).
2009-04-21 * Use foreach in a lot of places.Eelco Dolstra1-10/+5
2008-08-04 * nix-env --dry-run: show the total size of the substituterEelco Dolstra1-1/+5
downloads.
2008-08-04 * querySubstitutablePathInfo: work properly when run via the daemon.Eelco Dolstra1-6/+8
* --dry-run: print the paths that we don't know how to build/substitute.
2008-08-02 * Make nix-env --dry-run print the paths to be substituted correctlyEelco Dolstra1-3/+7
again. (After the previous substituter mechanism refactoring I didn't update the code that obtains the references of substitutable paths.) This required some refactoring: the substituter programs are now kept running and receive/respond to info requests via stdin/stdout.
2008-06-09 * Merged the no-bdb branch (-r10900:HEADEelco Dolstra1-1/+0
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2007-08-12 * Get rid of the substitutes database table (NIX-47). Instead, if weEelco Dolstra1-4/+3
need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2006-11-30 * More remote operations.Eelco Dolstra1-2/+2
* Added new operation hasSubstitutes(), which is more efficient than querySubstitutes().size() > 0.
2006-11-30 * Put building in the store API.Eelco Dolstra1-2/+1
2006-11-30 * Refactoring. There is now an abstract interface class StoreAPIEelco Dolstra1-9/+9
containing functions that operate on the Nix store. One implementation is LocalStore, which operates on the Nix store directly. The next step, to enable secure multi-user Nix, is to create a different implementation RemoteStore that talks to a privileged daemon process that uses LocalStore to perform the actual operations.
2006-09-04 * Remove unnecessary inclusions of aterm2.h.Eelco Dolstra1-0/+2
2006-09-04 * Use a proper namespace.Eelco Dolstra1-0/+9
* Optimise header file usage a bit. * Compile the parser as C++.
2006-03-06 * `nix-env (-i|-u) --dry-run' now shows exactly which missing pathsEelco Dolstra1-1/+46
will be built or substituted.
2005-12-13 * Change `referer' to `referrer' throughout. In particular, theEelco Dolstra1-1/+1
nix-store query options `--referer' and `--referer-closure' have been changed to `--referrer' and `--referrer-closure' (but the old ones are still accepted for compatibility).
2005-02-14 * It is now possible to add store derivations or paths directly to aEelco Dolstra1-0/+9
user environment, e.g., $ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv or $ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1 This is useful because it allows Nix expressions to be bypassed entirely. For instance, if only a nix-pull manifest is provided, plus the top-level path of some component, it can be installed without having to supply the Nix expression (e.g., for obfuscation, or to be independent of Nix expression language changes or context dependencies).
2005-02-08 * Updated `nix-store --verify' to the new schema.Eelco Dolstra1-2/+2
2005-01-25 * Maintain the references/referers relation also for derivations.Eelco Dolstra1-41/+0
This simplifies garbage collection and `nix-store --query --requisites' since we no longer need to treat derivations specially. * Better maintaining of the invariants, e.g., setReferences() can only be called on a valid/substitutable path.
2005-01-25 * In nix-store: added query `--referers-closure' that returns theEelco Dolstra1-3/+6
closure of the referers relation rather than the references relation, i.e., the set of all paths that directly or indirectly refer to the given path. Note that contrary to the references closure this set is not fixed; it can change as paths are added to or removed from the store.
2005-01-20 * Another change to low-level derivations. The last one this year, IEelco Dolstra1-2/+5
promise :-) This allows derivations to specify on *what* output paths of input derivations they are dependent. This helps to prevent unnecessary downloads. For instance, a build might be dependent on the `devel' and `lib' outputs of some library component, but not the `docs' output.
2005-01-19 * Renamed `normalise.cc' -> `build.cc', `storeexprs.cc' ->Eelco Dolstra1-1/+1
`derivations.cc', etc. * Store the SHA-256 content hash of store paths in the database after they have been built/added. This is so that we can check whether the store has been messed with (a la `rpm --verify'). * When registering path validity, verify that the closure property holds.
2005-01-19 * Change extension `.store' to `.drv'.Eelco Dolstra1-51/+25
* Re-enable `nix-store --query --requisites'.
2005-01-19 * Started removing closure store expressions, i.e., the explicitEelco Dolstra1-6/+23
representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-17 * Removed the `id' attribute hack.Eelco Dolstra1-2/+3
* Formalise the notion of fixed-output derivations, i.e., derivations for which a cryptographic hash of the output is known in advance. Changes to such derivations should not propagate upwards through the dependency graph. Previously this was done by specifying the hash component of the output path through the `id' attribute, but this is insecure since you can lie about it (i.e., you can specify any hash and then produce a completely different output). Now the responsibility for checking the output is moved from the builder to Nix itself. A fixed-output derivation can be created by specifying the `outputHash' and `outputHashAlgo' attributes, the latter taking values `md5', `sha1', and `sha256', and the former specifying the actual hash in hexadecimal or in base-32 (auto-detected by looking at the length of the attribute value). MD5 is included for compatibility but should be considered deprecated. * Removed the `drvPath' pseudo-attribute in derivation results. It's no longer necessary. * Cleaned up the support for multiple output paths in derivation store expressions. Each output now has a unique identifier (e.g., `out', `devel', `docs'). Previously there was no way to tell output paths apart at the store expression level. * `nix-hash' now has a flag `--base32' to specify that the hash should be printed in base-32 notation. * `fetchurl' accepts parameters `sha256' and `sha1' in addition to `md5'. * `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a flag to specify the hash.)
2004-06-18 * Big refactoring. Move to a much more explicitly state machine basedEelco Dolstra1-0/+72
approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).