depot - monorepo for the virus lounge

Age	Commit message (Collapse)	Author	Files	Lines
2016-09-16	Make computeFSClosure() single-threaded again	Eelco Dolstra	1	-40/+64
	The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-07-21	Store::queryMissing(): Use a thread pool	Eelco Dolstra	1	-85/+104
	For one particular NixOS configuration, this cut the runtime of "nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in the size calculation that was causing certain paths to be counted twice, e.g. before: these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked): and after: these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21	Store::computeFSClosure(): Use thread pool	Eelco Dolstra	1	-27/+48
	This speeds up queries against the binary cache.
2016-06-03	Consistent quotes	Eelco Dolstra	1	-1/+1

2016-06-03	Show both cycle ends	Nikolay Amiantov	1	-5/+5

2016-06-02	Make derivationFromPath work on diverted stores	Eelco Dolstra	1	-8/+0

2016-04-19	Move path info caching from BinaryCacheStore to Store	Eelco Dolstra	1	-8/+9
	Caching path info is generally useful. For instance, it speeds up "nix path-info -rS /run/current-system" (i.e. showing the closure sizes of all paths in the closure of the current system) from 5.6s to 0.15s. This also eliminates some APIs like Store::queryDeriver() and Store::queryReferences().
2016-02-04	StoreAPI -> Store	Eelco Dolstra	1	-4/+4
	Calling a class an API is a bit redundant...
2016-02-04	Eliminate the "store" global variable	Eelco Dolstra	1	-58/+50
	Also, move a few free-standing functions into StoreAPI and Derivation. Also, introduce a non-nullable smart pointer, ref<T>, which is just a wrapper around std::shared_ptr ensuring that the pointer is never null. (For reference-counted values, this is better than passing a "T&", because the latter doesn't maintain the refcount. Usually, the caller will have a shared_ptr keeping the value alive, but that's not always the case, e.g., when passing a reference to a std::thread via std::bind.)
2015-07-17	OCD: foreach -> C++11 ranged for	Eelco Dolstra	1	-43/+43

2015-06-04	Allow substitutes for builds that have preferLocalBuild set	Eelco Dolstra	1	-2/+2
	Not substituting builds with "preferLocalBuild = true" was a bad idea, because it didn't take the cost of dependencies into account. For instance, if we can't substitute a fetchgit call, then we have to download/build git and all its dependencies. Partially reverts 5558652709f27e8a887580b77b93c705659d7a4b and adds a new derivation attribute "allowSubstitutes" to specify whether a derivation may be substituted.
2014-08-20	Use proper quotes everywhere	Eelco Dolstra	1	-2/+2

2014-04-08	If a .drv cannot be parsed, show its path	Eelco Dolstra	1	-1/+1
	Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2013-06-20	Don't substitute derivations that have preferLocalBuild set	Eelco Dolstra	1	-2/+3
	In particular this means that "trivial" derivations such as writeText are not substituted, reducing the number of GET requests to the binary cache by about 200 on a typical NixOS configuration.
2012-12-20	nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings	Eelco Dolstra	1	-17/+37
	So if a path is not garbage solely because it's reachable from a root due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store -q --roots’ now shows that root.
2012-11-26	queryMissing(): Handle partially valid derivations	Eelco Dolstra	1	-5/+6

2012-11-26	Only substitute wanted outputs of a derivation	Eelco Dolstra	1	-10/+17
	If a derivation has multiple outputs, then we only want to download those outputs that are actuallty needed. So if we do "nix-build -A openssl.man", then only the "man" output should be downloaded. Likewise if another package depends on ${openssl.man}. The tricky part is that different derivations can depend on different outputs of a given derivation, so we may need to restart the corresponding derivation goal if that happens.
2012-11-26	Make "nix-build -A <derivation>.<output>" do the right thing	Eelco Dolstra	1	-4/+7
	For example, given a derivation with outputs "out", "man" and "bin": $ nix-build -A pkg produces ./result pointing to the "out" output; $ nix-build -A pkg.man produces ./result-man pointing to the "man" output; $ nix-build -A pkg.all produces ./result, ./result-man and ./result-bin; $ nix-build -A pkg.all -A pkg2 produces ./result, ./result-man, ./result-bin and ./result-2.
2012-07-30	Refactor settings processing	Eelco Dolstra	1	-13/+11
	Put all Nix configuration flags in a Settings object.
2012-07-06	download-from-binary-cache: parallelise fetching of NAR info files	Eelco Dolstra	1	-27/+79
	Getting substitute information using the binary cache substituter has non-trivial latency overhead. A package or NixOS system configuration can have hundreds of dependencies, and in the worst case (when the local info cache is empty) we have to do a separate HTTP request for each of these. If the ping time to the server is t, getting N info files will take tN seconds; e.g., with a ping time of 0.1s to nixos.org, sequentially downloading 1000 info files (a typical NixOS config) will take at least 100 seconds. To fix this problem, the binary cache substituter can now perform requests in parallel. This required changing the substituter interface to support a function querySubstitutablePathInfos() that queries multiple paths at the same time, and rewriting queryMissing() to take advantage of parallelism. (Due to local caching, parallelising queryMissing() is sufficient for most use cases, since it's almost always called before building a derivation and thus fills the local info cache.) For example, parallelism speeds up querying all 1056 paths in a particular NixOS system configuration from 116s to 2.6s. It works so well because the eccentricity of the top-level derivation in the dependency graph is only 9. So we only need 10 round-trips (when using an unlimited number of parallel connections) to get everything. Currently we do a maximum of 150 parallel connections to the server. Thus it's important that the binary cache server (e.g. nixos.org) has a high connection limit. Alternatively we could use HTTP pipelining, but WWW::Curl doesn't support it and libcurl has a hard-coded limit of 5 requests per pipeline.
2012-04-30	* Add an option ‘build-use-substitutes’, which can be set to ‘false’	Eelco Dolstra	1	-1/+3
	to disable use of substitutes; i.e., force building from source. Fixes Nix/221.
2011-12-30	* Oops.	Eelco Dolstra	1	-1/+1

2011-12-30	* Move topoSortPaths() out of gc.cc.	Eelco Dolstra	1	-0/+36

2011-08-31	* Eliminate all uses of the global variable ‘store’ from libstore.	Eelco Dolstra	1	-15/+15
	This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2010-11-17	* Before a build, show the disk space that the downloaded store paths	Eelco Dolstra	1	-2/+3
	will approximately require.
2010-05-12	* Sync with the trunk.	Eelco Dolstra	1	-5/+1

2010-04-19	* Don't use the ATerm library for parsing/printing .drv files.	Eelco Dolstra	1	-5/+1

2010-02-22	* Get derivation outputs from the database instead of the .drv file,	Eelco Dolstra	1	-4/+4
	which requires more I/O.
2010-01-25	* Made `nix-store -qR --include-outputs' much faster if there are	Eelco Dolstra	1	-2/+9
	multiple paths specified on the command line (from O(n * m) to O(n + m), where n is the number of arguments and m is the size of the closure).
2009-04-21	* Use foreach in a lot of places.	Eelco Dolstra	1	-10/+5

2008-08-04	* nix-env --dry-run: show the total size of the substituter	Eelco Dolstra	1	-1/+5
	downloads.
2008-08-04	* querySubstitutablePathInfo: work properly when run via the daemon.	Eelco Dolstra	1	-6/+8
	* --dry-run: print the paths that we don't know how to build/substitute.
2008-08-02	* Make nix-env --dry-run print the paths to be substituted correctly	Eelco Dolstra	1	-3/+7
	again. (After the previous substituter mechanism refactoring I didn't update the code that obtains the references of substitutable paths.) This required some refactoring: the substituter programs are now kept running and receive/respond to info requests via stdin/stdout.
2008-06-09	* Merged the no-bdb branch (-r10900:HEAD	Eelco Dolstra	1	-1/+0
	https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2007-08-12	* Get rid of the substitutes database table (NIX-47). Instead, if we	Eelco Dolstra	1	-4/+3
	need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2006-11-30	* More remote operations.	Eelco Dolstra	1	-2/+2
	* Added new operation hasSubstitutes(), which is more efficient than querySubstitutes().size() > 0.
2006-11-30	* Put building in the store API.	Eelco Dolstra	1	-2/+1

2006-11-30	* Refactoring. There is now an abstract interface class StoreAPI	Eelco Dolstra	1	-9/+9
	containing functions that operate on the Nix store. One implementation is LocalStore, which operates on the Nix store directly. The next step, to enable secure multi-user Nix, is to create a different implementation RemoteStore that talks to a privileged daemon process that uses LocalStore to perform the actual operations.
2006-09-04	* Remove unnecessary inclusions of aterm2.h.	Eelco Dolstra	1	-0/+2

2006-09-04	* Use a proper namespace.	Eelco Dolstra	1	-0/+9
	* Optimise header file usage a bit. * Compile the parser as C++.
2006-03-06	* `nix-env (-i\|-u) --dry-run' now shows exactly which missing paths	Eelco Dolstra	1	-1/+46
	will be built or substituted.
2005-12-13	* Change `referer' to `referrer' throughout. In particular, the	Eelco Dolstra	1	-1/+1
	nix-store query options `--referer' and `--referer-closure' have been changed to `--referrer' and `--referrer-closure' (but the old ones are still accepted for compatibility).
2005-02-14	* It is now possible to add store derivations or paths directly to a	Eelco Dolstra	1	-0/+9
	user environment, e.g., $ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv or $ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1 This is useful because it allows Nix expressions to be bypassed entirely. For instance, if only a nix-pull manifest is provided, plus the top-level path of some component, it can be installed without having to supply the Nix expression (e.g., for obfuscation, or to be independent of Nix expression language changes or context dependencies).
2005-02-08	* Updated `nix-store --verify' to the new schema.	Eelco Dolstra	1	-2/+2

2005-01-25	* Maintain the references/referers relation also for derivations.	Eelco Dolstra	1	-41/+0
	This simplifies garbage collection and `nix-store --query --requisites' since we no longer need to treat derivations specially. * Better maintaining of the invariants, e.g., setReferences() can only be called on a valid/substitutable path.
2005-01-25	* In nix-store: added query `--referers-closure' that returns the	Eelco Dolstra	1	-3/+6
	closure of the referers relation rather than the references relation, i.e., the set of all paths that directly or indirectly refer to the given path. Note that contrary to the references closure this set is not fixed; it can change as paths are added to or removed from the store.
2005-01-20	* Another change to low-level derivations. The last one this year, I	Eelco Dolstra	1	-2/+5
	promise :-) This allows derivations to specify on what output paths of input derivations they are dependent. This helps to prevent unnecessary downloads. For instance, a build might be dependent on the `devel' and `lib' outputs of some library component, but not the `docs' output.
2005-01-19	* Renamed `normalise.cc' -> `build.cc', `storeexprs.cc' ->	Eelco Dolstra	1	-1/+1
	`derivations.cc', etc. * Store the SHA-256 content hash of store paths in the database after they have been built/added. This is so that we can check whether the store has been messed with (a la `rpm --verify'). * When registering path validity, verify that the closure property holds.
2005-01-19	* Change extension `.store' to `.drv'.	Eelco Dolstra	1	-51/+25
	* Re-enable `nix-store --query --requisites'.
2005-01-19	* Started removing closure store expressions, i.e., the explicit	Eelco Dolstra	1	-6/+23
	representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.