about summary refs log tree commit diff
path: root/src/libexpr/primops.cc
AgeCommit message (Collapse)AuthorFilesLines
2012-08-13 Add a primop ‘concatLists’Eelco Dolstra1-0/+10
This can serve as a generic efficient list builder. For instance, the function ‘catAttrs’ in Nixpkgs can be rewritten from attr: l: fold (s: l: if hasAttr attr s then [(getAttr attr s)] ++ l else l) [] l to attr: l: builtins.concatLists (map (s: if hasAttr attr s then [(getAttr attr s)] else []) l) Statistics before: time elapsed: 1.08683 size of a value: 24 environments allocated: 1384376 (35809568 bytes) list elements: 6946783 (55574264 bytes) list concatenations: 37434 values allocated: 1760440 (42250560 bytes) attribute sets allocated: 392040 right-biased unions: 186334 values copied in right-biased unions: 591137 symbols in symbol table: 18273 number of thunks: 1297673 number of thunks avoided: 1380759 number of attr lookups: 430802 number of primop calls: 628912 number of function calls: 1333544 Statistics after (including new catAttrs): time elapsed: 0.959854 size of a value: 24 environments allocated: 1010198 (26829296 bytes) list elements: 1984878 (15879024 bytes) list concatenations: 30488 values allocated: 1589760 (38154240 bytes) attribute sets allocated: 392040 right-biased unions: 186334 values copied in right-biased unions: 591137 symbols in symbol table: 18274 number of thunks: 1040925 number of thunks avoided: 1038428 number of attr lookups: 438419 number of primop calls: 474844 number of function calls: 959366
2012-08-13 Provide an efficient implementation of ‘elem’Eelco Dolstra1-2/+17
The one in Nixpkgs is O(n^2), this one is O(n). Big reduction in the number of list allocations. Statistics before (on a NixOS system config): time elapsed: 1.17982 size of a value: 24 environments allocated: 1543334 (39624560 bytes) list elements: 9612638 (76901104 bytes) list concatenations: 37434 values allocated: 1854933 (44518392 bytes) attribute sets allocated: 392040 right-biased unions: 186334 values copied in right-biased unions: 591137 symbols in symbol table: 18272 number of thunks: 1392467 number of thunks avoided: 1507311 number of attr lookups: 430801 number of primop calls: 691600 number of function calls: 1492502 Statistics after: time elapsed: 1.08683 size of a value: 24 environments allocated: 1384376 (35809568 bytes) list elements: 6946783 (55574264 bytes) list concatenations: 37434 values allocated: 1760440 (42250560 bytes) attribute sets allocated: 392040 right-biased unions: 186334 values copied in right-biased unions: 591137 symbols in symbol table: 18273 number of thunks: 1297673 number of thunks avoided: 1380759 number of attr lookups: 430802 number of primop calls: 628912 number of function calls: 1333544
2012-08-13 Add a "filter" primopEelco Dolstra1-0/+25
Evaluation of a NixOS configuration spends quite a lot of time in the "filter" function in Nixpkgs. As implemented in Nixpkgs, this is a O(n^2) operation, so it's a good candidate for providing a more efficient (i.e. primop) implementation. Using it gives a ~10% speed increase and a significant reduction in the number of evaluations. Statistics before (on a NixOS system config): time elapsed: 1.3258 size of a value: 24 environments allocated: 1980939 (50127080 bytes) list elements: 14679308 (117434464 bytes) list concatenations: 50828 values allocated: 2098938 (50374512 bytes) attribute sets allocated: 392040 right-biased unions: 186334 values copied in right-biased unions: 591137 symbols in symbol table: 18271 number of thunks: 1645752 number of thunks avoided: 1921196 number of attr lookups: 430798 number of primop calls: 838807 number of function calls: 1930107 Statistics after: time elapsed: 1.17982 size of a value: 24 environments allocated: 1543334 (39624560 bytes) list elements: 9612638 (76901104 bytes) list concatenations: 37434 values allocated: 1854933 (44518392 bytes) attribute sets allocated: 392040 right-biased unions: 186334 values copied in right-biased unions: 591137 symbols in symbol table: 18272 number of thunks: 1392467 number of thunks avoided: 1507311 number of attr lookups: 430801 number of primop calls: 691600 number of function calls: 1492502
2012-08-12 Add some basic profiling support to the evaluatorEelco Dolstra1-1/+2
Setting the environment variable NIX_COUNT_CALLS to 1 enables some basic profiling in the evaluator. It will count calls to functions and primops as well as evaluations of attributes. For example, to see where evaluation of a NixOS configuration spends its time: $ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 ./src/nix-instantiate/nix-instantiate '<nixos>' -A system --readonly-mode ... calls to 39 primops: 239532 head 233962 tail 191252 hasAttr ... calls to 1595 functions: 224157 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:19' 221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:14' 221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:10' ... evaluations of 7088 attributes: 167377 undefined position 132459 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:119:41' 47322 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:13:21' ...
2012-07-25 prim_import: When importing .drvs, allocate the intermediate attrset on the ↵Shea Levy1-1/+1
heap just in case it escapes the stack frame.
2012-07-25 import: If the path is a valid .drv file, parse it and generate a derivation ↵Shea Levy1-1/+25
attrset. The generated attrset has drvPath and outPath with the right string context, type 'derivation', outputName with the right name, all with a list of outputs, and an attribute for each output. I see three uses for this (though certainly there may be more): * Using derivations generated by something besides nix-instantiate (e.g. guix) * Allowing packages provided by channels to be used in nix expressions. If a channel installed a valid deriver for each package it provides into the store, then those could be imported and used as dependencies or installed in environment.systemPackages, for example. * Enable hydra to be consistent in how it treats inputs that are outputs of another build. Right now, if an input is passed as an argument to the job, it is passed as a derivation, but if it is accessed via NIX_PATH (i.e. through the <> syntax), then it is a path that can be imported. This is problematic because the build being depended upon may have been built with non-obvious arguments passed to its jobset file. With this feature, hydra can just set the name of that input to the path to its drv file in NIX_PATH
2012-07-12 builtins.storePath: resolve symlinksEelco Dolstra1-1/+5
Needed for Charon/Hydra interaction.
2012-06-27 nix-store -r: do substitutions in parallelEelco Dolstra1-1/+1
I.e. when multiple non-derivation arguments are passed to ‘nix-store -r’ to be substituted, do them in parallel.
2012-02-04 * Inline some functions and get rid of the indirection throughEelco Dolstra1-0/+1
EvalState::eval(). This gives a 12% speedup on ‘nix-instantiate /etc/nixos/nixos/ -A system --readonly-mode’ (from 1.01s to 0.89s).
2012-01-26 * Fix importing a derivation. This gave a segfault.Eelco Dolstra1-8/+23
2012-01-19 * Add some debug output to print the derivation name once it's known.Eelco Dolstra1-1/+4
This makes it easier to pinpoint the source of a crash.
2012-01-03 * Drop the inefficient "Path" suffix in output attribute names.Eelco Dolstra1-3/+1
2012-01-03 * Move the implementation of the ‘derivation’ primop into a separateEelco Dolstra1-23/+1
file.
2011-12-21 * The ‘foo.drvPath’ feature was already broken in read-only mode.Eelco Dolstra1-1/+2
Since it's rarely used and fixing it is too much work right now, just document it.
2011-12-21 * Simplify the context handling logic.Eelco Dolstra1-24/+13
2011-12-16 * Sync with the trunk.Eelco Dolstra1-14/+3
2011-12-02 * Move parseHash16or32 into libutil, and use in nix-hash.Eelco Dolstra1-11/+1
2011-11-06 There's no need to mess with drvPath at allShea Levy1-6/+4
2011-11-06 Fix faulty reversion of my changes to unsafeDiscardOutputDependencyShea Levy1-1/+1
2011-11-06 Respect all outputs passed to the derivation, not just the last oneShea Levy1-1/+3
2011-11-06 Embed output name into the context of the *OutPath attributes and extract it ↵Shea Levy1-2/+9
for input derivations Multiple outputs test passes!
2011-11-06 Include all outputs of derivations in the closure of explicitly-passed ↵Shea Levy1-2/+8
derivation paths This required adding a queryOutputDerivationNames function in the store API
2011-11-06 The 'insert output between = signs' approach was not helpfulShea Levy1-9/+2
2011-09-16 Remove the current output metadata from the string for ↵Shea Levy1-1/+8
unsaveDiscardOutputDependency
2011-09-16 Add information about which output is active to drvPath's contextShea Levy1-4/+6
This will break things that depend on being able to just strip away an equals sign, so those have to be updated next
2011-09-16 Add a currentOutput attribute to derivations keep track of which output is ↵Shea Levy1-1/+4
active
2011-09-14 First attempt at the output-as-derivation semanticsShea Levy1-9/+27
For each output, this adds a corresponding attribute to the derivation that is the same as the derivation except for outPath, which is set to the path specific to that output. Additionally, an "all" attribute is added that is a list of all of the output derivations. This has to be done outside of derivationStrict as each output is itself a derivation that contains itself (and all other outputs) as an attribute. The derivation itself is equivalent to the first output in the outputs list (or "out" if that list isn't set).
2011-08-31 * Eliminate all uses of the global variable ‘store’ from libstore.Eelco Dolstra1-4/+4
This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-06 * Refactoring: move parseExprFromFile() and parseExprFromString() intoEelco Dolstra1-2/+1
the EvalState class.
2011-07-20 * Don't allow derivations with fixed and non-fixed outputs.Eelco Dolstra1-43/+38
2011-07-20 * Fix a huuuuge security hole in the Nix daemon. It didn't check thatEelco Dolstra1-61/+3
derivations added to the store by clients have "correct" output paths (meaning that the output paths are computed by hashing the derivation according to a certain algorithm). This means that a malicious user could craft a special .drv file to build *any* desired path in the store with any desired contents (so long as the path doesn't already exist). Then the attacker just needs to wait for a victim to come along and install the compromised path. For instance, if Alice (the attacker) knows that the latest Firefox derivation in Nixpkgs produces the path /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1 then (provided this path doesn't already exist) she can craft a .drv file that creates that path (i.e., has it as one of its outputs), add it to the store using "nix-store --add", and build it with "nix-store -r". So the fake .drv could write a Trojan to the Firefox path. Then, if user Bob (the victim) comes along and does $ nix-env -i firefox $ firefox he executes the Trojan injected by Alice. The fix is to have the Nix daemon verify that derivation outputs are correct (in addValidPath()). This required some refactoring to move the hash computation code to libstore.
2011-07-18 * Support multiple outputs. A derivation can declare multiple outputsEelco Dolstra1-45/+84
by setting the ‘outputs’ attribute. For example: stdenv.mkDerivation { name = "aterm-2.5"; src = ...; outputs = [ "out" "tools" "dev" ]; configureFlags = "--bindir=$(tools)/bin --includedir=$(dev)/include"; } This derivation creates three outputs, named like this: /nix/store/gcnqgllbh01p3d448q8q6pzn2nc2gpyl-aterm-2.5 /nix/store/gjf1sgirwfnrlr0bdxyrwzpw2r304j02-aterm-2.5-tools /nix/store/hp6108bqfgxvza25nnxfs7kj88xi2vdx-aterm-2.5-dev That is, the symbolic name of the output is suffixed to the store path (except for the ‘out’ output). Each path is passed to the builder through the corresponding environment variable, e.g., ${tools}. The main reason for multiple outputs is to allow parts of a package to be distributed and garbage-collected separately. For instance, most packages depend on Glibc for its libraries, but don't need its header files. If these are separated into different store paths, then a package that depends on the Glibc libraries only causes the libraries and not the headers to be downloaded. The main problem with multiple outputs is that if one output exists while the others have been garbage-collected (or never downloaded in the first place), and we want to rebuild the other outputs, then this isn't possible because we can't clobber a valid output (it might be in active use). This currently gives an error message like: error: derivation `/nix/store/1s9zw4c8qydpjyrayxamx2z7zzp5pcgh-aterm-2.5.drv' is blocked by its output paths There are two solutions: 1) Do the build in a chroot. Then we don't need to overwrite the existing path. 2) Use hash rewriting (see the ASE-2005 paper). Scary but it should work. This is not finished yet. There is not yet an easy way to refer to non-default outputs in Nix expressions. Also, mutually recursive outputs aren't detected yet and cause the garbage collector to crash.
2011-01-14 * builtins.substring: if "start" is beyond the end of the string,Eelco Dolstra1-1/+1
return the empty string.
2010-10-24 * When allocating an attribute set, reserve enough space for allEelco Dolstra1-11/+12
elements. This prevents the vector from having to resize itself.
2010-10-24 * Keep attribute sets in sorted order to speed up attribute lookups.Eelco Dolstra1-8/+24
* Simplify the representation of attributes in the AST. * Change the behaviour of listToAttrs() in case of duplicate names.
2010-10-24 * Don't create thunks for variable lookups (if possible). ThisEelco Dolstra1-1/+1
significantly reduces the number of values allocated (e.g. from 8.7m to 4.9m for the Bittorrent test).
2010-10-24 * Store attribute sets as a vector instead of a map (i.e. a red-blackEelco Dolstra1-28/+37
tree). This saves a lot of memory. The vector should be sorted so that names can be looked up using binary search, but this is not the case yet. (Surprisingly, looking up attributes using linear search doesn't have a big impact on performance.) Memory consumption for $ nix-instantiate /etc/nixos/nixos/tests -A bittorrent.test --readonly-mode on x86_64-linux with GC enabled is now 185 MiB (compared to 946 MiB on the trunk).
2010-10-23 * Remove allocValues().Eelco Dolstra1-16/+7
2010-10-22 * Store Value nodes outside of attribute sets. I.e., Attr now storesEelco Dolstra1-36/+34
a pointer to a Value, rather than the Value directly. This improves the effectiveness of garbage collection a lot: if the Value is stored inside the set directly, then any live pointer to the Value causes all other attributes in the set to be live as well.
2010-10-04 * Make sure that config.h is included before the system headers,Eelco Dolstra1-1/+1
because it defines _FILE_OFFSET_BITS. Without this, on OpenSolaris the system headers define it to be 32, and then the 32-bit stat() ends up being called with a 64-bit "struct stat", or vice versa. This also ensures that we get 64-bit file sizes everywhere. * Remove the redundant call to stat() in parseExprFromFile(). The file cannot be a symlink because that's the exit condition of the loop before.
2010-08-02 * intersectAttrs: optimise for the case where the second set is largerEelco Dolstra1-7/+7
than the first set. (That's usually the case with callPackage.)
2010-06-01 * Turn build errors during evaluation into EvalErrors.Eelco Dolstra1-1/+5
2010-05-12 * Implemented tryEval, the last missing primop in the fast-evalEelco Dolstra1-12/+6
branch. Also added a test for tryEval.
2010-05-07 * Sync with the trunk.Eelco Dolstra1-1/+1
2010-05-07 * Updated addErrorContext.Eelco Dolstra1-6/+7
2010-05-07 * Keep track of the source positions of attributes.Eelco Dolstra1-30/+36
2010-04-21 * Fixed builtins.genericClosure.Eelco Dolstra1-22/+54
2010-04-21 * Store user environment manifests as a Nix expression inEelco Dolstra1-14/+17
$out/manifest.nix rather than as an ATerm. (Hm, I thought I committed this two days ago...)
2010-04-19 * Don't use the ATerm library for parsing/printing .drv files.Eelco Dolstra1-1/+1
2010-04-16 * Updated some more primops.Eelco Dolstra1-48/+23
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-10-07T16·47+0000