about summary refs log tree commit diff
path: root/src/libstore/store-api.hh
AgeCommit message (Collapse)AuthorFilesLines
2018-03-16 Reduce substitution memory consumptionEelco Dolstra1-1/+6
copyStorePath() now pipes the output of srcStore->narFromPath() directly into dstStore->addToStore(). The sink used by the former is converted into a source usable by the latter using boost::coroutine2. This is based on [1]. This reduces the maximum resident size of $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs from 418592 KiB to 53416 KiB. (The previous commit also reduced the runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will be to download files into a Sink. [1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18 Issue #1969.
2018-02-07 Merge pull request #1816 from shlevy/add-pathEelco Dolstra1-3/+3
Add path primop.
2018-02-06 Add path primop.Shea Levy1-3/+3
builtins.path allows specifying the name of a path (which makes paths with store-illegal names now addable), allows adding paths with flat instead of recursive hashes, allows specifying a filter (so is a generalization of filterSource), and allows specifying an expected hash (enabling safe path adding in pure mode).
2018-02-05 Allow substituters to be marked as trustedEelco Dolstra1-0/+2
This is needed by nixos-install, which uses the Nix store on the installation CD as a substituter. We don't want to disable signature checking entirely because substitutes from cache.nixos.org should still be checked. So now we can pas "local?trusted=1" to mark only the Nix store in /nix as not requiring signatures. Fixes #1819.
2018-02-03 Remove nix-build --hashEelco Dolstra1-1/+1
Instead, if a fixed-output derivation produces has an incorrect output hash, we now unconditionally move the outputs to the path corresponding with the actual hash and register it as valid. Thus, after correcting the hash in the Nix expression (e.g. in a fetchurl call), the fixed-output derivation doesn't have to be built again. It would still be good to have a command for reporting the actual hash of a fixed-output derivation (instead of throwing an error), but "nix-build --hash" didn't do that.
2018-01-31 document ability to set NIX_REMOTE=unix://path/to/socketSpencer Baugh1-0/+3
2018-01-12 import, builtins.readFile: Handle diverted storesEelco Dolstra1-2/+8
Fixes #1791
2017-10-23 Turn $NIX_REMOTE into a configuration optionEelco Dolstra1-2/+3
2017-07-30 Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim1-1/+1
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-14 nix path-info: Show download sizes for binary cache storesEelco Dolstra1-1/+1
E.g. $ nix path-info --json --store https://cache.nixos.org nixpkgs.thunderbird -S ... "downloadHash": "sha256:1jlixpzi225wwa0f4xdrwrqgi47ip1qpj9p06fyxxg07sfmyi4q0", "downloadSize": 43047620, "closureDownloadSize": 84745960 } ]
2017-07-14 nix path-info: Don't barf on invalid pathsEelco Dolstra1-1/+3
Now you get [ { "path": "/nix/store/fzvliz4j5xzvnd0w5zgw2l0ksqh578yk-bla", "valid": false } ]
2017-07-04 Sort substituters by priorityEelco Dolstra1-0/+5
Fixes #1438.
2017-07-03 Replace a few bool flags with enumsEelco Dolstra1-11/+27
Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-05-11 Change the meaning of info.ultimateEelco Dolstra1-3/+2
It now means "paths that were built locally". It no longer includes paths that were added locally. For those we don't need info.ultimate, since we have the content-addressability assertion (info.ca).
2017-05-02 build-remote: Fix fallback to other machines when connecting failsEelco Dolstra1-0/+4
Opening an SSHStore or LegacySSHStore does not actually establish a connection, so the try/catch block here did nothing. Added a Store::connect() method to test whether a connection can be established.
2017-05-01 build-remote: Don't require signaturesEelco Dolstra1-5/+2
This restores the old behaviour.
2017-05-01 Support arbitrary store URIs in nix.machinesEelco Dolstra1-6/+17
For backwards compatibility, if the URI is just a hostname, ssh:// (i.e. LegacySSHStore) is prepended automatically. Also, all fields except the URI are now optional. For example, this is a valid nix.machines file: local?root=/tmp/nix This is useful for testing the remote build machinery since you don't have to mess around with ssh.
2017-05-01 Add a dummy Store::buildPaths() methodEelco Dolstra1-7/+13
This default implementation of buildPaths() does nothing if all requested paths are already valid, and throws an "unsupported operation" error otherwise. This fixes a regression introduced by c30330df6f67c81986dfb124631bc756c8e58c0d in binary cache and legacy SSH stores.
2017-04-13 Add a Config class to simplify adding configuration settingsEelco Dolstra1-5/+20
The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-10 Minor cleanupEelco Dolstra1-6/+1
Also, possible fix for #1310 on 32-bit systems.
2017-04-06 Implement RemoteStore::queryMissing()Eelco Dolstra1-1/+1
This provides a significant speedup, e.g. 64 s -> 12 s for nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test on a cold local and CloudFront cache. The alternative is to use lots of concurrent daemon connections but that seems wasteful.
2017-04-06 Add a method to allow hydra-queue-runner to flush the path info cacheEelco Dolstra1-5/+7
2017-04-06 nix-daemon: Disable path info cacheEelco Dolstra1-1/+6
This is useless because the client also caches path info, and can cause problems for long-running clients like hydra-queue-runner (i.e. it may return cached info about paths that have been garbage-collected).
2017-03-16 copyPaths(): Use queryValidPaths() to reduce SSH latencyEelco Dolstra1-3/+5
2017-03-16 LegacySSHStore: Provide a faster implementation of computeFSClosure()Eelco Dolstra1-1/+1
This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-15 Store: Add a method for getting build logsEelco Dolstra1-0/+8
This allows various Store implementations to provide different ways to get build logs. For example, BinaryCacheStore can get the build logs from the binary cache. Also, remove the log-servers option since we can use substituters for this.
2017-03-03 build-remote: Don't use a SSH masterEelco Dolstra1-0/+2
This is unnecessary because we make only one connection.
2017-02-22 RemoteStore::addToStore(): Pass content-addressability assertionEelco Dolstra1-1/+6
... and use this in Downloader::downloadCached(). This fixes $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-07 Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nixEelco Dolstra1-1/+1
2017-02-07 copyStorePath(): Don't require signatures for "trusted" storesEelco Dolstra1-0/+4
For example, SSH stores could be trusted.
2017-02-07 Provide default implementations for a couple of Store methodsEelco Dolstra1-7/+7
2017-01-26 exportReferencesGraph: Export more complete info in JSON formatEelco Dolstra1-0/+14
This writes info about every path in the closure in the same format as ‘nix path-info --json’. Thus it also includes NAR hashes and sizes. Example: [ { "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl", "narSize": 197648, "references": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24" ], "closureSize": 20939776 }, { "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24", "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3", "narSize": 20742128, "references": [ "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24" ], "closureSize": 20742128 } ] Fixes #1134.
2017-01-20 nix-copy-closure: Implement in C++.Shea Levy1-1/+1
Tests fail currently because the database is not given proper hashes in the VM
2017-01-19 Merge pull request #981 from shlevy/build-remote-c++Eelco Dolstra1-0/+2
build-remote: Implement in C++
2016-12-07 Keep track of the exact build start/stop timesEelco Dolstra1-1/+3
2016-12-07 Add an option to make non-determinism non-fatalEelco Dolstra1-0/+11
That is, when build-repeat > 0, and the output of two rounds differ, then print a warning rather than fail the build. This is primarily to let Hydra check reproducibility of all packages.
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-19/+19
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-19/+19
2016-11-10 Store::computeFSClosure(): Support a set of pathsEelco Dolstra1-3/+7
This way, callers can exploits the parallelism of computeFSClosure() when they have multiple paths that they need the (combined) closure of.
2016-11-10 build-remote: Implement in C++Shea Levy1-0/+2
2016-11-09 copyClosure() / copyStorePath(): Expose dontCheckSigsEelco Dolstra1-2/+2
Needed by Hydra.
2016-11-09 Merge branch 'ssh-store' of https://github.com/shlevy/nixEelco Dolstra1-4/+10
2016-10-26 Restore the CachedFailure status codeEelco Dolstra1-0/+4
The removal of CachedFailure caused the value of TimedOut to change, which broke timed-out handling in Hydra (so timed-out builds would show up as "aborted" and would be retried, e.g. at http://hydra.nixos.org/build/42537427).
2016-10-21 Remove addPathToAccessorEelco Dolstra1-12/+2
2016-10-21 BinaryCacheStore: Optionally write a NAR listingEelco Dolstra1-1/+1
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-07 Implement generic Store::queryValidPaths()Eelco Dolstra1-1/+1
2016-10-07 Add copyClosure utility function for HydraEelco Dolstra1-0/+5
2016-10-07 importPaths(): Fix accessor support for HydraEelco Dolstra1-0/+11
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-1/+8
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-02 Factor out the unix domain socket-specific code from RemoteStoreShea Levy1-1/+1
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-09-08T13·09+0000