about summary refs log tree commit diff
path: root/src/libstore/binary-cache-store.cc
AgeCommit message (Collapse)AuthorFilesLines
2018-08-06 Fix Brotli decompression in 'nix log'Eelco Dolstra1-12/+11
This didn't work anymore since decompression was only done in the non-coroutine case. Decompressors are now sinks, just like compressors. Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK rather than BZ_OK), which we didn't notice because there was a missing 'throw': if (ret != BZ_OK) CompressionError("error while compressing bzip2 file");
2018-06-05 Don't require --fallback to recover from disappeared binary cache NARsEelco Dolstra1-1/+5
2018-05-30 Make LocalBinaryCacheStore::narFromPath() run in constant memoryEelco Dolstra1-9/+27
This reduces memory consumption of nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192 MiB to 53 MiB for a bzipped binary cache. It may also be faster because fetching can happen concurrently with decompression/writing. Continuation of 48662d151bdf4a38670897beacea9d1bd750376a. Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 Simplify the callback mechanismEelco Dolstra1-16/+21
2018-03-16 decompress(): Use a Source and SinkEelco Dolstra1-12/+8
This allows decompression to happen in O(1) memory.
2018-02-07 move the parallel-compression setting to binary-cache-store, the settingAmineChikhaoui1-1/+1
can be done now from the url e.g s3://nix-cache?parallel-compression=1 instead of nix.conf.
2018-02-07 make multi threaded compression configurable and use single threadedAmineChikhaoui1-1/+1
by default.
2017-12-07 Provide random access to cached NARsEelco Dolstra1-8/+4
E.g. $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \ /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null real 0m4.139s $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \ /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null real 0m0.024s (Before, the second call took ~0.220s.) This will use a NAR listing in /tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata, including the offsets of regular files inside the NAR. Thus, we don't need to read the entire NAR. (We do read the entire listing, but that's generally pretty small. We could use a SQLite DB by borrowing some more code from nixos-channel-scripts/file-cache.hh.) This is primarily useful when Hydra is serving files from an S3 binary cache, in particular when you have giant NARs. E.g. we had some 12 GiB NARs, so accessing individuals files was pretty slow.
2017-11-14 nix sign-paths: Support binary cachesEelco Dolstra1-13/+34
2017-11-14 nix ls-{nar,store} --json: Respect -REelco Dolstra1-1/+1
2017-11-14 nix ls-{store,nar}: Add --json flagEelco Dolstra1-34/+1
2017-10-19 Revert "Let's not populate the NAR cache from hydra-queue-runner for now"Eelco Dolstra1-2/+2
This reverts commit 908590dc6cfcca3a98755b194d93b2da39aee95c. Since hydra-server can have a different store URI from hydra-queue-runner now, we don't really need this.
2017-10-18 Let's not populate the NAR cache from hydra-queue-runner for nowEelco Dolstra1-2/+2
2017-10-17 BinaryCacheStore::addToStore(): Add NARs to the local cacheEelco Dolstra1-2/+6
2017-10-17 BinaryCacheStore: Support local caching of NARsEelco Dolstra1-1/+1
This speeds up commands like "nix cat-store". For example: $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf real 0m4.336s $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf real 0m0.045s The primary motivation is to allow hydra-server to serve files from S3 binary caches. Previously Hydra had a hack to do "nix-store -r <path>", but that fetches the entire closure so is prohibitively expensive. There is no garbage collection of the NAR cache yet. Also, the entire NAR is read when accessing a single member file. We could generate the NAR listing to provide random access. Note: the NAR cache is indexed by the store path hash, not the content hash, so NAR caches should not be shared between binary caches, unless you're sure that all your builds are binary-reproducible.
2017-10-17 Remove BinaryCacheStoreAccessorEelco Dolstra1-61/+1
Probably as a result of a bad merge in 4b8f1b0ec066a5b994747b1afd050f5f62d857f6, we had both a BinaryCacheStoreAccessor and a RemoteFSAccessor. BinaryCacheStore::getFSAccessor() returned the latter, but BinaryCacheStore::addToStore() checked for the former. This probably caused hydra-queue-runner to download paths that it just uploaded.
2017-08-31 Add an activity for binary cache queriesEelco Dolstra1-0/+7
2017-07-30 Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim1-9/+9
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-04 Support base-64 hashesEelco Dolstra1-1/+1
Also simplify the Hash API. Fixes #1437.
2017-07-03 Replace a few bool flags with enumsEelco Dolstra1-5/+5
Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-05-01 Add a dummy Store::buildPaths() methodEelco Dolstra1-5/+0
This default implementation of buildPaths() does nothing if all requested paths are already valid, and throws an "unsupported operation" error otherwise. This fixes a regression introduced by c30330df6f67c81986dfb124631bc756c8e58c0d in binary cache and legacy SSH stores.
2017-04-13 Add a Config class to simplify adding configuration settingsEelco Dolstra1-3/+0
The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-03-15 Compress NAR listings using the "text-compression" methodEelco Dolstra1-1/+1
So if "text-compression=br", the .ls file in S3 will get a Content-Encoding of "br". Brotli appears to compress better than xz for this kind of file and is natively supported by browsers.
2017-03-15 S3BinaryCacheStore: Set Content-TypeEelco Dolstra1-4/+4
This is necessary for serving log files to browsers.
2017-03-15 S3BinaryCacheStore: Support compression of narinfo and log filesEelco Dolstra1-0/+1
You can now set the store parameter "text-compression=br" to compress textual files in the binary cache (i.e. narinfo and logs) using Brotli. This sets the Content-Encoding header; the extension of compressed files is unchanged. You can separately specify the compression of log files using "log-compression=br". This is useful when you don't want to compress narinfo files for backward compatibility.
2017-03-15 BinaryCacheStore: Implement getBuildLog()Eelco Dolstra1-0/+24
We assume that build logs are stored under log/<drv>, e.g. /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv maps to https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-9/+9
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-9/+9
2016-11-09 Merge branch 'ssh-store' of https://github.com/shlevy/nixEelco Dolstra1-4/+3
2016-10-21 Remove addPathToAccessorEelco Dolstra1-75/+77
2016-10-21 BinaryCacheStore: Optionally write a NAR listingEelco Dolstra1-9/+64
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-07 importPaths(): Fix accessor support for HydraEelco Dolstra1-0/+8
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-6/+26
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-02 Factor a general remote FS accessor out of BinaryCacheStoreShea Levy1-64/+2
2016-08-10 FSAccessor: Throw InvalidPathEelco Dolstra1-1/+1
2016-07-26 makeFixedOutputPath(): Drop superfluous HashType argumentEelco Dolstra1-1/+1
2016-06-01 Make the store directory a member variable of StoreEelco Dolstra1-7/+8
2016-05-31 nix-copy-closure / build-remote.pl: Disable signature checkingEelco Dolstra1-1/+2
This restores the Nix 1.11 behaviour.
2016-05-30 Re-implement the WantMassQuery property of binary cachesEelco Dolstra1-1/+20
2016-05-30 BinaryCacheStore: Remove buildPaths() / ensurePath()Eelco Dolstra1-61/+2
2016-05-04 Add a Store::addToStore() variant that accepts a NAREelco Dolstra1-91/+14
As a side effect, this ensures that signatures are propagated when copying paths between stores. Also refactored import/export to make use of this.
2016-05-04 Remove OpenSSL-based signingEelco Dolstra1-6/+3
2016-04-29 BinaryCacheStore: Support bzip2 compressionEelco Dolstra1-1/+4
2016-04-29 Better error messageEelco Dolstra1-1/+6
2016-04-29 BinaryCacheStore: Support "none" compression methodEelco Dolstra1-23/+18
2016-04-29 BinaryCacheStore: Make the signing key a parameterEelco Dolstra1-1/+2
2016-04-29 BinaryCacheStore::queryPathInfo: Don't check signaturesEelco Dolstra1-9/+1
Other stores don't do this either. It's up to the caller to check signatures.
2016-04-28 Fix error messageEelco Dolstra1-1/+1
2016-04-22 BinaryCacheStore: When adding a path, ensure the references are validEelco Dolstra1-0/+11
This prevents copying a partial closure to a binary cache.
2016-04-21 Implement S3BinaryCacheStore::queryAllValidPaths()Eelco Dolstra1-4/+4
This allows commands like "nix verify --all" or "nix path-info --all" to work on S3 caches. Unfortunately, this requires some ugly hackery: when querying the contents of the bucket, we don't want to have to read every .narinfo file. But the S3 bucket keys only include the hash part of each store path, not the name part. So as a special exception queryAllValidPaths() can now return store paths *without* the name part, and queryPathInfo() accepts such store paths (returning a ValidPathInfo object containing the full name).
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-21T23·40+0000