about summary refs log tree commit diff
path: root/src/libstore/download.hh (follow)
AgeCommit message (Collapse)AuthorFilesLines
2018-08-06 Fix Brotli decompression in 'nix log'Eelco Dolstra1-3/+0
This didn't work anymore since decompression was only done in the non-coroutine case. Decompressors are now sinks, just like compressors. Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK rather than BZ_OK), which we didn't notice because there was a missing 'throw': if (ret != BZ_OK) CompressionError("error while compressing bzip2 file");
2018-08-06 Fix reporting of HTTP body size when a result callback is usedEelco Dolstra1-0/+1
2018-06-05 Further improve upload messagesEelco Dolstra1-0/+5
2018-05-30 Make HttpBinaryCacheStore::narFromPath() run in constant memoryEelco Dolstra1-0/+5
This reduces memory consumption of nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 176 MiB to 82 MiB. (The remaining memory is probably due to xz decompression overhead.) Issue https://github.com/NixOS/nix/issues/1681. Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 Simplify the callback mechanismEelco Dolstra1-2/+1
2018-05-09 make sure not to use cached channels for nix-channel --updateYorick van Pelt1-1/+2
fixes #1964
2018-03-12 Wrap thread local in function for CygwinAsad Saeeduddin1-1/+1
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-01-31 CleanupEelco Dolstra1-2/+2
2018-01-26 HttpBinaryCacheStore: Support upsertFile with PUT.Shea Levy1-2/+4
Some servers, such as Artifactory, allow uploading with PUT and BASIC auth. This allows nix copy to work to upload binaries to those servers. Worked on together with @adelbertc
2017-09-18 Don't remove Content-Encoding in fetchurl / nix-prefetch-urlEelco Dolstra1-0/+1
Fixes #1568.
2017-08-25 Allow activities to be nestedEelco Dolstra1-1/+3
In particular, this allows more relevant activities ("substituting X") to supersede inferior ones ("downloading X").
2017-05-16 Improve progress indicatorEelco Dolstra1-1/+0
2017-04-10 Default to 5 download retriesDan Peebles1-1/+1
This should help certain downloaders that don't request anything special for the number of retries, like nix-channel.
2017-03-15 S3BinaryCacheStore: Support compression of narinfo and log filesEelco Dolstra1-0/+3
You can now set the store parameter "text-compression=br" to compress textual files in the binary cache (i.e. narinfo and logs) using Brotli. This sets the Content-Encoding header; the extension of compressed files is unchanged. You can separately specify the compression of log files using "log-compression=br". This is useful when you don't want to compress narinfo files for backward compatibility.
2017-02-14 Add support for s3:// URIsEelco Dolstra1-1/+1
This adds support for s3:// URIs in all places where Nix allows URIs, e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix> and NIX_PATH. It allows fetching resources from private S3 buckets, using credentials obtained from the standard places (i.e. AWS_* environment variables, ~/.aws/credentials and the EC2 metadata server). This may not be super-useful in general, but since we already depend on aws-sdk-cpp, it's a cheap feature to add.
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-1/+1
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-1/+1
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-4/+6
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-14 Enable HTTP/2 supportEelco Dolstra1-7/+28
The binary cache store can now use HTTP/2 to do lookups. This is much more efficient than HTTP/1.1 due to multiplexing: we can issue many requests in parallel over a single TCP connection. Thus it's no longer necessary to use a bunch of concurrent TCP connections (25 by default). For example, downloading 802 .narinfo files from https://cache.nixos.org/, using a single TCP connection, takes 11.8s with HTTP/1.1, but only 0.61s with HTTP/2. This did require a fairly substantial rewrite of the Downloader class to use the curl multi interface, because otherwise curl wouldn't be able to do multiplexing for us. As a bonus, we get connection reuse even with HTTP/1.1. All downloads are now handled by a single worker thread. Clients call Downloader::enqueueDownload() to tell the worker thread to start the download, getting a std::future to the result.
2016-08-31 download.hh: Fix conflicts from nix-channel-c++ mergeShea Levy1-6/+1
2016-08-31 Merge branch 'nix-channel-c++'Shea Levy1-0/+6
2016-08-30 Increase the sleep time between download retriesEelco Dolstra1-0/+1
2016-08-15 builtins.fetch{url,tarball}: Allow name attributeShea Levy1-1/+1
2016-08-11 nix-channel: implement in c++Shea Levy1-0/+6
2016-08-10 HttpBinaryCacheStore: Retry on transient HTTP errorsEelco Dolstra1-5/+6
This makes us more robust against 500 errors from CloudFront or S3 (assuming the 500 error isn't cached by CloudFront...).
2016-07-26 builtins.{fetchurl,fetchTarball}: Support a sha256 attributeEelco Dolstra1-1/+3
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a hash is specified.
2016-04-15 BinaryCacheStore::readFile(): Return a shared_ptr to a stringEelco Dolstra1-1/+2
This allows readFile() to indicate that a file doesn't exist, and might eliminate some large string copying.
2016-02-29 Add an HTTP binary cache storeEelco Dolstra1-4/+19
Allowing stuff like NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 or NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-04 StoreAPI -> StoreEelco Dolstra1-2/+2
Calling a class an API is a bit redundant...
2016-02-04 Eliminate the "store" global variableEelco Dolstra1-1/+4
Also, move a few free-standing functions into StoreAPI and Derivation. Also, introduce a non-nullable smart pointer, ref<T>, which is just a wrapper around std::shared_ptr ensuring that the pointer is never null. (For reference-counted values, this is better than passing a "T&", because the latter doesn't maintain the refcount. Usually, the caller will have a shared_ptr keeping the value alive, but that's not always the case, e.g., when passing a reference to a std::thread via std::bind.)
2015-10-21 Show progress indicator for builtin fetchurlEelco Dolstra1-0/+1
2015-10-21 Disable TLS verification for builtin fetchurlEelco Dolstra1-1/+7
This makes it consistent with the Nixpkgs fetchurl and makes it work in chroots. We don't need verification because the hash of the result is checked anyway.
2015-07-20 Make <nix/fetchurl.nix> a builtin builderEelco Dolstra1-0/+22
This ensures that 1) the derivation doesn't change when Nix changes; 2) the derivation closure doesn't contain Nix and its dependencies; 3) we don't have to rely on ugly chroot hacks.