about summary refs log tree commit diff
path: root/src/libstore/download.cc
AgeCommit message (Collapse)AuthorFilesLines
2017-04-10 Add CURLE_WRITE_ERROR as a transient error conditionDan Peebles1-0/+5
We've observed it failing downloads in the wild and retrying the same URL a few moments later seemed to fix it.
2017-04-06 Retry downloads on transient SSL errors tooDan Peebles1-0/+2
2017-03-31 Retry curl error 16Eelco Dolstra1-0/+1
2017-03-15 runProgram(): Distinguish between empty input and no inputEelco Dolstra1-1/+1
For example, if we call brotli with an empty input, it shouldn't read from the caller's stdin.
2017-03-15 S3BinaryCacheStore: Support compression of narinfo and log filesEelco Dolstra1-5/+11
You can now set the store parameter "text-compression=br" to compress textual files in the binary cache (i.e. narinfo and logs) using Brotli. This sets the Content-Encoding header; the extension of compressed files is unchanged. You can separately specify the compression of log files using "log-compression=br". This is useful when you don't want to compress narinfo files for backward compatibility.
2017-03-15 Add support for brotli compressionEelco Dolstra1-2/+19
Build logs on cache.nixos.org are compressed using Brotli (since this allows them to be decompressed automatically by Chrome and Firefox), so it's handy if "nix log" can decompress them.
2017-03-12 Merge branch 'configurable-aws-region'Shea Levy1-1/+4
2017-03-06 Properly set the caFile for aws-sdk-cpp s3Shea Levy1-2/+1
2017-03-03 Add aws-region param to S3 store URLsShea Levy1-1/+4
2017-02-28 Fix building against older curl versionsEelco Dolstra1-1/+4
http://hydra.nixos.org/build/49490928
2017-02-27 Retry downloads on HTTP/2 stream errorsEelco Dolstra1-1/+3
Issue #1254.
2017-02-22 RemoteStore::addToStore(): Pass content-addressability assertionEelco Dolstra1-0/+1
... and use this in Downloader::downloadCached(). This fixes $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-21 Handle CURLE_RECV_ERROR as a transient errorEelco Dolstra1-1/+1
This fixes unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21 Fix building without S3 supportEelco Dolstra1-1/+1
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-16 Move netrcFile to SettingsEelco Dolstra1-5/+2
Also get rid of Settings::processEnvironment(), it appears to be useless.
2017-02-16 Include "curl" in the User-Agent headerEelco Dolstra1-1/+1
Some sites (e.g. BitBucket) give a helpful 401 error when trying to download a private archive if the User-Agent contains "curl", but give a redirect to a login page otherwise (so for instance "nix-prefetch-url" will succeed but produce useless output).
2017-02-16 Merge pull request #1215 from k0001/netrc-1Eelco Dolstra1-0/+8
Add netrc-file support
2017-02-14 Add support for s3:// URIsEelco Dolstra1-1/+27
This adds support for s3:// URIs in all places where Nix allows URIs, e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix> and NIX_PATH. It allows fetching resources from private S3 buckets, using credentials obtained from the standard places (i.e. AWS_* environment variables, ~/.aws/credentials and the EC2 metadata server). This may not be super-useful in general, but since we already depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-09 Documentation.Renzo Carbonara1-0/+3
2017-02-02 Add netrc-file supportRenzo Carbonara1-0/+5
2017-01-26 On HTTP errors, also show the curl errorEelco Dolstra1-1/+1
This is a hopefully temporary measure to diagnose the intermittent "HTTP error 200" failures.
2017-01-24 Enable verbose curl outputEelco Dolstra1-1/+14
Closes #1182.
2017-01-17 Handle SIGINT etc. via a sigwait() signal handler threadEelco Dolstra1-5/+15
This allows other threads to install callbacks that run in a regular, non-signal context. In particular, we can use this to signal the downloader thread to quit. Closes #1183.
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-10/+10
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-10/+10
2016-10-21 BinaryCacheStore: Optionally write a NAR listingEelco Dolstra1-1/+1
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-19 Merge branch 'priorityqueue' of https://github.com/groxxda/nixEelco Dolstra1-12/+17
2016-10-19 Fix uninitialised variableEelco Dolstra1-1/+1
2016-10-18 downloader: use priority_queueAlexander Ried1-12/+17
2016-10-14 CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUMEelco Dolstra1-4/+3
http://hydra.nixos.org/build/42025230
2016-10-13 Fix HTTP/2 supportEelco Dolstra1-3/+3
Commit 86e8c67efc33cf756500a1dec7fd6313658f2664 broke it, because CURL_* are not actually #defines.
2016-10-13 SSL_CERT_FILE -> NIX_SSL_CERT_FILEEelco Dolstra1-1/+2
This prevents collisions with the "native" OpenSSL, in particular on OS X. Fixes #921.
2016-10-05 Use std::random_deviceEelco Dolstra1-0/+1
2016-10-04 Fix build with older versions of libcurlEelco Dolstra1-0/+6
2016-09-21 printMsg(lvlError, ...) -> printError(...) etc.Eelco Dolstra1-5/+5
2016-09-20 TweakEelco Dolstra1-5/+1
2016-09-20 Improve robustnessEelco Dolstra1-2/+6
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-11/+27
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-14 Revive binary-caches-parallel-connectionsEelco Dolstra1-2/+8
It's a slight misnomer now because it actually limits *all* downloads, not just binary cache lookups. Also add a "enable-http2" option to allow disabling use of HTTP/2 (enabled by default).
2016-09-14 Enable HTTP/2 supportEelco Dolstra1-171/+379
The binary cache store can now use HTTP/2 to do lookups. This is much more efficient than HTTP/1.1 due to multiplexing: we can issue many requests in parallel over a single TCP connection. Thus it's no longer necessary to use a bunch of concurrent TCP connections (25 by default). For example, downloading 802 .narinfo files from https://cache.nixos.org/, using a single TCP connection, takes 11.8s with HTTP/1.1, but only 0.61s with HTTP/2. This did require a fairly substantial rewrite of the Downloader class to use the curl multi interface, because otherwise curl wouldn't be able to do multiplexing for us. As a bonus, we get connection reuse even with HTTP/1.1. All downloads are now handled by a single worker thread. Clients call Downloader::enqueueDownload() to tell the worker thread to start the download, getting a std::future to the result.
2016-08-31 download.hh: Fix conflicts from nix-channel-c++ mergeShea Levy1-9/+5
2016-08-31 Merge branch 'nix-channel-c++'Shea Levy1-3/+17
2016-08-30 Increase the sleep time between download retriesEelco Dolstra1-1/+1
2016-08-15 builtins.fetch{url,tarball}: Allow name attributeShea Levy1-4/+5
2016-08-11 nix-channel: implement in c++Shea Levy1-3/+17
2016-08-10 HttpBinaryCacheStore: Retry on transient HTTP errorsEelco Dolstra1-9/+26
This makes us more robust against 500 errors from CloudFront or S3 (assuming the 500 error isn't cached by CloudFront...).
2016-08-10 Show HTTP error codesEelco Dolstra1-2/+6
2016-07-26 makeFixedOutputPath(): Drop superfluous HashType argumentEelco Dolstra1-2/+2
2016-07-26 builtins.{fetchurl,fetchTarball}: Support a sha256 attributeEelco Dolstra1-8/+27
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a hash is specified.
2016-07-21 Fix random failures caused by CurlDownloader modifying a shared stringEelco Dolstra1-1/+1
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-15T22·37+0000