about summary refs log tree commit diff
path: root/src/libstore/download.cc
AgeCommit message (Collapse)AuthorFilesLines
2017-02-22 Explicitly model all settings and fail on unrecognized onesDan Peebles1-3/+3
Previously, the Settings class allowed other code to query for string properties, which led to a proliferation of code all over the place making up new options without any sort of central registry of valid options. This commit pulls all those options back into the central Settings class and removes the public get() methods, to discourage future abuses like that. Furthermore, because we know the full set of options ahead of time, we now fail loudly if someone enters an unrecognized option, thus preventing subtle typos. With some template fun, we could probably also dump the full set of options (with documentation, defaults, etc.) to the command line, but I'm not doing that yet here.
2017-02-21 Handle CURLE_RECV_ERROR as a transient errorEelco Dolstra1-1/+1
This fixes unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21 Fix building without S3 supportEelco Dolstra1-1/+1
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-16 Move netrcFile to SettingsEelco Dolstra1-5/+2
Also get rid of Settings::processEnvironment(), it appears to be useless.
2017-02-16 Include "curl" in the User-Agent headerEelco Dolstra1-1/+1
Some sites (e.g. BitBucket) give a helpful 401 error when trying to download a private archive if the User-Agent contains "curl", but give a redirect to a login page otherwise (so for instance "nix-prefetch-url" will succeed but produce useless output).
2017-02-16 Merge pull request #1215 from k0001/netrc-1Eelco Dolstra1-0/+8
Add netrc-file support
2017-02-14 Add support for s3:// URIsEelco Dolstra1-1/+27
This adds support for s3:// URIs in all places where Nix allows URIs, e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix> and NIX_PATH. It allows fetching resources from private S3 buckets, using credentials obtained from the standard places (i.e. AWS_* environment variables, ~/.aws/credentials and the EC2 metadata server). This may not be super-useful in general, but since we already depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-09 Documentation.Renzo Carbonara1-0/+3
2017-02-02 Add netrc-file supportRenzo Carbonara1-0/+5
2017-01-26 On HTTP errors, also show the curl errorEelco Dolstra1-1/+1
This is a hopefully temporary measure to diagnose the intermittent "HTTP error 200" failures.
2017-01-24 Enable verbose curl outputEelco Dolstra1-1/+14
Closes #1182.
2017-01-17 Handle SIGINT etc. via a sigwait() signal handler threadEelco Dolstra1-5/+15
This allows other threads to install callbacks that run in a regular, non-signal context. In particular, we can use this to signal the downloader thread to quit. Closes #1183.
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-10/+10
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-10/+10
2016-10-21 BinaryCacheStore: Optionally write a NAR listingEelco Dolstra1-1/+1
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-19 Merge branch 'priorityqueue' of https://github.com/groxxda/nixEelco Dolstra1-12/+17
2016-10-19 Fix uninitialised variableEelco Dolstra1-1/+1
2016-10-18 downloader: use priority_queueAlexander Ried1-12/+17
2016-10-14 CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUMEelco Dolstra1-4/+3
http://hydra.nixos.org/build/42025230
2016-10-13 Fix HTTP/2 supportEelco Dolstra1-3/+3
Commit 86e8c67efc33cf756500a1dec7fd6313658f2664 broke it, because CURL_* are not actually #defines.
2016-10-13 SSL_CERT_FILE -> NIX_SSL_CERT_FILEEelco Dolstra1-1/+2
This prevents collisions with the "native" OpenSSL, in particular on OS X. Fixes #921.
2016-10-05 Use std::random_deviceEelco Dolstra1-0/+1
2016-10-04 Fix build with older versions of libcurlEelco Dolstra1-0/+6
2016-09-21 printMsg(lvlError, ...) -> printError(...) etc.Eelco Dolstra1-5/+5
2016-09-20 TweakEelco Dolstra1-5/+1
2016-09-20 Improve robustnessEelco Dolstra1-2/+6
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-11/+27
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-14 Revive binary-caches-parallel-connectionsEelco Dolstra1-2/+8
It's a slight misnomer now because it actually limits *all* downloads, not just binary cache lookups. Also add a "enable-http2" option to allow disabling use of HTTP/2 (enabled by default).
2016-09-14 Enable HTTP/2 supportEelco Dolstra1-171/+379
The binary cache store can now use HTTP/2 to do lookups. This is much more efficient than HTTP/1.1 due to multiplexing: we can issue many requests in parallel over a single TCP connection. Thus it's no longer necessary to use a bunch of concurrent TCP connections (25 by default). For example, downloading 802 .narinfo files from https://cache.nixos.org/, using a single TCP connection, takes 11.8s with HTTP/1.1, but only 0.61s with HTTP/2. This did require a fairly substantial rewrite of the Downloader class to use the curl multi interface, because otherwise curl wouldn't be able to do multiplexing for us. As a bonus, we get connection reuse even with HTTP/1.1. All downloads are now handled by a single worker thread. Clients call Downloader::enqueueDownload() to tell the worker thread to start the download, getting a std::future to the result.
2016-08-31 download.hh: Fix conflicts from nix-channel-c++ mergeShea Levy1-9/+5
2016-08-31 Merge branch 'nix-channel-c++'Shea Levy1-3/+17
2016-08-30 Increase the sleep time between download retriesEelco Dolstra1-1/+1
2016-08-15 builtins.fetch{url,tarball}: Allow name attributeShea Levy1-4/+5
2016-08-11 nix-channel: implement in c++Shea Levy1-3/+17
2016-08-10 HttpBinaryCacheStore: Retry on transient HTTP errorsEelco Dolstra1-9/+26
This makes us more robust against 500 errors from CloudFront or S3 (assuming the 500 error isn't cached by CloudFront...).
2016-08-10 Show HTTP error codesEelco Dolstra1-2/+6
2016-07-26 makeFixedOutputPath(): Drop superfluous HashType argumentEelco Dolstra1-2/+2
2016-07-26 builtins.{fetchurl,fetchTarball}: Support a sha256 attributeEelco Dolstra1-8/+27
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a hash is specified.
2016-07-21 Fix random failures caused by CurlDownloader modifying a shared stringEelco Dolstra1-1/+1
2016-04-29 Add fetchgit builtinEelco Dolstra1-1/+1
The function builtins.fetchgit fetches Git repositories at evaluation time, similar to builtins.fetchTarball. (Perhaps the name should be changed, being confusing with respect to Nixpkgs's fetchgit function, with works at build time.) Example: (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello or (import (builtins.fetchgit { url = git://github.com/NixOS/nixpkgs-channels; rev = "nixos-16.03"; }) {}).hello Note that the result does not contain a .git directory.
2016-04-20 Cache path info lookups in SQLiteEelco Dolstra1-1/+1
This re-implements the binary cache database in C++, allowing it to be used by other Store backends, in particular the S3 backend.
2016-04-15 BinaryCacheStore::readFile(): Return a shared_ptr to a stringEelco Dolstra1-4/+5
This allows readFile() to indicate that a file doesn't exist, and might eliminate some large string copying.
2016-04-14 Support channel:<channel-name> URIsEelco Dolstra1-5/+14
For convenience, you can now say $ nix-env -f channel:nixos-16.03 -iA hello instead of $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello Similarly, $ nix-shell -I channel:nixpkgs-unstable -p hello $ nix-build channel:nixos-15.09 -A hello Abstracting over the NixOS/Nixpkgs channels location also allows us to use a more efficient transport (e.g. Git) in the future.
2016-04-06 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http://Eelco Dolstra1-2/+1
The 304 Not Modified was not handled correctly, so the empty result from the conditional request would overwrite the previous tarball.
2016-03-30 CurlDownloader: Fix HTTP error processingEelco Dolstra1-1/+0
2016-02-29 Add an HTTP binary cache storeEelco Dolstra1-49/+75
Allowing stuff like NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 or NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-22 curl: Set CURLOPT_NOSIGNALEelco Dolstra1-0/+2
Otherwise using curl is not safe in multi-threaded applications because it installs a SIGALRM handler.
2016-02-04 StoreAPI -> StoreEelco Dolstra1-1/+1
Calling a class an API is a bit redundant...
2016-02-04 Eliminate the "store" global variableEelco Dolstra1-1/+1
Also, move a few free-standing functions into StoreAPI and Derivation. Also, introduce a non-nullable smart pointer, ref<T>, which is just a wrapper around std::shared_ptr ensuring that the pointer is never null. (For reference-counted values, this is better than passing a "T&", because the latter doesn't maintain the refcount. Usually, the caller will have a shared_ptr keeping the value alive, but that's not always the case, e.g., when passing a reference to a std::thread via std::bind.)
2015-10-29 int2String() -> std::to_string()Eelco Dolstra1-1/+1
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-21T03·41+0000