about summary refs log tree commit diff
path: root/src/libstore/http-binary-cache-store.cc
AgeCommit message (Collapse)AuthorFilesLines
2019-09-03 Add some noexceptsEelco Dolstra1-1/+1
This is to assert that callback functions should never throw (since the context in which they're called may not be able to handle the exception).
2019-09-03 Ensure that Callback is called only onceEelco Dolstra1-5/+7
Also, make Callback movable but uncopyable.
2019-07-10 HttpBinaryCacheStore: Use default number of retries for NARsEelco Dolstra1-1/+0
2019-07-10 Revert "Fix 'error 9 while decompressing xz file'"Eelco Dolstra1-41/+16
This reverts commit 78fa47a7f08a4cb6ee7061bf0bd86a40e1d6dc91.
2019-06-24 Add '--no-net' convenience flagEelco Dolstra1-1/+0
This flag * Disables substituters. * Sets the tarball-ttl to infinity (ensuring e.g. that the flake registry and any downloaded flakes are considered current). * Disables retrying downloads and sets the connection timeout to the minimum. (So it doesn't completely disable downloads at the moment.) (cherry picked from commit 8ea842260b4fd93315d35c5ba94b1ff99ab391d8)
2019-06-24 Fix 'error 9 while decompressing xz file'Eelco Dolstra1-16/+41
Once we've started writing data to a Sink, we can't restart a download request, because then we end up writing duplicate data to the Sink. Therefore we shouldn't handle retries in Downloader but at a higher level (in particular, in copyStorePath()). Fixes #2952. (cherry picked from commit a67cf5a3585c41dd9f219a2c7aa9cf67fa69520b)
2018-09-07 If fallback is enabled, temporarily disable substituters after a failureEelco Dolstra1-1/+40
Otherwise, we just keep asking the substituter for other .narinfo files, which can take a very long time due to retries/timeouts.
2018-06-01 Improve binary cache upload messagesEelco Dolstra1-1/+1
Don't say "download" when we mean "upload".
2018-05-30 Make HttpBinaryCacheStore::narFromPath() run in constant memoryEelco Dolstra1-2/+20
This reduces memory consumption of nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 176 MiB to 82 MiB. (The remaining memory is probably due to xz decompression overhead.) Issue https://github.com/NixOS/nix/issues/1681. Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 Simplify the callback mechanismEelco Dolstra1-11/+7
2018-01-31 CleanupEelco Dolstra1-3/+4
2018-01-26 HttpBinaryCacheStore: Support upsertFile with PUT.Shea Levy1-1/+7
Some servers, such as Artifactory, allow uploading with PUT and BASIC auth. This allows nix copy to work to upload binaries to those servers. Worked on together with @adelbertc
2017-07-30 Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim1-1/+1
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-05-16 Improve progress indicatorEelco Dolstra1-2/+0
2017-03-15 S3BinaryCacheStore: Set Content-TypeEelco Dolstra1-1/+3
This is necessary for serving log files to browsers.
2016-11-26 Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra1-1/+1
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25 Get rid of unicode quotes (#1140)Guillaume Maudoux1-1/+1
2016-10-19 Shut up clang warningEelco Dolstra1-1/+1
2016-09-20 Fix "Promise already satisfied" errorEelco Dolstra1-1/+3
2016-09-16 Make computeFSClosure() single-threaded againEelco Dolstra1-8/+17
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-14 Enable HTTP/2 supportEelco Dolstra1-17/+9
The binary cache store can now use HTTP/2 to do lookups. This is much more efficient than HTTP/1.1 due to multiplexing: we can issue many requests in parallel over a single TCP connection. Thus it's no longer necessary to use a bunch of concurrent TCP connections (25 by default). For example, downloading 802 .narinfo files from https://cache.nixos.org/, using a single TCP connection, takes 11.8s with HTTP/1.1, but only 0.61s with HTTP/2. This did require a fairly substantial rewrite of the Downloader class to use the curl multi interface, because otherwise curl wouldn't be able to do multiplexing for us. As a bonus, we get connection reuse even with HTTP/1.1. All downloads are now handled by a single worker thread. Clients call Downloader::enqueueDownload() to tell the worker thread to start the download, getting a std::future to the result.
2016-08-30 Increase the sleep time between download retriesEelco Dolstra1-1/+2
2016-08-10 HttpBinaryCacheStore: Retry on transient HTTP errorsEelco Dolstra1-0/+2
This makes us more robust against 500 errors from CloudFront or S3 (assuming the 500 error isn't cached by CloudFront...).
2016-06-01 HttpBinaryCacheStore: Fix caching of WantMassQueryEelco Dolstra1-2/+4
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 Make the store directory a member variable of StoreEelco Dolstra1-3/+3
2016-05-30 Re-implement the WantMassQuery property of binary cachesEelco Dolstra1-3/+8
2016-05-30 BinaryCacheStore: Remove buildPaths() / ensurePath()Eelco Dolstra1-4/+3
2016-04-29 BinaryCacheStore: Make the signing key a parameterEelco Dolstra1-4/+3
2016-04-29 Allow parameters in store URIsEelco Dolstra1-1/+4
This is to allow store-specific configuration, e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-20 Cache path info lookups in SQLiteEelco Dolstra1-2/+13
This re-implements the binary cache database in C++, allowing it to be used by other Store backends, in particular the S3 backend.
2016-04-15 BinaryCacheStore::readFile(): Return a shared_ptr to a stringEelco Dolstra1-2/+8
This allows readFile() to indicate that a file doesn't exist, and might eliminate some large string copying.
2016-03-30 HttpBinaryCacheStore: Treat 403 errors as 404Eelco Dolstra1-1/+3
2016-03-24 HttpBinaryCacheStore: Make thread-safeEelco Dolstra1-2/+6
2016-03-15 Fix Darwin buildEelco Dolstra1-1/+1
http://hydra.nixos.org/build/33279996
2016-03-04 Add option binary-cache-secret-key-file for signing binary cachesEelco Dolstra1-1/+2
2016-03-04 BinaryCacheStore: Remove publicKeyFile argumentEelco Dolstra1-4/+3
The public key can be derived from the secret key, so there's no need for the user to supply it separately.
2016-03-03 Add file missing from 201b48de60751979835037a4b4f78128ba3fb7b3Eelco Dolstra1-0/+76