about summary refs log tree commit diff
path: root/src/libstore/download.cc
AgeCommit message (Collapse)AuthorFilesLines
2019-08-08 conf: stalled-download-timeout: make tunableGraham Christensen1-3/+1
Make curl's low speed limit configurable via stalled-download-timeout. Before, this limit was five minutes without receiving a single byte. This is much too long as if the remote end may not have even acknowledged the HTTP request.
2019-07-10 Resume NAR downloadsEelco Dolstra1-7/+28
This is a much simpler fix to the 'error 9 while decompressing xz file' problem than 78fa47a7f08a4cb6ee7061bf0bd86a40e1d6dc91. We just do a ranged HTTP request starting after the data that we previously wrote into the sink. Fixes #2952, #379.
2019-07-10 Downloader: Use warn()Eelco Dolstra1-3/+3
2019-07-10 Revert "Fix 'error 9 while decompressing xz file'"Eelco Dolstra1-12/+55
This reverts commit 78fa47a7f08a4cb6ee7061bf0bd86a40e1d6dc91.
2019-07-02 build: add exit code for hash and check mismatchesDaiderd Jordan1-1/+2
Makes it easier to identify the failure reason in other tooling, eg. differentiate between a non-deterministic --check vs a failed build. $ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000 hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org': wanted: sha256:0000000000000000000000000000000000000000000000000000 got: sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm $ echo $? 102 $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs $ echo $? 104
2019-06-28 Fix `http2 = false` having no effect. Fixes #2971.Aniket Deshpande1-0/+2
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf) had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar packages using HTTP/2. In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was being explicitly set when `downloadSettings.enableHttp2` was `true`, but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when `downloadSettings.enableHttp2` was `false`. This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states: "You have to set this option if you want to use libcurl's HTTP/2 support." but, also, in the changelog, states: "DEFAULT Since curl 7.62.0: CURL_HTTP_VERSION_2TLS Before that: CURL_HTTP_VERSION_1_1" So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0. In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to `CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config setting is `false`. This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-24 Fix 32-bit overflow with --no-netEelco Dolstra1-1/+1
--no-net causes tarballTtl to be set to the largest 32-bit integer, which causes comparison like 'time + tarballTtl < other_time' to fail on 32-bit systems. So cast them to 64-bit first. https://hydra.nixos.org/build/95076624 (cherry picked from commit 29ccb2e9697ee2184012dd13854e487928ae4441)
2019-06-24 Refactor downloadCached() interfaceEelco Dolstra1-22/+22
(cherry picked from commit df3f5a78d5ab0a1f2dc9d288b271b38a9b8b33b5)
2019-06-24 Add '--no-net' convenience flagEelco Dolstra1-17/+1
This flag * Disables substituters. * Sets the tarball-ttl to infinity (ensuring e.g. that the flake registry and any downloaded flakes are considered current). * Disables retrying downloads and sets the connection timeout to the minimum. (So it doesn't completely disable downloads at the moment.) (cherry picked from commit 8ea842260b4fd93315d35c5ba94b1ff99ab391d8)
2019-06-24 Fix 'error 9 while decompressing xz file'Eelco Dolstra1-55/+12
Once we've started writing data to a Sink, we can't restart a download request, because then we end up writing duplicate data to the Sink. Therefore we shouldn't handle retries in Downloader but at a higher level (in particular, in copyStorePath()). Fixes #2952. (cherry picked from commit a67cf5a3585c41dd9f219a2c7aa9cf67fa69520b)
2019-06-24 Downloader: Propagate exceptions from decompressionSink->finish()Eelco Dolstra1-2/+7
(cherry picked from commit 15fa70cd1b853f5e62662b99ccb9ef3da6cfadff)
2019-06-24 CachedDownloadResult: Include store pathEelco Dolstra1-0/+2
Also, make fetchGit and fetchMercurial update allowedPaths properly. (Maybe the evaluator, rather than the caller of the evaluator, should apply toRealPath(), but that's a bigger change.) (cherry picked from commit 5c34d665386f4053d666b0899ecca0639e500fbd)
2019-06-24 downloadCached: Return ETagEelco Dolstra1-4/+13
(cherry picked from commit 529add316c5356a8060c35f987643b7bf5c796dc)
2019-02-26 fix indentationDzmitry Zaitsau1-2/+2
2019-02-25 Apply param values on S3Helper initializationDzmitry Zaitsau1-1/+8
2019-02-25 Extend S3 URL parsing with parameters extractionDzmitry Zaitsau1-5/+17
2018-12-12 libstore: improve hash mismatch error messageszimbatm1-2/+2
Use the same output ordering and format everywhere. This is such a common issue that we trade the single-line error message for more readability. Old message: ``` fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m' ``` New message: ``` hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com': wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m got: sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm ```
2018-12-07 s3: make scheme configurableDaiderd Jordan1-1/+1
This enables using for http for S3 request for debugging or implementations that don't have https configured. This is not a problem for binary caches since they should not contain sensitive information. Both package signatures and AWS auth already protect against tampering.
2018-11-20 Merge pull request #2216 from dtzWill/fix/curl-760-timeoutEelco Dolstra1-1/+2
download: if there are active requests, never sleep for 10s
2018-09-26 Ensure download thread livenessEelco Dolstra1-19/+26
* Don't wait forever for the client to remove data from the buffer. This does mean that the buffer can grow without bounds (e.g. when downloading is faster than writing to disk), but meh. * Don't hold the state lock while calling the sink. The sink could take any amount of time to process the data (in particular when it's actually a coroutine), so we don't want to block the download thread.
2018-09-03 download: fix size reported to progress barWill Dietz1-1/+1
2018-08-06 Fix Brotli decompression in 'nix log'Eelco Dolstra1-24/+44
This didn't work anymore since decompression was only done in the non-coroutine case. Decompressors are now sinks, just like compressors. Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK rather than BZ_OK), which we didn't notice because there was a missing 'throw': if (ret != BZ_OK) CompressionError("error while compressing bzip2 file");
2018-08-06 Fix reporting of HTTP body size when a result callback is usedEelco Dolstra1-1/+4
2018-08-02 s3 binary cache: support specifying an endpointGraham Christensen1-1/+1
Works for uploading and not downloading.
2018-07-31 Fix race downloading into a sinkEelco Dolstra1-6/+8
This fixes 'error 10 while decompressing xz file'. https://hydra.nixos.org/build/78308551
2018-06-18 Detect redirect loopsEelco Dolstra1-0/+2
Fixes #2225.
2018-06-08 download: if there are active requests, never sleep for 10sWill Dietz1-1/+2
In some versions/configurations libcurl doesn't handle timeouts (especially DNS timeouts) in a way that wakes curl_multi_wait. This doesn't appear to be a problem if using c-ares, FWIW.
2018-06-05 Handle HTTP 410 Gone errorsEelco Dolstra1-1/+1
2018-06-05 Improve binary cache fallback testEelco Dolstra1-0/+5
2018-06-05 Further improve upload messagesEelco Dolstra1-8/+6
2018-06-01 download.cc: fix readCallback return type (int -> size_t)Will Dietz1-2/+2
I'm not sure if curl ever asks for enough data at once for truncation to occur but better safe than sorry.
2018-06-01 Improve binary cache upload messagesEelco Dolstra1-4/+14
Don't say "download" when we mean "upload".
2018-05-30 getDownloader(): SimplifyEelco Dolstra1-4/+2
2018-05-30 Make HttpBinaryCacheStore::narFromPath() run in constant memoryEelco Dolstra1-1/+91
This reduces memory consumption of nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 176 MiB to 82 MiB. (The remaining memory is probably due to xz decompression overhead.) Issue https://github.com/NixOS/nix/issues/1681. Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 Simplify the callback mechanismEelco Dolstra1-17/+20
2018-05-30 Move some Download-specific settings to download.ccEelco Dolstra1-8/+24
2018-05-25 libstore/download: remove unused function "getTime()"Will Dietz1-7/+0
2018-05-09 make sure not to use cached channels for nix-channel --updateYorick van Pelt1-2/+1
fixes #1964
2018-04-23 Merge branch 'hash-mismatch' of git://github.com/LnL7/nixShea Levy1-2/+2
2018-04-21 Merge pull request #1707 from dtzWill/fix/issue-1692Peter Simons1-1/+5
ignore "interrupted" exception in progress callback
2018-04-20 download: make hash mismatch error consistent with fetchurlDaiderd Jordan1-2/+2
2018-03-22 download: improve error for hash mismatch ("store mismatch")Will Dietz1-2/+7
Fixes #1905
2018-03-22 download.cc: fix path for cached filesWill Dietz1-1/+1
2018-03-16 Don't retry on CURLE_SSL_CACERT_BADFILEEelco Dolstra1-0/+1
The certificates won't get any better if we retry.
2018-03-14 Catch more possible instances of passing NULL to memcpy.Shea Levy1-0/+1
Actually fixes #1976.
2018-02-23 Fix downloadCached() with a chroot storeEelco Dolstra1-2/+2
E.g. nix run --store ~/my-nix -f channel:nixos-17.03 hello -c hello This problem was mentioned in #1897.
2018-01-31 Fix building without aws-sdk-cppEelco Dolstra1-4/+5
2018-01-31 Merge branch 'http-binary-cache-put-upsert' of https://github.com/adelbertc/nixEelco Dolstra1-1/+27
2018-01-31 Indent properlyEelco Dolstra1-3/+3
2018-01-26 HttpBinaryCacheStore: Support upsertFile with PUT.Shea Levy1-1/+27
Some servers, such as Artifactory, allow uploading with PUT and BASIC auth. This allows nix copy to work to upload binaries to those servers. Worked on together with @adelbertc
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-18T18·27+0000