depot - monorepo for the virus lounge

Age	Commit message (Collapse)	Author	Files	Lines
2017-03-15	More precise compression settings	Eelco Dolstra	1	-5/+8

2017-03-15	Compress NAR listings using the "text-compression" method	Eelco Dolstra	1	-2/+2
	So if "text-compression=br", the .ls file in S3 will get a Content-Encoding of "br". Brotli appears to compress better than xz for this kind of file and is natively supported by browsers.
2017-03-15	S3BinaryCacheStore: Set Content-Type	Eelco Dolstra	1	-5/+9
	This is necessary for serving log files to browsers.
2017-03-15	S3BinaryCacheStore: Support compression of narinfo and log files	Eelco Dolstra	1	-3/+25
	You can now set the store parameter "text-compression=br" to compress textual files in the binary cache (i.e. narinfo and logs) using Brotli. This sets the Content-Encoding header; the extension of compressed files is unchanged. You can separately specify the compression of log files using "log-compression=br". This is useful when you don't want to compress narinfo files for backward compatibility.
2017-03-12	Merge branch 'configurable-aws-region'	Shea Levy	1	-4/+5

2017-03-06	Properly set the caFile for aws-sdk-cpp s3	Shea Levy	1	-0/+1

2017-03-03	Add aws-region param to S3 store URLs	Shea Levy	1	-4/+5

2017-02-21	Log AWS retries	Eelco Dolstra	1	-0/+14

2017-02-14	Add support for s3:// URIs	Eelco Dolstra	1	-61/+80
	This adds support for s3:// URIs in all places where Nix allows URIs, e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix> and NIX_PATH. It allows fetching resources from private S3 buckets, using credentials obtained from the standard places (i.e. AWS_* environment variables, ~/.aws/credentials and the EC2 metadata server). This may not be super-useful in general, but since we already depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-08	Include config.h implicitly with '-include config.h' in CFLAGS	Tuomas Tynkkynen	1	-2/+0
	Because config.h can #define things like _FILE_OFFSET_BITS=64 and not every compilation unit includes config.h, we currently compile half of Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS unset. This causes major havoc with the Settings class on e.g. 32-bit ARM, where different compilation units disagree with the struct layout. E.g.: diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc @@ -166,6 +166,8 @@ void Settings::update() _get(useSubstitutes, "build-use-substitutes"); + fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes); _get(buildUsersGroup, "build-users-group"); diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc +++ b/src/libstore/remote-store.cc @@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn) void RemoteStore::setOptions(Connection & conn) { + fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes); conn.to << wopSetOptions Gave me: at Settings::update(): &useSubstitutes = 0xb6e5c5cb at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7 That was not a fun one to debug!
2016-12-22	Call Aws::InitAPI	Eelco Dolstra	1	-1/+17
	This is required now.
2016-12-08	S3BinaryCacheStore: Ensure it only builds on Linux	Eelco Dolstra	1	-0/+2

2016-12-08	Fix S3BinaryCacheStore	Eelco Dolstra	1	-0/+9
	It failed with AWS error uploading ‘6gaxphsyhg66mz0a00qghf9nqf7majs2.ls.xz’: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header. possibly because the istringstream_nocopy introduced in 0d2ebb4373e509521f27a6e8f16bfd39d05b2188 doesn't supply the seek method that the AWS library expects. So bring back the old version, but only for S3BinaryCacheStore.
2016-11-26	Revert "Get rid of unicode quotes (#1140)"	Eelco Dolstra	1	-11/+11
	This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25	Get rid of unicode quotes (#1140)	Guillaume Maudoux	1	-11/+11

2016-11-16	S3BinaryCacheStore:: Eliminate a string copy while uploading	Eelco Dolstra	1	-1/+1
	This cuts hydra-queue-runner's peak memory usage by about a third.
2016-10-06	Fix getS3Stats()	Eelco Dolstra	1	-1/+1

2016-09-21	printMsg(lvlError, ...) -> printError(...) etc.	Eelco Dolstra	1	-1/+1

2016-09-16	Make computeFSClosure() single-threaded again	Eelco Dolstra	1	-26/+30
	The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-06-01	HttpBinaryCacheStore: Fix caching of WantMassQuery	Eelco Dolstra	1	-1/+1
	Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01	Make the store directory a member variable of Store	Eelco Dolstra	1	-4/+4

2016-05-31	Shut up some clang warnings	Eelco Dolstra	1	-6/+6

2016-05-30	Fix build	Eelco Dolstra	1	-1/+1

2016-05-30	Re-implement the WantMassQuery property of binary caches	Eelco Dolstra	1	-1/+1

2016-05-30	BinaryCacheStore: Remove buildPaths() / ensurePath()	Eelco Dolstra	1	-4/+3

2016-05-04	Make the aws-cpp-sdk dependency optional	Eelco Dolstra	1	-0/+6

2016-04-29	nix verify --all: Support local binary caches	Eelco Dolstra	1	-2/+2

2016-04-29	BinaryCacheStore: Make the signing key a parameter	Eelco Dolstra	1	-4/+3

2016-04-29	Allow parameters in store URIs	Eelco Dolstra	1	-1/+4
	This is to allow store-specific configuration, e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-21	Implement S3BinaryCacheStore::queryAllValidPaths()	Eelco Dolstra	1	-1/+34
	This allows commands like "nix verify --all" or "nix path-info --all" to work on S3 caches. Unfortunately, this requires some ugly hackery: when querying the contents of the bucket, we don't want to have to read every .narinfo file. But the S3 bucket keys only include the hash part of each store path, not the name part. So as a special exception queryAllValidPaths() can now return store paths without the name part, and queryPathInfo() accepts such store paths (returning a ValidPathInfo object containing the full name).
2016-04-21	Move S3BinaryCacheStore from Hydra	Eelco Dolstra	1	-0/+218
	This allows running arbitrary Nix commands against an S3 binary cache. To do: make this a compile time option to prevent a dependency on aws-sdk-cpp.