depot - monorepo for the virus lounge

Age	Commit message (Collapse)	Author	Files	Lines
2019-10-28	fix(server): Thread request context to all relevant places	Vincent Ambo	6	-23/+20
	Previously background contexts where created where necessary (e.g. in GCS interactions). Should I begin to use request timeouts or other context-dependent things in the future, it's useful to have the actual HTTP request context around. This threads the request context through the application to all places that need it.
2019-10-28	chore(server): Move cache miss log statement to debug level	Vincent Ambo	2	-2/+2
	This is very annoying otherwise.
2019-10-28	fix(server): Correctly construct filesystem paths for layer serving	Vincent Ambo	1	-6/+8

2019-10-28	fix(server): Ensure paths exist when renaming in filesystem storage	Vincent Ambo	1	-1/+7
	The point at which files are moved happens to also (initially) be the point where the `layers` directory is created. For this reason renaming must ensure that all path components exist, which this commit takes care of.
2019-10-28	fix(server): Ensure error messages are correctly printed in logs	Vincent Ambo	1	-0/+7
	I assumed (incorrectly) that logrus would already take care of surfacing error messages in human-readable form.
2019-10-28	feat(server): Add filesystem storage backend config options	Vincent Ambo	3	-1/+11
	The filesystem storage backend can be enabled by setting `NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk location from which Nixery can serve files.
2019-10-28	refactor(server): Pass HTTP request to storage.ServeLayer	Vincent Ambo	4	-4/+6
	The request object is required for some serving methods (e.g. the filesystem one).
2019-10-28	feat(server): Implement initial filesystem storage backend	Vincent Ambo	1	-0/+68
	This allows users to store and serve layers from a local filesystem path.
2019-10-28	refactor(server): Change setup to create new storage backends	Vincent Ambo	5	-15/+44

2019-10-28	feat(server): Implement GCS storage backend with new interface	Vincent Ambo	1	-0/+206
	Logical implementation is mostly identical to the previous one, but adhering to the new storage.Backend interface.
2019-10-28	refactor(server): Introduce pluggable interface for storage backends	Vincent Ambo	5	-229/+111
	This abstracts over the functionality of Google Cloud Storage and other potential underlying storage backends to make it possible to replace these in Nixery. The GCS backend is not yet reimplemented.
2019-10-28	refactor(build): Do not expose nixery-server attribute	Vincent Ambo	1	-5/+3
	In most cases this is not useful for users without the wrapper script, so users should always build nixery-bin anyways.
2019-10-27	docs: Update GCS signing key documentation	Vincent Ambo	2	-12/+14
	This key is now taken straight from the configured service account key.
2019-10-27	refactor(server): Load GCS signing key from service account key	Vincent Ambo	1	-12/+16
	The JSON file generated for service account keys already contains the required information for signing URLs in GCS, thus the environment variables for toggling signing behaviour have been removed. Signing is now enabled automatically in the presence of service account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-11	fix(build): Only take the first matching hash for source hashing	Vincent Ambo	1	-1/+1
	Some Nix download mechanisms will add a second hash in the store path, which had been added to the source hash output (breaking argument interpolation).
2019-10-11	refactor(server): Only compress symlink forest layer once	Vincent Ambo	2	-16/+24
	Instead of compressing & decompressing again to get the underlying tar hash, use a similar mechanism as for store path layers for the symlink layer and only compress it once while uploading.
2019-10-11	fix(server): Use uncompressed tarball hashes in image config	Vincent Ambo	4	-16/+42
	Docker expects hashes of compressed tarballs in the manifest (as these are used to fetch from the content-addressable layer store), but for some reason it expects hashes in the configuration layer to be of uncompressed tarballs. To achieve this an additional SHA256 hash is calculcated while creating the layer tarballs, but before passing them to the gzip writer. In the current constellation the symlink layer is first compressed and then decompressed again to calculate its hash. This can be refactored in a future change.
2019-10-11	feat(server): Apply GZIP compression to all image layers	Vincent Ambo	4	-9/+15
	This fixes #62
2019-10-11	chore(build): Use separate GCS bucket for CI runs	Vincent Ambo	1	-1/+1
	This has become an issue recently with changes such as GZIP compression, where CI runs no longer work because they conflict with the production bucket for the public instance.
2019-10-06	refactor(server): Use logrus convenience functions for logs	Vincent Ambo	5	-128/+59
	Makes use of the `.WithError` and `.WithField` convenience functions in logrus to simplify log statement construction. This has the added benefit of making it easier to correctly log errors.
2019-10-06	fix(build-image): Import 'match' from builtins	Vincent Ambo	1	-0/+1

2019-10-06	refactor(server): Convert existing log entries to structured format	Vincent Ambo	7	-54/+243
	This rewrites all existing log statements into the structured logrus format. For consistency, all errors are always logged separately from the primary message in a field called `error`. Only the "info", "error" and "warn" severities are used.
2019-10-06	feat(server): Add log level to severity mapping	Vincent Ambo	2	-3/+36
	The output format now writes a `severity` field that follows that format that should be recognised by Stackdriver Logging.
2019-10-06	feat(server): Use hash of Nixery source as version	Vincent Ambo	4	-11/+48
	Uses a hash of Nixery's sources as the version displayed when Nixery launches or logs an error. This makes it possible to distinguish between errors logged from different versions. The source hashes should be reproducible between different checkouts of the same source tree.
2019-10-06	feat(server): Initial Stackdriver-compatible log formatter	Vincent Ambo	1	-0/+68
	This formatter has basic support for the Stackdriver Error Reporting format, but several things are still lacking: * the service version (preferably git commit?) needs to be included in the server somehow * log streams should be split between stdout/stderr as that is how AppEngine (and several other GCP services?) seemingly differentiate between info/error logs
2019-10-06	fix(server): Amend package path for Go tooling compatibility	Vincent Ambo	6	-10/+10
	With these changes it is possible to keep Nixery in $GOPATH and build the server in there, while still having things work correctly via Nix.
2019-10-06	refactor(server): Replace log calls with logrus	Vincent Ambo	8	-7/+17
	This introduces a structured logging library that can be used (next step) to attach additional metadata to log entries.
2019-10-03	refactor(server): Always include 'cacert' & 'iana-etc'	Vincent Ambo	2	-10/+19
	These two packages almost always end up being required by programs, but people don't necessarily consider them. They will now always be added and their popularity is artificially inflated to ensure they end up at the top of the layer list.
2019-10-03	fix(server): Ensure build cache objects are written to GCS	Vincent Ambo	1	-0/+5
	Cache writes might not be flushed without this call.
2019-10-03	feat(server): Order layers in image manifest based on merge rating	Vincent Ambo	3	-4/+20
	Image layers in manifests are now sorted in a stable (descending) order based on their merge rating, meaning that layers more likely to be shared between images come first. The reason for this change is Docker's handling of image layers on overlayfs2: Images are condensed into a single representation on disk after downloading. Due to this Docker will constantly redownload all layers that are applied in a different order in different images (layer order matters in imperatively created images), based on something it calls the 'ChainID'. Sorting the layers this way raises the likelihood of a long chain of matching layers at the beginning of an image. This relates to #39.
2019-10-03	chore(build-image): Remove nixery-build-layers	Vincent Ambo	3	-70/+10
	This functionality has been rolled into the server component and is no longer required.
2019-10-03	feat(server): Implement creation of layer tarballs in the server	Vincent Ambo	2	-79/+158
	This will create, upload and hash the layer tarballs in one disk read.
2019-10-03	fix(server): Do not invoke layer build if no layers are missing	Vincent Ambo	1	-1/+5
	This previously invoked a Nix derivation that spent a few seconds on making an empty object in JSON ...
2019-10-03	feat(server): Fetch popularity data on launch	Vincent Ambo	1	-0/+37
	The last missing puzzle piece for #50!
2019-10-03	feat(server): Reimplement local manifest cache backed by files	Vincent Ambo	5	-51/+70
	Implements a local manifest cache that uses the temporary directory to cache manifest builds. This is necessary due to the size of manifests: Keeping them entirely in-memory would quickly balloon the memory usage of Nixery, unless some mechanism for cache eviction is implemented.
2019-10-03	refactor(builder): Streamline layer creation & reintroduce caching	Vincent Ambo	1	-28/+53
	The functions used for layer creation are now easier to follow and have clear points at which the layer cache is checked and populated. This relates to #50.
2019-10-03	refactor: Remove remaining MD5-hash mentions and computations	Vincent Ambo	5	-30/+18

2019-10-03	refactor(server): Cache manifest entries for layer builds	Vincent Ambo	2	-31/+27
	MD5 hash checking is no longer performed by Nixery (it does not seem to be necessary), hence the layer cache now only keeps the SHA256 hash and size in the form of the manifest entry. This makes it possible to restructure the builder code to perform cache-fetching and cache-populating for layers in the same place.
2019-10-03	feat(server): Reintroduce manifest caching to GCS	Vincent Ambo	2	-29/+29
	The new builder now caches and reads cached manifests to/from GCS. The in-memory cache is disabled, as manifests are no longer written to local file and the caching of file paths does not work (unless we reintroduce reading/writing from temp files as part of the local cache).
2019-10-03	refactor(server): Clean up cache implementation	Vincent Ambo	3	-27/+26
	A couple of minor fixes and improvements to the cache implementation.
2019-10-03	chore(server): Remove "layer seen" cache	Vincent Ambo	1	-29/+5
	This cache is no longer required as it is implicit because the layer cache (mapping store path hashes to layer hashes) implies that a layer has been seen.
2019-10-03	fix(server): Upload symlink layer created by first Nix build	Vincent Ambo	1	-1/+6
	This layer is needed in addition to those that are built in the second Nix build.
2019-10-03	fix(server): Specify correct authentication scope for GCS	Vincent Ambo	1	-1/+4
	When retrieving tokens for service service accounts, some methods of retrieval require a scope to be specified.
2019-10-03	feat(server): Implement new build process core	Vincent Ambo	1	-58/+52
	Implements the new build process to the point where it can actually construct and serve image manifests. It is worth noting that this build process works even if the Nix sandbox is enabled! It is also worth nothing that none of the caching functionality that the new build process enables (such as per-layer build caching) is actually in use yet, hence running Nixery at this commit is prone to doing more work than previously. This relates to #50.
2019-10-03	fix(server): Minor fixes to updated new builder code	Vincent Ambo	2	-8/+7

2019-10-03	fix(nix): Minor fixes to derivations for new build process	Vincent Ambo	2	-2/+3

2019-10-03	feat(server): Implement package for creating image manifests	Vincent Ambo	1	-0/+114
	The new manifest package creates image manifests and their configuration. This previously happened in Nix, but is now part of the server's workload. This relates to #50.
2019-10-03	feat(server): Reimplement creation & uploading of layers	Vincent Ambo	3	-87/+250
	The new build process can now call out to Nix to create layers and upload them to the bucket if necessary. The layer cache is populated, but not yet used.
2019-10-03	refactor(server): Introduce a state type to carry runtime state	Vincent Ambo	2	-11/+31
	The state type contains things such as the bucket handle and Nixery's configuration which need to be passed around in the builder. This is only added for convenience.
2019-10-03	feat(server): Add cache for layer builds in GCS & local cache	Vincent Ambo	1	-0/+86
	This cache is going to be used for looking up whether a layer build has taken place already (based on a hash of the layer contents). See the caching section in the updated documentation for details. Relates to #50.