about summary refs log tree commit diff
path: root/tools
AgeCommit message (Collapse)AuthorFilesLines
2019-10-28 refactor(build): Do not expose nixery-server attributeVincent Ambo1-5/+3
In most cases this is not useful for users without the wrapper script, so users should always build nixery-bin anyways.
2019-10-27 docs: Update GCS signing key documentationVincent Ambo2-12/+14
This key is now taken straight from the configured service account key.
2019-10-27 refactor(server): Load GCS signing key from service account keyVincent Ambo1-12/+16
The JSON file generated for service account keys already contains the required information for signing URLs in GCS, thus the environment variables for toggling signing behaviour have been removed. Signing is now enabled automatically in the presence of service account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-11 fix(build): Only take the first matching hash for source hashingVincent Ambo1-1/+1
Some Nix download mechanisms will add a second hash in the store path, which had been added to the source hash output (breaking argument interpolation).
2019-10-11 refactor(server): Only compress symlink forest layer onceVincent Ambo2-16/+24
Instead of compressing & decompressing again to get the underlying tar hash, use a similar mechanism as for store path layers for the symlink layer and only compress it once while uploading.
2019-10-11 fix(server): Use uncompressed tarball hashes in image configVincent Ambo4-16/+42
Docker expects hashes of compressed tarballs in the manifest (as these are used to fetch from the content-addressable layer store), but for some reason it expects hashes in the configuration layer to be of uncompressed tarballs. To achieve this an additional SHA256 hash is calculcated while creating the layer tarballs, but before passing them to the gzip writer. In the current constellation the symlink layer is first compressed and then decompressed again to calculate its hash. This can be refactored in a future change.
2019-10-11 feat(server): Apply GZIP compression to all image layersVincent Ambo4-9/+15
This fixes #62
2019-10-11 chore(build): Use separate GCS bucket for CI runsVincent Ambo1-1/+1
This has become an issue recently with changes such as GZIP compression, where CI runs no longer work because they conflict with the production bucket for the public instance.
2019-10-06 refactor(server): Use logrus convenience functions for logsVincent Ambo5-128/+59
Makes use of the `.WithError` and `.WithField` convenience functions in logrus to simplify log statement construction. This has the added benefit of making it easier to correctly log errors.
2019-10-06 fix(build-image): Import 'match' from builtinsVincent Ambo1-0/+1
2019-10-06 refactor(server): Convert existing log entries to structured formatVincent Ambo7-54/+243
This rewrites all existing log statements into the structured logrus format. For consistency, all errors are always logged separately from the primary message in a field called `error`. Only the "info", "error" and "warn" severities are used.
2019-10-06 feat(server): Add log level to severity mappingVincent Ambo2-3/+36
The output format now writes a `severity` field that follows that format that should be recognised by Stackdriver Logging.
2019-10-06 feat(server): Use hash of Nixery source as versionVincent Ambo4-11/+48
Uses a hash of Nixery's sources as the version displayed when Nixery launches or logs an error. This makes it possible to distinguish between errors logged from different versions. The source hashes should be reproducible between different checkouts of the same source tree.
2019-10-06 feat(server): Initial Stackdriver-compatible log formatterVincent Ambo1-0/+68
This formatter has basic support for the Stackdriver Error Reporting format, but several things are still lacking: * the service version (preferably git commit?) needs to be included in the server somehow * log streams should be split between stdout/stderr as that is how AppEngine (and several other GCP services?) seemingly differentiate between info/error logs
2019-10-06 fix(server): Amend package path for Go tooling compatibilityVincent Ambo6-10/+10
With these changes it is possible to keep Nixery in $GOPATH and build the server in there, while still having things work correctly via Nix.
2019-10-06 refactor(server): Replace log calls with logrusVincent Ambo8-7/+17
This introduces a structured logging library that can be used (next step) to attach additional metadata to log entries.
2019-10-03 refactor(server): Always include 'cacert' & 'iana-etc'Vincent Ambo2-10/+19
These two packages almost always end up being required by programs, but people don't necessarily consider them. They will now always be added and their popularity is artificially inflated to ensure they end up at the top of the layer list.
2019-10-03 fix(server): Ensure build cache objects are written to GCSVincent Ambo1-0/+5
Cache writes might not be flushed without this call.
2019-10-03 feat(server): Order layers in image manifest based on merge ratingVincent Ambo3-4/+20
Image layers in manifests are now sorted in a stable (descending) order based on their merge rating, meaning that layers more likely to be shared between images come first. The reason for this change is Docker's handling of image layers on overlayfs2: Images are condensed into a single representation on disk after downloading. Due to this Docker will constantly redownload all layers that are applied in a different order in different images (layer order matters in imperatively created images), based on something it calls the 'ChainID'. Sorting the layers this way raises the likelihood of a long chain of matching layers at the beginning of an image. This relates to #39.
2019-10-03 chore(build-image): Remove nixery-build-layersVincent Ambo3-70/+10
This functionality has been rolled into the server component and is no longer required.
2019-10-03 feat(server): Implement creation of layer tarballs in the serverVincent Ambo2-79/+158
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 fix(server): Do not invoke layer build if no layers are missingVincent Ambo1-1/+5
This previously invoked a Nix derivation that spent a few seconds on making an empty object in JSON ...
2019-10-03 feat(server): Fetch popularity data on launchVincent Ambo1-0/+37
The last missing puzzle piece for #50!
2019-10-03 feat(server): Reimplement local manifest cache backed by filesVincent Ambo5-51/+70
Implements a local manifest cache that uses the temporary directory to cache manifest builds. This is necessary due to the size of manifests: Keeping them entirely in-memory would quickly balloon the memory usage of Nixery, unless some mechanism for cache eviction is implemented.
2019-10-03 refactor(builder): Streamline layer creation & reintroduce cachingVincent Ambo1-28/+53
The functions used for layer creation are now easier to follow and have clear points at which the layer cache is checked and populated. This relates to #50.
2019-10-03 refactor: Remove remaining MD5-hash mentions and computationsVincent Ambo5-30/+18
2019-10-03 refactor(server): Cache manifest entries for layer buildsVincent Ambo2-31/+27
MD5 hash checking is no longer performed by Nixery (it does not seem to be necessary), hence the layer cache now only keeps the SHA256 hash and size in the form of the manifest entry. This makes it possible to restructure the builder code to perform cache-fetching and cache-populating for layers in the same place.
2019-10-03 feat(server): Reintroduce manifest caching to GCSVincent Ambo2-29/+29
The new builder now caches and reads cached manifests to/from GCS. The in-memory cache is disabled, as manifests are no longer written to local file and the caching of file paths does not work (unless we reintroduce reading/writing from temp files as part of the local cache).
2019-10-03 refactor(server): Clean up cache implementationVincent Ambo3-27/+26
A couple of minor fixes and improvements to the cache implementation.
2019-10-03 chore(server): Remove "layer seen" cacheVincent Ambo1-29/+5
This cache is no longer required as it is implicit because the layer cache (mapping store path hashes to layer hashes) implies that a layer has been seen.
2019-10-03 fix(server): Upload symlink layer created by first Nix buildVincent Ambo1-1/+6
This layer is needed in addition to those that are built in the second Nix build.
2019-10-03 fix(server): Specify correct authentication scope for GCSVincent Ambo1-1/+4
When retrieving tokens for service service accounts, some methods of retrieval require a scope to be specified.
2019-10-03 feat(server): Implement new build process coreVincent Ambo1-58/+52
Implements the new build process to the point where it can actually construct and serve image manifests. It is worth noting that this build process works even if the Nix sandbox is enabled! It is also worth nothing that none of the caching functionality that the new build process enables (such as per-layer build caching) is actually in use yet, hence running Nixery at this commit is prone to doing more work than previously. This relates to #50.
2019-10-03 fix(server): Minor fixes to updated new builder codeVincent Ambo2-8/+7
2019-10-03 fix(nix): Minor fixes to derivations for new build processVincent Ambo2-2/+3
2019-10-03 feat(server): Implement package for creating image manifestsVincent Ambo1-0/+114
The new manifest package creates image manifests and their configuration. This previously happened in Nix, but is now part of the server's workload. This relates to #50.
2019-10-03 feat(server): Reimplement creation & uploading of layersVincent Ambo3-87/+250
The new build process can now call out to Nix to create layers and upload them to the bucket if necessary. The layer cache is populated, but not yet used.
2019-10-03 refactor(server): Introduce a state type to carry runtime stateVincent Ambo2-11/+31
The state type contains things such as the bucket handle and Nixery's configuration which need to be passed around in the builder. This is only added for convenience.
2019-10-03 feat(server): Add cache for layer builds in GCS & local cacheVincent Ambo1-0/+86
This cache is going to be used for looking up whether a layer build has taken place already (based on a hash of the layer contents). See the caching section in the updated documentation for details. Relates to #50.
2019-10-03 feat(nix): Add derivation to create layer tars from a store path setVincent Ambo3-9/+69
This introduces a new Nix derivation that, given an attribute set of layer hashes mapped to store paths, will create a layer tarball for each of the store paths. This is going to be used by the builder to create layers that are not present in the cache. Relates to #50.
2019-10-03 docs(caching): Add information about Nixery's caching strategiesVincent Ambo2-0/+71
2019-10-03 feat(server): Introduce function to hash contents of a layerVincent Ambo1-0/+13
This creates a cache key which can be used to check if a layer has already been built.
2019-10-03 refactor(server): Expose layer grouping logic via a functionVincent Ambo1-55/+24
Refactors the layer grouping package (which previously compiled to a separate binary) to expose the layer grouping logic via a function instead. This is the next step towards creating layers inside of the server component instead of in Nix. Relates to #50.
2019-10-03 chore(server): Import layer grouping logic into server componentVincent Ambo2-0/+361
2019-10-03 chore(build-image): Simplify wrapper build & remove layer groupingVincent Ambo5-452/+21
Simplifies the wrapper script used to invoke Nix builds from Nixery to just contain the essentials, since the layer grouping logic is moving into the server itself.
2019-10-03 refactor(build-image): Do not assemble image layers in NixVincent Ambo1-162/+57
This is the first step towards a more granular build process where some of the build responsibility moves into the server component. Rather than assembling all layers inside of Nix, it will only create the symlink forest and return information about the runtime paths required by the image. The server is then responsible for grouping these paths into layers, and assembling the layers themselves. Relates to #50.
2019-09-22 refactor(build): Add group-layers to top-level Nix derivationsVincent Ambo1-5/+7
This makes CI build the group-layers tool (and cache it to Cachix!)
2019-09-22 refactor(build-image): Minor tweak to layer construction scriptVincent Ambo1-2/+2
2019-09-21 fix(build): Ensure launch script compatibility with other runtimesVincent Ambo1-1/+2
Fixes two launch script compatibility issues with other container runtimes (such as gvisor): * don't fail if /tmp already exists * don't fail if the environment becomes unset
2019-09-21 feat(server): Log Nix output live during the buildsVincent Ambo1-3/+12
Instead of dumping all Nix output as one at the end of the build process, stream it live as the lines come in. This is a lot more useful for debugging stuff like where manifest retrievals get stuck.