about summary refs log tree commit diff
path: root/tools/nixery
AgeCommit message (Collapse)AuthorFilesLines
2019-11-09 feat(build): Integration test on both CPU architecturesVincent Ambo1-1/+22
2019-11-09 feat(build): Include arm64 in build matrixVincent Ambo1-0/+4
2019-11-09 fix(build-image): Allow "cross-builds" of images for different archVincent Ambo1-4/+9
Imports the package set twice in the builder expression: Once configured for the target system, once configured for the native system. This makes it possible to fetch the actual image contents for the required architecture, but use local tools to assemble the symlink layer and metadata.
2019-11-09 feat(builder): Support 'arm64' meta-packageVincent Ambo1-12/+29
Specifying this meta-package toggles support for ARM64 images, for example: # Pull a default x86_64 image docker pull nixery.dev/hello # Pull an ARM64 image docker pull nixery.dev/arm64/hello
2019-11-09 refactor(builder): Parameterise CPU architecture to use for imagesVincent Ambo3-6/+33
Adds the CPU architecture to the image configuration. This will make it possible to let users toggle architecture via meta-packages. Relates to #13
2019-11-03 chore(build): Add nixery-popcount to top-level package setVincent Ambo2-0/+28
2019-11-03 feat(popcount): Cache seen narinfos on diskVincent Ambo1-0/+14
2019-11-03 chore: Add missing copyright headers to source filesVincent Ambo8-0/+110
2019-11-03 feat(popcount): Add new narinfo-based popcount implementationVincent Ambo4-67/+256
Adds an implementation of popcount that, instead of realising derivations locally, just queries the cache's narinfo files. The downside of this is that calculating popularity for arbitrary Nix package sets is not possible with this implementation. The upside is that calculating the popularity for an entire Nix channel can now be done in ~10 seconds[0]. This fixes #65. [0]: Assuming a /fast/ internet connection.
2019-10-29 chore(server): Remove outdated TODOVincent Ambo1-5/+1
Real-life experience has shown that the weighting of the metric produced here is appropriate.
2019-10-29 refactor(server): Handle non-error errors safelyVincent Ambo1-5/+6
This case should not be possible unless something manually constructs a logrus entry with a non-error value in the log.ErrorKey field, but it's better to be safe than sorry.
2019-10-28 chore(build): Configure build to use new GCS configuration optionsVincent Ambo1-3/+2
2019-10-28 fix(server): Use correct scope for GCS tokensVincent Ambo1-1/+1
2019-10-28 docs(under-the-hood): Update builder & storage backend informationVincent Ambo1-28/+51
Both of these no longer matched the reality of what was actually going on in Nixery.
2019-10-28 docs: Add storage configuration options to READMEVincent Ambo1-1/+25
2019-10-28 fix(server): Thread request context to all relevant placesVincent Ambo6-23/+20
Previously background contexts where created where necessary (e.g. in GCS interactions). Should I begin to use request timeouts or other context-dependent things in the future, it's useful to have the actual HTTP request context around. This threads the request context through the application to all places that need it.
2019-10-28 chore(server): Move cache miss log statement to debug levelVincent Ambo2-2/+2
This is very annoying otherwise.
2019-10-28 fix(server): Correctly construct filesystem paths for layer servingVincent Ambo1-6/+8
2019-10-28 fix(server): Ensure paths exist when renaming in filesystem storageVincent Ambo1-1/+7
The point at which files are moved happens to also (initially) be the point where the `layers` directory is created. For this reason renaming must ensure that all path components exist, which this commit takes care of.
2019-10-28 fix(server): Ensure error messages are correctly printed in logsVincent Ambo1-0/+7
I assumed (incorrectly) that logrus would already take care of surfacing error messages in human-readable form.
2019-10-28 feat(server): Add filesystem storage backend config optionsVincent Ambo3-1/+11
The filesystem storage backend can be enabled by setting `NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk location from which Nixery can serve files.
2019-10-28 refactor(server): Pass HTTP request to storage.ServeLayerVincent Ambo4-4/+6
The request object is required for some serving methods (e.g. the filesystem one).
2019-10-28 feat(server): Implement initial filesystem storage backendVincent Ambo1-0/+68
This allows users to store and serve layers from a local filesystem path.
2019-10-28 refactor(server): Change setup to create new storage backendsVincent Ambo5-15/+44
2019-10-28 feat(server): Implement GCS storage backend with new interfaceVincent Ambo1-0/+206
Logical implementation is mostly identical to the previous one, but adhering to the new storage.Backend interface.
2019-10-28 refactor(server): Introduce pluggable interface for storage backendsVincent Ambo5-229/+111
This abstracts over the functionality of Google Cloud Storage and other potential underlying storage backends to make it possible to replace these in Nixery. The GCS backend is not yet reimplemented.
2019-10-28 refactor(build): Do not expose nixery-server attributeVincent Ambo1-5/+3
In most cases this is not useful for users without the wrapper script, so users should always build nixery-bin anyways.
2019-10-27 docs: Update GCS signing key documentationVincent Ambo2-12/+14
This key is now taken straight from the configured service account key.
2019-10-27 refactor(server): Load GCS signing key from service account keyVincent Ambo1-12/+16
The JSON file generated for service account keys already contains the required information for signing URLs in GCS, thus the environment variables for toggling signing behaviour have been removed. Signing is now enabled automatically in the presence of service account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-11 fix(build): Only take the first matching hash for source hashingVincent Ambo1-1/+1
Some Nix download mechanisms will add a second hash in the store path, which had been added to the source hash output (breaking argument interpolation).
2019-10-11 refactor(server): Only compress symlink forest layer onceVincent Ambo2-16/+24
Instead of compressing & decompressing again to get the underlying tar hash, use a similar mechanism as for store path layers for the symlink layer and only compress it once while uploading.
2019-10-11 fix(server): Use uncompressed tarball hashes in image configVincent Ambo4-16/+42
Docker expects hashes of compressed tarballs in the manifest (as these are used to fetch from the content-addressable layer store), but for some reason it expects hashes in the configuration layer to be of uncompressed tarballs. To achieve this an additional SHA256 hash is calculcated while creating the layer tarballs, but before passing them to the gzip writer. In the current constellation the symlink layer is first compressed and then decompressed again to calculate its hash. This can be refactored in a future change.
2019-10-11 feat(server): Apply GZIP compression to all image layersVincent Ambo4-9/+15
This fixes #62
2019-10-11 chore(build): Use separate GCS bucket for CI runsVincent Ambo1-1/+1
This has become an issue recently with changes such as GZIP compression, where CI runs no longer work because they conflict with the production bucket for the public instance.
2019-10-06 refactor(server): Use logrus convenience functions for logsVincent Ambo5-128/+59
Makes use of the `.WithError` and `.WithField` convenience functions in logrus to simplify log statement construction. This has the added benefit of making it easier to correctly log errors.
2019-10-06 fix(build-image): Import 'match' from builtinsVincent Ambo1-0/+1
2019-10-06 refactor(server): Convert existing log entries to structured formatVincent Ambo7-54/+243
This rewrites all existing log statements into the structured logrus format. For consistency, all errors are always logged separately from the primary message in a field called `error`. Only the "info", "error" and "warn" severities are used.
2019-10-06 feat(server): Add log level to severity mappingVincent Ambo2-3/+36
The output format now writes a `severity` field that follows that format that should be recognised by Stackdriver Logging.
2019-10-06 feat(server): Use hash of Nixery source as versionVincent Ambo4-11/+48
Uses a hash of Nixery's sources as the version displayed when Nixery launches or logs an error. This makes it possible to distinguish between errors logged from different versions. The source hashes should be reproducible between different checkouts of the same source tree.
2019-10-06 feat(server): Initial Stackdriver-compatible log formatterVincent Ambo1-0/+68
This formatter has basic support for the Stackdriver Error Reporting format, but several things are still lacking: * the service version (preferably git commit?) needs to be included in the server somehow * log streams should be split between stdout/stderr as that is how AppEngine (and several other GCP services?) seemingly differentiate between info/error logs
2019-10-06 fix(server): Amend package path for Go tooling compatibilityVincent Ambo6-10/+10
With these changes it is possible to keep Nixery in $GOPATH and build the server in there, while still having things work correctly via Nix.
2019-10-06 refactor(server): Replace log calls with logrusVincent Ambo8-7/+17
This introduces a structured logging library that can be used (next step) to attach additional metadata to log entries.
2019-10-03 refactor(server): Always include 'cacert' & 'iana-etc'Vincent Ambo2-10/+19
These two packages almost always end up being required by programs, but people don't necessarily consider them. They will now always be added and their popularity is artificially inflated to ensure they end up at the top of the layer list.
2019-10-03 fix(server): Ensure build cache objects are written to GCSVincent Ambo1-0/+5
Cache writes might not be flushed without this call.
2019-10-03 feat(server): Order layers in image manifest based on merge ratingVincent Ambo3-4/+20
Image layers in manifests are now sorted in a stable (descending) order based on their merge rating, meaning that layers more likely to be shared between images come first. The reason for this change is Docker's handling of image layers on overlayfs2: Images are condensed into a single representation on disk after downloading. Due to this Docker will constantly redownload all layers that are applied in a different order in different images (layer order matters in imperatively created images), based on something it calls the 'ChainID'. Sorting the layers this way raises the likelihood of a long chain of matching layers at the beginning of an image. This relates to #39.
2019-10-03 chore(build-image): Remove nixery-build-layersVincent Ambo3-70/+10
This functionality has been rolled into the server component and is no longer required.
2019-10-03 feat(server): Implement creation of layer tarballs in the serverVincent Ambo2-79/+158
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 fix(server): Do not invoke layer build if no layers are missingVincent Ambo1-1/+5
This previously invoked a Nix derivation that spent a few seconds on making an empty object in JSON ...
2019-10-03 feat(server): Fetch popularity data on launchVincent Ambo1-0/+37
The last missing puzzle piece for #50!
2019-10-03 feat(server): Reimplement local manifest cache backed by filesVincent Ambo5-51/+70
Implements a local manifest cache that uses the temporary directory to cache manifest builds. This is necessary due to the size of manifests: Keeping them entirely in-memory would quickly balloon the memory usage of Nixery, unless some mechanism for cache eviction is implemented.