about summary refs log tree commit diff
path: root/tools
AgeCommit message (Collapse)AuthorFilesLines
2020-02-26 fix(popcount): Fix nix-build -A nixery-popcountFlorian Klink1-8/+6
Previously, this was failing as follows: ``` these derivations will be built: /nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'... building warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1 error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed ```
2020-01-19 fix(builder): Fix minor logging switcharooVincent Ambo1-1/+1
2019-11-27 refactor: Reshuffle file structure for better code layoutVincent Ambo21-114/+83
This gets rid of the package called "server" and instead moves everything into the project root, such that Go actually builds us a binary called `nixery`. This is the first step towards factoring out CLI-based functionality for Nixery.
2019-11-27 fix(builder): Ensure "solo-metapackages" do not break buildsVincent Ambo1-1/+7
The previous logic failed because single meta-packages such as "nixery.dev/shell" would not end up removing the meta-package itself from the list of packages passed to Nix, causing a build failure. This was a regression introduced in 827468a.
2019-11-27 test(builder): Add test coverage for name->image conversionVincent Ambo1-0/+123
Adds tests to cover that packages & metapackages are parsed into image names correctly.
2019-11-09 chore(build): Use significantly fewer layers for Nixery itselfVincent Ambo2-3/+5
Nixery itself is built with the buildLayeredImage system, which takes some time to create large numbers of layers. This adjusts the default number of image layers from 96 to 20. Additionally Nixery's image is often loaded with `docker load -i`, which ignores layer cache hits anyways. Additionaly the CI build is configured to use only 1, which speeds up CI runs.
2019-11-09 feat(build): Integration test on both CPU architecturesVincent Ambo1-1/+22
2019-11-09 feat(build): Include arm64 in build matrixVincent Ambo1-0/+4
2019-11-09 fix(build-image): Allow "cross-builds" of images for different archVincent Ambo1-4/+9
Imports the package set twice in the builder expression: Once configured for the target system, once configured for the native system. This makes it possible to fetch the actual image contents for the required architecture, but use local tools to assemble the symlink layer and metadata.
2019-11-09 feat(builder): Support 'arm64' meta-packageVincent Ambo1-12/+29
Specifying this meta-package toggles support for ARM64 images, for example: # Pull a default x86_64 image docker pull nixery.dev/hello # Pull an ARM64 image docker pull nixery.dev/arm64/hello
2019-11-09 refactor(builder): Parameterise CPU architecture to use for imagesVincent Ambo3-6/+33
Adds the CPU architecture to the image configuration. This will make it possible to let users toggle architecture via meta-packages. Relates to #13
2019-11-03 chore(build): Add nixery-popcount to top-level package setVincent Ambo2-0/+28
2019-11-03 feat(popcount): Cache seen narinfos on diskVincent Ambo1-0/+14
2019-11-03 chore: Add missing copyright headers to source filesVincent Ambo8-0/+110
2019-11-03 feat(popcount): Add new narinfo-based popcount implementationVincent Ambo4-67/+256
Adds an implementation of popcount that, instead of realising derivations locally, just queries the cache's narinfo files. The downside of this is that calculating popularity for arbitrary Nix package sets is not possible with this implementation. The upside is that calculating the popularity for an entire Nix channel can now be done in ~10 seconds[0]. This fixes #65. [0]: Assuming a /fast/ internet connection.
2019-10-29 chore(server): Remove outdated TODOVincent Ambo1-5/+1
Real-life experience has shown that the weighting of the metric produced here is appropriate.
2019-10-29 refactor(server): Handle non-error errors safelyVincent Ambo1-5/+6
This case should not be possible unless something manually constructs a logrus entry with a non-error value in the log.ErrorKey field, but it's better to be safe than sorry.
2019-10-28 chore(build): Configure build to use new GCS configuration optionsVincent Ambo1-3/+2
2019-10-28 fix(server): Use correct scope for GCS tokensVincent Ambo1-1/+1
2019-10-28 docs(under-the-hood): Update builder & storage backend informationVincent Ambo1-28/+51
Both of these no longer matched the reality of what was actually going on in Nixery.
2019-10-28 docs: Add storage configuration options to READMEVincent Ambo1-1/+25
2019-10-28 fix(server): Thread request context to all relevant placesVincent Ambo6-23/+20
Previously background contexts where created where necessary (e.g. in GCS interactions). Should I begin to use request timeouts or other context-dependent things in the future, it's useful to have the actual HTTP request context around. This threads the request context through the application to all places that need it.
2019-10-28 chore(server): Move cache miss log statement to debug levelVincent Ambo2-2/+2
This is very annoying otherwise.
2019-10-28 fix(server): Correctly construct filesystem paths for layer servingVincent Ambo1-6/+8
2019-10-28 fix(server): Ensure paths exist when renaming in filesystem storageVincent Ambo1-1/+7
The point at which files are moved happens to also (initially) be the point where the `layers` directory is created. For this reason renaming must ensure that all path components exist, which this commit takes care of.
2019-10-28 fix(server): Ensure error messages are correctly printed in logsVincent Ambo1-0/+7
I assumed (incorrectly) that logrus would already take care of surfacing error messages in human-readable form.
2019-10-28 feat(server): Add filesystem storage backend config optionsVincent Ambo3-1/+11
The filesystem storage backend can be enabled by setting `NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk location from which Nixery can serve files.
2019-10-28 refactor(server): Pass HTTP request to storage.ServeLayerVincent Ambo4-4/+6
The request object is required for some serving methods (e.g. the filesystem one).
2019-10-28 feat(server): Implement initial filesystem storage backendVincent Ambo1-0/+68
This allows users to store and serve layers from a local filesystem path.
2019-10-28 refactor(server): Change setup to create new storage backendsVincent Ambo5-15/+44
2019-10-28 feat(server): Implement GCS storage backend with new interfaceVincent Ambo1-0/+206
Logical implementation is mostly identical to the previous one, but adhering to the new storage.Backend interface.
2019-10-28 refactor(server): Introduce pluggable interface for storage backendsVincent Ambo5-229/+111
This abstracts over the functionality of Google Cloud Storage and other potential underlying storage backends to make it possible to replace these in Nixery. The GCS backend is not yet reimplemented.
2019-10-28 refactor(build): Do not expose nixery-server attributeVincent Ambo1-5/+3
In most cases this is not useful for users without the wrapper script, so users should always build nixery-bin anyways.
2019-10-27 docs: Update GCS signing key documentationVincent Ambo2-12/+14
This key is now taken straight from the configured service account key.
2019-10-27 refactor(server): Load GCS signing key from service account keyVincent Ambo1-12/+16
The JSON file generated for service account keys already contains the required information for signing URLs in GCS, thus the environment variables for toggling signing behaviour have been removed. Signing is now enabled automatically in the presence of service account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-11 fix(build): Only take the first matching hash for source hashingVincent Ambo1-1/+1
Some Nix download mechanisms will add a second hash in the store path, which had been added to the source hash output (breaking argument interpolation).
2019-10-11 refactor(server): Only compress symlink forest layer onceVincent Ambo2-16/+24
Instead of compressing & decompressing again to get the underlying tar hash, use a similar mechanism as for store path layers for the symlink layer and only compress it once while uploading.
2019-10-11 fix(server): Use uncompressed tarball hashes in image configVincent Ambo4-16/+42
Docker expects hashes of compressed tarballs in the manifest (as these are used to fetch from the content-addressable layer store), but for some reason it expects hashes in the configuration layer to be of uncompressed tarballs. To achieve this an additional SHA256 hash is calculcated while creating the layer tarballs, but before passing them to the gzip writer. In the current constellation the symlink layer is first compressed and then decompressed again to calculate its hash. This can be refactored in a future change.
2019-10-11 feat(server): Apply GZIP compression to all image layersVincent Ambo4-9/+15
This fixes #62
2019-10-11 chore(build): Use separate GCS bucket for CI runsVincent Ambo1-1/+1
This has become an issue recently with changes such as GZIP compression, where CI runs no longer work because they conflict with the production bucket for the public instance.
2019-10-06 refactor(server): Use logrus convenience functions for logsVincent Ambo5-128/+59
Makes use of the `.WithError` and `.WithField` convenience functions in logrus to simplify log statement construction. This has the added benefit of making it easier to correctly log errors.
2019-10-06 fix(build-image): Import 'match' from builtinsVincent Ambo1-0/+1
2019-10-06 refactor(server): Convert existing log entries to structured formatVincent Ambo7-54/+243
This rewrites all existing log statements into the structured logrus format. For consistency, all errors are always logged separately from the primary message in a field called `error`. Only the "info", "error" and "warn" severities are used.
2019-10-06 feat(server): Add log level to severity mappingVincent Ambo2-3/+36
The output format now writes a `severity` field that follows that format that should be recognised by Stackdriver Logging.
2019-10-06 feat(server): Use hash of Nixery source as versionVincent Ambo4-11/+48
Uses a hash of Nixery's sources as the version displayed when Nixery launches or logs an error. This makes it possible to distinguish between errors logged from different versions. The source hashes should be reproducible between different checkouts of the same source tree.
2019-10-06 feat(server): Initial Stackdriver-compatible log formatterVincent Ambo1-0/+68
This formatter has basic support for the Stackdriver Error Reporting format, but several things are still lacking: * the service version (preferably git commit?) needs to be included in the server somehow * log streams should be split between stdout/stderr as that is how AppEngine (and several other GCP services?) seemingly differentiate between info/error logs
2019-10-06 fix(server): Amend package path for Go tooling compatibilityVincent Ambo6-10/+10
With these changes it is possible to keep Nixery in $GOPATH and build the server in there, while still having things work correctly via Nix.
2019-10-06 refactor(server): Replace log calls with logrusVincent Ambo8-7/+17
This introduces a structured logging library that can be used (next step) to attach additional metadata to log entries.
2019-10-03 refactor(server): Always include 'cacert' & 'iana-etc'Vincent Ambo2-10/+19
These two packages almost always end up being required by programs, but people don't necessarily consider them. They will now always be added and their popularity is artificially inflated to ensure they end up at the top of the layer list.
2019-10-03 fix(server): Ensure build cache objects are written to GCSVincent Ambo1-0/+5
Cache writes might not be flushed without this call.
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-12T10·48+0000