about summary refs log tree commit diff
path: root/tvix/store (follow)
AgeCommit message (Collapse)AuthorFilesLines
2022-12-28 r/5527 feat(tvix/store): make blobstore stream chunksFlorian Klink3-172/+171
This changes the RPC methods to return/consume a stream of chunks, instead of a very big message containing the whole blob, to keep message sizes in manageable sizes (less than 4MiB). Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5525 chore(tvix/store/protos): add PathInfoService::CalculateNAR()Florian Klink3-39/+222
Expose the NAR calculation to a separate `CalculateNAR` method, which responds with the NAR size and sha256 hash. Contrary to what cl/7618 and cl/7620 initially did, don't add different other request types. In the CalculateNARResponse message, there's now some duplication in the (optional) `narinfo` field of a PathInfo, but I'm not entirely sure if we want to drop the fields from there yet. Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5524 chore(tvix/store/protos): drop non-sha256 NAR hashesFlorian Klink2-196/+41
Nix actually doesn't support anything else than sha256 NAR hashes, so there's little reason to communicate anything else in here. Co-Authored-By: edef <edef@unfathomable.blue> Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5523 chore(tvix/store/protos): move (Root)Node into separate messageFlorian Klink2-124/+191
This is gonna get used in another place in a second. Co-Authored-By: edef <edef@unfathomable.blue> Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5520 feat(tvix/store): implement Directory::validate()Florian Klink2-1/+329
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5519 chore(tvix/store): use DUMMY_DIGEST, tooFlorian Klink1-1/+1
Missed from cl/7649. Change-Id: Id2d382567f91cba9cdd792fe44bc0a83d040ec69 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7656 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5518 chore(tvix/store): use Directory::default in some more placesFlorian Klink1-18/+6
Change-Id: Iebad5e0b38f9506933fa855e1bfb2611d27ff572 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7655 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 r/5510 chore(tvix/store): use lazy_static and dummy digestsFlorian Klink2-2/+11
While there's currently nothing in here checking the size of the digest, we should use something that passes the to-be-introduced validate() function. Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5509 feat(tvix/store/protos): implement Directory.Validate()Florian Klink2-0/+297
Validate thecks the Directory message for invalid data, such as: - violations of name restrictions - invalid digest lengths - not properly sorted lists - duplicate names in the three lists Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5508 feat(tvix/store/protos): implement Size() and Digest() for DirectoryFlorian Klink4-0/+251
This adds Size() and Digest() functions for the golang version. Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5496 feat(txix/store/protos): init go.mod, generate .pb.go filesFlorian Klink9-0/+2390
This allows importing the generated .pb.go files into other go projects. I initially looked at buildGo.protos, but it doesn't work for multi-.proto files, and actually having LSP support for the generated structs is nice, too. Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5495 feat(tvix/store/protos): add go_package optionFlorian Klink5-0/+10
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321 Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2022-12-26 r/5494 docs(tvix/store): add README, document services and store modelFlorian Klink4-2/+391
These are intended to help digest the protocol definitions for tvix- store, and how they tie into the whole concept. Co-Authored-By: Vincent Ambo <mail@tazj.in> Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5492 feat(tvix/store): use tonic-build for protosFlorian Klink3-17/+22
While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-22 r/5471 feat(tvix/proto): add rpc_directory.protoFlorian Klink3-0/+48
This provides an interface to retrieve and upload single Directory messages, or a DAG of them. Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 r/5470 feat(tvix/proto): add rpc_pathinfoFlorian Klink3-0/+44
This defines a service that can be used to upload and retrieve metadata of nix paths. Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-15 r/5422 feat(tvix/proto): add rpc_blobstoreFlorian Klink3-0/+34
This defines a service that can be used to get and put content-addressed chunks of data. Change-Id: I36cf2278ed1daf71848c04fdfd14450b2268c5de Reviewed-on: https://cl.tvl.fyi/c/depot/+/7135 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-15 r/5420 refactor(tvix): build Rust projects using crate2nixVincent Ambo1-9/+17
Introduces granular dependency builds using crate2nix, bootstrapped off the generated configuration from the newly introduced workspace (see cl/7533). This commit checks in the generated Cargo.nix file which can be regenerated with a parameterless invocation of `crate2nix generate` in `//tvix`. I tried generating this in IFD, but it turned out to be harder than what seemed worthwhile for now. In this setup, the various build targets for Rust projects end up being attributes of the imported `Cargo.nix` file at the `tvix.crates` attribute. These still lack configuration, however, which has been fixed in the various `default.nix` files of individual projects. Note that we (temporarily) lose the ability to build tvix-eval's benchmarks in CI. I haven't figured out what magic incantation summons them from the void again ... The `eval-okay-readDir` tests from both test suites have been disabled because they fail for unknown reasons when run in this new derivation. Somebody will have to debug it! Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-15 r/5419 refactor(tvix): share a Cargo.lock file between Rust projectsVincent Ambo2-525/+1
This relates to the (abandoned) cl/7256. Introduces a Cargo workspace at //tvix that is primarily intended to be used as a workaround for the annoying Nix+Rust tooling while having a consistent set of dependencies. This is driven in part by a desire to adopt crate2nix and get more granular Nix builds for Tvix's Rust projects, and in part by a need to split //tvix/eval into something providing the CLI (REPL etc.), and a library providing eval, without significantly altering the structure of build targets. To accomplish this the workspace has been designed to allow projects to remain independent build targets. I want to avoid lumping all the projects together - something like //tvix/eval should always be independent of other parts of tvix. A helper function in //tvix/default.nix lets downstream naersk projects construct a sparse root for the project which combines the workspace's `Cargo.lock` with the project's own `Cargo.toml`. Note that cargo commands in the workspace itself require the build dependencies of _all_ projects to be present, which is currently a bit annoying to accomplish. This introduces some breakage: 1. It breaks usage of rust-analyser without being in a shell with the dependencies of *all* Tvix projects, as it is not capable of respecting only the subset of dependencies for a part of the workspace. 2. It is no longer possible to run tests using `cargo test`, as the test generation crate we use does not work with workspaces: https://github.com/frehberg/test-generator/issues/6 This still works in the Nix build as we construct a Cargo project that looks like it's not in a workspace there. Until somebody fixes that crate / writes a new macro / does something else with the test suite, the way to run the tests is through the Nix build. Long-term we'll probably want to get rid of cargo completely, it's just a big wart and most tooling works without it if correctly configured, but we don't have time for that now. Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-04 r/5386 feat(tvix/store): add size() and digest() functions for DirectoryFlorian Klink5-3/+307
Change-Id: I3a9e3c77da1d5d7bfb93da0cd7bce8d48dbc8cd4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7254 Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-04 r/5385 feat(tvix/store): build rust proto filesFlorian Klink4-0/+343
This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-04 r/5384 feat(tvix/store): initFlorian Klink4-0/+35
Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-04 r/5383 feat(tvix/store): add pathinfo.protoFlorian Klink1-0/+74
This adds the PathInfo message, which hosts information about a Nix Store path, mapping to either of a {Directory,File,Symlink}Node. Change-Id: I79d871b6fad450d6a4ae4101fb72c51f9a83471f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7132 Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-04 r/5382 chore(tvix/store): move castore.protoFlorian Klink2-0/+80
This moves the castore.proto file into the //tvix/store/protos directory. Per- component protos might make more sense, than a "tvix-wide" proto directory. Change-Id: Ie728210174b041e9285a0e2ac605d715d0f0cbda Reviewed-on: https://cl.tvl.fyi/c/depot/+/7285 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-07-19T00·19+0000