about summary refs log tree commit diff
path: root/tvix/Cargo.nix (follow)
AgeCommit message (Collapse)AuthorFilesLines
2023-03-16 r/6013 chore(tvix): run `cargo update`Vincent Ambo1-397/+303
Change-Id: I710f0920da118cb1b94cd8ff364adadbeef998bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8315 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-03-15 r/6009 refactor(tvix/nix-compat): -derivation::Hash, +NixHashFlorian Klink1-0/+4
This stops using our own custom Hash structure, which was mostly only used because we had to parse the JSON representation somehow. Since cl/8217, there's a `NixHash` struct, which is better suited to hold this data. Converting the format requires a bit of serde labor though, but that only really matters when interacting with JSON representations (which we mostly don't). Change-Id: Idc5ee511e36e6726c71f66face8300a441b0bf4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8304 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 r/6007 chore(tvix/store): drop separate bin packageFlorian Klink1-18/+5
Apparently, having multiple packages with the same path is a bad thing: ``` The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`. Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html The output filenames should be unique. This is a known bug where multiple crates with the same name use the same path; see <https://github.com/rust-lang/cargo/issues/6313>. ``` Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 r/5988 chore(tvix/eval): prune some dependencies & featuresVincent Ambo1-435/+22
* We no longer need backtrace-on-stack-overflow, as we no longer overflow the stack with the recent eval refactorings. This was weird voodoo anyways, introduced earlier to debug some cases where stack overflows occured. * default features of genawaiter crate are not needed, as we don't use their proc macros Change-Id: I346fc5a18d7f117ee805909a8be8f535b96be76c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8263 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 r/5959 refactor(tvix/store): use read_all_and_chunk in gRPC blobserviceFlorian Klink1-3/+8
This was the last piece of code using BlobWriter. We can also use `read_all_and_chunk`, it's just requires a bit more plumbing: - The data coming from the client (stream) needs to be mapped (we extract the .data field). - The stream needs to be turned into an (async) reader - The reader needs to be made sync, and that code using the sync reader needs to be in a `task::spawn_blocking`. Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-11 r/5953 refactor(tvix/store): bump fastcdc depFlorian Klink1-2/+2
This removes the use of Box::new, switching fastcdc to version 3.0.2 with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed. Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5939 feat(tvix/store): use rayon to upload chunks concurrentlyFlorian Klink1-0/+4
Look at the data that's written to us, and upload all chunks but the rest in parallel, using rayon. This required moving `upload_chunk` outside the struct, and accepting a ChunkService to use for upload (which it was previously getting from `self.chunk_service`). This doesn't speed up things too much for now, because things are still mostly linear. Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5936 refactor(tvix/store): move entrypoint to src/bin/tvix-store.rsFlorian Klink1-2/+2
Change-Id: Ibb83be75a2be27debd9e85b43c1b824f59e54dab Reviewed-on: https://cl.tvl.fyi/c/depot/+/8165 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5932 feat(tvix/store): add import::import_pathFlorian Klink1-0/+4
This imports the contents at a given Path into the tvix store. It doesn't register the contents at a Path in the store itself, that's up to the PathInfoService. Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5925 feat(tvix/store): bump fastcdc, use v2020 versionFlorian Klink1-2/+2
This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5924 feat(tvix/store): add --json argFlorian Klink1-1/+41
This configures logging as JSON. Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5918 feat(tvix/store): add NARCalculationService trait and implFlorian Klink1-0/+23
This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-07 r/5899 feat(tvix/eval): introduce generators moduleVincent Ambo1-5/+202
This module contains the request/response types for generators requesting actions from the VM. For most of these, an async helper function is added that will be used inside of generator functions to make use of these requests/responses instead of constructing them directly. Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-02-04 r/5836 fix(tvix/cli): use tvlfyi/wu-manber fork for refscannerVincent Ambo1-25/+395
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-02 r/5825 refactor(tvix/cli): use Wu-Manber string scanning for drv referencesVincent Ambo1-0/+19
Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 r/5819 chore(tvix): update Cargo.lockFlorian Klink1-247/+272
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5794 feat(tvix/store): add write_nar functionFlorian Klink1-22/+0
This adds a function that consumes a [proto::node::Node] pointing to the root of a (store) path, and writes the contents in NAR serialization to the passed [std::io::Write]. We need this in various places: - tvix-store's calculate_nar() RPC method needs to render a NAR stream to get the nar hash, which is necessary to give things imported in the store a "NAR-based" store path. - communication with (remote) Nix (via daemon protocol) needs a NAR representation. - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a NAR representation. Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5791 refactor(tvix/nix-compat): absorb //tvix/derivationFlorian Klink1-71/+17
Put this in its src/derivation. Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-31 r/5788 refactor(tvix): introduce nix-compat crateFlorian Klink1-8/+58
Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5783 refactor(tvix/cli): absorb construct_output_hashFlorian Klink1-6/+0
This helper function only was created because populate_output_configuration was hard to test before cl/7939. With that out of the way, we can pull it in. Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-27 r/5773 refactor(tvix/cli): describe errors with thiserrorFlorian Klink1-0/+4
This is much less code, and makes it much easier to read. Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-27 r/5772 fix(tvix/cli): handle SRI hashes in outputHashFlorian Klink1-13/+305
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`, `fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash` attribute, being an SRI hash. In that case, `builtin.derivation` is called with `outputHashAlgo` being an empty string, and `outputHash` being an SRI hash string. In other cases, an SRI hash is passed as outputHash, but outputHashAlgo is set too. Nix does modify these values in (single, fixed) output specification it serializes to ATerm, but keeps it unharmed in `env`. Move this into a construct_output_hash helper function, that can be tested better in isolation. Change-Id: Id9d716a119664c44ea7747540399966752e20187 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-25 r/5754 feat(tvix/eval): implement builtins.fromTOMLFlorian Klink1-0/+141
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML testcase from nix_tests. It uses the `toml` crate, and the serde integration it brings with it. Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-25 r/5753 feat(tvix/eval): use lexical-core to format floatFlorian Klink1-0/+249
Apparently our naive implementation of float formatting, which simply used {:.5}, and trimmed trailing "0" strings not sufficient. It wrongly trimmed numbers with zeroes but no decimal point, like `10000` got trimmed to `1`. Nix uses `std::to_string` on the double, which according to https://en.cppreference.com/w/cpp/string/basic_string/to_string is equivalent to `std::sprintf(buf, "%f", value)`. https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated like this: > Precision specifies the exact number of digits to appear after > the decimal point character. The default precision is 6. In the > alternative implementation decimal point character is written even if > no digits follow it. For infinity and not-a-number conversion style > see notes. This doesn't seem to be the case though, and Nix uses scientific notation in some cases. There's a whole bunch of strategies to determine which is a more compact notation, and which notation should be used for a given number. https://github.com/rust-lang/rust/issues/24556 provides some pointers into various rabbit holes for those interested. This gist seems to be that currently a different formatting is not exposed in rust directly, at least not for public consumption. There is the [lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate though, which provides a way to format floats with various strategies and formats. Change our implementation of `TotalDisplay` for the `Value::Float` case to use that. We still need to do some post-processing, because Nix always adds the sign in scientific notation (and there's no way to configure lexical-core to do that), and lexical-core in some cases keeps the trailing zeros. Even with all that in place, there as a difference in `eval-okay- fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated the fixture to a less problematic number. With this, the testsuite passes again, and does for the upcoming CL introducing builtins.fromTOML, and enabling the nix testsuite bits for it, too. Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5727 chore(tvix/store): add fastcdc crateFlorian Klink1-0/+14
This is used for content-defined chunking. Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5720 chore(tvix/store): add tonic-mockFlorian Klink1-3/+214
Upstream seems to be dead, so we're using https://github.com/tyrchen/ tonic-mock/pull/3 here. According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment- 1241164173, we might not need this crate at all, but for now, it gets the job done and is less code to write in the tests. Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-20 r/5709 feat(tvix/cli): add `errors` module with drv construction errorsVincent Ambo1-0/+4
These will be threaded through to eval through the new `TvixError` variant. Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-18 r/5701 feat(tvix/derivation): more checks for output hashes and encodingFlorian Klink1-0/+4
Change-Id: Idebad60c3bf9daf94d04a36bb73ac0dd767f9e79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7856 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-17 r/5670 refactor(tvix/value): use proptest strategies from imbl crateVincent Ambo1-5/+125
Instead of going through Vec/BTreeMap for generating our internal types, use the proptest strategies from imbl. The one thing I couldn't figure out in the previous implementation is where the ranges/sizes of generated collections came from. The strategies in proptest use different types (Range, with an unknown default value, and SizeRange with 0..100). I've opted to specify 0..100 directly, but we can probably make it configurable. Change-Id: I749bc4c703fe424099240cab822b1642e5216361 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-16 r/5668 refactor(tvix/derivation): expose proper ValidateDerivationErrorFlorian Klink1-0/+4
Use proper errors, instead of anyhow. Change-Id: I6db14c72a6319b389b0136aac7b84f50a30fb366 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7847 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-16 r/5664 feat(tvix/eval): implement builtins.toXMLVincent Ambo1-0/+16
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5649 feat(tvix/store): implement PathInfoService with sledFlorian Klink1-3/+360
This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-11 r/5643 feat(tvix/cli): implement initial refscan moduleVincent Ambo1-0/+4
This module implements a ReferenceScanner struct which uses the aho_corasick crate to scan string inputs for known, non-overlapping candidates (store paths, in our case). I experimented with several different APIs, and landed on this version with an initial accumulator in the scanner. The scanner is instantiated from the candidates and "fed" all the strings, then consumed by the caller to retrieve the result. Right now only things that look vaguely like bytestrings can be fed to the scanner, there is no streaming support in the API yet. Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-10 r/5640 feat(tvix/eval): implement serde::Deserialize for ValueRyan Lahfa1-1/+15
Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-06 r/5616 fix(tvix/derivation): fix build after StorePath renameVincent Ambo1-58/+58
This project was not previously covered by CI (fixed in this commit), so we didn't catch breakage due to a renamed module. This was noticed while rebasing a CL that has a dependency on this crate in its Nix build. Change-Id: Ic48570b9313e5f73e14daab50cf7ea70918c94d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7778 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-04 r/5591 feat(tvix/derivation): implement Derivation::validate()Florian Klink1-0/+83
Change-Id: I87dfadda872439e108e5f678a5da63dd5b1915d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7732 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-03 r/5576 test(tvix/serde): add initial set of deserialisation testsVincent Ambo1-0/+1
Change-Id: I0a9779edf0296c25d37fb5f75f8fc5852fe00121 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7715 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de>
2023-01-03 r/5569 chore(tvix/store): make importableFlorian Klink1-6/+7
This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-02 r/5566 feat(tvix/serde): initial Nix->serde::Deserialize implVincent Ambo1-0/+32
This will make it possible fairly easily use Nix to represent arbitrary data structures, e.g. for using Nix as a config language. Only pure Nix (i.e. no `import` etc.) is supported for now. Not all types, specifically no struct traversal, are implemented in this commit. Change-Id: I9ac91a229a0d12bf818e6e3249f3e5a691599a2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7712 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 r/5558 feat(tvix/store): add logging with tracingFlorian Klink1-1/+215
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5557 feat(tvix/store): implement reflectionFlorian Klink1-2/+60
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5554 feat(tvix/store): initial dummy implementationFlorian Klink1-12/+32
This replaces the hello world example from tvix-store with an actual gRPC endpoint, implementing all of BlobService, DirectoryService and PathInfoService. All RPC methods currently respond with the unimplemented gRPC status. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5550 feat(tvix/store): add nixbase32 modFlorian Klink1-30/+92
This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 r/5541 refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrsVincent Ambo1-50/+39
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory between different iterations of a map. This slightly speeds up eval, but not significantly. Future work might include benchmarking whether using a `HashMap` and only ordering in cases where order is actually required would help. This switches to a fork of `im` that fixes some bugs with its OrdMap implementation. Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-12-29 r/5536 feat(tvix/derivation): serialize Nix DerivationJürgen Hahn1-0/+61
This adds a Derivation structure and allows to write it to a structure that implements std::fmt:Write. The implementation is based on the go-nix version. Change-Id: Ib54e1202b5c67f5d206b21bc109a751e971064cf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7659 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 r/5534 refactor(tvix/eval): use im::Vector for NixList representationVincent Ambo1-0/+114
This is a persistent, structurally sharing data structure which is more efficient in some of our use-cases. I have verified the efficiency improvement using `hyperfine` repeatedly over expressions on nixpkgs. Lists are not the most performance-critical structure in Nix (that would be attribute sets), but we can already see a small (~5-10%) improvement. Note that there are a handful of cases where we still go via `Vec` that need to be fixed, most notable for `builtins.sort` which can not currently be implemented directly using `im::Vector` because of a restrictive type bound. Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 r/5532 chore(tvix/store): trim Cargo.tomlFlorian Klink1-4/+6
lazy_static is only used in tests, and anyhow isn't used at all (yet). This can be dropped. Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5520 feat(tvix/store): implement Directory::validate()Florian Klink1-6/+14
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5510 chore(tvix/store): use lazy_static and dummy digestsFlorian Klink1-0/+4
While there's currently nothing in here checking the size of the digest, we should use something that passes the to-be-introduced validate() function. Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-26 r/5492 feat(tvix/store): use tonic-build for protosFlorian Klink1-3/+1775
While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>