about summary refs log tree commit diff
path: root/tvix/Cargo.lock
AgeCommit message (Collapse)AuthorFilesLines
2023-03-11 r/5953 refactor(tvix/store): bump fastcdc depFlorian Klink1-2/+2
This removes the use of Box::new, switching fastcdc to version 3.0.2 with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed. Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5939 feat(tvix/store): use rayon to upload chunks concurrentlyFlorian Klink1-0/+1
Look at the data that's written to us, and upload all chunks but the rest in parallel, using rayon. This required moving `upload_chunk` outside the struct, and accepting a ChunkService to use for upload (which it was previously getting from `self.chunk_service`). This doesn't speed up things too much for now, because things are still mostly linear. Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5932 feat(tvix/store): add import::import_pathFlorian Klink1-0/+1
This imports the contents at a given Path into the tvix store. It doesn't register the contents at a Path in the store itself, that's up to the PathInfoService. Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5925 feat(tvix/store): bump fastcdc, use v2020 versionFlorian Klink1-2/+2
This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5924 feat(tvix/store): add --json argFlorian Klink1-0/+13
This configures logging as JSON. Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5918 feat(tvix/store): add NARCalculationService trait and implFlorian Klink1-0/+8
This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-07 r/5899 feat(tvix/eval): introduce generators moduleVincent Ambo1-3/+77
This module contains the request/response types for generators requesting actions from the VM. For most of these, an async helper function is added that will be used inside of generator functions to make use of these requests/responses instead of constructing them directly. Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-02-04 r/5836 fix(tvix/cli): use tvlfyi/wu-manber fork for refscannerVincent Ambo1-16/+46
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-02 r/5825 refactor(tvix/cli): use Wu-Manber string scanning for drv referencesVincent Ambo1-1/+6
Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 r/5819 chore(tvix): update Cargo.lockFlorian Klink1-230/+230
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5793 refactor(tvix/nix-compat): absorb nar writerFlorian Klink1-4/+0
Expose it at `nix_compat::nar::writer`. Change-Id: I98a8acfa6277074f24608d64fb6c0082b41d10f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7969 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5791 refactor(tvix/nix-compat): absorb //tvix/derivationFlorian Klink1-17/+4
Put this in its src/derivation. Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-31 r/5788 refactor(tvix): introduce nix-compat crateFlorian Klink1-2/+14
Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5783 refactor(tvix/cli): absorb construct_output_hashFlorian Klink1-1/+0
This helper function only was created because populate_output_configuration was hard to test before cl/7939. With that out of the way, we can pull it in. Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-27 r/5773 refactor(tvix/cli): describe errors with thiserrorFlorian Klink1-0/+1
This is much less code, and makes it much easier to read. Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-27 r/5772 fix(tvix/cli): handle SRI hashes in outputHashFlorian Klink1-7/+122
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`, `fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash` attribute, being an SRI hash. In that case, `builtin.derivation` is called with `outputHashAlgo` being an empty string, and `outputHash` being an SRI hash string. In other cases, an SRI hash is passed as outputHash, but outputHashAlgo is set too. Nix does modify these values in (single, fixed) output specification it serializes to ATerm, but keeps it unharmed in `env`. Move this into a construct_output_hash helper function, that can be tested better in isolation. Change-Id: Id9d716a119664c44ea7747540399966752e20187 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-25 r/5754 feat(tvix/eval): implement builtins.fromTOMLFlorian Klink1-0/+53
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML testcase from nix_tests. It uses the `toml` crate, and the serde integration it brings with it. Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-25 r/5753 feat(tvix/eval): use lexical-core to format floatFlorian Klink1-0/+72
Apparently our naive implementation of float formatting, which simply used {:.5}, and trimmed trailing "0" strings not sufficient. It wrongly trimmed numbers with zeroes but no decimal point, like `10000` got trimmed to `1`. Nix uses `std::to_string` on the double, which according to https://en.cppreference.com/w/cpp/string/basic_string/to_string is equivalent to `std::sprintf(buf, "%f", value)`. https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated like this: > Precision specifies the exact number of digits to appear after > the decimal point character. The default precision is 6. In the > alternative implementation decimal point character is written even if > no digits follow it. For infinity and not-a-number conversion style > see notes. This doesn't seem to be the case though, and Nix uses scientific notation in some cases. There's a whole bunch of strategies to determine which is a more compact notation, and which notation should be used for a given number. https://github.com/rust-lang/rust/issues/24556 provides some pointers into various rabbit holes for those interested. This gist seems to be that currently a different formatting is not exposed in rust directly, at least not for public consumption. There is the [lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate though, which provides a way to format floats with various strategies and formats. Change our implementation of `TotalDisplay` for the `Value::Float` case to use that. We still need to do some post-processing, because Nix always adds the sign in scientific notation (and there's no way to configure lexical-core to do that), and lexical-core in some cases keeps the trailing zeros. Even with all that in place, there as a difference in `eval-okay- fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated the fixture to a less problematic number. With this, the testsuite passes again, and does for the upcoming CL introducing builtins.fromTOML, and enabling the nix testsuite bits for it, too. Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5727 chore(tvix/store): add fastcdc crateFlorian Klink1-0/+7
This is used for content-defined chunking. Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5720 chore(tvix/store): add tonic-mockFlorian Klink1-0/+64
Upstream seems to be dead, so we're using https://github.com/tyrchen/ tonic-mock/pull/3 here. According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment- 1241164173, we might not need this crate at all, but for now, it gets the job done and is less code to write in the tests. Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-20 r/5709 feat(tvix/cli): add `errors` module with drv construction errorsVincent Ambo1-0/+1
These will be threaded through to eval through the new `TvixError` variant. Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-18 r/5701 feat(tvix/derivation): more checks for output hashes and encodingFlorian Klink1-0/+1
Change-Id: Idebad60c3bf9daf94d04a36bb73ac0dd767f9e79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7856 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-17 r/5670 refactor(tvix/value): use proptest strategies from imbl crateVincent Ambo1-1/+46
Instead of going through Vec/BTreeMap for generating our internal types, use the proptest strategies from imbl. The one thing I couldn't figure out in the previous implementation is where the ranges/sizes of generated collections came from. The strategies in proptest use different types (Range, with an unknown default value, and SizeRange with 0..100). I've opted to specify 0..100 directly, but we can probably make it configurable. Change-Id: I749bc4c703fe424099240cab822b1642e5216361 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-16 r/5668 refactor(tvix/derivation): expose proper ValidateDerivationErrorFlorian Klink1-0/+1
Use proper errors, instead of anyhow. Change-Id: I6db14c72a6319b389b0136aac7b84f50a30fb366 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7847 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-16 r/5664 feat(tvix/eval): implement builtins.toXMLVincent Ambo1-0/+7
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5649 feat(tvix/store): implement PathInfoService with sledFlorian Klink1-0/+123
This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-11 r/5643 feat(tvix/cli): implement initial refscan moduleVincent Ambo1-0/+1
This module implements a ReferenceScanner struct which uses the aho_corasick crate to scan string inputs for known, non-overlapping candidates (store paths, in our case). I experimented with several different APIs, and landed on this version with an initial accumulator in the scanner. The scanner is instantiated from the candidates and "fed" all the strings, then consumed by the caller to retrieve the result. Right now only things that look vaguely like bytestrings can be fed to the scanner, there is no streaming support in the API yet. Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-10 r/5640 feat(tvix/eval): implement serde::Deserialize for ValueRyan Lahfa1-0/+1
Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-06 r/5616 fix(tvix/derivation): fix build after StorePath renameVincent Ambo1-14/+14
This project was not previously covered by CI (fixed in this commit), so we didn't catch breakage due to a renamed module. This was noticed while rebasing a CL that has a dependency on this crate in its Nix build. Change-Id: Ic48570b9313e5f73e14daab50cf7ea70918c94d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7778 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-04 r/5591 feat(tvix/derivation): implement Derivation::validate()Florian Klink1-0/+1
Change-Id: I87dfadda872439e108e5f678a5da63dd5b1915d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7732 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-04 r/5580 feat(tvix/derivation): add nix drv path generation to DerivationJürgen Hahn1-0/+23
This adds a function to generate the derivation path. The computation is based on the Go implementation. Change-Id: Iae89db4976f5fd9208f0453f73688689a245cd66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7729 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-03 r/5569 chore(tvix/store): make importableFlorian Klink1-1/+1
This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-02 r/5566 feat(tvix/serde): initial Nix->serde::Deserialize implVincent Ambo1-0/+8
This will make it possible fairly easily use Nix to represent arbitrary data structures, e.g. for using Nix as a config language. Only pure Nix (i.e. no `import` etc.) is supported for now. Not all types, specifically no struct traversal, are implemented in this commit. Change-Id: I9ac91a229a0d12bf818e6e3249f3e5a691599a2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7712 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 r/5558 feat(tvix/store): add logging with tracingFlorian Klink1-0/+68
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5557 feat(tvix/store): implement reflectionFlorian Klink1-0/+16
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5555 feat(tvix/store): refactor digest conversionJürgen Hahn1-1/+1
This refactors how the original digest type (Vec<u8>) is converted to [u8; PATH_HASH_SIZE]. Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 r/5554 feat(tvix/store): initial dummy implementationFlorian Klink1-8/+12
This replaces the hello world example from tvix-store with an actual gRPC endpoint, implementing all of BlobService, DirectoryService and PathInfoService. All RPC methods currently respond with the unimplemented gRPC status. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5550 feat(tvix/store): add nixbase32 modFlorian Klink1-0/+30
This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 r/5541 refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrsVincent Ambo1-21/+16
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory between different iterations of a map. This slightly speeds up eval, but not significantly. Future work might include benchmarking whether using a `HashMap` and only ordering in cases where order is actually required would help. This switches to a fork of `im` that fixes some bugs with its OrdMap implementation. Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-12-29 r/5536 feat(tvix/derivation): serialize Nix DerivationJürgen Hahn1-0/+10
This adds a Derivation structure and allows to write it to a structure that implements std::fmt:Write. The implementation is based on the go-nix version. Change-Id: Ib54e1202b5c67f5d206b21bc109a751e971064cf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7659 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 r/5534 refactor(tvix/eval): use im::Vector for NixList representationVincent Ambo1-0/+43
This is a persistent, structurally sharing data structure which is more efficient in some of our use-cases. I have verified the efficiency improvement using `hyperfine` repeatedly over expressions on nixpkgs. Lists are not the most performance-critical structure in Nix (that would be attribute sets), but we can already see a small (~5-10%) improvement. Note that there are a handful of cases where we still go via `Vec` that need to be fixed, most notable for `builtins.sort` which can not currently be implemented directly using `im::Vector` because of a restrictive type bound. Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-28 r/5520 feat(tvix/store): implement Directory::validate()Florian Klink1-6/+8
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5510 chore(tvix/store): use lazy_static and dummy digestsFlorian Klink1-0/+1
While there's currently nothing in here checking the size of the digest, we should use something that passes the to-be-introduced validate() function. Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-26 r/5492 feat(tvix/store): use tonic-build for protosFlorian Klink1-0/+537
While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-22 r/5474 feat(tvix/cli): implement `NixCompatIO` helper typeVincent Ambo1-0/+1
This type allows for temporarily compatibility with the C++ Nix store, specifically (for now) it gives us the store directory used by Nix and imports files the same way. Change-Id: I4767794ef2863eba49661315c63c4e17de946d60 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5450 chore(tvix/nar): add CI build targetVincent Ambo1-0/+4
Change-Id: Ic73a391da9a733cade26114ab1127907c8d62a57 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7598 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5442 fix(tvix/eval): use test-generator fork that supports workspacesVincent Ambo1-2/+1
This should make no difference in Nix builds, but allows running tests locally again with `cargo test` for //tvix/eval. Change-Id: I97d61840143d5c14db61d5862781bf635f9a28e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7590 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 r/5441 chore(tvix): upgrade to clap 4.0Vincent Ambo1-87/+87
In //tvix/eval: * criterion bumped to 4.0, which at least depends on clap 3.x instead of 2.x, which is less incompatible In //tvix/cli: * no changes required In //tvix/nix_cli: * some minor changes for compatibility with clap 4.0, no functionality changes Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5440 refactor(tvix): split binary (REPL etc.) out from evaluator libraryVincent Ambo1-2/+10
The tvix-eval project is independent from any *uses* of the evaluator, such as the tvix-repl. This functionality has been split out into a separate "tvix-cli" crate. Note that this doesn't have to mean that this CLI crate is the "final" CLI crate for tvix, the point of this is not "getting the CLI structure right" but rather "getting the evaluator structure right". This reshuffling is part of restructuring the way that functionality like store communication is injected into language evaluation. Note that at this commit the new CLI crate is not at feature-parity. Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-15 r/5419 refactor(tvix): share a Cargo.lock file between Rust projectsVincent Ambo1-0/+1763
This relates to the (abandoned) cl/7256. Introduces a Cargo workspace at //tvix that is primarily intended to be used as a workaround for the annoying Nix+Rust tooling while having a consistent set of dependencies. This is driven in part by a desire to adopt crate2nix and get more granular Nix builds for Tvix's Rust projects, and in part by a need to split //tvix/eval into something providing the CLI (REPL etc.), and a library providing eval, without significantly altering the structure of build targets. To accomplish this the workspace has been designed to allow projects to remain independent build targets. I want to avoid lumping all the projects together - something like //tvix/eval should always be independent of other parts of tvix. A helper function in //tvix/default.nix lets downstream naersk projects construct a sparse root for the project which combines the workspace's `Cargo.lock` with the project's own `Cargo.toml`. Note that cargo commands in the workspace itself require the build dependencies of _all_ projects to be present, which is currently a bit annoying to accomplish. This introduces some breakage: 1. It breaks usage of rust-analyser without being in a shell with the dependencies of *all* Tvix projects, as it is not capable of respecting only the subset of dependencies for a part of the workspace. 2. It is no longer possible to run tests using `cargo test`, as the test generation crate we use does not work with workspaces: https://github.com/frehberg/test-generator/issues/6 This still works in the Nix build as we construct a Cargo project that looks like it's not in a workspace there. Until somebody fixes that crate / writes a new macro / does something else with the test suite, the way to run the tests is through the Nix build. Long-term we'll probably want to get rid of cargo completely, it's just a big wart and most tooling works without it if correctly configured, but we don't have time for that now. Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-26T17·42+0000