about summary refs log tree commit diff
path: root/tvix/Cargo.nix
AgeCommit message (Collapse)AuthorFilesLines
2024-11-13 r/8917 feat(nix-daemon): Implement more nix daemon operations.Vova Kryachko1-2/+8
In particular QueryPathFromHashPart, QueryValidPaths, QueryValidDerivers Change-Id: Ie6ad83cec5ce9580044b85e201e4e23394f87075 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12762 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de>
2024-11-12 r/8907 feat(nix-daemon): Implement client handler.Vova Kryachko1-4/+97
This change includes only the basic nix handshake protocol handling and sets up a client session. The only supported operation at this point is SetOptions. Additional operations will be implemented in subsequent cls. Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743 Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Reviewed-by: Brian Olsen <me@griff.name>
2024-11-11 r/8900 refactor(tvix/[ca]store): rename store composition feature flagsFlorian Klink1-3/+3
tvix-castore already supports composition without any additional feature flags, the only thing that can be explicitly enabled is referring to other stores via an anonymous url. Rename that feature flag to "xp-composition-url-refs". tvix-store effectively only controls the CLI surface, so rename this to "xp-composition-cli". The "store" in the feature name was dropped, as it's already apparent from being in the tvix-[ca]store crate. Change-Id: I1175dc6280cbba4cbcdfb7fd4b35fce713b45fc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12747 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev>
2024-11-04 r/8893 feat(tvix/nix-compat): Add nix serialization supportBrian Olsen1-0/+12
This change implements the serialization part that is needed to implement the nix daemon protocol. Previously was add deserialization and derivers for that and this then adds the other part of that equation so that you can write types that can then be read using deserialization. Change-Id: I2917de634980a93822a4f5a8ad38897b9ce16d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12729 Autosubmit: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-30 r/8878 feat(tvix/nix-daemon): Initial skeleton for nix-daemonVova Kryachko1-0/+76
The goal is to create a drop-in replacement nix-daemon that nix-cpp can use as a `daemon` store. Change-Id: Ie092047dcc6a24a3b8d8d1b808f3e6fd2c493bf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12711 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-23 r/8854 chore(tvix/cli): drop direct nix-compat dependencyFlorian Klink1-4/+0
This is only using nix-compat through tvix-glue. Change-Id: Ib6391987954a28cc05a2c27c1c21585185d82d1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12681 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-10-23 r/8853 chore(tvix/cli): drop direct castore dependencyFlorian Klink1-4/+0
This is only using tvix-castore through tvix-store. Change-Id: I342ee1e18f8c3946b712cfb446520b732091d58f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12680 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-18 r/8835 refactor(tvix/[ca]store): use auto_implFlorian Klink1-0/+35
This implements BS, DS, PS for Box'ed or Arc'ed variants of it with less code, and less potential to accidentially forget to proxy default trait methods for blanked impls, as fixed in cl/12658. Change-Id: If2cdbb563a73792038ebe7bff45d6f880214855b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12661 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2024-10-17 r/8825 refactor(tvix/store): remove use of lazy_staticFlorian Klink1-4/+0
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. Change-Id: Ifca792f4d2dbc36b703de4a4dfa406015ab86da7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12614 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-10-17 r/8824 refactor(tvix/castore): remove remaining lazy_static usageFlorian Klink1-4/+0
Change-Id: I86480cf625a457c4aa8153262f829d34c230b084 Co-authored-by: edef <edef@edef.eu> Reviewed-on: https://cl.tvl.fyi/c/depot/+/12613 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-10-13 r/8805 refactor(tvix/tracing): remove use of lazy_staticFlorian Klink1-4/+0
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. Change-Id: I14bee4068dc73c948321481b5a4e1fc922a89a27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12611 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2024-10-13 r/8804 refactor(tvix/build): remove use of lazy_staticFlorian Klink1-4/+0
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. Change-Id: Ie2af74beda9fcf8aa19fca7d844bcbe732f05bf8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12610 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-10-13 r/8803 refactor(tvix/glue): remove use of lazy_staticVincent Ambo1-4/+0
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. Change-Id: Ibb3be8458b8a8912ea04c9360d64c5cf914254d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12609 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-13 r/8802 refactor(tvix/nix-compat): remove use of lazy_staticVincent Ambo1-4/+0
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. I found at least one dead variable along the way, which I deleted. Change-Id: I8600c87c49078fb5ff72671994c77b919259e67b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12608 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 r/8797 refactor(tvix/eval): remove usage of lazy_staticVincent Ambo1-4/+0
Equivalent logic is now in the standard library, and this dependency is no longer needed for eval. Change-Id: Iaa4410d89fdaa5b84cbd9e6bc6ae479c659d92f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12602 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-10-12 r/8791 chore(tvix): bump crate dependenciesFlorian Klink1-956/+1063
tonic-build deprecated their `compile()` function, it's now called `compile_protos()`. Change-Id: I8cacd7f01a251c207401e4e226b0e880744e96e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12596 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de>
2024-10-11 r/8786 chore(tvix/build): Bump oci-spec dependencyIlan Joselevich1-135/+101
Our oci-spec was a bit oudated and there were some renamings in one of the release, which made building tvix-build fail if it's a dependency. I encountered this issue while working on tvix-eval-jobs. Change-Id: I6d982965176b83170a07445e351d3f5e5679ed2e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12586 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-10-10 r/8785 feat(tvix/composition): allow urls as anonymous storesYureka1-2/+2
This allows specifying an url in place of a named reference to another composition entry, if the castore crate has been compiled with the xp-store-composition feature. Example: `--directory-service-addr cache://?near=memory://&far=memory://` This would be equivalent to the instantiation via toml file: ```toml [memory1] type = "memory" [memory2] type = "memory" [default] type = "cache" near = "memory1" far = "memory2" ``` Note that each anonymous url causes a distinct instance to be created. Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-04 r/8756 feat(tvix/[ca]store): Add logging support to redbIlan Joselevich1-0/+8
We wanted to enable this earlier but the log level of many of the messages in redb were too high, they've now been downgraded, so we can enable logs from redb. Context on the fix and release: - https://github.com/cberner/redb/pull/828 - https://github.com/cberner/redb/releases/tag/v2.1.2 Change-Id: I8635e8a0bcb01a7d0b580387ac9134ccdd0205f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12568 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-03 r/8752 chore(tvix/[ca]store): Drop sled support completely in favor of redbIlan Joselevich1-228/+9
Over the past couple of months we've been using redb instead of sled as the default filesystem-based database in PS and DS. I am confident that we can get rid of sled completely now, and just keep redb. Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567 Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-01 r/8744 feat(tvix/build), add OciBuildService, the old wayFlorian Klink1-8/+373
This is just patchset 10 of CL10855, before the color_eyre changes, rebased to the tvix_castore api. Change-Id: If4b42412ff8568058908cda971ad7d6f2d9f9b7b --- This provides a build service invoking runc. It can be used by using the `oci://$path_to_some_tempdir` builder URL for now. For now, it can be tested as such: ``` BUILD_SERVICE_ADDR=oci://$PWD/bundles target/debug/tvix let pkgs = (import <nixpkgs> {}); in builtins.readDir pkgs.perl ``` readDir is to actually trigger IO into the store path (which triggers the builds). For now it fails due to missing reference scanning (see followup CLs). Change-Id: I09b40e410114ce69966a41a0e3c33281b859e443 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12526 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-09-30 r/8737 feat(tvix/nar-bridge): implement range request for NARsFlorian Klink1-1/+233
With an implementation of AsyncRead + AsyncSeek, axum-range can answer range requests. We only use it if a range has been requested, as it uses more memory than the linear variant. Change-Id: I0072b0a09b328f3e932f14567a2caa3a49abcbf7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12509 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: yuka <yuka@yuka.dev>
2024-09-28 r/8725 chore(tvix/castore/fuse): update fuse-backend-rsYureka1-6/+10
This release includes support for overlay filesystems Change-Id: I946cbf346df045209afaea2d720bb57fc2f2659f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12538 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-09-27 r/8724 refactor(tvix): move refscan module to castoreYureka1-8/+12
This is required to add the optional refscanner parameter to the ingest functions. Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-26 r/8716 feat(tvix/store): Add a signing PathInfoServicesinavir1-0/+8
- Add a new PathInfoService implementation that wraps transparently around another except that it dynamically signs all the incoming path-infos with the provided signer. - Add a ServiceBuilder for this PathInfoService that provides a SigningPathInfoService with a keyfile signer Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12032 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: sinavir <tvix@sinavir.fr> Tested-by: BuildkiteCI
2024-09-25 r/8713 feat(tvix/eval): Use thiserror for ErrorKind and CatchableErrorKindIlan Joselevich1-0/+4
thiserror is much more easier to maintain than manually implementing Error and Display. Change-Id: Ibf13e2d8a96fba69c8acb362b7515274a593dfd6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12452 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-30 r/8619 fix(tvix): Follow-up fixing no-default-features and commentsIlan Joselevich1-6/+0
In https://cl.tvl.fyi/c/depot/+/12389 I accidentally deleted some comments. Also fixed some useless no-default-features with futures crate, making it match the previous feature combination. Change-Id: I72bb2cfb88719ff2f8812d90193de2bd49149cce Reviewed-on: https://cl.tvl.fyi/c/depot/+/12395 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-29 r/8610 chore(tvix): Migrate members to inherit deps from workspaceIlan Joselevich1-11/+26
From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-29 r/8609 chore(tvix): Bump versions of all compatible deps via cargo upgradeIlan Joselevich1-314/+760
This updates all the dependencies and their "minimum" versions in Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s `cargo upgrade` command that will eventually be merged into `cargo update`. Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-28 r/8608 chore(tvix/[ca]store): bump bigtable_rsFlorian Klink1-1447/+186
This bumps bigtable_rs to https://github.com/liufuyang/bigtable_rs/pull/86, allowing us to drop our second set of prost/tonic/http/axum crates. Change-Id: I70f9150289c3e8611ebe8a7d99490e3dfd085a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12384 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-28 r/8607 chore(tvix/[ca]store): bump hyper-util to 0.1.7Florian Klink1-5/+5
Change-Id: Ia5eaf7f4614701ad4dd06114476a73e1a041d830 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12383 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-28 r/8606 chore(tvix): bump tonic[-build] from 0.12.1 to 0.12.2Florian Klink1-26/+27
Change-Id: If7d1ae4491d3c70f323cf872d1a70afe6238f78a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12382 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-28 r/8602 fix(tvix/nix-compat-derive): Get rid of external feature flagBrian Olsen1-5/+1
The external feature flag was there because I couldn't find a way to refer to crate and nix-compat with the same name so that the generated code could be the same. In essence `use nix_compat::nix_daemon::de::NixDeserialize` is an error when used inside nix_compat crate. So my best fix was the external feature flag until I found the solution used here which also removes the flag completely. Change-Id: Ia3e89c6c350c3fb22ca87f974a39c21542aae152 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12376 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Brian Olsen <me@griff.name>
2024-08-27 r/8593 fix(tvix/store): restore v1alpha reflection endpointFlorian Klink1-2/+2
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection protocol. However, most clients, like Postman, Kreya and evans don't support that one yet. Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support alongside the v1 version of it, registering both services. This fixes the example documented in tvix/store/README.md, it was previously failing as evans couldn't find the v1alpha reflection service. See https://github.com/hyperium/tonic/pull/1888 for details. Change-Id: I55438877317f82dc39face13afeb9594cda07a4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-25 r/8586 feat(tvix/nix-compat-derive): Add deriver for NixDeserializeBrian Olsen1-2/+170
This adds a nix-compat-derive derive crate that implements a deriver for NixDeserialize implementations. This is to reduce the amount of code needed to implement deserialization for all the types used by the Nix daemon protocol. Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022 Autosubmit: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-25 r/8585 feat(nix-compat): Add NixDeserialize and NixRead traitsBrian Olsen1-79/+82
Add a trait for deserializing a type from a daemon worker connection. This adds the NixDeserialize trait which is kind of like the serde Deserialize trait in that individual types are meant to implement it and it can potentially be derived in the future. The NixDeserialize trait takes something that implements NixRead as input so that you can among other things mock the reader. Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990 Autosubmit: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-21 r/8547 feat(nix-compat/nix_http): init parse_nar[info]_strFlorian Klink1-0/+4
This moves the URL component parsing code we had in nar-bridge to nix-compat. We change the function signature to return an Option, not a Result<_, StatusCode>. This allows returning more appropriate error codes, as we can ok_or(…) at the callsite, which we now do: on an upload to an invalid path, we now return "unauthorized", while on a GET/HEAD, we return "not found". This also adds support to parse compression suffixes. While not supported in nar-bridge, other users of nix-compat might very well want to parse these paths. Also fix the error message when parsing NAR urls, it mentioned 32, not 52, which is a copypasta error from the narinfo URL parsing code. Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-19 r/8540 feat(nix-compat/narinfo/signature): generalize name fieldFlorian Klink1-4/+8
Requiring `name` to be a `&str` means it'll get annoying to pass around `Signature`, but being able to pass them around in an owned fashion is kinda a requirement for a stronger typed `PathInfo` struct, where we want to have full ownership. Rework the `Signature` struct to become generic over the type of the `name` field. This means, it becomes possible to have owned versions of it. We don't want to impose `String` or `SmolStr` for example, but want to leave it up to the nix-compat user to decide. Provide a type alias for the existing `&str` variant (`SignatureRef`), and use it where we previously used the non-generic `Signature` one. Add some tests to ensure it's possible to *use* `Signature` with both `String` and `SmolStr` (but only pull in `smol_str` as dev dependency for the tests). Also, add some more docstrings, these were a bit sparse. Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 r/8521 refactor(tvix/eval): remove use of imbl::OrdMapVincent Ambo1-250/+2
Removes imbl::OrdMap in favour of an Rc over the standard library's BTreeMap, which allows us to drop the imbl dependency completely. In my local tests this is actually slightly faster for `hello` and `firefox`. Change-Id: Ic9597ead4e98bf9530f290c6a94a3c5c3efd0acc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12201 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-08-19 r/8519 refactor(tvix/eval): ensure VM operations fit in a single byteVincent Ambo1-0/+15
This replaces the OpCode enum with a new Op enum which is guaranteed to fit in a single byte. Instead of carrying enum variants with data, every variant that has runtime data encodes it into the `Vec<u8>` that a `Chunk` now carries. This has several advantages: * Less stack space is required at runtime, and fewer allocations are required while compiling. * The OpCode doesn't need to carry "weird" special-cased data variants anymore. * It is faster (albeit, not by much). On my laptop, results consistently look approximately like this: Benchmark 1: ./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings Time (mean ± σ): 8.224 s ± 0.272 s [User: 7.149 s, System: 0.688 s] Range (min … max): 7.759 s … 8.583 s 10 runs Benchmark 2: ./after -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings Time (mean ± σ): 8.000 s ± 0.198 s [User: 7.036 s, System: 0.633 s] Range (min … max): 7.718 s … 8.334 s 10 runs See notes below for why the performance impact might be less than expected. * It is faster while at the same time dropping some optimisations we previously performed. This has several disadvantages: * The code is closer to how one would write it in C or Go. * Bit shifting! * There is (for now) slightly more code than before. On performance I have the following thoughts at the moment: In order to prepare for adding GC, there's a couple of places in Tvix where I'd like to fence off certain kinds of complexity (such as mutating bytecode, which, for various reaons, also has to be part of data that is subject to GC). With this change, we can drop optimisations like retroactively modifying existing bytecode and *still* achieve better performance than before. I believe that this is currently worth it to pave the way for changes that are more significant for performance. In general this also opens other avenues of optimisation: For example, we can profile which argument sizes actually exist and remove the copy overhead of varint decoding (which does show up in profiles) by using more adequately sized types for, e.g., constant indices. Known regressions: * Op::Constant is no longer printing its values in disassembly (this can be fixed, I just didn't get around to it, will do separately). Change-Id: Id9b3a4254623a45de03069dbdb70b8349e976743 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12191 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-16 r/8502 feat(tvix/glue): Add refscanner pattern and AsyncReadBrian Olsen1-0/+4
This splits the existing ReferenceScanner into a ReferenceScanner and ReferencePattern as well as adds an AsyncRead implementation that can do a scan while you read from it. The reason to split the scanner in two is that generating the pattern is expensive and when ingesting build results with multiple outputs you want to do several independant scans that look for the same pattern. The reader is for scanning files without having to load the entire file into memory. Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052 Autosubmit: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-16 r/8500 chore(tvix): bump object_storeFlorian Klink1-7/+28
Fix for RUSTSEC-2024-0358. Change-Id: Id9e8c6ae67845d95a52c884035fcea02d2d71505 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12214 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8496 chore(tvix/nix-compat): bump curve25519-dalekFlorian Klink1-21/+236
Fix for RUSTSEC-2024-0344. Change-Id: I9e9cac5f0dac9e5b2e5a934c99762d982136c0e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12210 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-10 r/8474 feat(tvix/eval): Store hash in key of internerAspen Smith1-0/+17
Rather than storing the leaked allocation for the string as the key in the interner, store the hash (using NoHashHashBuilder). I thought this would improve performance, but it doesn't: hello outpath time: [736.85 ms 748.42 ms 760.42 ms] change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05) No change in performance detected. but it at least doesn't *hurt* performance, and it *does* avoid an `unsafe`, so it's probably net good. Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-08-09 r/8465 fix(third_party/overlays): Patch crate2nix to use mkDerivation for testsIlan Joselevich1-39/+28
The problem with using runCommand and recreating the src directory with lndir is that it changes the file types of individual files, they will now be a symlink instead of a regular file. If you have a crate that tests that a file is of regular type then it will fail inside the crate2nix derivation. Also regenerate Cargo.nix for //tvix as it will be needed in the next commit. Change-Id: I9275602cc17a428f9fdf0e55daf12cd673bbc030 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12131 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich1-81/+86
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-07 r/8453 feat(tvix/eval): Forbid Hash{Map,Set}, use Fx insteadAspen Smith1-4/+26
Per https://nnethercote.github.io/perf-book/hashing.html, we have basically no reason to use the default hasher over a faster, non-DoS-resistant hasher. This gives a nice perf boost basically for free: hello outpath time: [704.76 ms 714.91 ms 725.63 ms] change: [-7.2391% -6.1018% -4.9189%] (p = 0.00 < 0.05) Performance has improved. Change-Id: If5587f444ed3af69f8af4eead6af3ea303b4ae68 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12046 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: aspen <root@gws.fyi>
2024-08-05 r/8443 feat(tvix/eval): Leak strings (with flag to disable)Aspen Smith1-1/+1
Default to always leaking strings, and copying strings by copying pointers rather than cloning the underlying allocation. This (somewhat bafflingly) doesn't seem to affect any benchmarks, but paves the way for some tricks around string allocation that do. Unfortunately, we can't do this (yet?) for contextful strings, for reasons I don't currently understand but which I will address later, when I address contextful strings more holistically. I've left a flag in here to disable this, both to test the cloning logic for strings for when/if we decide to bring this back, and to allow people who care more about memory usage than perf to disable leaking. Change-Id: Iec44bcbfe9b3d20389d2450b9a551792a79b9b26 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12045 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich1-0/+39
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8403 feat(tvix/store): add grpc healthcheck service to daemonYureka1-0/+53
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>