about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
13 days r/8906 fix(tvix): use tracing::field::Empty for all indicatif.pb_showFlorian Klink1-1/+1
This prevents the field from getting printed to stderr, apparently. Change-Id: Ia9860e4ff37224003154db88ee5f83103060e626 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12756 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
14 days r/8902 refactor(tvix/store): move Cache to cache modFlorian Klink2-4/+2
This being in combinators makes it harder to find. Change-Id: If7984bdbd43f164c670548639bb4846d859f6695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12750 Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
14 days r/8901 docs(tvix/[ca]store): improve docstrings, remove wildcard importsFlorian Klink3-11/+27
Extend the docstrings of `add_default_services`, and add one for `addrs_to_configs` as well as the module-wide one at `tvix_store::composition`. Change-Id: Ie9b449988eb210cd65b19b174094bbe0c4af2fd6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12748 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev>
14 days r/8900 refactor(tvix/[ca]store): rename store composition feature flagsFlorian Klink3-8/+8
tvix-castore already supports composition without any additional feature flags, the only thing that can be explicitly enabled is referring to other stores via an anonymous url. Rename that feature flag to "xp-composition-url-refs". tvix-store effectively only controls the CLI surface, so rename this to "xp-composition-cli". The "store" in the feature name was dropped, as it's already apparent from being in the tvix-[ca]store crate. Change-Id: I1175dc6280cbba4cbcdfb7fd4b35fce713b45fc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12747 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev>
14 days r/8899 refactor(tvix/store/composition): rename 'default' to 'root'Florian Klink3-10/+10
This becomes the root of the composition. `default` implies we can directly access anything else, which we cannot. `root` makes this more understandable, and it's all internal only anyways. Change-Id: I297511bc05a7c32c59510b9d192b40d1bd937b5f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12746 Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-11-10 r/8898 refactor(nix-compat): Move serialization machinery into wire.Vova Kryachko1-1/+1
This groups most `wire` feature gated logic into a single module. The nix_daemon module will be gated by a feature that adds nix-compat-derive as a dependency. All of this is a way to break the crate2nix dependency cycle between nix-compat and nix-compat-derive(which depends on nix-compat for its doctests). Change-Id: I95938a6f280c11967371ff21f8b5a19e6d3d3805 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12761 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-11-08 r/8896 feat(tvix): add instance_name to instrumentation of *ServicesBob van der Linden10-57/+123
Currently it is not possible to distinguish between tracing of the same *Service type whenever there are multiple of them. Now the instance_name of ServiceBuilder is passed into the *Service and used in the existing instrument as the `instance_name` field. Places that did not already have a instance_name in its context use `"default"`. In tests I used `"test"`. Change-Id: Ia20bf2a7bb849a781e370d087ba7ddb3be79f654 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12739 Tested-by: BuildkiteCI Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com> Reviewed-by: flokli <flokli@flokli.de>
2024-10-30 r/8875 refactor(tvix/store): drop log level cli argFlorian Klink1-8/+1
This does the same as cl/12585 did for tvix-cli. Change-Id: I35215947f3616acd65654c2bbdb19d0c360b568b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12714 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-10-27 r/8860 chore(3p/sources): bump channels & overlays (2024-10-23)Vincent Ambo1-1/+1
* ops/modules: remove deprecated headscale DNS setting * users/aspen: use stable julia again (upstream ticket is resolved, and 16 has been removed because it's long deprecated) * users/wpcarro: utillinux -> util-linux * 3p/overlays: temporarily restore utillinux until upstream breakage is fixed * users/tazjin: noto-cjk -> noto-cjk-sans * tvix: regenerate Go protobufs Change-Id: I9dbec812306cd4e66030d14bbea3805e8224e0d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12679 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: aspen <root@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-10-19 r/8838 refactor(tvix/store/pathinfo/signing_wrapper): remove cloneFlorian Klink1-9/+17
Construct the owned signature in a separate scope, so all borrows to the original PathInfo are already dropped again, and we can modify the PathInfo without having to clone it. Change-Id: I03e7390540c2cfe7a2c61850bdbe8a33d213a5d9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12663 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-10-19 r/8837 refactor(tvix): make indicatif.pb_show=1 more explicitFlorian Klink2-39/+109
This pushes generating spans with pb_show up to the caller. They usually have more context on how to present things, if at all. Change-Id: Icfcaa64a8a57dce50c0261f2d06e7c051e3946c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12657 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-10-19 r/8836 refactor(tvix/castore): add try_into_anonymous_node, rename to try_*Florian Klink2-3/+3
We have two places where we parse protos and want their names to be empty: - Receiving a root node in a nar-bridge NAR request - Processing the CalculateNAR gRPC call We don't have any place where we want to keep a name as bytes::Bytes around, yet we used the `into_name_bytes_and_node` method. It was also a bit wrongly named - it wasn't very clear the name was not validated, and that the function may fail. This moves the "splitting off the name as bytes::Bytes" part into a private helper, only leaving the `try_into_name_and_node` and `try_into_anonymous_node` methods around. Change-Id: I2c7fd9871d49ec67450d7efa6a30d96197fb319c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12664 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-10-18 r/8835 refactor(tvix/[ca]store): use auto_implFlorian Klink6-36/+16
This implements BS, DS, PS for Box'ed or Arc'ed variants of it with less code, and less potential to accidentially forget to proxy default trait methods for blanked impls, as fixed in cl/12658. Change-Id: If2cdbb563a73792038ebe7bff45d6f880214855b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12661 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2024-10-18 r/8833 fix(tvix/store/pathinfo): use gRPC NarCalculationServiceFlorian Klink2-1/+14
If the PathInfoService is using gRPC, expose it via the nar_calculation_service() method in the PathInfoService. Also pass nar_calculation_service through in tha blanket impl. This now causes a `tvix-store import` using the default config to not fall back to `SimpleRenderer`, which will calculate the NAR hash and size by downloading the uploaded blobs (and blobs it didn't need to upload) locally, making such imports faster. Change-Id: If2c3fe6584e9093cba322d2360f355a3923904ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/12658 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
2024-10-18 r/8832 fix(tvix/store/grpc_wrapper): don't validate node nameFlorian Klink1-4/+7
Clients can (and do) send an empty name in here, and we discard it anyways. Change-Id: Iec135d9cba48cc066464e5639319c845079787e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12659 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-10-17 r/8825 refactor(tvix/store): remove use of lazy_staticFlorian Klink6-117/+119
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. Change-Id: Ifca792f4d2dbc36b703de4a4dfa406015ab86da7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12614 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-10-17 r/8822 refactor(nix-compat/store_path): drop build_nar_based_store_pathFlorian Klink1-3/+5
This is the only (remaining) occurence of it, and not really more code than just calling store_path::build_ca_path with `CAHash::Nar(NixHash::Sha256(…))`, especially considering we need the CAHash in the PathInfo struct later anyways - so let's remove this function. Change-Id: Ia82212086062c366e0280ca0823d9e68a3f91d3a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12632 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-15 r/8811 refactor(tvix/glue/builtins/import): refactorFlorian Klink2-56/+29
This removes all the intermediate helper functions and reorganizes the import code to only do the calculations where/when needed, and hopefully makes things easier to understand as well. Change-Id: I7e4c89c742bf8569b45e303523f7f801da7127ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/12627 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Jörg Thalheim <joerg@thalheim.io> Reviewed-by: edef <edef@edef.eu>
2024-10-12 r/8791 chore(tvix): bump crate dependenciesFlorian Klink1-1/+1
tonic-build deprecated their `compile()` function, it's now called `compile_protos()`. Change-Id: I8cacd7f01a251c207401e4e226b0e880744e96e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12596 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de>
2024-10-11 r/8790 test(tvix/store/signing_wrapper): restructureFlorian Klink1-14/+16
Move things around a bit to make it easier to understand what's going on: - We first validate our fixture invariants - We then insert into the PathInfoService - Do all comparisons and checks we can on the returned PathInfo struct - Only convert to the NarInfo variant to calculate the fingerprint, and don't keep intermediate let bindings for this Before cl/12588, this was arguably much harder to do that way, as we relied on some of the conversions done in the to_narinfo() function. Change-Id: Iaddbf1079f73ce566ef6d56f69a823e080b2e006 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12595 Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: sinavir <tvix@sinavir.fr>
2024-10-11 r/8787 refactor(tvix/store): use strictly typed PathInfo structMarijan Petričević19-874/+594
This switches the PathInfoService trait from using the proto-derived PathInfo struct to a more restrictive struct, and updates all implementations to use it. It removes a lot of the previous conversion and checks, as invalid states became nonrepresentable, and validations are expressed on the type level. PathInfoService implementations consuming protobuf need to convert and do the verification internally, and can only return the strongly typed variant. The nix_compat::narinfo::NarInfo conversions for the proto PathInfo are removed, we only keep a version showing a NarInfo representation for the strong struct. Converting back to a PathInfo requires the root node now, but is otherwise trivial, so left to the users. Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-10 r/8785 feat(tvix/composition): allow urls as anonymous storesYureka3-5/+8
This allows specifying an url in place of a named reference to another composition entry, if the castore crate has been compiled with the xp-store-composition feature. Example: `--directory-service-addr cache://?near=memory://&far=memory://` This would be equivalent to the instantiation via toml file: ```toml [memory1] type = "memory" [memory2] type = "memory" [default] type = "cache" near = "memory1" far = "memory2" ``` Note that each anonymous url causes a distinct instance to be created. Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-10 r/8783 feat(tvix/pathinfo/nixhttp): use ingest stores from urlYureka1-2/+18
This still defaults to the "default" services, but allows users to tell the nix+http pathinfoservice to ingest the castore nodes into a non-default blob-/directoryservice when used with the experimental store composition. Change-Id: I5c0f683ce95d888eadf3f302520a47f42f1a481d Reviewed-on: https://cl.tvl.fyi/c/depot/+/12148 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-04 r/8756 feat(tvix/[ca]store): Add logging support to redbIlan Joselevich1-1/+1
We wanted to enable this earlier but the log level of many of the messages in redb were too high, they've now been downgraded, so we can enable logs from redb. Context on the fix and release: - https://github.com/cberner/redb/pull/828 - https://github.com/cberner/redb/releases/tag/v2.1.2 Change-Id: I8635e8a0bcb01a7d0b580387ac9134ccdd0205f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12568 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-03 r/8752 chore(tvix/[ca]store): Drop sled support completely in favor of redbIlan Joselevich5-213/+0
Over the past couple of months we've been using redb instead of sled as the default filesystem-based database in PS and DS. I am confident that we can get rid of sled completely now, and just keep redb. Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567 Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-01 r/8741 feat(castore/fs): optional refscanner for ingestYureka2-4/+6
Change-Id: Ieca06de4c2e2680d89fe05a380079fafa5454837 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12529 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-09-26 r/8716 feat(tvix/store): Add a signing PathInfoServicesinavir4-0/+215
- Add a new PathInfoService implementation that wraps transparently around another except that it dynamically signs all the incoming path-infos with the provided signer. - Add a ServiceBuilder for this PathInfoService that provides a SigningPathInfoService with a keyfile signer Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12032 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: sinavir <tvix@sinavir.fr> Tested-by: BuildkiteCI
2024-09-25 r/8715 feat(tvix/store): seekable nar rendererYureka4-0/+535
Co-authored-by: edef <edef@edef.eu> Change-Id: I233206e8aae35504ca0519ac88178dfc5596bedb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12439 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-09-25 r/8714 refactor(tvix/store/tests): combine tests into one parametrized functionYureka2-179/+103
Change-Id: I9ff43b29be68b9840c58286da96fa52927691804 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12507 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-24 r/8711 fix(tvix/nar-bridge): Remove name check for root node in nar generationsinavir1-2/+2
Nar-bridge tried to parse the name of the protobuf node encoded in the URL into a PathComponent but this name was empty, leading to an error when the user tried to retrieve the nar file. This was an oversight from the conversion to stricter types (some of the CLs in the serious containing cl/12217). We need a version converting a protobuf without a name to our stricter types, but an empty PathComponent cannot be constructed. So we need a into_name_and_node() version that returns the name as Bytes, not PathComponent. Change-Id: I2996cdd2e0107133e502748947298f512f1cc521 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12504 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-29 r/8610 chore(tvix): Migrate members to inherit deps from workspaceIlan Joselevich1-56/+48
From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-29 r/8609 chore(tvix): Bump versions of all compatible deps via cargo upgradeIlan Joselevich1-30/+30
This updates all the dependencies and their "minimum" versions in Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s `cargo upgrade` command that will eventually be merged into `cargo update`. Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-28 r/8608 chore(tvix/[ca]store): bump bigtable_rsFlorian Klink1-1/+3
This bumps bigtable_rs to https://github.com/liufuyang/bigtable_rs/pull/86, allowing us to drop our second set of prost/tonic/http/axum crates. Change-Id: I70f9150289c3e8611ebe8a7d99490e3dfd085a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12384 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-28 r/8607 chore(tvix/[ca]store): bump hyper-util to 0.1.7Florian Klink1-1/+1
Change-Id: Ia5eaf7f4614701ad4dd06114476a73e1a041d830 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12383 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-27 r/8593 fix(tvix/store): restore v1alpha reflection endpointFlorian Klink2-6/+13
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection protocol. However, most clients, like Postman, Kreya and evans don't support that one yet. Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support alongside the v1 version of it, registering both services. This fixes the example documented in tvix/store/README.md, it was previously failing as evans couldn't find the v1alpha reflection service. See https://github.com/hyperium/tonic/pull/1888 for details. Change-Id: I55438877317f82dc39face13afeb9594cda07a4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-20 r/8545 refactor(nix-compat/store_path): make StorePath generic on SFlorian Klink4-11/+16
Similar to how cl/12253 already did this for `Signature`, we apply the same logic to `StorePath`. `StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less redundant code for the two different implementation. `.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a `StorePath<String>` (for now). I briefly thought about only publicly exporting `StorePath<String>` as `StorePath`, but the diff is not too large and this will make it easier to gradually introduce more flexibility in which store paths to accept. Also, remove some silliness in `StorePath::from_absolute_path_full`, which now doesn't allocate anymore. Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-19 r/8540 feat(nix-compat/narinfo/signature): generalize name fieldFlorian Klink1-1/+1
Requiring `name` to be a `&str` means it'll get annoying to pass around `Signature`, but being able to pass them around in an owned fashion is kinda a requirement for a stronger typed `PathInfo` struct, where we want to have full ownership. Rework the `Signature` struct to become generic over the type of the `name` field. This means, it becomes possible to have owned versions of it. We don't want to impose `String` or `SmolStr` for example, but want to leave it up to the nix-compat user to decide. Provide a type alias for the existing `&str` variant (`SignatureRef`), and use it where we previously used the non-generic `Signature` one. Add some tests to ensure it's possible to *use* `Signature` with both `String` and `SmolStr` (but only pull in `smol_str` as dev dependency for the tests). Also, add some more docstrings, these were a bit sparse. Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 r/8538 feat(tvix): drop usage of sparseTree in favor of lib.sourceByRegexIlan Joselevich2-14/+9
We can avoid depending on things outside //tvix by just using a similar util from nixpkgs. Change-Id: I9ea3e1f0a8a059ea10caaec173569ba9f316aec6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12247 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink4-9/+15
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8505 refactor(tvix/castore): drop {Directory,File,Symlink}NodeFlorian Klink4-62/+64
Add a `SymlinkTarget` type to represent validated symlink targets. With this, no invalid states are representable, so we can make `Node` be just an enum of all three kind of types, and allow access to these fields directly. Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink13-163/+138
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-15 r/8494 chore(tvix): Retain original passthru in Rust buildsBrian Olsen1-1/+1
When using the runTests feature of crate2nix the derivation that runs the tests is put into passthru.test but all default.nix files for Rust crates in Tvix threw that away. This commit retains passthru so that you can get access to the test derivation. Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink10-20/+18
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka13-173/+179
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 r/8483 docs(tvix/store/utils): add docstrings for ServiceUrls*Florian Klink1-2/+15
Describe what these structs are used for, and for each of it, explain which usecases it's used for. Change-Id: I8b7857bc68ec2b37df9f5163e06d028a64a12c79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich2-0/+6
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-04 r/8442 feat(tvix/store): Sled{PathInfo,Directory}Service -> Redb…Ilan Joselevich1-2/+2
Use redb instead of sled for the default filesystem implementation of PathInfoService and DirectoryService. In the future we'll also drop sled support completely. Change-Id: I513ff0c2ff953d59714aa50b9aa1301b02f53d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12085 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-29 r/8424 fix(tvix/store): Immediately return an error when using sled on /Ilan Joselevich1-0/+6
We already do this for redb and for sled in SledDirectoryService. Change-Id: I34c7178257a6a04e9f12ed4037a4ef585d7b0d54 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12060 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-29 r/8422 fix(tvix/store) RedbPathInfoService: improve logs and errorsIlan Joselevich1-6/+10
Add more logging and remove context from errors because that's already provided by the logs (Errors also need to be refactored anyway, there's also confusion about StorageError vs InvalidRequest, there's no consistency) Change-Id: Ia43c0d237d9075152490c635b05fb3fb343abcc8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12058 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8420 fix(tvix/store): adjust from_addr redb test to do what it saysIlan Joselevich1-2/+2
Change-Id: If15f161d5c7aba05ac1d8e2a4f6fac5a7053943a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12040 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-25T10·14+0000