about summary refs log tree commit diff
path: root/tvix/store/src/pathinfoservice
AgeCommit message (Collapse)AuthorFilesLines
2024-08-20 r/8545 refactor(nix-compat/store_path): make StorePath generic on SFlorian Klink1-1/+2
Similar to how cl/12253 already did this for `Signature`, we apply the same logic to `StorePath`. `StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less redundant code for the two different implementation. `.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a `StorePath<String>` (for now). I briefly thought about only publicly exporting `StorePath<String>` as `StorePath`, but the diff is not too large and this will make it easier to gradually introduce more flexibility in which store paths to accept. Also, remove some silliness in `StorePath::from_absolute_path_full`, which now doesn't allocate anymore. Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink1-4/+4
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink4-20/+33
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink3-4/+3
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka4-19/+23
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-29 r/8424 fix(tvix/store): Immediately return an error when using sled on /Ilan Joselevich1-0/+6
We already do this for redb and for sled in SledDirectoryService. Change-Id: I34c7178257a6a04e9f12ed4037a4ef585d7b0d54 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12060 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-29 r/8422 fix(tvix/store) RedbPathInfoService: improve logs and errorsIlan Joselevich1-6/+10
Add more logging and remove context from errors because that's already provided by the logs (Errors also need to be refactored anyway, there's also confusion about StorageError vs InvalidRequest, there's no consistency) Change-Id: Ia43c0d237d9075152490c635b05fb3fb343abcc8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12058 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8420 fix(tvix/store): adjust from_addr redb test to do what it saysIlan Joselevich1-2/+2
Change-Id: If15f161d5c7aba05ac1d8e2a4f6fac5a7053943a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12040 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8419 docs(tvix/store): Document redb in from_addrIlan Joselevich1-0/+5
Change-Id: Id2ef4ee1b22c20e5b79156f40821578979105ddc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12039 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich4-0/+229
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8396 fix(tvix/store): Fix narinfo compression selectionsinavir1-2/+2
Parsing of the narinfo file sets the compression field to None instead of Some("none"). The mapping selecting the decompression reader expected the former in //tvix/store/src/pathinfoservice/nix_http.rs. Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8394 refactor(tvix/nix-compat): rename PubKey to VerifyingKeyFlorian Klink1-3/+3
Align these with the way it's called in the ed25519 crates. Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-1/+2
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka9-180/+421
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-03 r/8341 test(tvix/store): test listing endpoint tooFlorian Klink1-1/+9
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-03 r/8340 fix(tvix/store/pathinfo/bigtable): fix listing endpointFlorian Klink1-1/+3
We were wrongly comparing the raw row_key with the store path digest, but we hexlower-encode the digest in the row key (via derive_pathinfo_key). Update the logic to fix that. Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-02 r/8339 feat(tvix/tracing): http trace propagationSimon Hauser1-2/+4
Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink1-8/+2
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser3-12/+36
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8288 feat(tvix/store): display progress for NAR calculationFlorian Klink1-2/+7
This is currently still taking a noticeable amount of time, so make sure we show it is happening. Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-11 r/8248 refactor(tvix/store/nar/import): add ingest_nar_and_hashFlorian Klink1-22/+5
This wraps ingest_nar, but also keeps track of the number of bytes read, and calculates the sha256 digest of it. Make use of it in the NixHTTPPathInfoService, where this code is coming from. Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser1-2/+1
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-05-20 r/8157 refactor(tvix/store/pathinfo): test with PathInfoService directlyFlorian Klink4-71/+58
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar` anymore, so most of them don't actually have a handle to a {Blob,Directory}Service anymore. This means, we can simplify the construction of them for test cases a lot. Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-13 r/8144 feat(tvix/store/pathinfo): add a Cache combinatorFlorian Klink2-0/+113
This allows querying two PathInfoService implementations sequentially, and inserts into the "near" one if it's not there yet. There is no negative cache, and put / listing is not implemented (for now). Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-12 r/8117 feat(tvix/store/pathinfo/*): add more instrumentationFlorian Klink6-11/+20
Add instrumentation to the get() and put() implementations of all PathInfoService. Use the nixbase32 representation of the digest, not the base64 one. Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8112 refactor(tvix/store/pathinfo/memory): tokio RwLock, improve list()Florian Klink1-14/+14
We don't want to use the std::sync::RwLock here, as it blocks. This also means we don't need to deal with the error cases anymore. The list() implementation is updated to use try_stream, which means we can now avoid collecting everything into a Vec before returning from it. Change-Id: I9057dcc410dc553e6b1be3f20d5ee830569e8218 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11611 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8106 feat(tvix/store): add LruPathInfoServiceFlorian Klink2-0/+127
This provides an implementation of PathInfoService storing PathInfo in memory up to a certain capacity, then evicting these that have been used the least recently. Change-Id: I9d738687caf4f181a957f72245f26b92832313cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8105 refactor(tvix/store/pathinfo/sled): drop {blob,directory}_serviceFlorian Klink2-31/+10
These are not used anymore. Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8104 refactor(tvix/store/pathinfo/memory): drop {blob,directory}_serviceFlorian Klink3-26/+6
These are not used anymore. Change-Id: I6c16b4d80ddaabcb75fec3ea3e32b923b7719485 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11620 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8103 refactor(tvix/store): drop calculate_nar from PathInfoServiceFlorian Klink7-88/+46
This shouldn't be part of the PathInfoService trait. Pretty much none of the PathInfoServices do implement it, and requiring them to implement it means they also cannot make use of this calculation already being done by other PathInfoServices. Move it out into its own NarCalculationService trait, defined somewhere at tvix_store::nar, and have everyone who wants to trigger nar calculation use nar_calculation_service directly, which now is an additional field in TvixStoreIO for example. It being moved outside the PathInfoService trait doesn't prohibit specific implementations to implement it (like the GRPC client for the `PathInfoService` does. This is currently wired together in a bit of a hacky fashion - as of now, everything uses the naive implementation that traverses blob and directoryservice, rather than composing it properly. I want to leave that up to a later CL, dealing with other parts of store composition too. Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8102 fix(tvix/store/pathinfo/sled): use spawn_blockingFlorian Klink1-22/+45
This does IO, which might take a longer amount of time than what we want to be blocking the normal executor. Use spawn_blocking instead. I didn't add it for the constructors, as we only call these once. Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-11 r/8099 refactor(tvix/store/pathinfo/grpc): simplify pingpongFlorian Klink2-76/+7
We already have the same code in make_grpc_path_info_service_client. Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-06 r/8081 fix(tvix/store/pathinfo/nix_http): update error messageFlorian Klink1-1/+1
Autosubmit was too fast and submitted this before my push went through. Flagged in https://cl.tvl.fyi/c/depot/+/11580/comment/40a56824_7be73334/ Change-Id: I1f835ae60c2c6fd2db6654c1b1c71d90bee141af Reviewed-on: https://cl.tvl.fyi/c/depot/+/11595 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-06 r/8080 feat(tvix/store): use async nar reader for ingest_narFlorian Klink1-108/+72
Rename read_nar to ingest_nar, and have it use the async nar reader version, and the ingest_entries machinery. This means we can now drop all code dealing with manually assembling castore nodes. Update our consumer, NixHTTPPathInfoService to use the new API. As we now accept an AsyncRead, we don't need to do any blocking here anymore, and can use the same async-compression crate as in the fetching logic (and support some more compression formats out of the box). Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-03 r/8071 refactor(tvix/store/pathinfo/sled): cleanup, add instrumentationFlorian Klink1-57/+42
Write this a bit more compact, by using map_err(|e| …) and ?. Ideally we'd get rid of the error mapping entirely, by using proper error types, but that's left for a followup. Change-Id: I68dc72b162ac89c5ff82d8c2bc26e1c808a0affd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11584 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-03 r/8070 test(tvix/[ca]store): enable bigtable only with "integration" featureFlorian Klink2-3/+3
The emulator and bigtable client are quite big. Remove them from the default //tvix:shell. Put the tests behind a `integration` feature flag, and add a variant with that enabled to CI, and drop the bigtable tools from //tvix:shell. Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-25 r/8008 fix(tvix): typoedef1-1/+1
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7965 chore(tvix/store): migrate import.rs and tests/pathinfo.rs to rstestFlorian Klink2-4/+4
Also, rename the DUMMY_NAME constant in the fixtures to DUMMY_PATH, which aligns more with the ToString representation and from_bytes conversions we have on StorePath[Ref]. Change-Id: I39763c9dfa84c5d86f2fd0171b3a4d36fd72f267 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11464 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-19 r/7964 chore(tvix/store): migrate from test_case to rstestFlorian Klink1-43/+37
Change-Id: Id673cd9c574b00a739de78aab2ee86f9d9b1ed40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11463 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-15 r/7912 feat(tvix/castore/fs): support extended attributesFlorian Klink1-0/+2
This exposes `user.tvix.castore.{blob,directory}.digest` xattr keys for files and directories: ``` ❯ getfattr -d /tmp/tvix/06jrrv6wwp0nc1m7fr5bgdw012rfzfx2-nano-7.2-info getfattr: Removing leading '/' from absolute path names user.tvix.castore.directory.digest="b3:SuYDcUM9RpWcnA40tYB1BtYpR0xw72v3ymhKDQbBfe4=" ❯ getfattr -d /tmp/tvix/156a89x10c3kaby9rgf3fi4k0p6r9wl1-etc-shells getfattr: Removing leading '/' from absolute path names user.tvix.castore.blob.digest="b3:pZkwZoHN+/VQ8wkaX0wYVXZ0tV/HhtKlSqiaWDK7uRs=" ``` It's currently mostly used for debugging, though it might be useful for tvix-castore-aware syncing programs using the filesystem too. Change-Id: I26ac3cb9fe51ffbf7f880519f26741549cb5ab6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11422 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 r/7906 feat(tvix/*store/bigtable): limit retries connecting to cbtemulatorFlorian Klink1-1/+3
This kept retrying indefinitely if the socket didn't appear. Change-Id: I4d4ef61df73cef6abda698501432f370abc8a82c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11406 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-12 r/7887 feat(tvix/store/pathinfo/grpc): instrument functionsFlorian Klink1-0/+6
Change-Id: Idb7d7144be1917fbaf83e9fd76c5b2ebb3df98d2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11400 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-09 r/7884 feat(tvix/store): add bigtable pathinfoservice backendFlorian Klink4-1/+451
Put behind the "cloud" backend, like in the `tvix-castore` crate. Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7793 refactor(tvix/store/pathinfo/from_addr): stop using gen_*_serviceFlorian Klink1-7/+10
Remove usage of the gen_{blob,directory}_service() helper functions from utils. We populate Memory{Blob,Directory}Services here directly, as test_case and rstest doesn't compose well. Change-Id: I0fb48aadb8c818f508b18ceb83c85eb91359442a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11278 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7792 refactor(tvix/pathinfo/grpc): use rstest fixturesFlorian Klink1-6/+12
Change-Id: Ib114a4d141ca829520aed34600678d613994f875 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11277 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7788 feat(tvix/store): add rstest-based PathInfoService testsFlorian Klink3-0/+175
This introduces rstest-based tests. We also add fixtures for creating some BlobService / DirectoryService out of thin air. To test a PathInfoService, we don't really care too much about its internal storage - ensuring they work is up to the castore tests. Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-28 r/7787 refactor(tvix/store/tests/fixtures): use [u8; 20]Florian Klink1-1/+1
This makes the fixture more use-able when interacting with the trait, the Bytes are only useful for the gRPC version. Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-27 r/7786 feat(tvix/store): AsRef<dyn PathInfoService> impl PathInfoServiceFlorian Klink1-0/+25
Change-Id: I7fc06ae97a50d04b8c36292b3457c112242a7a70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11270 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-20 r/7747 refactor(tvix/store/pathinfo/from_addr): use match guardsFlorian Klink1-59/+62
This will allow feature-flagging some of the backends. Change-Id: Ie92914c3e2ad870eee87e73b3b5abe605fb56fe7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11202 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-18 r/7732 refactor(tvix/store/pathinfo/nix_http): NixHash to display NARHashFlorian Klink1-3/+5
This will give us the sha256: prefix, and hashes we're more used to, in that context. Change-Id: I72e42fe685e365ba9baa7cd81001387d239fa7c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11194 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI