about summary refs log tree commit diff
path: root/tvix/store/src
AgeCommit message (Collapse)AuthorFilesLines
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink10-20/+18
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka13-173/+179
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 r/8483 docs(tvix/store/utils): add docstrings for ServiceUrls*Florian Klink1-2/+15
Describe what these structs are used for, and for each of it, explain which usecases it's used for. Change-Id: I8b7857bc68ec2b37df9f5163e06d028a64a12c79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich1-0/+5
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-04 r/8442 feat(tvix/store): Sled{PathInfo,Directory}Service -> Redb…Ilan Joselevich1-2/+2
Use redb instead of sled for the default filesystem implementation of PathInfoService and DirectoryService. In the future we'll also drop sled support completely. Change-Id: I513ff0c2ff953d59714aa50b9aa1301b02f53d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12085 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-29 r/8424 fix(tvix/store): Immediately return an error when using sled on /Ilan Joselevich1-0/+6
We already do this for redb and for sled in SledDirectoryService. Change-Id: I34c7178257a6a04e9f12ed4037a4ef585d7b0d54 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12060 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-29 r/8422 fix(tvix/store) RedbPathInfoService: improve logs and errorsIlan Joselevich1-6/+10
Add more logging and remove context from errors because that's already provided by the logs (Errors also need to be refactored anyway, there's also confusion about StorageError vs InvalidRequest, there's no consistency) Change-Id: Ia43c0d237d9075152490c635b05fb3fb343abcc8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12058 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8420 fix(tvix/store): adjust from_addr redb test to do what it saysIlan Joselevich1-2/+2
Change-Id: If15f161d5c7aba05ac1d8e2a4f6fac5a7053943a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12040 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8419 docs(tvix/store): Document redb in from_addrIlan Joselevich1-0/+5
Change-Id: Id2ef4ee1b22c20e5b79156f40821578979105ddc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12039 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich4-0/+229
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8403 feat(tvix/store): add grpc healthcheck service to daemonYureka1-0/+3
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8400 feat(tvix): add experimental-store-composition optionYureka2-36/+29
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 r/8399 refactor(tvix): move service addrs into shared clap structYureka2-106/+94
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-21 r/8396 fix(tvix/store): Fix narinfo compression selectionsinavir1-2/+2
Parsing of the narinfo file sets the compression field to None instead of Some("none"). The mapping selecting the decompression reader expected the former in //tvix/store/src/pathinfoservice/nix_http.rs. Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8394 refactor(tvix/nix-compat): rename PubKey to VerifyingKeyFlorian Klink1-3/+3
Align these with the way it's called in the ed25519 crates. Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-1/+2
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka13-248/+559
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-20 r/8374 fix(tvix/store): add missing nar_info::ca::Hash::TextSha256Florian Klink1-21/+26
Also reorder to how it's ordered in the .proto file. Change-Id: I87c422feac4d12f6a84f99aa889f9c524a9878b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11985 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-18 r/8368 refactor(tvix): use composition & registry for from_addrYureka2-14/+14
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-03 r/8341 test(tvix/store): test listing endpoint tooFlorian Klink1-1/+9
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-03 r/8340 fix(tvix/store/pathinfo/bigtable): fix listing endpointFlorian Klink1-1/+3
We were wrongly comparing the raw row_key with the store path digest, but we hexlower-encode the digest in the row key (via derive_pathinfo_key). Update the logic to fix that. Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-02 r/8339 feat(tvix/tracing): http trace propagationSimon Hauser1-2/+4
Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 r/8336 feat(tvix/store): add --remote-path-info-service-addr optionFlorian Klink1-0/+26
This is allows adding a cache in front of tvix-store daemon, and is less code duplication than cl/11902, means we can probably land that until we have proper store composition config. It can be used to provide a tvix-store daemon interface for a Nix HTTP Binary cache, saving all calculated PathInfo to another PathInfoService after ingestion. Change-Id: If141d718c2635f66aa90d46a80fd79c86c07d9ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11903 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 r/8335 feat(tvix/store): use tokio-listener for tvix-store daemon commandFlorian Klink1-16/+15
This allows binding on unix sockets, as well as systemd socket activation. Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8306 refactor(tvix/glue): take &CAHash, not CAHashFlorian Klink1-2/+2
We use a bit less cloning that way. Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink1-8/+2
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser5-19/+57
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8297 feat(tvix/tracing): optional progressbarSimon Hauser1-1/+1
Disable the progressbar on default and provide a interface for optionally enabling the progressbar. Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8295 feat(tvix/store/utils): detect gRPC NAR calculation serviceFlorian Klink1-5/+26
We were currently always using SimpleRenderer, which would mean the client would download every blob locally to calculate the checksum, which of course is very slow. Detect this special case and create a second instance (and client) for now. Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-17 r/8289 feat(tvix/tvix-store): improve progress barsFlorian Klink1-13/+1
Don't show an empty spinner for daemon commands. Move the bar to the right, so the text is better aligned between spinner progress and bar progress styles. Generally, push progress bars a bit more down to the place where we can track progress. This includes adding one in the upload_blob span. Introduce another progress style template for transfers, which interprets the counter as bytes (not just a plain integer), and also a data rate. Use it for here and in the fetching code, and also make the progress bar itself a bit less wide. Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 r/8288 feat(tvix/store): display progress for NAR calculationFlorian Klink2-2/+14
This is currently still taking a noticeable amount of time, so make sure we show it is happening. Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8287 docs(tvix/store/import): update import_path_as_nar_ca docstrFlorian Klink1-4/+5
Change-Id: I697172220efe73b2a215185027df09f815d32627 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8286 refactor(tvix/store): drop FUSE multithread workaround for DarwinFlorian Klink1-8/+1
The underlying issue in macFUSE has been fixed in https://github.com/osxfuse/osxfuse/issues/974. Bump our `macfuse` in nixpkgs to a version containing the fix. This can be removed while our nixpkgs pin is bumped past https://github.com/NixOS/nixpkgs/pull/320197. Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 r/8285 fix(tvix/store/bin): fix shutdown behaviour for FUSEFlorian Klink1-11/+17
Both umounts happening from another process, as well as tvix-store itself calling umount() on FuseDaemon will cause the FUSE worker threads to terminate. So far there was no nice way to wait on these threads to be terminated from multiple places, causing the `tvix-store mount` command to only be terminated if interrupted via ctrl-c, not via an external umount. Update FuseDaemon to use a ThreadPool, which gives us a join primitive over all threads, that can also be called from multiple places. Await on a join() from there to end the program, not the ctrl-c signal handler as it was before. Using FuseDaemon from multiple tasks requires Arc<>-ing both the ThreadPool as well as the inner FuseSession (which also needs to be inside a Mutex if we want to unmount), but now we can clone FuseDaemon around and use it in two places. We could probably also have used an Option and drop the FuseSession after the first umount, but this looks cleaner. Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-14 r/8271 feat(tvix/tracing): correctly close otlp on exitSimon Hauser1-19/+36
Provide a new interface for forcing a flush of otlp traces and use this interface to shutdown otlp prior to exiting tvix-store, either if the tool was stopped with a SIGTERM or ended regularly. This also fixes an issue where traces were not even exported if for example we just imported 10 paths and never even emitted more than 256 traces. The implementation uses a mpsc channel so a flush can be done without having to wait for it to complete. If you want to wait for a flush to complete you can provide a oneshot channel which will receive a message once flushing is complete. Because of a otlp bug `force_flush` as well as `shutdown_tracer_provider` need to be executed using `spawn_blocking` otherwise the function will deadlock. See https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335 Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-11 r/8248 refactor(tvix/store/nar/import): add ingest_nar_and_hashFlorian Klink3-23/+46
This wraps ingest_nar, but also keeps track of the number of bytes read, and calculates the sha256 digest of it. Make use of it in the NixHTTPPathInfoService, where this code is coming from. Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser2-100/+12
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-06 r/8220 feat(tvix/store/bin): add progress bar infrastructureFlorian Klink1-16/+64
This adds the tracing-indicatif crate, and configures it as a layer in our tracing_subscriber pipeline to emit progress for every span that's configured so. It also moves from using std::io::stderr to write logs to using their writer, to avoid clobbering output. Progress bar styles are defined in a lazy_static, moving this into a general tracing is left for later. This adds some usage of this to the `imports` and `copy` commands. The output can still be improved a bit - we should probably split each task up into a smaller (instrumented) helper functions, so we can create a progress bar for each task. Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-20 r/8159 feat(tvix/store): concurrently upload small blobs during nar ingestionConnor Brewster1-12/+22
Currently all blobs are uploaded serially when ingesting NARs. If a NAR contains many, small blobs, ingestion may become slow if there is a lot of round-trip latency to the blob service. This makes the NAR ingester use the ConcurrentBlobUploader which allows for buffering small blobs in memory so they can be uploaded concurrently to the blob service without blocking further deserialization. Change-Id: I093a73770232df12d9a11e5d901b99c08505c3cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11694 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-20 r/8157 refactor(tvix/store/pathinfo): test with PathInfoService directlyFlorian Klink4-71/+58
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar` anymore, so most of them don't actually have a handle to a {Blob,Directory}Service anymore. This means, we can simplify the construction of them for test cases a lot. Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-13 r/8144 feat(tvix/store/pathinfo): add a Cache combinatorFlorian Klink2-0/+113
This allows querying two PathInfoService implementations sequentially, and inserts into the "near" one if it's not there yet. There is no negative cache, and put / listing is not implemented (for now). Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-12 r/8117 feat(tvix/store/pathinfo/*): add more instrumentationFlorian Klink6-11/+20
Add instrumentation to the get() and put() implementations of all PathInfoService. Use the nixbase32 representation of the digest, not the base64 one. Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8112 refactor(tvix/store/pathinfo/memory): tokio RwLock, improve list()Florian Klink1-14/+14
We don't want to use the std::sync::RwLock here, as it blocks. This also means we don't need to deal with the error cases anymore. The list() implementation is updated to use try_stream, which means we can now avoid collecting everything into a Vec before returning from it. Change-Id: I9057dcc410dc553e6b1be3f20d5ee830569e8218 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11611 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8106 feat(tvix/store): add LruPathInfoServiceFlorian Klink2-0/+127
This provides an implementation of PathInfoService storing PathInfo in memory up to a certain capacity, then evicting these that have been used the least recently. Change-Id: I9d738687caf4f181a957f72245f26b92832313cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8105 refactor(tvix/store/pathinfo/sled): drop {blob,directory}_serviceFlorian Klink2-31/+10
These are not used anymore. Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8104 refactor(tvix/store/pathinfo/memory): drop {blob,directory}_serviceFlorian Klink3-26/+6
These are not used anymore. Change-Id: I6c16b4d80ddaabcb75fec3ea3e32b923b7719485 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11620 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8103 refactor(tvix/store): drop calculate_nar from PathInfoServiceFlorian Klink13-111/+155
This shouldn't be part of the PathInfoService trait. Pretty much none of the PathInfoServices do implement it, and requiring them to implement it means they also cannot make use of this calculation already being done by other PathInfoServices. Move it out into its own NarCalculationService trait, defined somewhere at tvix_store::nar, and have everyone who wants to trigger nar calculation use nar_calculation_service directly, which now is an additional field in TvixStoreIO for example. It being moved outside the PathInfoService trait doesn't prohibit specific implementations to implement it (like the GRPC client for the `PathInfoService` does. This is currently wired together in a bit of a hacky fashion - as of now, everything uses the naive implementation that traverses blob and directoryservice, rather than composing it properly. I want to leave that up to a later CL, dealing with other parts of store composition too. Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8102 fix(tvix/store/pathinfo/sled): use spawn_blockingFlorian Klink1-22/+45
This does IO, which might take a longer amount of time than what we want to be blocking the normal executor. Use spawn_blocking instead. I didn't add it for the constructors, as we only call these once. Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-11 r/8099 refactor(tvix/store/pathinfo/grpc): simplify pingpongFlorian Klink2-76/+7
We already have the same code in make_grpc_path_info_service_client. Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-08 r/8095 refactor(nix-compat/nar/reader): reuse prev_name allocationedef1-1/+1
We reuse the prev_name allocation for Entry, instead of allocating and returning a separate Vec. We encode the `prev_name: None` case as an empty vector, since we don't allow empty names anyway, and the sorting is equivalent. Change-Id: I975b37ff873805f5ff099bc82128706891052247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607 Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T10·25+0000