about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2024-08-15 r/8494 chore(tvix): Retain original passthru in Rust buildsBrian Olsen1-1/+1
When using the runTests feature of crate2nix the derivation that runs the tests is put into passthru.test but all default.nix files for Rust crates in Tvix threw that away. This commit retains passthru so that you can get access to the test derivation. Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink10-20/+18
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka13-173/+179
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 r/8483 docs(tvix/store/utils): add docstrings for ServiceUrls*Florian Klink1-2/+15
Describe what these structs are used for, and for each of it, explain which usecases it's used for. Change-Id: I8b7857bc68ec2b37df9f5163e06d028a64a12c79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich2-0/+6
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-04 r/8442 feat(tvix/store): Sled{PathInfo,Directory}Service -> Redb…Ilan Joselevich1-2/+2
Use redb instead of sled for the default filesystem implementation of PathInfoService and DirectoryService. In the future we'll also drop sled support completely. Change-Id: I513ff0c2ff953d59714aa50b9aa1301b02f53d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12085 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-29 r/8424 fix(tvix/store): Immediately return an error when using sled on /Ilan Joselevich1-0/+6
We already do this for redb and for sled in SledDirectoryService. Change-Id: I34c7178257a6a04e9f12ed4037a4ef585d7b0d54 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12060 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-29 r/8422 fix(tvix/store) RedbPathInfoService: improve logs and errorsIlan Joselevich1-6/+10
Add more logging and remove context from errors because that's already provided by the logs (Errors also need to be refactored anyway, there's also confusion about StorageError vs InvalidRequest, there's no consistency) Change-Id: Ia43c0d237d9075152490c635b05fb3fb343abcc8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12058 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8420 fix(tvix/store): adjust from_addr redb test to do what it saysIlan Joselevich1-2/+2
Change-Id: If15f161d5c7aba05ac1d8e2a4f6fac5a7053943a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12040 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 r/8419 docs(tvix/store): Document redb in from_addrIlan Joselevich1-0/+5
Change-Id: Id2ef4ee1b22c20e5b79156f40821578979105ddc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12039 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich5-0/+230
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8404 test(tvix/store): add xp-store-composition to feature powersetYureka1-1/+1
Change-Id: Ibbd9a7585ec2f70c8f0f4d25ad858aa5ebc031dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12023 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 r/8403 feat(tvix/store): add grpc healthcheck service to daemonYureka2-0/+4
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8402 fix(store): add toml dependency for feature xp-store-compositionYureka1-1/+1
Fixes 'use of undeclared crate or module ' with --features xp-store-composition Change-Id: I44dce86e656094d180b91a00f385f685d21f3fbd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12021 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8400 feat(tvix): add experimental-store-composition optionYureka3-36/+31
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 r/8399 refactor(tvix): move service addrs into shared clap structYureka2-106/+94
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-21 r/8396 fix(tvix/store): Fix narinfo compression selectionsinavir1-2/+2
Parsing of the narinfo file sets the compression field to None instead of Some("none"). The mapping selecting the decompression reader expected the former in //tvix/store/src/pathinfoservice/nix_http.rs. Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8394 refactor(tvix/nix-compat): rename PubKey to VerifyingKeyFlorian Klink1-3/+3
Align these with the way it's called in the ed25519 crates. Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka2-10/+12
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka13-248/+559
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-20 r/8374 fix(tvix/store): add missing nar_info::ca::Hash::TextSha256Florian Klink1-21/+26
Also reorder to how it's ordered in the .proto file. Change-Id: I87c422feac4d12f6a84f99aa889f9c524a9878b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11985 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-18 r/8368 refactor(tvix): use composition & registry for from_addrYureka2-14/+14
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-09 r/8360 fix(tvix/{store,castore,build}): Compile tonic w/o config in build.rsIlan Joselevich1-7/+3
Previously we had to make a mutable Config instance and set bytes and other values in it because they were not exposed to the builder pattern (https://github.com/hyperium/tonic/issues/908) but now they are, so we just set them through the builder. Change-Id: I8904c6b93f09173b56586024b1ced59d622bce66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11966 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 r/8357 refactor(tvix): point SSL_CERT_FILE to /dev/nullFlorian Klink1-3/+3
reqwest wants to be able to read a file of trust roots when constructed, but as it doesn't actually do any HTTPS connections inside the nix build, an empty list of trust roots is totally sufficient. Thankfully /dev/null provides such a file. Change-Id: I9bd1619b2c9f8ff2a6640d2ac410d4de5b20c2ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/11961 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-07-03 r/8341 test(tvix/store): test listing endpoint tooFlorian Klink1-1/+9
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-03 r/8340 fix(tvix/store/pathinfo/bigtable): fix listing endpointFlorian Klink1-1/+3
We were wrongly comparing the raw row_key with the store path digest, but we hexlower-encode the digest in the row key (via derive_pathinfo_key). Update the logic to fix that. Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-02 r/8339 feat(tvix/tracing): http trace propagationSimon Hauser2-3/+6
Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 r/8336 feat(tvix/store): add --remote-path-info-service-addr optionFlorian Klink1-0/+26
This is allows adding a cache in front of tvix-store daemon, and is less code duplication than cl/11902, means we can probably land that until we have proper store composition config. It can be used to provide a tvix-store daemon interface for a Nix HTTP Binary cache, saving all calculated PathInfo to another PathInfoService after ingestion. Change-Id: If141d718c2635f66aa90d46a80fd79c86c07d9ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11903 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 r/8335 feat(tvix/store): use tokio-listener for tvix-store daemon commandFlorian Klink2-17/+16
This allows binding on unix sockets, as well as systemd socket activation. Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 r/8334 chore(tvix): bump bigtable_rs to latest releaseFlorian Klink1-3/+1
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a while, no need to use our own checkout here. Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-07-01 r/8331 chore(tvix): bump to data-encoding 2.6.0Florian Klink1-1/+1
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 r/8306 refactor(tvix/glue): take &CAHash, not CAHashFlorian Klink1-2/+2
We use a bit less cloning that way. Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink1-8/+2
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser6-20/+59
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8297 feat(tvix/tracing): optional progressbarSimon Hauser1-1/+1
Disable the progressbar on default and provide a interface for optionally enabling the progressbar. Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8295 feat(tvix/store/utils): detect gRPC NAR calculation serviceFlorian Klink1-5/+26
We were currently always using SimpleRenderer, which would mean the client would download every blob locally to calculate the checksum, which of course is very slow. Detect this special case and create a second instance (and client) for now. Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-17 r/8289 feat(tvix/tvix-store): improve progress barsFlorian Klink1-13/+1
Don't show an empty spinner for daemon commands. Move the bar to the right, so the text is better aligned between spinner progress and bar progress styles. Generally, push progress bars a bit more down to the place where we can track progress. This includes adding one in the upload_blob span. Introduce another progress style template for transfers, which interprets the counter as bytes (not just a plain integer), and also a data rate. Use it for here and in the fetching code, and also make the progress bar itself a bit less wide. Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 r/8288 feat(tvix/store): display progress for NAR calculationFlorian Klink2-2/+14
This is currently still taking a noticeable amount of time, so make sure we show it is happening. Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8287 docs(tvix/store/import): update import_path_as_nar_ca docstrFlorian Klink1-4/+5
Change-Id: I697172220efe73b2a215185027df09f815d32627 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8286 refactor(tvix/store): drop FUSE multithread workaround for DarwinFlorian Klink1-8/+1
The underlying issue in macFUSE has been fixed in https://github.com/osxfuse/osxfuse/issues/974. Bump our `macfuse` in nixpkgs to a version containing the fix. This can be removed while our nixpkgs pin is bumped past https://github.com/NixOS/nixpkgs/pull/320197. Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 r/8285 fix(tvix/store/bin): fix shutdown behaviour for FUSEFlorian Klink1-11/+17
Both umounts happening from another process, as well as tvix-store itself calling umount() on FuseDaemon will cause the FUSE worker threads to terminate. So far there was no nice way to wait on these threads to be terminated from multiple places, causing the `tvix-store mount` command to only be terminated if interrupted via ctrl-c, not via an external umount. Update FuseDaemon to use a ThreadPool, which gives us a join primitive over all threads, that can also be called from multiple places. Await on a join() from there to end the program, not the ctrl-c signal handler as it was before. Using FuseDaemon from multiple tasks requires Arc<>-ing both the ThreadPool as well as the inner FuseSession (which also needs to be inside a Mutex if we want to unmount), but now we can clone FuseDaemon around and use it in two places. We could probably also have used an Option and drop the FuseSession after the first umount, but this looks cleaner. Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-15 r/8280 feat(tvix/cli,store): add tracy featureFlorian Klink1-0/+1
If compiled with this features, this emits packets compatible with the [Tracy](https://github.com/wolfpld/tracy) format. Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-14 r/8271 feat(tvix/tracing): correctly close otlp on exitSimon Hauser1-19/+36
Provide a new interface for forcing a flush of otlp traces and use this interface to shutdown otlp prior to exiting tvix-store, either if the tool was stopped with a SIGTERM or ended regularly. This also fixes an issue where traces were not even exported if for example we just imported 10 paths and never even emitted more than 256 traces. The implementation uses a mpsc channel so a flush can be done without having to wait for it to complete. If you want to wait for a flush to complete you can provide a oneshot channel which will receive a message once flushing is complete. Because of a otlp bug `force_flush` as well as `shutdown_tracer_provider` need to be executed using `spawn_blocking` otherwise the function will deadlock. See https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335 Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 r/8269 chore(tvix/docs): move [ca]store docs to tvix/docsFlorian Klink2-289/+1
Change-Id: Idd78ffae34b6ea7b93d13de73b98c61a348869fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11808 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2024-06-11 r/8248 refactor(tvix/store/nar/import): add ingest_nar_and_hashFlorian Klink3-23/+46
This wraps ingest_nar, but also keeps track of the number of bytes read, and calculates the sha256 digest of it. Make use of it in the NixHTTPPathInfoService, where this code is coming from. Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser3-108/+15
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-08 r/8227 docs(tvix): fix some typos across various documentsBen Webb1-1/+1
Fix some typos found while reading various documents, mostly those relating to the castore. Here is a summary of the edits. - fix broken link between documents in the store and castore directories - clarify expression in castore's data model document that indicates that the *name* of each child node of a directory must be unique across all three lists of children - add missing closing parenthesis in castore's data model document - replace "how" with "what" in the phrase "unclear how a ... would even look like" in castore's why-not-git-trees document - remove unnecessary articles in castore's blobstore chunking document - add missing "y" to "optionall" in eval's compilation of bindings document Change-Id: I1997ea91bb4e9c40abcd81e0cde9405968580ba6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11763 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-06 r/8220 feat(tvix/store/bin): add progress bar infrastructureFlorian Klink2-16/+66
This adds the tracing-indicatif crate, and configures it as a layer in our tracing_subscriber pipeline to emit progress for every span that's configured so. It also moves from using std::io::stderr to write logs to using their writer, to avoid clobbering output. Progress bar styles are defined in a lazy_static, moving this into a general tracing is left for later. This adds some usage of this to the `imports` and `copy` commands. The output can still be improved a bit - we should probably split each task up into a smaller (instrumented) helper functions, so we can create a progress bar for each task. Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-06-03 r/8211 feat(tvix): add CI targets for checking crate features powersetIlan Joselevich1-20/+24
Closes: https://b.tvl.fyi/issues/401 With this change all crate features (and their combinations) will be built and tested in CI. From now on, when adding/removing a Cargo feature for a crate, you will want to add it to the features power set that gets tested in CI. For each crate there's a default.nix with a `mkFeaturePowerset` invocation, modify the list to include/remove the feature. Note that you don't want to add "collection" features, such as `fs` for tvix-[ca]store or `default`. Change-Id: I966dde1413d057770787da3296cce9c1924570e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-20 r/8159 feat(tvix/store): concurrently upload small blobs during nar ingestionConnor Brewster1-12/+22
Currently all blobs are uploaded serially when ingesting NARs. If a NAR contains many, small blobs, ingestion may become slow if there is a lot of round-trip latency to the blob service. This makes the NAR ingester use the ConcurrentBlobUploader which allows for buffering small blobs in memory so they can be uploaded concurrently to the blob service without blocking further deserialization. Change-Id: I093a73770232df12d9a11e5d901b99c08505c3cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11694 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-30T19·07+0000