about summary refs log tree commit diff
path: root/tvix/store/Cargo.toml
AgeCommit message (Collapse)AuthorFilesLines
2024-08-29 r/8610 chore(tvix): Migrate members to inherit deps from workspaceIlan Joselevich1-56/+48
From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-29 r/8609 chore(tvix): Bump versions of all compatible deps via cargo upgradeIlan Joselevich1-30/+30
This updates all the dependencies and their "minimum" versions in Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s `cargo upgrade` command that will eventually be merged into `cargo update`. Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-28 r/8608 chore(tvix/[ca]store): bump bigtable_rsFlorian Klink1-1/+3
This bumps bigtable_rs to https://github.com/liufuyang/bigtable_rs/pull/86, allowing us to drop our second set of prost/tonic/http/axum crates. Change-Id: I70f9150289c3e8611ebe8a7d99490e3dfd085a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12384 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-28 r/8607 chore(tvix/[ca]store): bump hyper-util to 0.1.7Florian Klink1-1/+1
Change-Id: Ia5eaf7f4614701ad4dd06114476a73e1a041d830 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12383 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-27 r/8593 fix(tvix/store): restore v1alpha reflection endpointFlorian Klink1-1/+1
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection protocol. However, most clients, like Postman, Kreya and evans don't support that one yet. Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support alongside the v1 version of it, registering both services. This fixes the example documented in tvix/store/README.md, it was previously failing as evans couldn't find the v1alpha reflection service. See https://github.com/hyperium/tonic/pull/1888 for details. Change-Id: I55438877317f82dc39face13afeb9594cda07a4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich1-0/+1
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich1-0/+1
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8403 feat(tvix/store): add grpc healthcheck service to daemonYureka1-0/+1
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8402 fix(store): add toml dependency for feature xp-store-compositionYureka1-1/+1
Fixes 'use of undeclared crate or module ' with --features xp-store-composition Change-Id: I44dce86e656094d180b91a00f385f685d21f3fbd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12021 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8400 feat(tvix): add experimental-store-composition optionYureka1-0/+2
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-9/+10
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-02 r/8339 feat(tvix/tracing): http trace propagationSimon Hauser1-1/+2
Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 r/8335 feat(tvix/store): use tokio-listener for tvix-store daemon commandFlorian Klink1-1/+1
This allows binding on unix sockets, as well as systemd socket activation. Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 r/8334 chore(tvix): bump bigtable_rs to latest releaseFlorian Klink1-3/+1
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a while, no need to use our own checkout here. Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-07-01 r/8331 chore(tvix): bump to data-encoding 2.6.0Florian Klink1-1/+1
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser1-1/+2
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-15 r/8280 feat(tvix/cli,store): add tracy featureFlorian Klink1-0/+1
If compiled with this features, this emits packets compatible with the [Tracy](https://github.com/wolfpld/tracy) format. Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser1-8/+3
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-06 r/8220 feat(tvix/store/bin): add progress bar infrastructureFlorian Klink1-0/+2
This adds the tracing-indicatif crate, and configures it as a layer in our tracing_subscriber pipeline to emit progress for every span that's configured so. It also moves from using std::io::stderr to write logs to using their writer, to avoid clobbering output. Progress bar styles are defined in a lazy_static, moving this into a general tracing is left for later. This adds some usage of this to the `imports` and `copy` commands. The output can still be improved a bit - we should probably split each task up into a smaller (instrumented) helper functions, so we can create a progress bar for each task. Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-16 r/8152 chore(tvix): drop json feature for tracing-subscriberFlorian Klink1-1/+1
Drop the (unused) feature flags here, and get rid of some crate dependencies. Change-Id: Id64852b498725467d56abb12b548301cfba6a760 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11678 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-15 r/8149 chore(tvix): bump to tokio-listener 0.4.1Florian Klink1-1/+1
This gives multi-listener support, as well as listening on named socket FDs. Change-Id: I92b441f854e2faf80074463d0ca6bdc23cbd890a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11663 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de>
2024-05-15 r/8148 chore(tvix/store): bump otlp stackFlorian Klink1-5/+5
This aligns the tonic version we use directly in Tvix to the one pulled in by the OTLP stack. Change-Id: I658528c8dabb7cd6948f1207ddcdef1984e82037 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11666 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-05-13 r/8139 chore(tvix/clippy): configure clippy allow block_in_conditionsFlorian Klink1-0/+3
… in Cargo.toml. This gets an imperative `cargo clippy` run to pick up that config, so `-A clippy::blocks_in_conditions` doesn't need to be explicitly specified anymore. Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-05-11 r/8113 refactor(tvix/castore/blobservice/memory): use parking_lot RwLockFlorian Klink1-0/+1
This one doesn't require us to deal with poisoning, is upgradeable and the right thing to use when locking access to data, not IO resources. Change-Id: I78634953a73404500d28f51f1d93a87e215c8149 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8106 feat(tvix/store): add LruPathInfoServiceFlorian Klink1-0/+1
This provides an implementation of PathInfoService storing PathInfo in memory up to a certain capacity, then evicting these that have been used the least recently. Change-Id: I9d738687caf4f181a957f72245f26b92832313cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-06 r/8082 refactor(tvix): remove usage of async-recursionConnor Brewster1-1/+0
Rust 1.77 supports async recursion as long as there is some form of indirection (ie. `Box::pin`). This removes the need to use the async-recursion crate. Change-Id: Ic9613ab7f32016f0103032a861edff92e2fb8b41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11596 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-06 r/8080 feat(tvix/store): use async nar reader for ingest_narFlorian Klink1-1/+1
Rename read_nar to ingest_nar, and have it use the async nar reader version, and the ingest_entries machinery. This means we can now drop all code dealing with manually assembling castore nodes. Update our consumer, NixHTTPPathInfoService to use the new API. As we now accept an AsyncRead, we don't need to do any blocking here anymore, and can use the same async-compression crate as in the fetching logic (and support some more compression formats out of the box). Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-03 r/8070 test(tvix/[ca]store): enable bigtable only with "integration" featureFlorian Klink1-0/+4
The emulator and bigtable client are quite big. Remove them from the default //tvix:shell. Put the tests behind a `integration` feature flag, and add a variant with that enabled to CI, and drop the bigtable tools from //tvix:shell. Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7966 chore(tvix/store): remove test-case dependencyFlorian Klink1-1/+0
Change-Id: I62c3b93af438fd6397ded85dd4fe5acc01e0093a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11465 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-13 r/7900 feat(tvix/store/bin): add "copy" commandFlorian Klink1-0/+1
This allows copying individual store paths from the local /nix/store into tvix-store. As we don't support getting this information from Nix yet, we currently expect metadata to be provided externally: Nix' `exportReferencesGraph` feature contains pretty much all data we need for this. Expect a list of this information at a well-known key (`closure`, similar to nixpkgs' `pkgs/build-support/binary-cache/ default.nix`). We currently simply upload all store paths sequentially, without any parallelism or awareness in how the reference graph looks like. As long as the connected stores don't enforce this, this is fine, at least for now. Change-Id: Ib83c998465adddfdb110db994843c44e26b3d3d8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11397 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-12 r/7885 chore(tvix): bump rstest to 0.19.0Florian Klink1-1/+1
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-09 r/7884 feat(tvix/store): add bigtable pathinfoservice backendFlorian Klink1-1/+14
Put behind the "cloud" backend, like in the `tvix-castore` crate. Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7788 feat(tvix/store): add rstest-based PathInfoService testsFlorian Klink1-0/+2
This introduces rstest-based tests. We also add fixtures for creating some BlobService / DirectoryService out of thin air. To test a PathInfoService, we don't really care too much about its internal storage - ensuring they work is up to the castore tests. Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-20 r/7749 refactor(tvix/castore): introduce "cloud" feature flagFlorian Klink1-1/+2
This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-18 r/7723 feat(tvix/store): support RUST_LOG env varFlorian Klink1-1/+1
This allows selectively increasing the log level for only parts of the stack. For example, the following RUST_LOG env var enables "tracing" level logging for `tvix_store` and `tvix_castore`, while keeping it at "info" for the rest of the stack: export RUST_LOG='info,tvix_store=trace,tvix_castore=trace' It only affects logs, not traces (if enabled). Change-Id: Ib936bd132a405f216e75c843db83fbd71d20a18a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11182 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-16 r/7705 chore(tvix): bump tonic to 0.11.0Florian Klink1-4/+4
This bumps tonic and surrounding crates to 0.11.x. We added support for tonic 0.11.x into tokio-listener (https://github.com/vi/tokio-listener/pull/4), so that's bumped as well. Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-03 r/7642 feat(tvix/store): use reqwests' rustls-native-roots featureFlorian Klink1-1/+1
This makes reqwest honor `SSL_CERT_FILE` - previously it was using the chain bundled in webpki-roots. `object_store` pulls in `reqwest` with this feature, and the cargo solver will enable that feature globally as soon as we pull it in, as it assumes features are additive. This requires setting `SSL_CERT_FILE` when running tests, otherwise they'll fail with the unhelpful "NotFound" error. This was quite some fun to debug, why adding `object_store` to tvix-castore suddenly made tvix-store tests fail! Change-Id: I64fc82b4d994715480efdb1ffecb279716456ab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11090 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-02-20 r/7566 refactor(tvix/store/proto): use bstr::ByteSliceFlorian Klink1-0/+1
String::from_utf8_lossy simply discards invalid bytes, while bstr replaces them with their replacement character. Change-Id: Ib78ff36ca5faacc1ad60bc4ddde7b62773848c07 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10981 Tested-by: BuildkiteCI Reviewed-by: Peter Kolloch <info@eigenvalue.net> Reviewed-by: flokli <flokli@flokli.de>
2024-01-05 r/7350 chore(tvix): bump test-case dep to 3.3.1Florian Klink1-1/+1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-01 r/7308 feat(tvix/store): add opentelemetryFlorian Klink1-2/+7
This is behind the otlp feature flag (currently enabled by default). By default, this will try to push traces to a OTLP collector running at https://localhost:4317. You can make one available by running: ``` docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 --rm \ jaegertracing/all-in-one:1.42 ``` Started like that, jaeger brings a web interface at http://localhost:16686/search As documented in https://docs.rs/opentelemetry-otlp/latest/opentelemetry_otlp/, you can point this to another location by setting `OTEL_EXPORTER_OTLP_ENDPOINT`. Change-Id: Id1dca367d70027b2ea98bb70bcf99a68363ec2be Reviewed-on: https://cl.tvl.fyi/c/depot/+/8194 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: aaqaishtyaq <aaqaishtyaq@gmail.com>
2023-12-24 r/7261 chore(tvix): switch to upstream futures 0.3.30Florian Klink1-1/+1
The bugs have been fixed, https://github.com/rust-lang/futures-rs/pull/2801 and https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended up in that release. Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-24 r/7258 refactor(tvix/castore): add RootNode impl for BTreeMap, mv fs testsFlorian Klink1-3/+0
cl/10378 did already move store/fs to castore/fs, but we kept the tests in tvix-store, as they were populating a PathInfoService to make nodes appear in the mount root. Update these tests to now just insert root nodes into a BTreeMap, and ensure we can use that as a RootNodes too. Change-Id: Iad7d1ee4f9423eb6e3a1da33f433842c9ae0de1f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10410 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-22 r/7256 chore(tvix): move store/fs to castore/fsFlorian Klink1-46/+5
With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-16 r/7224 chore(tvix/store): switch back to fuse-backend-rs upstreamFlorian Klink1-3/+1
The fix mentioned in Cargo.toml did land in https://github.com/cloud-hypervisor/fuse-backend-rs/pull/162. Change-Id: I4451f910806f05400f7e0f83581ae786f170b9e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10375 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 r/7158 chore(tvix): drop sled compression supportFlorian Klink1-1/+1
It's been a while since the last sled release, and that one binds to a pretty old version of zstd, requiring workarounds like cl/10090. Upstream sled main branch currently has zstd halfway patched out (it's a no-op, but the feature flag and options are still there), and it's in that state for a year. Rather than maintaining our own fork of sled, let's just stop using the compression feature in sled, dropping the version pin to zstd that way, removing the need for cl/10090. This doesn't mean we won't reintroduce per-blob compression - but we probably just won't let sled take care of the compression, but do it ourselves - which is necessary for more chunked blob storage anyways. Even though we do drop the feature flag, we still need to explicitly use use_compression(false). Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-19 r/7038 feat(tvix/store/pathinfoservice): implement NixHTTPPathInfoServiceFlorian Klink1-0/+2
NixHTTPPathInfoService acts as a bridge in between the Nix HTTP Binary cache protocol provided by Nix binary caches such as cache.nixos.org, and the Tvix Store Model. It implements the [PathInfoService] trait in an interesting way: Every [PathInfoService::get] fetches the .narinfo and referred NAR file, inserting components into a [BlobService] and [DirectoryService], then returning a [PathInfo] struct with the root. Due to this being quite a costly operation, clients are expected to layer this service with store composition, so they're only ingested once. The client is expected to be (indirectly) using the same [BlobService] and [DirectoryService], so able to fetch referred Directories and Blobs. [PathInfoService::put] and [PathInfoService::nar] are not implemented and return an error if called. This behaves very similar to the nar-bridge-pathinfo code in nar-bridge, except it's now in Rust. Change-Id: Ia03d4fed9d0657965d100299af97cd917a03f2f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10069 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-14 r/6810 refactor(tvix/store): Upgrade tokio-listener to get tonic supportConnor Brewster1-2/+2
Tonic support was added to tokio-listener upstream which removes the need for use to have tonic compatibility wrapper types around it. See: https://github.com/vi/tokio-listener/pull/2 Fixes b/311 Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-12 r/6789 feat(tvix/store): enable tls[-roots] feature in tonicFlorian Klink1-1/+1
This teaches tonic to speak https, so `grpc+https` becomes a working connection string. Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6769 feat(tvix/store): use tvix_compat::nar::writer::asyncedef1-2/+3
Change-Id: Iad36872244df6f2225a2884f6b20cacd8f918b31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9619 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu>
2023-10-08 r/6743 test(tvix/store): use tokio-retry for exp backoffFlorian Klink1-0/+1
Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI