about summary refs log tree commit diff
path: root/tvix/castore
AgeCommit message (Collapse)AuthorFilesLines
2024-04-12 r/7886 feat(tvix/castore/directory/grpc): instrument functionsFlorian Klink1-1/+3
Change-Id: I9cc0a6a32184773597556ab5f9250257aa18ca4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11399 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-12 r/7885 chore(tvix): bump rstest to 0.19.0Florian Klink1-1/+1
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-09 r/7881 feat(tvix/castore/directory): add bigtable backendFlorian Klink7-1/+425
This adds a Directory service using https://cloud.google.com/bigtable/docs/ as a K/V store. Directory (closures) are put in individual keys. We don't do any bucketed upload of directory closures (yet), as castore/ fs does query individually, does not request recursively (and buffers). This will be addressed by store composition at some point. Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-07 r/7866 feat(tvix/castore): drop test-case crate depFlorian Klink1-1/+0
Change-Id: I5049a3682a58ce848d80f413b2964331025a90a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11370 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 r/7865 refactor(tvix/castore/directoryservice/from_addr): migrate to rstestFlorian Klink1-18/+19
Change-Id: I52b14652822766421d66e95bc646ed7baecc705f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11369 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Tested-by: BuildkiteCI
2024-04-07 r/7864 refactor(tvix/castore): migrate closure_validator to rstestFlorian Klink1-13/+14
Change-Id: I6c594d2e670a681484b858c3e04bc25b9e5a2077 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11368 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 r/7863 refactor(tvix/castore/tonic): migrate to rstestFlorian Klink1-12/+16
Change-Id: Ie088bf03c739bf64abf3432175362a8f92f501c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11367 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 r/7862 refactor(tvix/castore/hashing_reader): migrate to rstestFlorian Klink1-5/+6
Change-Id: I99ae0e27b4db4799db8af7cd6b9cc8d7f09227de Reviewed-on: https://cl.tvl.fyi/c/depot/+/11366 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 r/7861 refactor(tvix/castore/blobservice/object_store): drop individual testsFlorian Klink1-57/+3
This (and more) should now be covered by the generic testsuite (in crate::blobservice::tests). Change-Id: Ib3afc4f19f7e37a561b7398d43663dc941971f5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11365 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 r/7860 refactor(tvix/blobservice/from_addr): move from test_case to rstestFlorian Klink1-33/+30
This allows conditionalizing test cases on feature flags. Change-Id: Ic9ed9ef52f703c973fda2ca4aae5f425e33b67de Reviewed-on: https://cl.tvl.fyi/c/depot/+/11364 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Tested-by: BuildkiteCI
2024-03-28 r/7797 feat(tvix/castore/*service/tests): add objectstore to tests, sortFlorian Klink2-2/+3
Change-Id: If3a45d2f8008b75524ead704b05320364cc60d92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11282 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7796 feat(tvix/castore/directoryservice): log more TODOsFlorian Klink1-0/+3
We need to define behaviours and add tests for these. Change-Id: Id5825fafbf47897d8de42503ea6006eb131b1082 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11281 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-28 r/7795 refactor(tvix/castore/*): drop utils.rs and grpc directorysvc testsFlorian Klink9-604/+21
This drops pretty much all of castore/utils.rs. There were only two things left in there, both a bit messy and only used for tests: Some `gen_*_service()` helper functions. These can be expressed by `from_addr("memory://")`. The other thing was some plumbing code to test the gRPC layer, by exposing a in-memory implementation via gRPC, and then connecting to that channel via a gRPC client again. Previous CLs moved the connection setup code to {directory,blob}service::tests::utils, close to where we exercise them, the new rstest-based tests. The tests interacting directly on the gRPC types are removed, all scenarios that were in there show now be covered through the rstest ones on the trait level. Change-Id: I450ccccf983b4c62145a25d81c36a40846664814 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11223 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7788 feat(tvix/store): add rstest-based PathInfoService testsFlorian Klink2-2/+2
This introduces rstest-based tests. We also add fixtures for creating some BlobService / DirectoryService out of thin air. To test a PathInfoService, we don't really care too much about its internal storage - ensuring they work is up to the castore tests. Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-24 r/7779 feat(tvix/castore): add rstest-based BlobService testsFlorian Klink3-243/+294
Change-Id: Ifae9c41e1e3fa5e96f9b7e188181a44650ff8a04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11250 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-24 r/7778 feat(tvix/castore): AsRef<dyn BlobService> impl BlobServiceFlorian Klink1-0/+22
This allows us to use containers around BlobServices as BlobServices too. Change-Id: I3c7feb074f42b4e07c550fb8dfa63cf81d448ab5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11249 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-24 r/7777 feat(tvix/castore): add rstest-based DirectoryService testsFlorian Klink6-1/+277
This creates test scenarios (using the DirectoryService trait) that we want all DirectoryService implementations to pass. Some of these tests are ported from proto::tests::grpc_directoryservice, which tested this on the gRPC interface (rather than the trait), some others ensure certain behaviour for which we only recently introduced general checking logic (through ClosureValidator). We also borrow some code related to setting up a gRPC DirectoryService client (connecting to a server exposing a in-memory DiretoryService) from castore::utils, this will be deleted once it's all ported over. Change-Id: I6810215a76101f908e2aaecafa803c70d85bc552 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11247 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-24 r/7776 feat(tvix/castore): AsRef<dyn DirectoryService> impl DirectoryServiceFlorian Klink1-0/+25
This allows us to use containers around DirectoryServices as DirectoryServices too. Change-Id: I56cca27b3212858db8b12b874df0e567dd868711 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11248 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-24 r/7775 feat(tvix/castore/directory): add SledDirectoryPutterFlorian Klink1-3/+59
This uses DirectoryClosureValidator for validation and the sled batch API to insert multiple directories at once. Change-Id: I2d6dc513ccbc02e638f8d22173da5463e73182ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/11222 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-24 r/7774 refactor(tvix/castore/directory/grpc_wrapper): use ClosureValidatorFlorian Klink2-89/+19
This greatly simplifies the code in this function, replacing it with a much better tested (and more capable!) version of the validation logic. It also enables the gRPC server frontend to make use of the DirectoryPutter interface. While this might not be too visible in terms of latency thanks to gRPC streams bursting, it also enables further optimizations later (such as bucketing of directory closures). Change-Id: I21f805aa72377dd5266de3b525905d9f445337d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11221 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-24 r/7773 refactor(tvix/castore/directory): have SimplePutter use ValidatorFlorian Klink5-59/+82
This simplifies a bunch of code, and gets rid of some TODOs. Also, move it out of castore/utils, and into its own file. Change-Id: Ie63e05a6cdfb2a73e878cf7107f9172aed1cdf13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11224 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-24 r/7772 feat(tvix/castore/directory): add ClosureValidatorFlorian Klink2-0/+256
This can be used to validate a Directory closure (connected DAG of Directories), and their insertion order. Directories need to be inserted (via `add`), in an order from the leaves to the root. During insertion, we validate as much as we can at that time: - individual validation of Directory messages - validation of insertion order (no upload of not-yet-known Directories) - validation of size fields of referred Directories Internally it keeps all received Directories (and their sizes) in a HashMap, keyed by digest. Once all Directories have been inserted, a drain() function can be called to get a (deduplicated and) validated list of directories, in from-leaves-to-root order (to be stored somewhere). While assembling that list, a check for graph connectivity is performed too, to ensure there's no separate components being sent (and only one root). It adds a test suite for these cases, which is much nicer to test than where we previously had these checks (only in the gRPC server wrapper). Followup CLs will move the existing putters to use this. Change-Id: Ie88c832924c170a24626e9e3e91d868497b5d7a4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11220 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-23 r/7768 docs(tvix/castore): fix missing slash in docstringFlorian Klink1-1/+1
Change-Id: I5f39d0e613b651458b168cfd9df0693d7f01d704 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11246 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-23 r/7767 fix(tvix/castore/directory/tests): close upload handleFlorian Klink1-0/+2
We need to ensure the Directories are successfully uploaded before doing any testing with them. Change-Id: Iafa8deb86b3d5eb302ebfba3ced34385f67a7229 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11244 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-20 r/7757 feat(tvix/castore): derive Eq and Hash automaticallyFlorian Klink2-3/+2
This allows these messages to be put in HashSets. Change-Id: Ia58094cafe53eb624578821d3d8d969c5d21a1d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11219 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-03-20 r/7756 refactor(tvix/castore): instrument DirectoryPutter impls consistentlyFlorian Klink2-4/+5
Log the entire span with "trace" level, not just its `ret` level. The level of the error value event defaults to ERROR, so we don't loose these. B3Digest implements Debug and Display the same way, so we can omit the `(Display)` part in `ret(Display)` for them. Change-Id: Id00d123a5798e5bdc9820dd97ae2b4d4eb5455f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11218 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-20 r/7755 refactor(tvix/castore/directory): remove GRPCPutter::newFlorian Klink1-10/+3
This is no public API to construct this, there's exactly one caller, and it's perfectly fine to directly populate the struct there. Change-Id: Idae43a0162ee9bc687d21c550e0c9df33f12d263 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11217 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-20 r/7753 feat(tvix/castore): record errors for some failures in SimplePutterFlorian Klink1-0/+4
This makes it easier to see what's going wrong when uploading multiple Directories. Change-Id: Ieb71424b9761777c5f719b2f365962644de82baf Reviewed-on: https://cl.tvl.fyi/c/depot/+/11209 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-20 r/7751 feat(tvix/castore/blob): document missing objectstore+*:// URLFlorian Klink1-0/+1
Change-Id: I3cbc75e636efd467ddda7fc3c3c8f19ad42204ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/11206 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7750 refactor(tvix/castore/blob): drop simplefsFlorian Klink3-208/+1
This functionality is provided by the object store backend too (using `objectstore+file://$some_path`). This backend also supports content-defined chunking and compresses chunks with zstd. Change-Id: I5968c713112c400d23897c59db06b6c713c9d8cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11205 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7749 refactor(tvix/castore): introduce "cloud" feature flagFlorian Klink2-6/+23
This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-20 r/7748 docs(tvix/castore/directory): update docstring for get_recursiveFlorian Klink1-1/+4
The rust trait was missing to document the order of the elements in the stream. Document that, and also the reasoning behind this. Change-Id: I27ef0b2020082783fc41c2015233175e2b8e716d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11203 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-20 r/7746 refactor(tvix/castore/directory/from_addr): use match guardsFlorian Klink1-39/+42
This will allow feature-flagging some of the backends. Change-Id: Iddbdb89d3cf9c966a2c25b06b03e6917b284cae5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11201 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-20 r/7745 refactor(tvix/castore/blob/from_addr): use match guardsFlorian Klink1-53/+59
This will allow feature-flagging some of the backends. Change-Id: Idffbf8b3fd154f5a3d938225c3871feffea8ff8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11200 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-18 r/7730 refactor(tvix/castore/blobsvc): use B3Digest Display implFlorian Klink2-10/+2
We don't need to use BASE64 here on our own, B3Digest has a Display impl. This will also make sure the `b3:` digest is present in field values. Change-Id: I0ce6ee0f7e7e99fb9b16872953a1b742e99be291 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11192 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-18 r/7724 feat(tvix/blobservice/object_store) more loggingFlorian Klink1-1/+3
Have derive_{blob,chunk}_path emit trace-level events for both the values they're called with, as well as the return value. With RUST_LOG in place, it doesn't get lost in other unrelated noise. Change-Id: Id2451e3657324eff482841eb26a22d19e22bde30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11136 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-18 r/7722 feat(tvix/castore/blobsvc/grpc): read data in chunksFlorian Klink1-26/+54
Whenever this encounters an open_read(), it'll first check for more granular chunking. If there's more granular chunking data available, a ChunkedReader is constructed (which supports seeking backwards). This currently is still a bit stupid, and doesn't compose, as `ChunkedReader` uses `self` as the `BlobService` to ask for the individual chunks. In store composition future, we might want to compose this differently, essentially constructing `ChunkedReader` with another `BlobService` representing the entire hierarchy, so there's a chance to locally cache things, and do less requests. Change-Id: I22e0df4d6245f666d083b4f0b7114d3ac41d1dce Reviewed-on: https://cl.tvl.fyi/c/depot/+/11185 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 r/7720 refactor(tvix/castore/src/blobservice): remove useless else caseFlorian Klink1-4/+3
Change-Id: I09000371a1d8ff212ab46050d1a480509c6ffe70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11183 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-17 r/7719 feat(tvix/castore): impl Debug for B3DigestFlorian Klink1-1/+7
Use the same format as Display, b3: followed by the base64 representation. This makes the debug implementation of everything containing a b3 digest much nicer to read. Change-Id: I3ca3154d0b6fb07781c8f9c83ece3ff1a6957902 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11181 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-16 r/7705 chore(tvix): bump tonic to 0.11.0Florian Klink1-3/+3
This bumps tonic and surrounding crates to 0.11.x. We added support for tonic 0.11.x into tokio-listener (https://github.com/vi/tokio-listener/pull/4), so that's bumped as well. Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-16 r/7701 fix(tvix): don't emit rerun-if-changedFlorian Klink1-2/+2
`build.rs` emits rerun-if-changed statements for all proto files, as well as all include paths we pass it. Unfortunately, due to protobufs include path rules, we need to specify the path to the depot root itself as an include path, at least when building impurely with `cargo`. This causes cargo to essentially always rebuild, as it also puts its own temporary files in there. Unfortunately, tonic-build does not chase down to individual .proto files that are included. Disable emitting these `rerun-if-changed` statements for now. This could cause cargo to not rebuild protos every time, causing stale data until the next local `cargo clean`, but considering the protos change not that frequently, and it'll immediately surface if trying to build via Nix (either locally or in CI), it's a good-enough compromise. Change-Id: Ifd279a2216222ef3fc0e70c5a2fe6f87997f562e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11157 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2024-03-11 r/7685 feat(tvix/castore/blobsvc/from_addr): support object_storeFlorian Klink1-1/+21
The object_store crate supports a ton of different stores, with different schemes. For now, use a objectstore+ scheme prefix to enable these. Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-03-11 r/7684 feat(tvix/castore/blobsvc): add object storage implementationFlorian Klink4-0/+564
This uses the `object_store` crate to expose a tvix-castore BlobService backed by object storage. It's using FastCDC to chunk blobs into smaller chunks when writing to it. These are exposed at the .chunks() method. Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-03-09 r/7660 fix(tvix/castore/grpc/directory): skip_all fields in instrumentFlorian Klink1-15/+21
This only contains the outer metadata wrapping, and that's not too interesting: > Request { metadata: MetadataMap { headers: {"content-type": > "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers", > "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions: > Extensions } Drop these fields for now, and rely on the underlying implementations to add instrumentation for the application-specific fields. Clean up the error logging a bit. Change-Id: Ife1090ed411766a61e1fa60fd4c9570f38de1e98 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11102 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-09 r/7659 fix(tvix/castore/grpc/blob): skip_all fields in instrumentFlorian Klink1-5/+11
This only contains the outer metadata wrapping, and that's not too interesting: > Request { metadata: MetadataMap { headers: {"content-type": > "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers", > "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions: > Extensions } Drop these fields for now, and rely on the underlying implementations to add instrumentation for the application-specific fields. Log errors in some places where we didn't so far. Change-Id: Ia68d6c526987d3716be62a0809195401cf28512b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11101 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-03 r/7650 fix(tvix/castore): also set SSL_CERT_FILE for tests thereFlorian Klink1-0/+3
For everything using reqwest here during test cases, we also need to set SSL_CERT_FILE. Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7648 refactor(tvix/*/from_addr): improve test debuggabilityFlorian Klink2-4/+12
If there's an unexpected test failure, print it out, rather than just saying something is false even though it should be true. Use .expect() for this, which displays the error if it failed. We can't use expect_err(), as our stores are not display'able, so use an assertion with a message there. Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 r/7641 feat(tvix/castore): add HashingReader, B3HashingReaderFlorian Klink2-0/+90
HashingReader wraps an existing AsyncRead, and allows querying for the digest of all data read "through" it. The hash function is configurable by type parameter, and we define B3HashingReader. Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7640 feat(tvix/castore/digests): impl From digest::Output<_> for B3DigestFlorian Klink2-3/+11
This allows calling .into() to get a B3Digest. Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7631 refactor(tvix/castore/blobsvc/chunked_reader): refactor, documentFlorian Klink1-131/+139
The public-consumable thing here is ChunkedReader, not ChunkedBlob. ChunkedBlob is a helper that can be used to get a new AsyncRead, but not AsyncSeek. It is used internally by ChunkedReader whenever the client seeks. Make this more obvious, by extending the documentation, and putting ChunkedReader at the top of this file. Also make ChunkedBlob and its methods private, and give ChunkedReader a more useful constructor (from_chunks, instead of from_chunked_blob). Change-Id: I2399867591df923faa73927b924e7c116ad98dc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-12T08·08+0000