about summary refs log tree commit diff
path: root/tvix/castore/src
AgeCommit message (Collapse)AuthorFilesLines
2023-12-11 r/7158 chore(tvix): drop sled compression supportFlorian Klink3-3/+7
It's been a while since the last sled release, and that one binds to a pretty old version of zstd, requiring workarounds like cl/10090. Upstream sled main branch currently has zstd halfway patched out (it's a no-op, but the feature flag and options are still there), and it's in that state for a year. Rather than maintaining our own fork of sled, let's just stop using the compression feature in sled, dropping the version pin to zstd that way, removing the need for cl/10090. This doesn't mean we won't reintroduce per-blob compression - but we probably just won't let sled take care of the compression, but do it ourselves - which is necessary for more chunked blob storage anyways. Even though we do drop the feature flag, we still need to explicitly use use_compression(false). Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-09 r/7140 refactor(tvix/castore): address clippyFlorian Klink1-10/+7
We match to destructure a single pattern. Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-24 r/7053 fix(tvix/castore): correctly flag unreachable codesterni1-1/+1
Change-Id: Id09afa4b77c3c70fb5695f253f6df4aa88b61e19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10113 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-19 r/7037 refactor(tvix/castore/blobservice): rm AsyncBufRead from BlobReaderFlorian Klink1-4/+1
There's no need to already require this to be buffered here. Change-Id: Ib9a11b194e0754d87ab8d2ef0b8cb0f4edc01229 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10074 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-18 r/7027 feat(tvix/castore): fix tracing instrument in `MemoryBlobService`Ryan Lahfa1-2/+3
Change-Id: Iedba57e8b3e1a44f14f5baa1e981275d4b02eb56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10070 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-18 r/7026 feat(tvix/castore): impl From<std::io::Error> for ErrorFlorian Klink1-0/+10
Make it less annoying to convert from io::Error to this. We already have one direction, doesn't hurt to have the other too. Change-Id: I9fe2c6da608c9d54910ee8c397572aadb1d90d99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10068 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-17 r/7025 refactor(tvix/castore/tonic): use match in channel_from_urlFlorian Klink1-45/+48
Having random if blocks and returning from them is error-prone. Also, turns out we only need the unprefixed scheme in the fallback case, so move it down to there. Change-Id: Ifcb09279c963f8a39e0dbabe145990263f3d7cf9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10041 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-15 r/7023 refactor(tvix/castore/utils): drop unused DuplexStreamWrapperFlorian Klink1-13/+1
This wasn't used at all, let's remove it. Change-Id: I426e3d93c32ebe65247ae5cf8d05b5bf686be2d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10044 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-11-15 r/7022 refactor(tvix/castore/tonic): make async, support wait-connect=?Florian Klink8-157/+144
This moves the sync `channel::from_url` to a async `tonic::channel_from_url`. It now allows connecting non-lazily if `wait- connect=1` is set in the URL params. Also, make the pingpong tests for blobsvc and directorysvc use the wait- connect=1 codepath. Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7021 refactor(tvix/castore): remove DirectoryService::from_urlFlorian Klink5-187/+98
Make directoryservice::from_addr use the more specific constructors. Change-Id: I9fee2afed77692505988d631d9fe246d9843d25a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10029 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7020 refactor(tvix/castore/blobsvc): remove BlobService::from_urlFlorian Klink5-241/+98
Make blobservice::from_addr use the more specific constructors. Change-Id: Id9637e279d6910ce6d92ff0086a984be5c65a8c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10028 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7016 feat(tvix/castore/src/channel): move from_url testsFlorian Klink1-0/+60
These gRPC PathInfoService tests were actually not too useful in here, what we're mostly testing is the channel construction, so move it to there. Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-07 r/6972 fix(tvix/castore): B3Digest::{to_vec -> as_slice}edef6-15/+13
Not a single call site actually makes use of the Vec. Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-05 r/6954 feat(tvix/castore): extend blobstore protos for verified streamingFlorian Klink2-2/+3
This pdates the proto docstrings a bit, especially w.r.t. verified streaming. It also adds send_chunks, send_bao fields to StatBlobRequest (renamed from BlobMeta) Change-Id: I590cc8646d86b73bca9f38a9b6d9ea15e4df5cb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9951 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6946 feat(tvix/castore): bump [Directory,File]Node size to u64Florian Klink5-19/+19
Having more than 4GiB files is quite possible (think about the NixOS graphical installer, and an uncompressed iso of it). No wire format changes. Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-27 r/6894 feat(tvix/castore): carry name in ValidateNodeError::InvalidNameedef2-14/+14
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef2-7/+7
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6883 refactor(tvix): condense long bytestringsedef2-27/+12
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-17 r/6850 docs(tvix/castore/directorysvc): document drop without put and closeFlorian Klink1-0/+4
As seen in cl/9761, we only use put() and close() in some cases. Make sure this is documented, so we don't end up having implementations not able to deal with that. Change-Id: I406ad23f2f4c9604eca731d898ae41a4663fb846 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9786 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6842 fix(tvix/castore): Fix race when ingesting into castoreConnor Brewster1-0/+4
After finishing the ingestion, the directory putter was not being closed. This caused a race where the root directory node was accessed before the directory node had been flushed to the server. This patch makes it so we close the putter before returning the root node which should ensure that the root node exists on the directory service server before the `ingest_path` function returns. Fixes b/326 Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-16 r/6824 refactor(tvix/castore): clippyFlorian Klink1-1/+1
Change-Id: I8a792d6b238190a78390c5d8f638669a2681f70f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9746 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink5-88/+84
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink2-22/+18
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink2-7/+8
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6798 refactor(tvix/castore): simplify node validation checksFlorian Klink1-9/+8
We can just check the digest length to be correct, and move the symlink target checks to a single line. Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-13 r/6796 docs(tvix/castore): point out use of contents_firstLinus Heckemann1-0/+5
Change-Id: I7620d2abe01675ea7028a478d4f8447e36d5768b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9605 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-12 r/6794 refactor(tvix/castore): factor out node checksFlorian Klink2-33/+73
Implement `validate()` on `node::Node`, and call it from PathInfo's validate() too. Node-related errors are moved to a ValidateNodeError error type. This additionally adds some more validations for symlink targets (they must not be empty, and not contain null bytes). Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-12 r/6790 refactor(tvix/castore/utils): clippyFlorian Klink1-8/+4
Change-Id: Ib0fa1e1fb90d7f315f943a3f28266c0fb3714a31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9708 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-10 r/6778 fix(tvix/castore): use bstr for formatting names in errorsedef1-4/+4
Much friendlier than either Base64 or raw byte slices. Change-Id: I9b4cdd57c83ddc76c0be8103da4320207657a72b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9622 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-10 r/6777 fix(tvix/castore): handle Directory::size overflow explicitlyedef2-9/+90
We use checked arithmetic for computing the total size, and verify that size is in-bounds in Directory::validate. If an out-of-bounds size makes it to the "unchecked" size method, we either panic (in debug mode), or silently saturate to u32::MAX. No new panic sites are added, since overflows in debug mode already panic at the language level. Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-10 r/6776 test(tvix/castore): demonstrate that Directory::size may panicedef1-0/+31
Change-Id: Iebfa12df51c8c9d39b4d9eb2fccc6ffa312639cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9620 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-08 r/6744 test(tvix/castore): add uds pingpong test for directorysvcFlorian Klink1-2/+61
This was missing so far. Change-Id: Ic8c7781aa0e101342e8a7d94f206937b4163ebfa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9570 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 r/6742 test(tvix/castore): use tokio-retry for exp backoffFlorian Klink1-15/+13
Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 r/6740 refactor(tvix/castore): simplify test_valid_unix_path_ping_pongFlorian Klink1-65/+47
We don't need to spawn two tokio runtimes anymore, and can do the URL parsing at once, too. Change-Id: I38ab96978cb7f8c31ded2726262e0b1366655094 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9566 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-08 r/6735 refactor(tvix/castore): move tests to grpc client, rm tonic-mockFlorian Klink3-35/+63
Similar to gen_directorysvc_grpc_client, introduce a gen_blobsvc_grpc_client function that provides a gRPC client connected to a blobservice. The test is update to use that client to test against, rather than the server trait, removing the last usage of tonic_mock, so it's removed as well. Fixes b/243. Change-Id: If746e8600588da247eb53a63b70fe72f139e9e77 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9564 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-08 r/6729 docs(tvix/castore): remove TODOFlorian Klink1-1/+0
This probably was about passing around directory_putter at some point, which we do, so whatever this meant, it's not actionable anymore. Change-Id: I1b4e0cdd2119bf2b2a9cf06d186a3b476b0ff367 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9573 Reviewed-by: Linus Heckemann <git@sphalerite.org> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-08 r/6728 refactor(tvix/castore): use DuplexStream instead of unix socketFlorian Klink3-76/+47
We can use DuplexStream to create to bidirectional pairs, which avoids manually waiting for unix sockets to pop up and connect, and creating temporary directoires to create the unix sockets in. Turns out, we also don't actually need to spawn the server in a separate runtime, it works just fine these days. This might be due to all the sync barriers in between being gone. Change-Id: I6b79823bc6209cbcb343b7a498c64a2ba6e0aee7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9562 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 r/6725 refactor(tvix/castore): move gRPC directorysvc tests off tonic_mockFlorian Klink1-74/+90
The tests were using the gRPC Service/Server interface, not the gRPC clients. Use our new gen_directorysvc_grpc_client() function to spin up a server in a separate thread, and test via that client, rather than *just* the server implementation. This is more correct, because we're now really exercising the gRPC stack, and some of the stream implementations are also only implemented on the client. Change-Id: I99434a2399856a44c9e6988d0b720f7a429d0ad1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9560 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-08 r/6724 refactor(tvix/castore/directorysvc): factor out gRPC client genFlorian Klink2-173/+178
Move this code into a helper function, which we'll use in other places in a bit. Change-Id: Icae6f6dd2d4b2fa86fd2b836ddd7a4ca0e0354e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9559 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6704 refactor(tvix/castore): move magic number to B3_LEN constFlorian Klink2-5/+7
… and export it. Change-Id: I47d2dc2f5a8174da65c614b43801d648506e2d73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9544 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
2023-10-04 r/6689 fix(tvix/castore): explicitly name lifetimes in process_entryedef1-3/+3
Otherwise this produces absolutely inscrutable errors: note: hidden type `[async fn body@castore/src/import.rs:63:1: 63:94]` captures lifetime '_#24r Change-Id: If5d9626c9edf400de5bcec038bcaa5a3117561f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9506 Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de>
2023-09-26 r/6660 chore(tvix/[ca]store): allow building without tonic-reflectionFlorian Klink1-1/+1
- rename the feature to `tonic-reflection` in both tvix-store and tvix- castore. - set it to disabled in tvix-castore by default - enable it in tvix-store, and pull in tvix-castore with the feature enabled. Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-22 r/6630 chore(tvix/castore): remove dependency on anyhowFlorian Klink1-3/+1
We don't use this here, except in a test, where we don't really use it either. Change-Id: Ia6c45fccf663fe328942b1e1a2cd1c3a1b7f9ae5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9376 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink30-0/+4785
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-21T16·00+0000