about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-03-28 r/7793 refactor(tvix/store/pathinfo/from_addr): stop using gen_*_serviceFlorian Klink1-7/+10
Remove usage of the gen_{blob,directory}_service() helper functions from utils. We populate Memory{Blob,Directory}Services here directly, as test_case and rstest doesn't compose well. Change-Id: I0fb48aadb8c818f508b18ceb83c85eb91359442a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11278 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7792 refactor(tvix/pathinfo/grpc): use rstest fixturesFlorian Klink1-6/+12
Change-Id: Ib114a4d141ca829520aed34600678d613994f875 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11277 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7791 refactor(tvix/store/tests/nar_renderer): use rstest fixturesFlorian Klink1-70/+67
Change-Id: I975729cb97d69d080fb63fbb8f81b4ac46cbfb7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11276 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7790 refactor(tvix/store/nar/renderer): don't require AsRefFlorian Klink1-8/+6
Previous CLs ensured BlobService and DirectoryService is implemented on AsRef<dyn BlobService> and AsRef<dyn DirectoryService> respectively, so there's no need to require AsRef in here. Change-Id: I1e8b93188ca784e1b9a592f3d35fcf6b9df75c75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11275 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-28 r/7789 refactor(tvix/store/nar/import): use rstest fixtures from castoreFlorian Klink1-14/+18
This stops using the gen_{blob,directory}_service() functions from tvix_castore::utils in favor of the rstest-based fixtures. Change-Id: I4c2fff2926f34af63d3bd5c29c8e939dbdf7e062 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11274 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7788 feat(tvix/store): add rstest-based PathInfoService testsFlorian Klink10-4/+216
This introduces rstest-based tests. We also add fixtures for creating some BlobService / DirectoryService out of thin air. To test a PathInfoService, we don't really care too much about its internal storage - ensuring they work is up to the castore tests. Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-28 r/7787 refactor(tvix/store/tests/fixtures): use [u8; 20]Florian Klink4-10/+9
This makes the fixture more use-able when interacting with the trait, the Bytes are only useful for the gRPC version. Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-27 r/7786 feat(tvix/store): AsRef<dyn PathInfoService> impl PathInfoServiceFlorian Klink1-0/+25
Change-Id: I7fc06ae97a50d04b8c36292b3457c112242a7a70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11270 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-27 r/7785 feat(tvix/nix-compat/store_path): derive Hash for StorePathRefIlan Joselevich1-1/+1
This allows StorePathRef as keys in a hashmap, by deriving Hash. The same is already done for StorePath. Change-Id: I3fc54c45787948116dcb27dfb5dc806b9b505835 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11269 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-27 r/7784 feat(tvix/nix-compat): worker protocol operation parserPicnoir5-4/+139
Change-Id: I7776635b17c44534223603d28cf59c7eebd976e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11229 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-24 r/7779 feat(tvix/castore): add rstest-based BlobService testsFlorian Klink3-243/+294
Change-Id: Ifae9c41e1e3fa5e96f9b7e188181a44650ff8a04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11250 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-24 r/7778 feat(tvix/castore): AsRef<dyn BlobService> impl BlobServiceFlorian Klink1-0/+22
This allows us to use containers around BlobServices as BlobServices too. Change-Id: I3c7feb074f42b4e07c550fb8dfa63cf81d448ab5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11249 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-24 r/7777 feat(tvix/castore): add rstest-based DirectoryService testsFlorian Klink8-1/+331
This creates test scenarios (using the DirectoryService trait) that we want all DirectoryService implementations to pass. Some of these tests are ported from proto::tests::grpc_directoryservice, which tested this on the gRPC interface (rather than the trait), some others ensure certain behaviour for which we only recently introduced general checking logic (through ClosureValidator). We also borrow some code related to setting up a gRPC DirectoryService client (connecting to a server exposing a in-memory DiretoryService) from castore::utils, this will be deleted once it's all ported over. Change-Id: I6810215a76101f908e2aaecafa803c70d85bc552 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11247 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-24 r/7776 feat(tvix/castore): AsRef<dyn DirectoryService> impl DirectoryServiceFlorian Klink1-0/+25
This allows us to use containers around DirectoryServices as DirectoryServices too. Change-Id: I56cca27b3212858db8b12b874df0e567dd868711 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11248 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-24 r/7775 feat(tvix/castore/directory): add SledDirectoryPutterFlorian Klink1-3/+59
This uses DirectoryClosureValidator for validation and the sled batch API to insert multiple directories at once. Change-Id: I2d6dc513ccbc02e638f8d22173da5463e73182ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/11222 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-24 r/7774 refactor(tvix/castore/directory/grpc_wrapper): use ClosureValidatorFlorian Klink2-89/+19
This greatly simplifies the code in this function, replacing it with a much better tested (and more capable!) version of the validation logic. It also enables the gRPC server frontend to make use of the DirectoryPutter interface. While this might not be too visible in terms of latency thanks to gRPC streams bursting, it also enables further optimizations later (such as bucketing of directory closures). Change-Id: I21f805aa72377dd5266de3b525905d9f445337d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11221 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-24 r/7773 refactor(tvix/castore/directory): have SimplePutter use ValidatorFlorian Klink5-59/+82
This simplifies a bunch of code, and gets rid of some TODOs. Also, move it out of castore/utils, and into its own file. Change-Id: Ie63e05a6cdfb2a73e878cf7107f9172aed1cdf13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11224 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-24 r/7772 feat(tvix/castore/directory): add ClosureValidatorFlorian Klink2-0/+256
This can be used to validate a Directory closure (connected DAG of Directories), and their insertion order. Directories need to be inserted (via `add`), in an order from the leaves to the root. During insertion, we validate as much as we can at that time: - individual validation of Directory messages - validation of insertion order (no upload of not-yet-known Directories) - validation of size fields of referred Directories Internally it keeps all received Directories (and their sizes) in a HashMap, keyed by digest. Once all Directories have been inserted, a drain() function can be called to get a (deduplicated and) validated list of directories, in from-leaves-to-root order (to be stored somewhere). While assembling that list, a check for graph connectivity is performed too, to ensure there's no separate components being sent (and only one root). It adds a test suite for these cases, which is much nicer to test than where we previously had these checks (only in the gRPC server wrapper). Followup CLs will move the existing putters to use this. Change-Id: Ie88c832924c170a24626e9e3e91d868497b5d7a4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11220 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-23 r/7768 docs(tvix/castore): fix missing slash in docstringFlorian Klink1-1/+1
Change-Id: I5f39d0e613b651458b168cfd9df0693d7f01d704 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11246 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-23 r/7767 fix(tvix/castore/directory/tests): close upload handleFlorian Klink1-0/+2
We need to ensure the Directories are successfully uploaded before doing any testing with them. Change-Id: Iafa8deb86b3d5eb302ebfba3ced34385f67a7229 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11244 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-23 r/7764 feat(tvix): add mdbook to shellFlorian Klink1-0/+3
With this, you can `mdbook build` inside the docs/ directory. Change-Id: I97df12ed839e3c7a8065563c039436e1a830281e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11243 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-23 r/7763 fix(nix-compat/nixhash): fix SRI string parsing with superfluous suffixFlorian Klink1-41/+38
We tried to be more strict than Nix, actually detecting if multiple hashes were specified, or other garbage at the end. However, Nix seems to just chop off at the end, so happily accepts anything afterwards. Example: https://github.com/NixOS/nixpkgs/pull/298041 Example: https://github.com/NixOS/nixpkgs/pull/298052 Change-Id: I2c1a49f51c8f8589a84df2fbf148e67e7380b550 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11234 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-22 r/7758 feat(tvix/nix-compat): add read_u32 wire primitivePicnoir1-1/+35
Change-Id: I87a40f79ee7e91bc4fe3dc0ee7818a533c729373 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11225 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-20 r/7757 feat(tvix/castore): derive Eq and Hash automaticallyFlorian Klink2-3/+2
This allows these messages to be put in HashSets. Change-Id: Ia58094cafe53eb624578821d3d8d969c5d21a1d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11219 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-03-20 r/7756 refactor(tvix/castore): instrument DirectoryPutter impls consistentlyFlorian Klink2-4/+5
Log the entire span with "trace" level, not just its `ret` level. The level of the error value event defaults to ERROR, so we don't loose these. B3Digest implements Debug and Display the same way, so we can omit the `(Display)` part in `ret(Display)` for them. Change-Id: Id00d123a5798e5bdc9820dd97ae2b4d4eb5455f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11218 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-20 r/7755 refactor(tvix/castore/directory): remove GRPCPutter::newFlorian Klink1-10/+3
This is no public API to construct this, there's exactly one caller, and it's perfectly fine to directly populate the struct there. Change-Id: Idae43a0162ee9bc687d21c550e0c9df33f12d263 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11217 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-20 r/7753 feat(tvix/castore): record errors for some failures in SimplePutterFlorian Klink1-0/+4
This makes it easier to see what's going wrong when uploading multiple Directories. Change-Id: Ieb71424b9761777c5f719b2f365962644de82baf Reviewed-on: https://cl.tvl.fyi/c/depot/+/11209 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-20 r/7752 fix(tvix): allow trace level log messages in debug buildsFlorian Klink2-3/+3
We had max_level_debug, not max_level_trace, so it was not possible to have `RUST_LOG` emit trace-level values (at least for our crates). release_max_level_info still controls the level for release builds, so that's not affected. Change-Id: I1f127ab63f13b1622d6616e06759a9414d9bb201 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11208 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-20 r/7751 feat(tvix/castore/blob): document missing objectstore+*:// URLFlorian Klink1-0/+1
Change-Id: I3cbc75e636efd467ddda7fc3c3c8f19ad42204ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/11206 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7750 refactor(tvix/castore/blob): drop simplefsFlorian Klink3-208/+1
This functionality is provided by the object store backend too (using `objectstore+file://$some_path`). This backend also supports content-defined chunking and compresses chunks with zstd. Change-Id: I5968c713112c400d23897c59db06b6c713c9d8cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11205 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7749 refactor(tvix/castore): introduce "cloud" feature flagFlorian Klink4-11/+31
This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-20 r/7748 docs(tvix/castore/directory): update docstring for get_recursiveFlorian Klink1-1/+4
The rust trait was missing to document the order of the elements in the stream. Document that, and also the reasoning behind this. Change-Id: I27ef0b2020082783fc41c2015233175e2b8e716d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11203 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-20 r/7747 refactor(tvix/store/pathinfo/from_addr): use match guardsFlorian Klink1-59/+62
This will allow feature-flagging some of the backends. Change-Id: Ie92914c3e2ad870eee87e73b3b5abe605fb56fe7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11202 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7746 refactor(tvix/castore/directory/from_addr): use match guardsFlorian Klink1-39/+42
This will allow feature-flagging some of the backends. Change-Id: Iddbdb89d3cf9c966a2c25b06b03e6917b284cae5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11201 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-03-20 r/7745 refactor(tvix/castore/blob/from_addr): use match guardsFlorian Klink1-53/+59
This will allow feature-flagging some of the backends. Change-Id: Idffbf8b3fd154f5a3d938225c3871feffea8ff8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11200 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-19 r/7741 feat(tvix/nix-compat): introduce magic hello/responsePicnoir2-1/+6
Setting the wire module as public to re-use it from tvix-daemon. Change-Id: I570cffc480c7b784d813663f77572bbe9d4e8259 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11197 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-03-19 r/7736 fix(tvix/eval): propagate errs from key comparison in genericClosuresterni3-7/+17
The accompanying test case shows that we need to bubble up the catchable error from the equality check if one is created. Change-Id: Ic9929a57aa7653c8aa5a72d1711cf3264798c731 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11159 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-19 r/7735 fix(tvix/eval): don't force lambda arg in map, mapAttrs & genListsterni8-4/+38
It is pretty pointless to force the function argument if we are going to use a suspended call later since forcing the function may fail in ways that are not covered by Catchables (non-recoverable errors, infinite recursions). From this, it kind of seems as if using #[catch] is never correct and should be replaced by #[lazy]. Also we should probably try to come up with more test cases for stuff where laziness gets us out of the jam as an equivalent to the catchable tests for nonrecoverable errors. Fixes b/386. Change-Id: Ia926df4ac1b440ec430403ab7b40924a0c97221b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11153 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2024-03-18 r/7733 refactor(tvix/store/import): use B3Digest in log_nodeFlorian Klink1-5/+3
Change-Id: I2347bbae8e7d4e19eeed4a3fb13729d0a94feedd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 r/7732 refactor(tvix/store/pathinfo/nix_http): NixHash to display NARHashFlorian Klink1-3/+5
This will give us the sha256: prefix, and hashes we're more used to, in that context. Change-Id: I72e42fe685e365ba9baa7cd81001387d239fa7c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11194 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-18 r/7731 refactor(tvix/store/nar): use B3Digest in RenderErrorFlorian Klink1-5/+4
Ensure consistent formatting of blake3 digests. It looks like we don't actually construct these error types anywhere, so no code needs to be refactored to it currently, but still good to be consistent. Change-Id: I49dc8a7f3cb4245ac06b9a6a44b72060434a3d32 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11193 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-18 r/7730 refactor(tvix/castore/blobsvc): use B3Digest Display implFlorian Klink2-10/+2
We don't need to use BASE64 here on our own, B3Digest has a Display impl. This will also make sure the `b3:` digest is present in field values. Change-Id: I0ce6ee0f7e7e99fb9b16872953a1b742e99be291 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11192 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-18 r/7729 feat(tvix/boot): make VM cmdline and asserted VM output configurableFlorian Klink1-2/+8
This can be used in the future to boot proper NixOS VMs, which want a init= in their cmdline. Change-Id: Iad4a25d5081f3a6af1c7f62f15853f3afaae4a0c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11190 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-18 r/7728 fix(tvix/boot/tests): disable otlpFlorian Klink2-3/+8
We don't have a OTLP collector inside the Nix build, this removes some noise from the log about not being able to reach it. Change-Id: I058839c8f214821b536306c8c30e1a8e7192153b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11189 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 r/7727 feat(tvix/boot): support seeding closures via nar-bridgeFlorian Klink1-42/+99
This updates the parameters mkBootTest can be called with. It now accepts a `path`, and then either importPathName needs to be set, or isClosure needs to be set to true. The former activates the existing functionality, tvix-store import is used to import contents as a NAR-addressed store path. The latter uploads the path as a closure (so including its references, and keeping the store paths intact) to tvix-store. We use nar-bridge, and the HTTP interface it provides to do this. As `nix copy` can't be used inside a Nix build, we use `pkgs.mkBinaryCache` to come up with the .narinfo and .nar files that would be in a binary cache, and then use a bit of GNU Parallel and bash to upload store paths ourselves. Change-Id: Icfa5c0af0c22ab5418686947aa2c060f5987b873 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11188 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-18 r/7726 docs(tvix/boot): document we could use a smaller kernel here.Florian Klink1-0/+2
We currently use a pretty big kernel to boot the tvix VMs, with a lot of drivers. It can probably be slimmed down significantly, at least on the hardware driver and filesystem front. Change-Id: I4c7f73fb2dafbf2bcdec8057d2b14a9b0e9b3275 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11187 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-03-18 r/7725 feat(tvix/boot/tests): make urls and paths configurableFlorian Klink1-16/+39
Fork off a long-living tvix-store daemon in the background, and have both the import, as well as the virtiofs invocation connect to that one, rather than use the underlying storage directly. This also opens up the potential to have other things seed the store (like nar-bridge). Change-Id: Ie13aedaf7ab31e5ebe78a7be42f144af3920fa9c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11186 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 r/7724 feat(tvix/blobservice/object_store) more loggingFlorian Klink1-1/+3
Have derive_{blob,chunk}_path emit trace-level events for both the values they're called with, as well as the return value. With RUST_LOG in place, it doesn't get lost in other unrelated noise. Change-Id: Id2451e3657324eff482841eb26a22d19e22bde30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11136 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-18 r/7723 feat(tvix/store): support RUST_LOG env varFlorian Klink4-20/+153
This allows selectively increasing the log level for only parts of the stack. For example, the following RUST_LOG env var enables "tracing" level logging for `tvix_store` and `tvix_castore`, while keeping it at "info" for the rest of the stack: export RUST_LOG='info,tvix_store=trace,tvix_castore=trace' It only affects logs, not traces (if enabled). Change-Id: Ib936bd132a405f216e75c843db83fbd71d20a18a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11182 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 r/7722 feat(tvix/castore/blobsvc/grpc): read data in chunksFlorian Klink1-26/+54
Whenever this encounters an open_read(), it'll first check for more granular chunking. If there's more granular chunking data available, a ChunkedReader is constructed (which supports seeking backwards). This currently is still a bit stupid, and doesn't compose, as `ChunkedReader` uses `self` as the `BlobService` to ask for the individual chunks. In store composition future, we might want to compose this differently, essentially constructing `ChunkedReader` with another `BlobService` representing the entire hierarchy, so there's a chance to locally cache things, and do less requests. Change-Id: I22e0df4d6245f666d083b4f0b7114d3ac41d1dce Reviewed-on: https://cl.tvl.fyi/c/depot/+/11185 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-10-31T20·44+0000