about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2023-10-09 r/6761 chore(tvix/*store): use google.golang.org/protobufFlorian Klink2-3/+3
The other one is deprecated, according to https://pkg.go.dev/github.com/golang/protobuf/proto. Change-Id: I502d5bfa7431df0dea3d564f1e0ecfea14c90e10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9597 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 r/6758 feat(tvix/nar-bridge): add ToNixNarInfo()Florian Klink2-3/+3
Convenience function, moves all code converting from a PathInfo struct to to go-nix's NarInfo. Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 r/6757 test(tvix/store/protos): add go tests for Validate()Florian Klink3-0/+130
We already have validation tests for Rust, let's add the missing ones for golang too. Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595 Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-08 r/6743 test(tvix/store): use tokio-retry for exp backoffFlorian Klink2-17/+16
Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 r/6741 refactor(tvix/store): simplify test_valid_unix_path_ping_pongFlorian Klink1-43/+31
We don't need to spawn two tokio runtimes anymore, and can do the URL parsing at once, too. Change-Id: I7885a894bb1250cd087d4e1893e3e73b631331da Reviewed-on: https://cl.tvl.fyi/c/depot/+/9567 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 r/6738 docs(tvix/store/bin): drop outdated TODOFlorian Klink1-1/+0
We already use store_path::build_nar_based_store_path as a helper function down there. Change-Id: I46842bb0a2f4cd5de7dcfc3c4b6aae399618667a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9571 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 r/6736 chore(tvix): bump tokio[-util], tonic, prostFlorian Klink1-7/+7
This bumps all these dependencies to their more recent versions. We have to enable the `macros` tonic feature, and looks like we should also enable `rt`, not only `rt-multi-thread`. Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 r/6726 feat(tvix/store/listener): implement connect_info()Florian Klink1-3/+19
For Unix and TCP streams, we can actually give more info than just (). Change-Id: I80ca261b9e7a880389ec2029ea5893180314f0b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9561 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 r/6723 chore(tvix/store): drop tonic-mock from dependenciesFlorian Klink1-1/+0
It's not used in here. Change-Id: If1dbf360ec37b649c27d3a9bf11f7bf0ca947631 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9558 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-07 r/6720 fix(tvix/store): Remove virtiofs from default featuresConnor Brewster2-2/+3
vhost-user-backend doesn't support macOS yet, so the virtiofs features will not work on macOS. This removes it as a default feature which makes `cargo build` work out of the box on macOS. The `virtiofs` feature is enabled for Linux when building via Nix, but if being built by cargo directly, the feature must be enabled via a cargo flag. Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6715 refactor(tvix/store/proto): use NamedNode traitFlorian Klink1-31/+28
This saves us writing the name parsing code three times. We can also delay parsing until we did other (cheaper) checks. Change-Id: I1abe3f20dba4215b38839cf7466297e028d64656 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9548 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6712 feat(tvix/store/protos): add Validate() method to pathinfo.goFlorian Klink3-9/+105
This is very similar to the Rust counterpart. Change-Id: I40d51aaac3fcf7f52e5896587e561bc2377f6269 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9549 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6707 test(tvix/store/pathinfo): add more tests for referencesFlorian Klink2-42/+101
This should cover all error cases produced. Change-Id: If31816d9b087551d86d7913df55df8f9f44bb554 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9546 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-05 r/6706 feat(tvix/store): add validation for referencesFlorian Klink2-6/+60
This validates the size of reference digests in the PathInfo message, as well as inside the narinfo submessage. If narinfo is set, they need to parse to StorePath, and have the same digest there as in the PathInfo message. `proto::tests::pathinfo::validate_references` needed to be updated, because we actually did not populate the proper references before. Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-05 r/6705 refactor(tvix/store): mv *Err*::Invalid{Node,}DigestLenFlorian Klink2-7/+7
There's other digests in the PathInfo structure, that also might have wrong digest lengths. Rename this to give some room for them, and update the error message a bit as we go. Change-Id: I06562664721156e658f2ed14ba1de907377d284b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-02 r/6687 fix(tvix/store): Fix FUSE support on MacOSBrian Olsen6-17/+37
This partially fixes b/312 and gets FUSE to work again on MacOS. It is mostly small type changes and an update to fuse-backend-rs because upstream currently doesn't work with MacFuse. It also sets the default FUSE thread count on MacOS to 1 because otherwise the mount command will hang when shutting down as only one thread gets ENODEV and all the others just keep blocking. Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2023-10-02 r/6686 fix(tvix/store): FuseDaemon unmount is blocking async runtimeBrian Olsen1-1/+1
The unmount method in FuseDaemon calls join on a bunch of threads and that is a blocking call but it is called from an async context in the tvix-store binary. This change wraps the call to unmount in a spawn_blocking. Change-Id: If89183b4a3f890874e75f5faf90cd24cb18da1e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9489 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-26 r/6663 fix(tvix/store): rename Read method in ReadmeFlorian Klink1-1/+1
This call was renamed to `Read` a while ago, but I forgot to update the README. Change-Id: Ibcb97fe2520a5e9bb8444ec1adade6890dda4a91 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9476 Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-26 r/6661 fix(tvix/store): allow building with Nix on DarwinFlorian Klink1-0/+2
Curently both fuse and virtiofs features fail to build on Darwin. Disable them for now until fixed. Change-Id: Ic99fa5e3ebec8a7072ba884101a5c18652ed409c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9473 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-09-26 r/6660 chore(tvix/[ca]store): allow building without tonic-reflectionFlorian Klink4-7/+7
- rename the feature to `tonic-reflection` in both tvix-store and tvix- castore. - set it to disabled in tvix-castore by default - enable it in tvix-store, and pull in tvix-castore with the feature enabled. Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 r/6658 chore(tvix/store): only depend on vhost-user-fs if virtiofs enabledFlorian Klink1-6/+11
Change-Id: Iaf88d43df27b87548794a8dddc9a1e08279b8ffb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9471 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 r/6654 chore(tvix/store): move fuse-backend-rs back to upstreamFlorian Klink1-2/+2
It's in no release yet, but this is the current rev in master. Change-Id: I9e4bc2fde26e85c4182bd66b3e566c2d957072ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/9469 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-09-25 r/6653 feat(tvix/store): Support listening on UNIX domain socketsConnor Brewster4-1/+121
This adds support for listening on UNIX domain sockets via the tokio-listener crate. The crate will automatically determine whether to start a TCP or UNIX domain socket server based on the listen address. Unfortunately, it's not compatible with tonic right out of the box so I added some wrapper types to implement the necessary traits to make things work. We should investigate upstreaming a `tonic` option to the tokio-listener crate which implements the relevant `tonic` traits. Example: ``` $ tvix-store daemon -l /run/tvix-store.sock INFO tvix_store: tvix-store listening on /run/tvix-store.sock $ tvix-store mount -l /mnt/tvix --blob-service-addr grpc+unix:///run/tvix-store.sock --directory-service-addr grpc+unix:///run/tvix-store.sock --path-info-service-addr grpc+unix:///run/tvix-store.sock $ ls /mnt/tvix ``` Change-Id: I91c4a4b0c5a177b3b90e6c01a4e5d263130e6bdb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9429 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-24 r/6642 feat(tvix/store/fs): Add support for virtiofs backendConnor Brewster4-1/+325
This adds a virtiofs daemon implementation which hooks into the existing tvix-store filesystem implementation that is used for FUSE. This allows adding the filesystem to a microvm without having to set up FUSE inside the guest. Change-Id: If80c36c9657f2289853e8d9a364bf4f1f7b7559c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9344 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-22 r/6632 chore(tvix): re-enable {ca,}store-protos-goFlorian Klink2-18/+10
This couldn't be enabled before until go get was able to fetch the modules again. Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink65-7481/+356
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6628 docs(tvix/castore): rename traverse_to -> descend_toFlorian Klink2-17/+13
With the move of this code out into castore it has become apparent this is a general descent inside the castore. Concerns like making sure the whole Directory closure has been fetched/is fetched initially is nothing this layer needs to worry about. We can handle this during substitution of a new PathInfo, once there's store composition. Closes b/270. Change-Id: I661ed08e54bc81478e032cfb9abeb23e5b337fbe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9373 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6627 chore(tvix): remove some unused crate dependenciesFlorian Klink1-4/+0
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6625 docs(tvix/store/pathinfosvc): fix copypasta errorFlorian Klink1-1/+0
This is not a simple get and put returning digest. You currently can get by output hash, and a put gives you back the (possibly modified) PathInfo message. Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster20-694/+729
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-20 r/6622 refactor(tvix/store/fs): Separate FUSE and filesystem codeConnor Brewster9-131/+138
In prepration for adding virtiofs support, I thought it would make sense to split out the filesystem implementation from FUSE itself. The `fs` module holds the tvix-store filesystem implemetation and the `fuse` module holds the code to spawn a FUSE daemon backed by multiple threads. Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 r/6621 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rsConnor Brewster7-345/+496
This switches the FUSE implementation from fuser to fuse-backend-rs. fuse-backend-rs is designed to work with both FUSE and virtiofs. Virtiofs support will make it possible to plug the tvix-store into a microvm and have `/nix/store` access without having to setup FUSE inside the guest. Additionally fuse-backend-rs has nice support for running multiple FUSE threads and has some async support. The goal of this commit is to mechanically switch over to fuse-backend-rs with minimal changes. I did have to add some locks here and there because fuse-backend-rs uses `&self` on all methods whereas fuser uses `&mut self`. `&self` is required for concurrent access to the FUSE server, so this makes sense. We can consider switching to concurrent maps and use some other techniques to reduce lock contention and critical section size. Issue: https://b.tvl.fyi/issues/305 Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-18 r/6608 chore(tvix/store/protos): drop unused fields for nowFlorian Klink5-218/+51
This wasn't removed yet, and no code is using/populating it so far. It's confusing, let's update it to the state of things now, and re- introduce it once we get there. Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-18 r/6607 fix(tvix/store/fuse): fix executable bitFlorian Klink2-1/+38
We were blindly returning 0o444 for all regular files, but regular files with executable bit need to be 0o555. This wasn't spotted because stat'ing executable files was not part of the test suite, it's now added. Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 r/6606 refactor(tvix/store/blobsvc): make BlobStore asyncFlorian Klink20-881/+1249
We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 r/6598 feat(tvix/store): improve error message formattingFlorian Klink1-1/+2
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can, we want this to display it as a string (and fall back to the base64 encoding if we can't). Before: > rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted After: > rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 r/6597 chore(tvix/store): allow non_snake_caseFlorian Klink1-1/+1
Some of the tonic-generated types create new warnings. Change-Id: Ib538bfd7d90d857d80a129fbbdf3a38c89cc04ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9349 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-15 r/6596 feat(tvix/store): add simple integration testFlorian Klink1-3/+29
This imports the docs folder into the tvix store, and ensures it comes up with the same store path as Nix. While we validate hashes in general through tvix-cli output path checks already, it doesn't include the tvix-store CLI itself. See https://cl.tvl.fyi/c/depot/+/9329/comment/339f0720_524f0104/ for context. Change-Id: I239ce5b6a07cb962b242142ab716693359b8674c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9338 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 r/6595 feat(tvix/store/import): print store path to stdoutFlorian Klink1-2/+13
This allows comparing the output in an integration test. Change-Id: I8bb2254e18e90005a4f1b30fd47ef69642e3732e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9337 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 r/6594 fix(tvix/store): log to stderr, not stdoutFlorian Klink1-4/+5
This leaves some space in stdout to provide actual meaningful output. Also, rename print_node to log_node because that's what it does, it's using the logger to log out more detailed info. Change-Id: Ic64a6330dbfcdc63eb4198067a5c5e47b841b9a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9336 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-09-12 r/6581 fix(tvix/store/blobservice/seeker): fix debug assertFlorian Klink1-2/+9
We were asserting absolute_offset > self.pos, but that's not true for both being zero. Ramp up the tracing bits a bit, so we actually can see this in the debug logs. Change-Id: I21693bcafab227549b19cd6f1215d2f8dee77ecc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9292 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-09 r/6572 fix(tvix/protos): support []byte node namesBrian McGee2-34/+35
The underlying protos were changed to return `[]byte` instead of `string`. Change-Id: I5d3e5d8de0ed7200325f7ab0d62e3c10d8eb1b7d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9285 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-08 r/6568 docs(tvix/store/fuse): add more comment to list_root argFlorian Klink1-0/+2
Forgot to squash this into cl/9217. Change-Id: Id5a87cbe729592f3b53e8c329b6890519f5da319 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9272 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-05 r/6558 refactor(tvix/store): use tokio::task::JoinHandleFlorian Klink3-37/+35
This makes the inside code a bit less verbose. I wasn't able to describe the type of the async move closure itself, which would allow us to remove the JoinHandle<_> type annotation entirely. Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-09-05 r/6557 feat(tvix/store): fix ctrl-c handling on mount commandFlorian Klink2-4/+25
This enables the tokio `signal` feature, and registers a ctrl_c signal handler, which will use the unmount handle to unmount in case a ctrl-c signal is received. This avoids having disconnected mountpoints when Ctrl-C'ing a `tvix-store mount` invocation. In case the filesystem is unmounted externally (via `umount /path/to/ mountpoint`), the future is waiting for the signal is never resolved and the task is stopped. Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-05 r/6556 feat(tvix/store/fuse): allow listingFlorian Klink3-4/+111
This provides an additional configuration flag to the tvix-store mount subcommand, and logic in the fuse module to request listing for the root of the mountpoint. Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: adisbladis <adisbladis@gmail.com> Tested-by: BuildkiteCI
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink9-41/+310
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-05 r/6554 refactor(tvix/store): infer more typesFlorian Klink1-51/+48
We don't need to explicitly describe the type of the task itself, describing the return type of the async closure is sufficient. Also, use io::Result<_> instead of Result<_, io::Error>. Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-05 r/6553 chore(tvix/store): drop walkdir workaround for symlinks at rootFlorian Klink2-21/+5
https://github.com/BurntSushi/walkdir/pull/170 got merged, meaning we don't need to keep our own logic in here anymore. Our test cases already cover this. Change-Id: Ied3043ee651c8aafa10271c1e1ca5d460fb6c0b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9269 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-04 r/6550 refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crateFlorian Klink4-367/+55
This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-11T13·34+0000