about summary refs log tree commit diff
path: root/tvix/store/src/fuse
AgeCommit message (Collapse)AuthorFilesLines
2023-09-20 r/6622 refactor(tvix/store/fs): Separate FUSE and filesystem codeConnor Brewster5-2453/+0
In prepration for adding virtiofs support, I thought it would make sense to split out the filesystem implementation from FUSE itself. The `fs` module holds the tvix-store filesystem implemetation and the `fuse` module holds the code to spawn a FUSE daemon backed by multiple threads. Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 r/6621 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rsConnor Brewster4-327/+470
This switches the FUSE implementation from fuser to fuse-backend-rs. fuse-backend-rs is designed to work with both FUSE and virtiofs. Virtiofs support will make it possible to plug the tvix-store into a microvm and have `/nix/store` access without having to setup FUSE inside the guest. Additionally fuse-backend-rs has nice support for running multiple FUSE threads and has some async support. The goal of this commit is to mechanically switch over to fuse-backend-rs with minimal changes. I did have to add some locks here and there because fuse-backend-rs uses `&self` on all methods whereas fuser uses `&mut self`. `&self` is required for concurrent access to the FUSE server, so this makes sense. We can consider switching to concurrent maps and use some other techniques to reduce lock contention and critical section size. Issue: https://b.tvl.fyi/issues/305 Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-18 r/6607 fix(tvix/store/fuse): fix executable bitFlorian Klink2-1/+38
We were blindly returning 0o444 for all regular files, but regular files with executable bit need to be 0o555. This wasn't spotted because stat'ing executable files was not part of the test suite, it's now added. Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 r/6606 refactor(tvix/store/blobsvc): make BlobStore asyncFlorian Klink2-170/+409
We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-05 r/6556 feat(tvix/store/fuse): allow listingFlorian Klink2-3/+100
This provides an additional configuration flag to the tvix-store mount subcommand, and logic in the fuse module to request listing for the root of the mountpoint. Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: adisbladis <adisbladis@gmail.com> Tested-by: BuildkiteCI
2023-09-03 r/6548 docs(tvix/store): address cargo doc warningsFlorian Klink1-3/+3
Fix some broken link references. Change-Id: I69c9b2b62af35bb777e4df1a01ba3181a368be47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9214 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-08-20 r/6500 refactor(tvix/store): cargo clippyFlorian Klink2-15/+14
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink3-9/+9
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-07-21 r/6437 feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>Florian Klink1-2/+2
This will save us some copies, because a clone will simply create an additional pointer to the same data. Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink4-23/+23
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-07-21 r/6435 feat(tvix/store/fuse): implement open explicitlyFlorian Klink1-31/+81
This "reverts" commit 9f600de22671ee1f88e6fb9e53a5a385b434871b (the initial revert of f5e291cf8328096d790f5416cf1968cb9164220a). Now with BlobService returning a BlobReader that implements io::Seek, we can actually just call blob_reader.seek(io::SeekFrom::Start(offset as u64)). This means, we currently will fail to seek backwards inside a file. Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 r/6433 feat(tvix/nix-compat): have StorePath accept bytesFlorian Klink1-1/+2
The primary constructor for this is now from_bytes, from_string is simply calling .as_bytes() on the string, passing it along. The InvalidName error now contains a Vec<u8>, to encode the invalid name (which might not be a string anymore). from_absolute_path now accepts a &[u8] (even though we might want to make this a OSString of some sort). StorePath::validate_name has been degraded to a pub(crate) function. It's still used in src/derivation, even though it probably shouldn't at all - that cleanup is left for cl/8412 though. Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-02 r/6385 fix(tvix/store/fuse): fix log level for a debug messageFlorian Klink1-1/+1
Change-Id: Ib801e46636901553d71455a739aed34e5828ca0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8888 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-02 r/6384 docs(tvix/store/fuse/inodes): clippy lintFlorian Klink1-2/+2
Change-Id: I7b1bcb9c0cca76f05271f25912d26b14152fe0c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8887 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 r/6371 feat(tvix/store/fuse): add test reading large fileFlorian Klink1-0/+59
Change-Id: Ic705d05909c59c764d68a730169e9cc2b2538d60 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8847 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 r/6368 fix(tvix/store/fuse): revert "implement open explicitly"Florian Klink1-72/+30
This reverts commit f5e291cf8328096d790f5416cf1968cb9164220a. The offsets are relative to the start of the file, and as long as we don't have BlobReaders implement seek, this will be very annoying to deal with. Change-Id: I05968f7c5c0ec0000597da90f451d6bb650c3e13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8882 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-30 r/6366 feat(tvix/store/fuse): implement open explicitlyFlorian Klink1-30/+72
This moves from stateless I/O to actually dealing with file handles, allowing the filesystem to keep reusing existing blobreaders, instead of opening a new reader on every read() call. Change-Id: I3fc35c071e4aee1021c8bbd58749d082b0abd188 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8834 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-20 r/6337 feat(tvix/store/fuse): initial implementationFlorian Klink5-2/+1804
This is a first implementation of a FUSE filesystem, mounting tvix-store to a given location. This is mostly meant as one additional lens into a store, and could be used for builds. It's not meant to be used as a general-purpose thing. It still has some rough edges: - It doesn't implement open/close, so it doesn't use file handles. Which means, we need to open blobs for partial reads over and over again. - It doesn't implement seek, as BlobReader doesn't implement seek yet. - It doesn't track "lifetimes" of inodes by listening on forget, meaning it might hold more data in memory than necessary. - As we don't have store composition (and a caching layer) yet, operations might be slow. Change-Id: Ib1812ed761dfaf6aeb548443ae939c87530b7be8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8667 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink1-5/+5
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6275 refactor(tvix/store/fuse): use Arc<dyn …> instead of genericsFlorian Klink1-10/+12
Change-Id: I5685379bd6f89d17da6843d31bef4c1fc4dc0a18 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8745 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-10 r/6263 feat(tvix/store): add mount command to entrypoint and fuse modFlorian Klink1-0/+24
`tvix-store mount PATH` will mount the tvix-store to the given path. Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>