about summary refs log tree commit diff
path: root/tvix/store/src/bin
AgeCommit message (Collapse)AuthorFilesLines
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster1-9/+4
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-20 r/6622 refactor(tvix/store/fs): Separate FUSE and filesystem codeConnor Brewster1-2/+7
In prepration for adding virtiofs support, I thought it would make sense to split out the filesystem implementation from FUSE itself. The `fs` module holds the tvix-store filesystem implemetation and the `fuse` module holds the code to spawn a FUSE daemon backed by multiple threads. Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 r/6621 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rsConnor Brewster1-14/+18
This switches the FUSE implementation from fuser to fuse-backend-rs. fuse-backend-rs is designed to work with both FUSE and virtiofs. Virtiofs support will make it possible to plug the tvix-store into a microvm and have `/nix/store` access without having to setup FUSE inside the guest. Additionally fuse-backend-rs has nice support for running multiple FUSE threads and has some async support. The goal of this commit is to mechanically switch over to fuse-backend-rs with minimal changes. I did have to add some locks here and there because fuse-backend-rs uses `&self` on all methods whereas fuser uses `&mut self`. `&self` is required for concurrent access to the FUSE server, so this makes sense. We can consider switching to concurrent maps and use some other techniques to reduce lock contention and critical section size. Issue: https://b.tvl.fyi/issues/305 Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-18 r/6606 refactor(tvix/store/blobsvc): make BlobStore asyncFlorian Klink1-5/+16
We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-15 r/6595 feat(tvix/store/import): print store path to stdoutFlorian Klink1-2/+13
This allows comparing the output in an integration test. Change-Id: I8bb2254e18e90005a4f1b30fd47ef69642e3732e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9337 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 r/6594 fix(tvix/store): log to stderr, not stdoutFlorian Klink1-4/+5
This leaves some space in stdout to provide actual meaningful output. Also, rename print_node to log_node because that's what it does, it's using the logger to log out more detailed info. Change-Id: Ic64a6330dbfcdc63eb4198067a5c5e47b841b9a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9336 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-09-08 r/6568 docs(tvix/store/fuse): add more comment to list_root argFlorian Klink1-0/+2
Forgot to squash this into cl/9217. Change-Id: Id5a87cbe729592f3b53e8c329b6890519f5da319 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9272 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-05 r/6557 feat(tvix/store): fix ctrl-c handling on mount commandFlorian Klink1-3/+24
This enables the tokio `signal` feature, and registers a ctrl_c signal handler, which will use the unmount handle to unmount in case a ctrl-c signal is received. This avoids having disconnected mountpoints when Ctrl-C'ing a `tvix-store mount` invocation. In case the filesystem is unmounted externally (via `umount /path/to/ mountpoint`), the future is waiting for the signal is never resolved and the task is stopped. Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-05 r/6556 feat(tvix/store/fuse): allow listingFlorian Klink1-1/+11
This provides an additional configuration flag to the tvix-store mount subcommand, and logic in the fuse module to request listing for the root of the mountpoint. Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: adisbladis <adisbladis@gmail.com> Tested-by: BuildkiteCI
2023-09-05 r/6554 refactor(tvix/store): infer more typesFlorian Klink1-51/+48
We don't need to explicitly describe the type of the task itself, describing the return type of the async closure is sufficient. Also, use io::Result<_> instead of Result<_, io::Error>. Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-04 r/6550 refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crateFlorian Klink1-12/+55
This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-02 r/6545 docs(tvix/store): update commentFlorian Klink1-1/+1
Since the refactor to use URIs for all three services, this actually does talk to a daemon by default. Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink1-4/+4
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-06-14 r/6305 refactor(tvix/store/blobsvc): make from_addr not asyncFlorian Klink1-3/+3
A previous iteration of this code did actually connect (in the gRPC client), which was why we had this function async. However, as the connection there is now lazy too, we can drop the asyncness in this function. Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 r/6304 refactor(tvix/store/pathinfosvc): add from_addrFlorian Klink1-17/+28
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6303 feat(tvix/store/directorysvc): add from_addrFlorian Klink1-20/+22
Add --directory-service-addr arg to tvix-store CLI. Change-Id: Iea1e6f08f27f7157b21ccf397297c68358bd78a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8743 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 r/6302 feat(tvix/store/blobsvc): add from_addrFlorian Klink1-17/+29
This allows constructing blob stores with a URL syntax at runtime, by passing the --blob-service-addr arg. We probably still want to have some builder pattern here, to allow additional schemes to be registered. Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink1-4/+5
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6275 refactor(tvix/store/fuse): use Arc<dyn …> instead of genericsFlorian Klink1-1/+5
Change-Id: I5685379bd6f89d17da6843d31bef4c1fc4dc0a18 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8745 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6273 refactor(tvix/store): use Arc instead of BoxFlorian Klink1-12/+10
This allows us to blob services without closing them before putting them in a box. We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers require Sync. Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink1-3/+6
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink1-9/+11
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink1-13/+6
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6269 feat(tvix/store): eliminate generics in BlobStoreFlorian Klink1-3/+4
To construct various stores at runtime, we need to eliminate associated types from the BlobService trait, and return Box<dyn …> instead of specific types. This also means we can't consume self in the close() method, so everything we write to is put in an Option<>, and during the first close we take from there. Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-10 r/6263 feat(tvix/store): add mount command to entrypoint and fuse modFlorian Klink1-0/+26
`tvix-store mount PATH` will mount the tvix-store to the given path. Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-30 r/6223 feat(tvix/store/bin): use sled for daemon, grpc for import cmdFlorian Klink1-8/+23
This now creates different store client, depending on the cli subcommand. The `import` command will connect to the gRPC service, and the `daemon` command will use the sled implementation. It might make sense to define some URI syntax to make this configurable by the user, via the CLI. Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 r/6221 feat(tvix/store/bin/import): process all path imports concurrentlyFlorian Klink1-36/+47
Change-Id: I3e1428a4725fc2e552e8f37bc0550121117fcef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8633 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 r/6206 refactor(tvix/store/bin): instantiating TvixStoreIO onceFlorian Klink1-9/+11
Instead of instantiating it once in every loop iteration, put it in an Arc, and clone that before passing it to the spawned task. Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 r/6198 refactor(tvix/store/bin): use std::ioFlorian Klink1-3/+4
Change-Id: I3f3dc9732d90790d92268c04c75eccbe92e7e05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8634 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-25 r/6197 fix(tvix/store/bin): use spawn_blocking to call import_pathFlorian Klink1-10/+14
This operation is blocking, so it should be run inside a blocking tokio task. Tokio panics if it detects a blocking operation inside a non- blocking task, so cl/8619 would cause it to panic (as the GRPC clients use spawn_blocking under the hood). As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614), we create a new instance of TvixStoreIO inside each loop iteration. Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6191 feat(tvix/store): implement TvixStoreIOFlorian Klink1-39/+15
This providesEvalIO, asking given PathInfoService, DirectoryService and BlobService. Change-Id: I32f210f5a7aa8173ad9a7d53e8a5ac03619f527a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8561 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6177 refactor(tvix/store/directorysvc): move from Vec<u8> to B3DigestFlorian Klink1-1/+2
This introduces a new struct, B3Digest, which internally holds a Vec<u8>, but only allows construction with 32 bytes. It also implements display, which will print the base64 representation. This should reduce some boilerplate when parsing Vec<u8>. Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-17 r/6149 refactor(tvix/store): rename import::{import_path -> ingest_path}Florian Klink1-2/+2
This distinguishes it better from the EvalIO::import_path method. Also update the docstring to explain what it does (and what it doesn't). Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-11 r/6133 refactor(tvix/store): remove ChunkServiceFlorian Klink1-12/+2
Whether chunking is involved or not, is an implementation detail of each Blobstore. Consumers of a whole blob shouldn't need to worry about that. It currently is not visible in the gRPC interface either. It shouldn't bleed into everything. Let the BlobService trait provide `open_read` and `open_write` methods, which return handles providing io::Read or io::Write, and leave the details up to the implementation. This means, our custom BlobReader module can go away, and all the chunking bits in there, too. In the future, we might still want to add more chunking-aware syncing, but as a syncing strategy some stores can expose, not as a fundamental protocol component. This currently needs "SyncReadIntoAsyncRead", taken and vendored in from https://github.com/tokio-rs/tokio/pull/5669. It provides a AsyncRead for a sync Read, which is necessary to connect our (sync) BlobReader interface to a GRPC server implementation. As an alternative, we could also make the BlobReader itself async, and let consumers of the trait (EvalIO) deal with the async-ness, but this is less of a change for now. In terms of vendoring, I initially tried to move our tokio crate to these commits, but ended up in version incompatibilities, so let's vendor it in for now. Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-19 r/6028 feat(tvix/store/bin): print store pathFlorian Klink1-0/+38
After ingestion of the contents into the store, this will use the NonCachingNARCalculationService to create a NAR stream or the contents of the path, and use our Derivation output path calculation machinery to determine the output path (using recursive hashing strategy). In a real-world scenario, we obviously want to cache these calculations, but this should be sufficient to tinker around with it. Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-12 r/5956 chore(tvix/store): allow(unused_mut)Florian Klink1-0/+1
When building tvix-store without default features, this variable doesn't need to be mutable. Silence the warning. Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5937 feat(tvix/store): move to daemon subcommand, add import subcommandFlorian Klink1-45/+106
This exposes the previous default behavior at the `tvix-store daemon` subcommand. It also adds a `tvix-store import` command, which will ingest a given path into the store. Change-Id: Ide14f1d409b9364e7f98090690c744326486e470 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5936 refactor(tvix/store): move entrypoint to src/bin/tvix-store.rsFlorian Klink1-0/+106
Change-Id: Ibb83be75a2be27debd9e85b43c1b824f59e54dab Reviewed-on: https://cl.tvl.fyi/c/depot/+/8165 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-12T22·28+0000