depot/tvix/cli/src, branch refs/r/6611 monorepo for the virus lounge http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F6611 2023-09-18T10:33:30+00:00 refactor(tvix/store/blobsvc): make BlobStore async 2023-09-18T10:33:30+00:00 Florian Klink flokli@flokli.de 2023-09-13T12:20:21+00:00 urn:sha1:da6cbb4a459d02111c44a67d3d0dd7e654abff23 We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate 2023-09-04T16:02:12+00:00 Florian Klink flokli@flokli.de 2023-09-02T18:16:35+00:00 urn:sha1:3c340b28bdc812b9c13393c9e2bb378ba7dd9ec8 This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> refactor(tvix/cli): cargo clippy 2023-08-20T21:43:02+00:00 Florian Klink flokli@flokli.de 2023-08-19T15:13:43+00:00 urn:sha1:146175b83d8a5422810ed628d46d00656ab7f95d Change-Id: I723bc9b29733f5ee4e3094b92696fbdeb4b15a33 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9094 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI refactor(tvix/cli): stop parsing NixHash twice 2023-08-20T19:44:19+00:00 Florian Klink flokli@flokli.de 2023-08-19T21:33:31+00:00 urn:sha1:9cb38c5ba5e51e8843ee2ab210842701c5a6d7ce We already have the parsed output_hash from above, no need to construct it again. Change-Id: Ie6d924ab446137c25c29fbeaf671aa7e5418262d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9110 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI chore(tvix/cli): delete unused NixCompatIO code 2023-08-14T11:50:00+00:00 sterni sternenseemann@systemli.org 2023-08-14T11:36:45+00:00 urn:sha1:08a5849e2b3efe141ec039130d2d6280a6f45f05 Change-Id: Icb91b102208fea512e04383ce9d65b0681af18ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9079 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> refactor(tvix/cli/refscan): use wu-manber crate with &[u8] support 2023-07-31T21:50:47+00:00 Florian Klink flokli@flokli.de 2023-07-29T22:38:55+00:00 urn:sha1:737a6ca01e3ee017834fd310cb942079b5d10446 PR'ed at https://github.com/tvlfyi/wu-manber/pull/1, and now merged. Change-Id: I8c71e359196396a1d42a3ea2ab7ac15b137b2db0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8992 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> refactor(tvix/nix-compat): support non-unicode Derivations 2023-07-31T21:41:38+00:00 Florian Klink flokli@flokli.de 2023-07-29T19:14:44+00:00 urn:sha1:79531c3dab1c24ff3171c0aa067004c8e6c92e3f Derivations can have non-unicode strings in their env values, so the ATerm representations are not necessarily String anymore, but Vec<u8>. Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> docs(tvix/cli/refscan): fix comment 2023-07-31T09:01:34+00:00 Florian Klink flokli@flokli.de 2023-07-29T22:36:21+00:00 urn:sha1:0dea3086f5ed15191d01e499d55e6edfa73805ee The comment still mentions Aho-Corasick, which isn't correct. Change-Id: I846a2ca9ea7075c2456ca6ef04a132ff6161227a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8991 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> feat(tvix/store/proto): use Bytes instead of Vec<u8> 2023-07-22T09:03:02+00:00 Florian Klink flokli@flokli.de 2023-07-19T15:52:50+00:00 urn:sha1:432222f098bfceb033e63e9a63687e35574457f9 Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> refactor(tvix/store): use bytes for node names and symlink targets 2023-07-21T19:01:49+00:00 Florian Klink flokli@flokli.de 2023-07-18T16:37:25+00:00 urn:sha1:72e82ffcb11b1aaf1f1cc8db4189ced5ec0aa42e Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<feed xmlns="http://www.w3.org/2005/Atom">
<title>depot/tvix/cli/src, branch refs/r/6611</title>
<subtitle>monorepo for the virus lounge</subtitle>
<id>http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F6611</id>
<link rel="self" href="http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F6611"/>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/"/>
<updated>2023-09-18T10:33:30+00:00</updated>
<entry>
<title>refactor(tvix/store/blobsvc): make BlobStore async</title>
<updated>2023-09-18T10:33:30+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-09-13T12:20:21+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=da6cbb4a459d02111c44a67d3d0dd7e654abff23"/>
<id>urn:sha1:da6cbb4a459d02111c44a67d3d0dd7e654abff23</id>
<content type="text"> We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> </content>
</entry>
<entry>
<title>refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate</title>
<updated>2023-09-04T16:02:12+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-09-02T18:16:35+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=3c340b28bdc812b9c13393c9e2bb378ba7dd9ec8"/>
<id>urn:sha1:3c340b28bdc812b9c13393c9e2bb378ba7dd9ec8</id>
<content type="text"> This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>refactor(tvix/cli): cargo clippy</title>
<updated>2023-08-20T21:43:02+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-08-19T15:13:43+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=146175b83d8a5422810ed628d46d00656ab7f95d"/>
<id>urn:sha1:146175b83d8a5422810ed628d46d00656ab7f95d</id>
<content type="text"> Change-Id: I723bc9b29733f5ee4e3094b92696fbdeb4b15a33 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9094 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>refactor(tvix/cli): stop parsing NixHash twice</title>
<updated>2023-08-20T19:44:19+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-08-19T21:33:31+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=9cb38c5ba5e51e8843ee2ab210842701c5a6d7ce"/>
<id>urn:sha1:9cb38c5ba5e51e8843ee2ab210842701c5a6d7ce</id>
<content type="text"> We already have the parsed output_hash from above, no need to construct it again. Change-Id: Ie6d924ab446137c25c29fbeaf671aa7e5418262d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9110 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>chore(tvix/cli): delete unused NixCompatIO code</title>
<updated>2023-08-14T11:50:00+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-08-14T11:36:45+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=08a5849e2b3efe141ec039130d2d6280a6f45f05"/>
<id>urn:sha1:08a5849e2b3efe141ec039130d2d6280a6f45f05</id>
<content type="text"> Change-Id: Icb91b102208fea512e04383ce9d65b0681af18ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9079 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>refactor(tvix/cli/refscan): use wu-manber crate with &[u8] support</title>
<updated>2023-07-31T21:50:47+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-29T22:38:55+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=737a6ca01e3ee017834fd310cb942079b5d10446"/>
<id>urn:sha1:737a6ca01e3ee017834fd310cb942079b5d10446</id>
<content type="text"> PR'ed at https://github.com/tvlfyi/wu-manber/pull/1, and now merged. Change-Id: I8c71e359196396a1d42a3ea2ab7ac15b137b2db0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8992 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> </content>
</entry>
<entry>
<title>refactor(tvix/nix-compat): support non-unicode Derivations</title>
<updated>2023-07-31T21:41:38+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-29T19:14:44+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=79531c3dab1c24ff3171c0aa067004c8e6c92e3f"/>
<id>urn:sha1:79531c3dab1c24ff3171c0aa067004c8e6c92e3f</id>
<content type="text"> Derivations can have non-unicode strings in their env values, so the ATerm representations are not necessarily String anymore, but Vec<u8>. Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>docs(tvix/cli/refscan): fix comment</title>
<updated>2023-07-31T09:01:34+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-29T22:36:21+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=0dea3086f5ed15191d01e499d55e6edfa73805ee"/>
<id>urn:sha1:0dea3086f5ed15191d01e499d55e6edfa73805ee</id>
<content type="text"> The comment still mentions Aho-Corasick, which isn't correct. Change-Id: I846a2ca9ea7075c2456ca6ef04a132ff6161227a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8991 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> </content>
</entry>
<entry>
<title>feat(tvix/store/proto): use Bytes instead of Vec<u8></title>
<updated>2023-07-22T09:03:02+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-19T15:52:50+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=432222f098bfceb033e63e9a63687e35574457f9"/>
<id>urn:sha1:432222f098bfceb033e63e9a63687e35574457f9</id>
<content type="text"> Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>refactor(tvix/store): use bytes for node names and symlink targets</title>
<updated>2023-07-21T19:01:49+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-18T16:37:25+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=72e82ffcb11b1aaf1f1cc8db4189ced5ec0aa42e"/>
<id>urn:sha1:72e82ffcb11b1aaf1f1cc8db4189ced5ec0aa42e</id>
<content type="text"> Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI </content>
</entry>
</feed>