depot/tvix/castore/src/import.rs, branch refs/r/8785 monorepo for the virus lounge http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F8785 2024-04-20T14:14:19+00:00 refactor(tvix/castore/import): make module, split off fs and error 2024-04-20T14:14:19+00:00 Florian Klink flokli@flokli.de 2024-04-20T12:01:49+00:00 urn:sha1:e9db0449e700154baee1470f914c3f09089442d0 Move error types and filesystem-specific functions to a separate file, and keep the fs:: namespace in public exports. Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> refactor(tvix/castore/import): rename ingest_entries arg 2024-04-20T14:14:19+00:00 Florian Klink flokli@flokli.de 2024-04-20T12:00:43+00:00 urn:sha1:c4cb099823dbd20f673b870b47e4fb27af6c139c This is not a stream of direntries anymore, but a stream of ingestion entries. Change-Id: I387f4497b6567066b24c58ca0262e710348180e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI refactor(tvix/castore): generalize store ingestion streams 2024-04-19T20:37:05+00:00 Connor Brewster cbrewster@hey.com 2024-04-18T18:51:28+00:00 urn:sha1:259d7a3cfa214e7eab7b0862024d595489e92592 Previously the store ingestion code was coupled to `walkdir::DirEntry`s produced by the `walkdir` crate which made it impossible to reuse ingesting from other sources like tarballs or NARs. This introduces a `IngestionEntry` which carries enough information for store ingestion and a future for computing the Blake3 digest of files. This allows the producer to perform file uploads in a way that makes sense for the source, ie. the filesystem upload could concurrently upload multiple files at the same time, while the NAR ingestor will need to ingest the entire blob before yielding the next blob in the stream. In the future we can buffer small blobs and upload them concurrently, but the full blob still needs to be read from the NAR before advancing. Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI refactor(tvix/castore): relax trait bounds on BlobService 2024-04-15T14:47:12+00:00 Florian Klink flokli@flokli.de 2024-04-13T22:02:07+00:00 urn:sha1:f1349caf3f6128e53880ef7829258de27c0a4481 We don't need to clone BlobService anymore. Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> refactor(tvix/castore/import): restructure directory uploader a bit 2024-04-15T14:47:12+00:00 Florian Klink flokli@flokli.de 2024-04-13T16:57:34+00:00 urn:sha1:6ebaa7b88a25636ee90f21b3a42a3674c98d00ef Have a Option<Box<dyn DirectoryPutter>>, which is lazily initialized whenever we first want to upload a directory. Have the loop explicitly break when it encounters the root_node, and deal with the flushing after the loop. Deal with the FUTUREWORK (assertion for root directory digest matching what the DirectoryPutter returns). Change-Id: Iefc4904d8b8387e868fb752d40e3e4e4218c7407 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11417 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> refactor(tvix/castore/import): put invariant checker into a .inspect() 2024-04-15T14:37:35+00:00 Florian Klink flokli@flokli.de 2024-04-13T16:30:47+00:00 urn:sha1:c088123d4e8c0b49cf0af568426d6d146a07f44c Separate this a bit stronger from the main application flow. Change-Id: I2e9bd3ec47cc6e37256ba6afc6e0586ddc9a051f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11416 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com> refactor(tvix/*/import): rename direntry_stream, entries_per_depths 2024-04-15T14:06:50+00:00 Florian Klink flokli@flokli.de 2024-04-13T16:22:37+00:00 urn:sha1:b70744fda627594cd42f8b6ae2d0dd1d7d037d61 Align these names and comments with the two users, to make it more obvious we're doing the same thing here, just use a different method to come up with entries_per_depths. Change-Id: I42058e397588b6b57a6299e87183bef27588b228 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11415 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> refactor(tvix/castore/import): inline process_entry 2024-04-15T14:05:49+00:00 Florian Klink flokli@flokli.de 2024-04-13T16:17:58+00:00 urn:sha1:bcc00fba8f65c767f831ab05f15278d23ea4b322 This did very little, and especially the part of relying on the outside caller to pass in a Directory if the type is a directory required having per-entry-type specific logic anyways. It's cleaner to just inline it. Change-Id: I997a8513ee91c67b0a2443cb5cd9e8700f69211e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11414 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> refactor(tvix/castore/import): move process_entry to the end of the file 2024-04-15T14:05:18+00:00 Florian Klink flokli@flokli.de 2024-04-13T15:58:16+00:00 urn:sha1:d47bd4f4bc7f650bbbc1b4a695434fe79b993b15 This makes it easier to understand the code. Change-Id: I0a9047433000551a6ba1f50a8c5c93527bc86216 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11413 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> feat(tvix/castore/import): remove copying in find_ancestor 2024-04-13T13:20:29+00:00 Florian Klink flokli@flokli.de 2024-04-13T12:53:14+00:00 urn:sha1:b70e01a4dbf3a4057a38593f6657893a1f9e5120 We don't need to copy if we explicitly say that the returned Option<Path> may hold onto bytes from the passed in &DirEntry. Change-Id: Ib46b6fd2f8f19a45f8bef79c4c1d2fa6b490cad7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11410 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<feed xmlns="http://www.w3.org/2005/Atom">
<title>depot/tvix/castore/src/import.rs, branch refs/r/8785</title>
<subtitle>monorepo for the virus lounge</subtitle>
<id>http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F8785</id>
<link rel="self" href="http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F8785"/>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/"/>
<updated>2024-04-20T14:14:19+00:00</updated>
<entry>
<title>refactor(tvix/castore/import): make module, split off fs and error</title>
<updated>2024-04-20T14:14:19+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-20T12:01:49+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=e9db0449e700154baee1470f914c3f09089442d0"/>
<id>urn:sha1:e9db0449e700154baee1470f914c3f09089442d0</id>
<content type="text"> Move error types and filesystem-specific functions to a separate file, and keep the fs:: namespace in public exports. Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>refactor(tvix/castore/import): rename ingest_entries arg</title>
<updated>2024-04-20T14:14:19+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-20T12:00:43+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=c4cb099823dbd20f673b870b47e4fb27af6c139c"/>
<id>urn:sha1:c4cb099823dbd20f673b870b47e4fb27af6c139c</id>
<content type="text"> This is not a stream of direntries anymore, but a stream of ingestion entries. Change-Id: I387f4497b6567066b24c58ca0262e710348180e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>refactor(tvix/castore): generalize store ingestion streams</title>
<updated>2024-04-19T20:37:05+00:00</updated>
<author>
<name>Connor Brewster</name>
<email>cbrewster@hey.com</email>
</author>
<published>2024-04-18T18:51:28+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=259d7a3cfa214e7eab7b0862024d595489e92592"/>
<id>urn:sha1:259d7a3cfa214e7eab7b0862024d595489e92592</id>
<content type="text"> Previously the store ingestion code was coupled to `walkdir::DirEntry`s produced by the `walkdir` crate which made it impossible to reuse ingesting from other sources like tarballs or NARs. This introduces a `IngestionEntry` which carries enough information for store ingestion and a future for computing the Blake3 digest of files. This allows the producer to perform file uploads in a way that makes sense for the source, ie. the filesystem upload could concurrently upload multiple files at the same time, while the NAR ingestor will need to ingest the entire blob before yielding the next blob in the stream. In the future we can buffer small blobs and upload them concurrently, but the full blob still needs to be read from the NAR before advancing. Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>refactor(tvix/castore): relax trait bounds on BlobService</title>
<updated>2024-04-15T14:47:12+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T22:02:07+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=f1349caf3f6128e53880ef7829258de27c0a4481"/>
<id>urn:sha1:f1349caf3f6128e53880ef7829258de27c0a4481</id>
<content type="text"> We don't need to clone BlobService anymore. Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>refactor(tvix/castore/import): restructure directory uploader a bit</title>
<updated>2024-04-15T14:47:12+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T16:57:34+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=6ebaa7b88a25636ee90f21b3a42a3674c98d00ef"/>
<id>urn:sha1:6ebaa7b88a25636ee90f21b3a42a3674c98d00ef</id>
<content type="text"> Have a Option<Box<dyn DirectoryPutter>>, which is lazily initialized whenever we first want to upload a directory. Have the loop explicitly break when it encounters the root_node, and deal with the flushing after the loop. Deal with the FUTUREWORK (assertion for root directory digest matching what the DirectoryPutter returns). Change-Id: Iefc4904d8b8387e868fb752d40e3e4e4218c7407 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11417 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> </content>
</entry>
<entry>
<title>refactor(tvix/castore/import): put invariant checker into a .inspect()</title>
<updated>2024-04-15T14:37:35+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T16:30:47+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=c088123d4e8c0b49cf0af568426d6d146a07f44c"/>
<id>urn:sha1:c088123d4e8c0b49cf0af568426d6d146a07f44c</id>
<content type="text"> Separate this a bit stronger from the main application flow. Change-Id: I2e9bd3ec47cc6e37256ba6afc6e0586ddc9a051f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11416 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com> </content>
</entry>
<entry>
<title>refactor(tvix/*/import): rename direntry_stream, entries_per_depths</title>
<updated>2024-04-15T14:06:50+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T16:22:37+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=b70744fda627594cd42f8b6ae2d0dd1d7d037d61"/>
<id>urn:sha1:b70744fda627594cd42f8b6ae2d0dd1d7d037d61</id>
<content type="text"> Align these names and comments with the two users, to make it more obvious we're doing the same thing here, just use a different method to come up with entries_per_depths. Change-Id: I42058e397588b6b57a6299e87183bef27588b228 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11415 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> </content>
</entry>
<entry>
<title>refactor(tvix/castore/import): inline process_entry</title>
<updated>2024-04-15T14:05:49+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T16:17:58+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=bcc00fba8f65c767f831ab05f15278d23ea4b322"/>
<id>urn:sha1:bcc00fba8f65c767f831ab05f15278d23ea4b322</id>
<content type="text"> This did very little, and especially the part of relying on the outside caller to pass in a Directory if the type is a directory required having per-entry-type specific logic anyways. It's cleaner to just inline it. Change-Id: I997a8513ee91c67b0a2443cb5cd9e8700f69211e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11414 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> </content>
</entry>
<entry>
<title>refactor(tvix/castore/import): move process_entry to the end of the file</title>
<updated>2024-04-15T14:05:18+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T15:58:16+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=d47bd4f4bc7f650bbbc1b4a695434fe79b993b15"/>
<id>urn:sha1:d47bd4f4bc7f650bbbc1b4a695434fe79b993b15</id>
<content type="text"> This makes it easier to understand the code. Change-Id: I0a9047433000551a6ba1f50a8c5c93527bc86216 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11413 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> </content>
</entry>
<entry>
<title>feat(tvix/castore/import): remove copying in find_ancestor</title>
<updated>2024-04-13T13:20:29+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2024-04-13T12:53:14+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=b70e01a4dbf3a4057a38593f6657893a1f9e5120"/>
<id>urn:sha1:b70e01a4dbf3a4057a38593f6657893a1f9e5120</id>
<content type="text"> We don't need to copy if we explicitly say that the returned Option<Path> may hold onto bytes from the passed in &DirEntry. Change-Id: Ib46b6fd2f8f19a45f8bef79c4c1d2fa6b490cad7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11410 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI </content>
</entry>
</feed>