about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-04-20 r/7987 refactor(tvix/castore): ingest filesystem entries in parallelFlorian Klink4-82/+53
Rather than carrying around an Future in the IngestionEntry::Regular, simply carry the plain B3Digest. Code reading through a non-seekable data stream has no choice but to read and upload blobs immediately, and code seeking through something seekable (like a filesystem) probably knows better what concurrency to pick when ingesting, rather than the consuming side. (Our only) one of these seekable source implementations is now doing exactly that. We produce a stream of futures, and then use [StreamExt::buffered] to process more than one, concurrently. We still keep the same order, to avoid shuffling things and violating the stream order. This also cleans up walk_path_for_ingestion in castore/import, as well as ingest_dir_entries in glue/tvix_store_io. Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-20 r/7986 fix(tvix): fix outdated comment and error in TvixStoreIO::openConnor Brewster1-3/+3
This function was originally called `read_to_string` but was changed to `open` to make it so that file contents aren't always held in memory. A comment and error message were not updated to reflect the new name of this method. Change-Id: I3d86e2f6d7006c2e1513121fc3c62efcb7e7b9bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11495 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7985 feat(tvix/castore): Fix build warnings in release modeConnor Brewster2-0/+2
Fixes some build warnings that only happen when building in release mode which disables `debug_assertions`. Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2024-04-20 r/7984 fix(tvix/castore): ensure all directories are present during ingestionConnor Brewster1-0/+8
`ingest_entries` requires that all directories referenced by entries in the ingestion stream have an explicit entry in the stream. For example, if the stream contains a file with path `foo/bar`, there must be an entry that comes later in the stream for the directory `foo`. Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488 Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-20 r/7983 feat(tvix/eval): Implement builtins.fetchTarballAspen Smith9-31/+305
Implement a first pass at the fetchTarball builtin. This uses much of the same machinery as fetchUrl, but has the extra complexity that tarballs have to be extracted and imported as store paths (into the directory- and blob-services) before hashing. That's reasonably involved due to the structure of those two services. This is (unfortunately) not easy to test in an automated way, but I've tested it manually for now and it seems to work: tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath => "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string Co-authored-by: Connor Brewster <cbrewster@hey.com> Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020 Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7982 feat(tvix/castore/import): only allow normal components in entry pathsFlorian Klink1-1/+10
Explicitly document and add a debug assertion for that. It's up to callers to ensure this doesn't happen. Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7981 refactor(tvix/castore/import): make module, split off fs and errorFlorian Klink7-197/+225
Move error types and filesystem-specific functions to a separate file, and keep the fs:: namespace in public exports. Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-20 r/7980 refactor(tvix/castore/import): rename ingest_entries argFlorian Klink1-6/+3
This is not a stream of direntries anymore, but a stream of ingestion entries. Change-Id: I387f4497b6567066b24c58ca0262e710348180e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7979 refactor(tvix/castore): generalize store ingestion streamsConnor Brewster5-258/+256
Previously the store ingestion code was coupled to `walkdir::DirEntry`s produced by the `walkdir` crate which made it impossible to reuse ingesting from other sources like tarballs or NARs. This introduces a `IngestionEntry` which carries enough information for store ingestion and a future for computing the Blake3 digest of files. This allows the producer to perform file uploads in a way that makes sense for the source, ie. the filesystem upload could concurrently upload multiple files at the same time, while the NAR ingestor will need to ingest the entire blob before yielding the next blob in the stream. In the future we can buffer small blobs and upload them concurrently, but the full blob still needs to be read from the NAR before advancing. Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7978 feat(tvix/castore): add convenience `add` method to DirectoryConnor Brewster2-1/+166
This adds `Directory::add` which is a convenience helper for adding nodes into a `Directory` while preserving sorted order. This implements `Ord` and `PartialOrd` for `FileNode`, `SymlinkNode`, and `DirectoryNode` so `binary_search` can be used. Change-Id: I94b86bdef5d0da55aa352e098988b9704cafca19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11481 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-19 r/7977 chore(tvix/glue): migrate from test_case to rstestFlorian Klink5-234/+218
Change-Id: Id1bac94e7b95960dabd55f095f04bdabfeee10ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/11476 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7976 chore(tvix/cli): remove unused dependencyFlorian Klink3-10/+0
Change-Id: I6a6cabde22d28bb9afc391b9bcd896a999090366 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11475 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7975 chore(nix-compat): remove unused test-case and test-generatorFlorian Klink5-318/+133
This is especially nice, as it allows us getting rid of the other versions of proc-macro2 and quote. Change-Id: I9fdd012ee6c0ded3e18ec30504b83ac2032d1390 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11474 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7974 chore(nix-compat/drv): migrate from test_{case,resources} to rstestFlorian Klink1-76/+88
Also, replace the useless read_file helper with std::fs::read. Change-Id: I704ab42d1ffb9089716533125ff390b7441a8591 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11473 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-19 r/7973 chore(nix-compat/derivation/parser): migrate from test_case to rstestFlorian Klink1-23/+37
Change-Id: I7c29e7d9e2b9cd66483ed44a9f2b3d291061f81d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11472 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-19 r/7972 chore(nix-compat/nixhash): migrate from test_case to rstestFlorian Klink1-12/+14
Change-Id: I053f2c7fc9fdf59de5d8697eab32644b860e40be Reviewed-on: https://cl.tvl.fyi/c/depot/+/11471 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-19 r/7971 chore(nix-compat/aterm): migrate from test_case to rstestFlorian Klink2-22/+30
Change-Id: If05fa3d0f41d45e3574e0c5ddff406cb33dc52f4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11470 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-19 r/7970 chore(tvix/nix-compat/store_path): migrate from test_case to rstestFlorian Klink1-16/+20
Change-Id: Ic466a27d61b95ca4d297abd6eb976c083e8b40af Reviewed-on: https://cl.tvl.fyi/c/depot/+/11469 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7969 chore(nix-compat/nixbase32): migrate from test_case to rstestFlorian Klink1-14/+17
Change-Id: I9ed5e728ff0706a37d0a238210a8dd23eb306033 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11468 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-19 r/7968 chore(nix-compat/narinfo): migrate from test_case to rstestFlorian Klink2-33/+42
Change-Id: Ibd3cd1bc0d9deaf018a66748cea56af51451703c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11467 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-19 r/7967 chore(tvix/build): migrate from test_case to rstestFlorian Klink5-39/+41
Change-Id: I75d8d61f836c76e8765e0e3b49022c056de84850 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11466 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7966 chore(tvix/store): remove test-case dependencyFlorian Klink3-7/+2
Change-Id: I62c3b93af438fd6397ded85dd4fe5acc01e0093a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11465 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-19 r/7965 chore(tvix/store): migrate import.rs and tests/pathinfo.rs to rstestFlorian Klink5-87/+72
Also, rename the DUMMY_NAME constant in the fixtures to DUMMY_PATH, which aligns more with the ToString representation and from_bytes conversions we have on StorePath[Ref]. Change-Id: I39763c9dfa84c5d86f2fd0171b3a4d36fd72f267 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11464 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-19 r/7964 chore(tvix/store): migrate from test_case to rstestFlorian Klink1-43/+37
Change-Id: Id673cd9c574b00a739de78aab2ee86f9d9b1ed40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11463 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-18 r/7955 feat(tvix/nix-compat/nix_daemon/version): add ProtocolVersionFlorian Klink4-22/+156
This provides a nice wrapper struct to deal with versions. Change-Id: I6acc03bc9f8d84a0583196073b52776c45d3fe92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11454 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-18 r/7954 feat(tvix/nix-compat): enable more features by defaultFlorian Klink1-0/+3
There's no point in not enabling these, and rust-analyzer is quite useless too. nix-compat users outside of here can still explicitly disable default features if they don't want to pull in these features. Change-Id: I8f2f24e7734ad152d1733bc3421098fc91af290a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11455 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-16 r/7951 feat(tvix/boot): boot nixos VMFlorian Klink1-0/+34
This makes use of the vmCmdline and assertVMOutput to boot a NixOS VM off of `tvix-store virtiofs` have it write a message on successful bootup, then have it shut down again. We check we see the message in the output. Seeding the store with the closure it's what's currently taking most of the time (as it's all linear still), so log a message before booting the VM. Change-Id: I6887d935f589eb0ad6507033c01337e1ec500a0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11191 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7950 fix(tvix/castore/blobservice/grpc): don't use NaiveSeeker for nowFlorian Klink1-9/+17
Userland likes to seek backwards, and until we have store composition and can serve chunks from a local cache, we need to buffer the individual chunks in memory. Change-Id: I66978a0722d5f55ed4a9a49d116cecb64a01995d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11448 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7949 fix(tvix/castore/src): record rq.handle field in read()Florian Klink1-1/+1
This makes it easier to separate concurrent requests on the same inode. Change-Id: I7637c1d889336beeb0d186182ce22fbf60fd16c3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11447 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7948 fix(tvix/castore/fs): use io::copy to fill kernel-provided bufferFlorian Klink1-3/+12
The docs state we must fill all of the buffer, except on EOF. Change-Id: Id977ba99c0b15132422474ebbf82bb92b79d55ba Reviewed-on: https://cl.tvl.fyi/c/depot/+/11446 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7947 feat(tvix/castore/blob/naive_seeker): add some more tracingFlorian Klink1-10/+17
Change-Id: Iecf4a82a7d84008a8620825570b34e9094e6d590 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11445 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7946 feat(tvix/castore/blob/chunked_reader): add some more tracesFlorian Klink1-2/+8
Change-Id: I2408707a7bc0e1c0cd8bd2933f8d68805b9e12c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11444 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7945 fix(tvix/castore/blobservice/chunk_rd): only skip *first* chunk bytesFlorian Klink1-3/+4
When (re)initializing a chunked reader, we were erroneously skipping the first n bytes from all chunks, not just the first one. Fix this, by passing in an enumerated list of chunks, and only calling SeekFrom::Start() on the first chunk in the stream. With this, I'm able to invoke b3sum on bin/bash successfully. Change-Id: I52ea480569267e093b0ac9d6bcd5c2d1b4db25f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11443 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7944 refactor(tvix/castore/blob/naive_seeker): rework skipping for clarityFlorian Klink1-42/+35
Increase the discard_buf to 4096 (as I've seen this size). Use the ready! macro to propagate pendings. Make it more clear what exactly should be skipped in total, and what during the current iteration. Also write down that poll_read call already takes care of updating self.pos, as I ran into that trap earlier (and added it here). Change-Id: I2d22e1c8a835c0f3dd0c648917009b2bad4fd57c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11442 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7943 feat(tvix/castore/blob/chunked_reader): only reassemble on real seekFlorian Klink1-16/+16
If the resulting offset equals to our current position, there's no need to recreate a reader. Change-Id: I855f0c79c514c16ca48a78e12978af2835fbbd6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11441 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-15 r/7938 docs(tvix/castore/blobservice): explain open_read for small blobs moreFlorian Klink1-0/+1
State that this case applies if the blob is small enough to fit inside a single chunk. Change-Id: I0383514729e686799599b629cf1303b284147bb4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11440 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7937 feat(tvix/castore/blobs/object_store): chunks() method for small blobsFlorian Klink1-4/+37
We previously returned Ok(None) when being asked for more granular chunking info, signalling the blob does not exist at all. This is however incorrect, we should return an empty Vec instead, as documented in the trait. Change-Id: I83ecc2027e0767134c7598792c2ee6d964853c66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11439 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7936 fix(tvix/castore/blob/object_store): tweak log levelsFlorian Klink1-3/+3
Don't log with info! here, bug debug!. Change-Id: I57bd5f2a45276090b893a4051fd175e3948ddfa4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11438 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7935 fix(tix/castore/blobservice): don't warn if chunk list is emptyFlorian Klink1-4/+1
It's perfectly normal if we ask for more granular chunking info and the backend responds it does not have it. Change-Id: I593ab3e53b4f4e70c99f39b266546d2ac8eb10c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11437 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7934 feat(tvix/castore/blob/grpc_wrapper): add blob.digest fieldFlorian Klink1-2/+3
We're receiving bytes over the wire, and encode them the same way B3Digest does internally, but don't use it for formatting, as we're discarding that string. In case the sent bytes don't have the right length, the string will be short, but it's better to still have it as a field, even if it's not a valid b3 digest. Change-Id: I6ef08275d51c8a0d98f5e46844b15dfd05d17cd8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11436 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7932 refactor(tvix/castore/fs): add parenthesis for readabilityFlorian Klink1-4/+4
As suggested in cl/11426. Change-Id: Ic2bb8cf2838bf0be09fb8bc62b8e598a3d153699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11434 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7931 refactor(tvix/castore/fs): remove From<Node> for InodeDataFlorian Klink2-107/+76
These were copying unnecessarily. Instead, have a InodeData::from_node(), which *consumes* the Node entirely, returns `InodeData` and the split-off name (which is not part of InodeData). Callers can then use the result in various helper functions, like: - InodeData::as_fuse_type - InodeData::as_fuse_file_attr - InodeData::as_fuse_entry … to prepare their replies to the kernel. This removes not only a bunch of clones, but also a lot of copy-pasted code. Change-Id: Idbca5f25cc29e96c1f4c614b33dff2becb0a8738 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11435 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7930 fix(tvix/castore/blobs): reply to has() for chunksFlorian Klink1-1/+8
We allow reading individual chunks via open_read(), it's inconsistent if a has() would return Ok(false). Change-Id: Ie713d968172ccd2687d2e6e0dfef89ee152ef511 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11420 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-15 r/7929 refactor(tvix/castore): relax trait bounds on BlobServiceFlorian Klink1-2/+2
We don't need to clone BlobService anymore. Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7928 fix(tvix/castore/directory): fix graph traversalFlorian Klink4-90/+109
Use a proper graph library to ensure all nodes are reachable from the root. We had a bit of that handrolled during add(), as well as later, which had an annoying bug: Redundant nodes were omitted during insert, but when returning the list during finalize, we did not properly account they need to be introduced before their parents are sent. We now simply populate a petgraph DiGraph during insert (skipping inserting nodes we already saw), and use petgraph's DfsPostOrder to traverse the graph during finalize. If the number of returned indices equals the total number of nodes in the graph, all nodes are reachable from the root, we can consume the graph and return the nodes as a vec, in the same order as the traversal (and insertion). Providing a regression test for the initial bug is challenging, as the current code uses a bunch of HashSets. I manually tested ingesting a full NixOS closure using this mechanism (via gRPC, which exposes this problem, as it validates twice), and it now works. Change-Id: Ic1d5e3e981f2993cc08c5c6b60ad895e578326dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11418 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-15 r/7927 refactor(tvix/castore/import): restructure directory uploader a bitFlorian Klink1-17/+51
Have a Option<Box<dyn DirectoryPutter>>, which is lazily initialized whenever we first want to upload a directory. Have the loop explicitly break when it encounters the root_node, and deal with the flushing after the loop. Deal with the FUTUREWORK (assertion for root directory digest matching what the DirectoryPutter returns). Change-Id: Iefc4904d8b8387e868fb752d40e3e4e4218c7407 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11417 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7926 refactor(tvix/castore/import): put invariant checker into a .inspect()Florian Klink1-18/+18
Separate this a bit stronger from the main application flow. Change-Id: I2e9bd3ec47cc6e37256ba6afc6e0586ddc9a051f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11416 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7925 refactor(tvix/*/import): rename direntry_stream, entries_per_depthsFlorian Klink2-8/+10
Align these names and comments with the two users, to make it more obvious we're doing the same thing here, just use a different method to come up with entries_per_depths. Change-Id: I42058e397588b6b57a6299e87183bef27588b228 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11415 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7924 refactor(tvix/castore/import): inline process_entryFlorian Klink1-118/+65
This did very little, and especially the part of relying on the outside caller to pass in a Directory if the type is a directory required having per-entry-type specific logic anyways. It's cleaner to just inline it. Change-Id: I997a8513ee91c67b0a2443cb5cd9e8700f69211e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11414 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7923 refactor(tvix/castore/import): move process_entry to the end of the fileFlorian Klink1-95/+95
This makes it easier to understand the code. Change-Id: I0a9047433000551a6ba1f50a8c5c93527bc86216 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11413 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-23T15·38+0000