about summary refs log tree commit diff
path: root/tvix/docs/components.md (unfollow)
AgeCommit message (Collapse)AuthorFilesLines
3 hours r/8048 refactor(tvix/castore/import): restructure error typesFlorian Klink7-72/+126
Have ingest_entries return an Error type with only three kinds: - Error while uploading a specific Directory - Error while finalizing the directory upload - Error from the producer Move all ingestion method-specific errors to the individual implementations. Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
5 hours r/8047 refactor(tvix/castore): remove IngestionEntry::UnknownFlorian Klink2-10/+1
We shouldn't try to represent non-representable things in the ingestion entries (only to throw an error). It's cleaner to throw the error directly in the part producing the stream. Change-Id: I6b6f6d8c2f677425210142a39f1829ddeefec812 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11556 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: firefly <firefly@firefly.nu>
5 hours r/8046 refactor(tvix/castore/import): move upload_blob_at_path into fs modFlorian Klink2-28/+27
This is only useful for when we have access to a filesystem, so it shouldn't be in the root. Change-Id: I9923aaed1aef9d3a1e8fad41f58821d51c2eb34b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11555 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: firefly <firefly@firefly.nu> Tested-by: BuildkiteCI
7 hours r/8045 fix(tvix/castore/import): symlink targets are Vec<u8>Florian Klink3-4/+9
These can be arbitrary bytes in theory. Some of our libraries might be more strict, or inconsistent w.r.t. their representation of path separators. Change-Id: I7981b74fc7d3dd79f5589cf2ef52ced7b71dd003 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11551 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
10 hours r/8044 docs(tvix/castore): fix tvix_castore::import sub-mod docstringsFlorian Klink2-2/+4
The one for `fs` was wrong, and ended up being attached to ingest_path, and the one for `archive` was missing entirely. Change-Id: I8a4c32fb5293badb1ea0764c278a88e4ca33c018 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11552 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
11 hours r/8043 fix(tvix/nix-compat/wire): RangeInclusive<usize> for memory size boundsedef2-32/+30
u64 is an inappropriate type for something memory-sized, and most callers end up with off-by-ones when using `..` rather than `..=`, including the tests for the module itself. Change-Id: If3b7bea27eb0a6c01e0a5d7e64966acbbb664268 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11550 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
11 hours r/8042 refactor(nix-compat/wire): drop primitive functionsedef6-112/+33
These may as well be inlined, and hardly need tests, since they just alias AsyncReadExt::read_u64_le / AsyncWriteExt::write_u64_le. Boolean reading is worth making explicit, since callers may differ on how they want to handle values other than 0 and 1. Boolean writing simplifies to `.write_u64_le(x as u64)`, which is also fine to inline. Change-Id: Ief9722fe886688693feb924ff0306b5bc68dd7a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11549 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
11 hours r/8041 refactor(nix-compat/wire/bytes): branchless padding computationedef1-7/+8
Change-Id: Ie07c2516a485c78afa6f9a3c8256e9708c4c42c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11548 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
11 hours r/8040 fix(tvix/nix-compat/wire/bytes): return UnexpectedEof for unexpected EOFedef1-4/+1
Change-Id: I8cb79f93742cc3a9088701bc8c2795de893eb6a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11547 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
14 hours r/8039 fix(tazj.in): fix redirectVincent Ambo1-1/+1
Change-Id: I0b695f0104bc587b1c5b7591c8d512a265d96873 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11534 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
20 hours r/8038 test(tvix): Fix tvix tests on macOSAlice Carroll3-7/+26
Prior to this, some tests would not build or would fail in an obscure way. Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
23 hours r/8037 feat(tvix/glue/tests): configure <nix NIX_PATHFlorian Klink1-0/+2
This is also needed to make `<nix/fetchurl.nix>` available in the testsuite. At some point, we might want to classify different types of tests excepting a different featureset, but for now, enabling this for all of glue is fine. Change-Id: Ided450bbb1f8eb7b66d454bd28bd19b17eb318e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11546 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
23 hours r/8036 feat(tvix/glue/tests): wrap TvixStoreIO with TvixIOFlorian Klink1-1/+6
This makes `<nix/fetchurl.nix>` available in the testsuite. Change-Id: I04657b227c539edec86e5b6033e1016adcfce244 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11537 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
27 hours r/8035 refactor(tvix/glue/fetchers): rename "nixhash" to just "hash"Florian Klink1-12/+12
Change-Id: I2e40b1bcbd9932469ec2af9bb9ce20bb267566ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/11536 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
28 hours r/8034 refactor(nix-compat/wire/bytes/reader): move size reading to async ctoredef1-137/+65
with_size only existed to allow separating the phases of reading size and reading data, and similarly the Size state only existed to allow folding size reading into poll_read. Neither of these are necessary if we make the constructor async, and handle the size reading there. As a bonus, this makes BytesReader::len sensibly typed, and permits implementing is_empty, as Clippy demands of us. Change-Id: I72173ec06d60b7998d16a3ecfc5e6ac5424bbed3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11542 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
29 hours r/8033 docs(tvix/glue/fetchers): update docstring on TarballFlorian Klink1-1/+2
The tarball can also not be compressed. Change-Id: Idbf1b7168cc9a43826355e3cb8b4140f5f53d780 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11535 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
29 hours r/8032 refactor(nix-compat/wire/bytes/reader): use fn for PhantomDataedef1-1/+1
We don't actually hold a Tag, we just want to bind the type. Change-Id: Ida67c026f852ed54c3f18df914cf5c31e6227fd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11541 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
29 hours r/8031 refactor(tvix/nix-compat/wire/bytes): use RangeInclusive for limitsedef1-14/+7
The (min, max) pair is already a RangeInclusive in essence, so we might as well represent it that way. Change-Id: I2f67f3c47dc36b87e866ff5dc2e0cd28f01fbb04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11540 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
29 hours r/8030 refactor(nix-compat/wire/bytes): fold TrailerReader into BytesReaderedef2-91/+41
The TrailerReader has no purpose separate from BytesReader, and the code gets a fair bit simpler this way. EOF handling is simplified, since we just rely on the implicit behaviour of the existing case. Change-Id: Id9b9f022c7c89fbc47968a96032fc43553af8290 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11539 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
29 hours r/8029 feat(nix-compat/wire/bytes/reader): expose the remaining data lengthedef2-0/+31
The API is a bit odd here, because we don't have a distinct type for a known-length reader. Change-Id: I4a1dd07fbed0a400004dbe4aa2095c51898ad3bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11538 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name>
31 hours r/8028 refactor(nix-compat/nar/reader): always enable poisoning *API*edef1-64/+66
The poisoning API is now always available, whether debug_assertions is enabled or not. When debug assertions are not enabled, it is equivalent to a unit struct, and is always considered ready and unpoisoned. Change-Id: I950237f4816d480330d9acab32661ed4f1663049 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11525 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
31 hours r/8027 refactor(tvix/glue): clone lessFlorian Klink1-30/+28
Prepare the NixAttr to return without an intermediate Vec<(String, NixString)>, and without into_iter(), and send off the unmoved Derivation struct to known_paths without having to clone it. Change-Id: Icdb9f78938e998a27d0313c5d9ab15b93af5821d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11531 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
31 hours r/8026 refactor(tvix/glue): move decompression into fetchers/ subdirFlorian Klink3-5/+7
This is specifically used for the fetcher code (only). Moving it to there for now. Change-Id: I1e1d0541b85340ef4ff3a4c6b3fa99b51853f539 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11532 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2 days r/8025 chore(3p/sources): Bump channels & overlayssterni6-24/+27
- //tvix: address new clippy lints - //users/tazjin: Satisfy gonic module's new need for a playlist folder. - //users/aspen/games: adjust for changed location of df's default init.txt and d_init.txt. Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2 days r/8024 feat(tools/depot-deps): include josh-filtersterni1-0/+1
This is occasionally useful for testing //views or preparing new subtrees or subtree updates. Change-Id: I6841e5c3d105406fb960b59e16019f8444da238b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11458 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2 days r/8023 refactor(3p): use josh from nixpkgssterni4-52/+3
The change we need has been released and propagated to nixos channels. Change-Id: Ib10a1d42d7ef6deaf5665a13b72ece345e83d7dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11457 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2 days r/8022 chore(tazj.in): add a convenience redirectVincent Ambo1-0/+5
Change-Id: Ia41fbff390a2b1df0926ab33e9f4f66b1fd92512 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11533 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
3 days r/8021 feat(sterni/machines/ingeborg): switch to quasselsterni3-1/+19
Change-Id: I2970f47e22e6984db49d455b31903814fa94ec76 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11529 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
3 days r/8020 chore(3p/rust-crates:libgit2-sys): 0.16.1+1.7.1 -> 0.16.2+1.7.2sterni1-2/+2
Change-Id: I21ab23177cbbc903eae02ea95dc3fe51f731f777 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11528 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: sterni <sternenseemann@systemli.org>
4 days r/8019 feat(tvix/cli): change log format to compact, add RUST_LOG supportFlorian Klink1-1/+8
This brings feature parity with the tvix-store CLI command, at least as far as logging (not tracing) is concerned. Alternative to cl/11482, RUST_LOG seems to be the more canonical way to influence this, and is consistent with how it's done in tvix-store. Change-Id: I923a0b0ae55dc49af7efdacdcf5b1f24e561b3c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11527 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
4 days r/8018 feat(tvix/store/bin): disable --json option, set log to compactFlorian Klink1-30/+10
This wasn't really used - to ingest logs in meachine-readable form, the OTLP infrastructure is more suitable to provide structured logs than parsing JSON from std{err}, as it also captures span information. Also, the non-JSON output is a bit too spammy, as remarked in cl/11483 - change it to `compact`. Change-Id: I48007b84ba076ab566abbb6131a02868fe0eb397 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11526 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
5 days r/8017 chore(nix-compat/nar/wire): space Tag::Buf docs properlyedef1-0/+1
Change-Id: If15b14b5328b9e7f85ea3fa7644d327dfb50e1a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11523 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
5 days r/8016 feat(nix-compat/wire/bytes/reader): split out reading the trailer blockedef1-56/+134
We separate ingesting the trailer block into a Future of its own, parametrised on the specifics of the trailer pattern. This is intended to be used for future work on an async NAR reader, which needs to read a terminating parenthesis as well as the regular padding. Thanks to @griff for suggesting separating the ingestion into its own Future. Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
5 days r/8015 refactor(nix-compat/wire/bytes/reader): drop random whitespaceedef1-2/+2
Change-Id: Ic683eab435576acc8f7e03f5684767ffa468851a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11521 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
5 days r/8014 refactor(nix-compat/wire/bytes/reader): drop Pin::get_mutedef2-4/+4
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to explicitly call get_mut. Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
5 days r/8013 refactor(nix-compat/wire): move BytesPacketPosition into writeredef2-20/+17
We don't use it in the reader anymore. Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
5 days r/8012 fix(nix-compat/bytes): make BytesReader less hazardousedef2-178/+317
We now *never* return the final bytes until we've read the padding in full, so read_exact is safe to use. This is implemented by TrailerReader, which splits the phases of reading (and validating) the final 8-byte block, and providing the contained payload bytes to the caller. Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
5 days r/8011 feat(nix-compat/wire/bytes): allow specifying a pre-read sizeedef1-0/+41
Change-Id: I9c94239c308cfbc2e6dae871ba77fb33507433c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11517 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
5 days r/8010 refactor(nix-compat/wire/bytes): drop pin_project, clean upedef1-63/+57
We already require R: Unpin in the constructor, so there's not much use to pin projection. Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
5 days r/8009 refactor(nix-compat/wire/bytes): don't parametrise on RangeBoundsedef1-12/+20
This is semantically a RangeInclusive, since we can only have 0..=u64::MAX at most, and monomorphising on the bounds doesn't buy us anything. Change-Id: Ib601d7fd77d703d6c8c5ec27ac9e67bb122ce1c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11515 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
5 days r/8008 fix(tvix): typoedef2-2/+2
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
5 days r/8007 test(tvix/nix-compat): add debug assertions for nar readertcmal2-84/+426
Adds debug assertions to ensure that the reader's variants are upheld. If any of the following happens, then the currently in use reader must be abandoned: * A directory or file reader encounters an error * A directory or file reader is dropped before being fully read from Additionally, a directory reader must not be read from again after it has returned None. These checks are only used when debug_assertions are on, so vanish in release mode. Resolves two TODO items added by edef Change-Id: I27bd9643a632798db5351957506c166b9bd5ca4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11508 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Autosubmit: Aria Shrimpton <me@aria.rip> Tested-by: BuildkiteCI
5 days r/8006 feat(ops/users): add caralice to userssterni1-0/+5
Change-Id: I3d907589c75939c86faa3c1276e4023126ad3d17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11513 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
5 days r/8005 docs(tvix): add a list of TODOs/ideasFlorian Klink2-0/+130
This contains a rough collection of ideas on the TODO list, trying to keep track of it somewhere. Change-Id: Ifc5b0cf9f7ac38f7a8e56515882bdf70e349544b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11512 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
6 days r/8004 docs(tvix/nix-compat/wire/bytes/reader): fix typoFlorian Klink1-1/+1
This is a ReadBuf, not a BufRead. Change-Id: Ie80e894f4b24b77cdd60409ddfaa66dae0ffeec9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11511 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
6 days r/8003 docs(tvix/nix-compat): add missing reference to BytesReaderFlorian Klink1-2/+2
Change-Id: Ideed83d191b55e131720e598b7591e8375a26cfd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11510 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
6 days r/8002 refactor(tvix/castore): add separate Error enum for archivesConnor Brewster3-33/+37
The `Error` enum for the `imports` crate has both filesystem and archive specific errors and was starting to get messy. This adds a separate `Error` enum for archive-specific errors and then keeps a single `Archive` variant in the top-level import `Error` for all archive errors. Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498 Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
7 days r/8001 feat(tvix/castore): upload blobs concurrently when ingesting archivesConnor Brewster2-12/+92
Ingesting tarballs with a lot of small files is very slow because of the round trip time to the `BlobService`. To mitigate this, small blobs can be buffered into memory and uploaded concurrently in the background. Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
7 days r/8000 refactor(tvix/castore): switch to `ingest_entries` for tarball ingestionConnor Brewster3-128/+239
With `ingest_entries` being more generalized, we can now use it for ingesting the directory entries generated from tarballs. Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
7 days r/7999 fix(users/picnoir/tvix-daemon): gitignore target dir, use cleaned srcFlorian Klink2-2/+4
This prevents the nix build copying the target/ dir into the store whenever this is built through Nix. Change-Id: I397228fd8e2e3265ed87d3400fe927bc505da090 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11496 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
generated by cgit-pink 1.4.1 (git 2.44.0) at 2024-04-30T20·32+0000