about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-05-02 r/8065 feat(tvix/castore/path): implement PathBuf::from_host_pathFlorian Klink1-1/+105
This allows converting from std::path::Path to castore PathBufs. A flag is present to control .. canonicalization, and the usual caveats about platform-specific differences apply. Currently only added for unix, we'll carefully consider other platforms on a case-by-case basis. Change-Id: If289a92f75a2e5c3eec132b6a91a28d225fc1989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11577 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-02 r/8064 refactor(nix-compat): derivation_or_fod_hash -> hash_derivation_moduloFlorian Klink4-48/+57
There's no need for us to come up with our own names for this. Also update the comments/docstrings a bit, and inline the intermediate hash_derivation_modulo calculation. Change-Id: I09dab8ffe1ebfb6601841e98119eee4ff25d8f39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11578 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-01 r/8063 feat(tvix/castore/path): impl AsRef<Path> for PathFlorian Klink1-0/+6
This allows using both Path and PathBuf in a function argument taking `impl AsRef<Path>`. Change-Id: Ibd3ba6fac538069d2fe729d1ef399fdef301668f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11574 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 r/8062 feat(tvix/castore): expose PathBuf tooFlorian Klink1-1/+1
Change-Id: I1538725766a73436eba9a92dcb4f3275ace2fa4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11573 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 r/8061 fix(tvix/castore/path): join, push -> try_{join,push}edef1-7/+7
These are fallible methods, so they should be named accordingly. Change-Id: I6dc271c42989dd6500173488190f65381835d6fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11572 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8060 feat(tvix/castore/path): add PathBuf::{new, with_capacity, push}edef1-12/+37
Change-Id: Ia64f4bda80e91adbdb804f4f26cef5ace8f5406a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11571 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8059 feat(tvix/castore/path): more conversionsedef1-2/+37
Change-Id: I3ee510b444848316df520dc8ca445d0f3c7d607f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11567 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8058 feat(tvix/castore/path): single-component paths are children of ROOTedef1-11/+21
The empty path (Path::ROOT) is explicitly a valid path, and "foo" is simply a child of "". The root itself is the only path without a parent. Change-Id: Iff00dc8aed89eaf98702b664c0df658bd5a1d88a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11569 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-01 r/8057 test(tvix/castore/path): test the null byte caseedef1-0/+1
Change-Id: Ia4ceaf56f6cb4d8ed1ad935c77b3898d711da73a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11568 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8056 feat(tvix/castore/path): use proto::validate_node_nameedef2-6/+13
Use the shared code for validating node names, since that is what path components represent. Change-Id: I12109c1306b224718faa66cf1f2874c78c1436a7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11566 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8055 chore(tvix/castore/path): drop now-duplicate testsedef1-24/+1
Since PathBuf doesn't have inherent methods anymore, these just forward to Path itself. Change-Id: I30f44adc9994337c367bad985ada0e8fcb98dd6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11570 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8054 feat(tvix/nar-bridge): support HTTP HEAD for narinfoEdwin Mackenzie-Owen1-28/+33
Change-Id: I4ec8e9108f52d0a84bc1af01a27bd51de4ab493f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11456 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-01 r/8053 fix(nix-compat/store_path): permit store paths with dots againedef1-10/+8
Nix 2.4 accidentally permitted this behaviour, but the revert came too late to beat Hyrum's law. It is now considered permissible. Link: https://github.com/NixOS/nix/pull/9867 Change-Id: Ie97777af6765fe1c12c8aa593afe1c9b69125775 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11553 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8052 feat(tvix/castore/path): implement Debug + Displayedef1-3/+33
We implement Debug explicitly, so that we don't just see raw integers. Change-Id: I11213094728f3e0c674562ee71c092a950041632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11565 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-01 r/8051 feat(tvix/castore/path): make Path unsized, repr-equivalent to [u8]edef1-51/+57
Change-Id: Ia1e6a10dba0bcda44e06e4cab7d4e52ca88b5859 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11564 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8050 feat(tvix/castore/path): add Path{Buf}Florian Klink2-0/+248
This contains Path and PathBuf, representing platform-independent paths representable by the castore model. These are always relative, and platform-independent, which distinguishes them from the ones provided in the standard library. A subsequent CL will move IngestionEntry (and more) to use them. Change-Id: Ib85857f4159ebc2f3c00192c95d4e5b54ffd4fcf Reviewed-on: https://cl.tvl.fyi/c/depot/+/11558 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-04-30 r/8049 feat(tvix/store): use local object_store for tvix-store daemon blobsFlorian Klink1-1/+5
The sled backend doesn't perform very well with blobs in there, especially as it's not doing any chunking. Switch to the `objectstore+file://` instead, which does do CDC. Change-Id: Ic0d8836c6fc811b80c7202e3ee7f44a05a4f8dfa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11554 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-30 r/8048 refactor(tvix/castore/import): restructure error typesFlorian Klink7-72/+126
Have ingest_entries return an Error type with only three kinds: - Error while uploading a specific Directory - Error while finalizing the directory upload - Error from the producer Move all ingestion method-specific errors to the individual implementations. Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 r/8047 refactor(tvix/castore): remove IngestionEntry::UnknownFlorian Klink2-10/+1
We shouldn't try to represent non-representable things in the ingestion entries (only to throw an error). It's cleaner to throw the error directly in the part producing the stream. Change-Id: I6b6f6d8c2f677425210142a39f1829ddeefec812 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11556 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: firefly <firefly@firefly.nu>
2024-04-30 r/8046 refactor(tvix/castore/import): move upload_blob_at_path into fs modFlorian Klink2-28/+27
This is only useful for when we have access to a filesystem, so it shouldn't be in the root. Change-Id: I9923aaed1aef9d3a1e8fad41f58821d51c2eb34b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11555 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: firefly <firefly@firefly.nu> Tested-by: BuildkiteCI
2024-04-30 r/8045 fix(tvix/castore/import): symlink targets are Vec<u8>Florian Klink3-4/+9
These can be arbitrary bytes in theory. Some of our libraries might be more strict, or inconsistent w.r.t. their representation of path separators. Change-Id: I7981b74fc7d3dd79f5589cf2ef52ced7b71dd003 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11551 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-04-30 r/8044 docs(tvix/castore): fix tvix_castore::import sub-mod docstringsFlorian Klink2-2/+4
The one for `fs` was wrong, and ended up being attached to ingest_path, and the one for `archive` was missing entirely. Change-Id: I8a4c32fb5293badb1ea0764c278a88e4ca33c018 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11552 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-04-30 r/8043 fix(tvix/nix-compat/wire): RangeInclusive<usize> for memory size boundsedef2-32/+30
u64 is an inappropriate type for something memory-sized, and most callers end up with off-by-ones when using `..` rather than `..=`, including the tests for the module itself. Change-Id: If3b7bea27eb0a6c01e0a5d7e64966acbbb664268 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11550 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 r/8042 refactor(nix-compat/wire): drop primitive functionsedef5-109/+28
These may as well be inlined, and hardly need tests, since they just alias AsyncReadExt::read_u64_le / AsyncWriteExt::write_u64_le. Boolean reading is worth making explicit, since callers may differ on how they want to handle values other than 0 and 1. Boolean writing simplifies to `.write_u64_le(x as u64)`, which is also fine to inline. Change-Id: Ief9722fe886688693feb924ff0306b5bc68dd7a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11549 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-30 r/8041 refactor(nix-compat/wire/bytes): branchless padding computationedef1-7/+8
Change-Id: Ie07c2516a485c78afa6f9a3c8256e9708c4c42c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11548 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 r/8040 fix(tvix/nix-compat/wire/bytes): return UnexpectedEof for unexpected EOFedef1-4/+1
Change-Id: I8cb79f93742cc3a9088701bc8c2795de893eb6a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11547 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-30 r/8038 test(tvix): Fix tvix tests on macOSAlice Carroll3-7/+26
Prior to this, some tests would not build or would fail in an obscure way. Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 r/8037 feat(tvix/glue/tests): configure <nix NIX_PATHFlorian Klink1-0/+2
This is also needed to make `<nix/fetchurl.nix>` available in the testsuite. At some point, we might want to classify different types of tests excepting a different featureset, but for now, enabling this for all of glue is fine. Change-Id: Ided450bbb1f8eb7b66d454bd28bd19b17eb318e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11546 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 r/8036 feat(tvix/glue/tests): wrap TvixStoreIO with TvixIOFlorian Klink1-1/+6
This makes `<nix/fetchurl.nix>` available in the testsuite. Change-Id: I04657b227c539edec86e5b6033e1016adcfce244 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11537 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 r/8035 refactor(tvix/glue/fetchers): rename "nixhash" to just "hash"Florian Klink1-12/+12
Change-Id: I2e40b1bcbd9932469ec2af9bb9ce20bb267566ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/11536 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-29 r/8034 refactor(nix-compat/wire/bytes/reader): move size reading to async ctoredef1-137/+65
with_size only existed to allow separating the phases of reading size and reading data, and similarly the Size state only existed to allow folding size reading into poll_read. Neither of these are necessary if we make the constructor async, and handle the size reading there. As a bonus, this makes BytesReader::len sensibly typed, and permits implementing is_empty, as Clippy demands of us. Change-Id: I72173ec06d60b7998d16a3ecfc5e6ac5424bbed3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11542 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 r/8033 docs(tvix/glue/fetchers): update docstring on TarballFlorian Klink1-1/+2
The tarball can also not be compressed. Change-Id: Idbf1b7168cc9a43826355e3cb8b4140f5f53d780 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11535 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 r/8032 refactor(nix-compat/wire/bytes/reader): use fn for PhantomDataedef1-1/+1
We don't actually hold a Tag, we just want to bind the type. Change-Id: Ida67c026f852ed54c3f18df914cf5c31e6227fd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11541 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 r/8031 refactor(tvix/nix-compat/wire/bytes): use RangeInclusive for limitsedef1-14/+7
The (min, max) pair is already a RangeInclusive in essence, so we might as well represent it that way. Change-Id: I2f67f3c47dc36b87e866ff5dc2e0cd28f01fbb04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11540 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 r/8030 refactor(nix-compat/wire/bytes): fold TrailerReader into BytesReaderedef2-91/+41
The TrailerReader has no purpose separate from BytesReader, and the code gets a fair bit simpler this way. EOF handling is simplified, since we just rely on the implicit behaviour of the existing case. Change-Id: Id9b9f022c7c89fbc47968a96032fc43553af8290 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11539 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 r/8029 feat(nix-compat/wire/bytes/reader): expose the remaining data lengthedef2-0/+31
The API is a bit odd here, because we don't have a distinct type for a known-length reader. Change-Id: I4a1dd07fbed0a400004dbe4aa2095c51898ad3bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11538 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name>
2024-04-29 r/8028 refactor(nix-compat/nar/reader): always enable poisoning *API*edef1-64/+66
The poisoning API is now always available, whether debug_assertions is enabled or not. When debug assertions are not enabled, it is equivalent to a unit struct, and is always considered ready and unpoisoned. Change-Id: I950237f4816d480330d9acab32661ed4f1663049 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11525 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 r/8027 refactor(tvix/glue): clone lessFlorian Klink1-30/+28
Prepare the NixAttr to return without an intermediate Vec<(String, NixString)>, and without into_iter(), and send off the unmoved Derivation struct to known_paths without having to clone it. Change-Id: Icdb9f78938e998a27d0313c5d9ab15b93af5821d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11531 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2024-04-29 r/8026 refactor(tvix/glue): move decompression into fetchers/ subdirFlorian Klink3-5/+7
This is specifically used for the fetcher code (only). Moving it to there for now. Change-Id: I1e1d0541b85340ef4ff3a4c6b3fa99b51853f539 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11532 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-28 r/8025 chore(3p/sources): Bump channels & overlayssterni3-2/+3
- //tvix: address new clippy lints - //users/tazjin: Satisfy gonic module's new need for a playlist folder. - //users/aspen/games: adjust for changed location of df's default init.txt and d_init.txt. Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-04-26 r/8019 feat(tvix/cli): change log format to compact, add RUST_LOG supportFlorian Klink1-1/+8
This brings feature parity with the tvix-store CLI command, at least as far as logging (not tracing) is concerned. Alternative to cl/11482, RUST_LOG seems to be the more canonical way to influence this, and is consistent with how it's done in tvix-store. Change-Id: I923a0b0ae55dc49af7efdacdcf5b1f24e561b3c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11527 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 r/8018 feat(tvix/store/bin): disable --json option, set log to compactFlorian Klink1-30/+10
This wasn't really used - to ingest logs in meachine-readable form, the OTLP infrastructure is more suitable to provide structured logs than parsing JSON from std{err}, as it also captures span information. Also, the non-JSON output is a bit too spammy, as remarked in cl/11483 - change it to `compact`. Change-Id: I48007b84ba076ab566abbb6131a02868fe0eb397 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11526 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 r/8017 chore(nix-compat/nar/wire): space Tag::Buf docs properlyedef1-0/+1
Change-Id: If15b14b5328b9e7f85ea3fa7644d327dfb50e1a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11523 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-26 r/8016 feat(nix-compat/wire/bytes/reader): split out reading the trailer blockedef1-56/+134
We separate ingesting the trailer block into a Future of its own, parametrised on the specifics of the trailer pattern. This is intended to be used for future work on an async NAR reader, which needs to read a terminating parenthesis as well as the regular padding. Thanks to @griff for suggesting separating the ingestion into its own Future. Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-26 r/8015 refactor(nix-compat/wire/bytes/reader): drop random whitespaceedef1-2/+2
Change-Id: Ic683eab435576acc8f7e03f5684767ffa468851a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11521 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 r/8014 refactor(nix-compat/wire/bytes/reader): drop Pin::get_mutedef2-4/+4
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to explicitly call get_mut. Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8013 refactor(nix-compat/wire): move BytesPacketPosition into writeredef2-20/+17
We don't use it in the reader anymore. Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-25 r/8012 fix(nix-compat/bytes): make BytesReader less hazardousedef2-178/+317
We now *never* return the final bytes until we've read the padding in full, so read_exact is safe to use. This is implemented by TrailerReader, which splits the phases of reading (and validating) the final 8-byte block, and providing the contained payload bytes to the caller. Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8011 feat(nix-compat/wire/bytes): allow specifying a pre-read sizeedef1-0/+41
Change-Id: I9c94239c308cfbc2e6dae871ba77fb33507433c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11517 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8010 refactor(nix-compat/wire/bytes): drop pin_project, clean upedef1-63/+57
We already require R: Unpin in the constructor, so there's not much use to pin projection. Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-17T05·04+0000