about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2024-08-20 r/8544 refactor(nix-compat/aterm): update function names a bitFlorian Klink3-26/+26
Don't call functions bstr or str when they return BString or String, it's confusing. Rename them to `string` and `bytes`. We might be able to generalize over this being BString or Vec<u8> later. Change-Id: I8198551ed3ba1cfc479bf7e3cbbc13a426faf4c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12257 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-20 r/8543 feat(tools/depotfmt): Get gofmt from pkgs.go instead of buildGoIlan Joselevich1-2/+2
This allows importing just //tools/depotfmt.nix when using a josh workspace. Change-Id: Idf7eb438be1cc1e60445aace982fdca1f4367595 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12254 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-20 r/8542 chore(aspen/system): Turn off disableWhileTyping for libinputAspen Smith1-6/+1
I... actually really hate this, it turns out. Plus it breaks video games, lol. Change-Id: I4b86e38ee78f5f3b6ffec5e7cc4f1dc054d91c56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12256 Reviewed-by: aspen <root@gws.fyi> Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-08-19 r/8541 feat(tvix/nix-compat): entry locator in listing structuresRyan Lahfa2-1/+113
This implements a simple DFS locator in listing structures. It is interoperable with the Rust standard library paths, we also build our own errors to restrict path values to reasonable secure defaults, e.g. relative paths with no `..` component. Tests are added for this new feature for a positive and a negative check. In addition, a path validation test was added. The Windows-style prefix is gated on the Windows platform as UNIX does not parse `C:\\` as a `Component::Prefix(_)` but as a `Component::Normal(_)`. Change-Id: Iae2a80bebd8138e41af94aa7d09f2842c3c5a786 Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz> Reviewed-on: https://cl.tvl.fyi/c/depot/+/12255 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-19 r/8540 feat(nix-compat/narinfo/signature): generalize name fieldFlorian Klink9-45/+117
Requiring `name` to be a `&str` means it'll get annoying to pass around `Signature`, but being able to pass them around in an owned fashion is kinda a requirement for a stronger typed `PathInfo` struct, where we want to have full ownership. Rework the `Signature` struct to become generic over the type of the `name` field. This means, it becomes possible to have owned versions of it. We don't want to impose `String` or `SmolStr` for example, but want to leave it up to the nix-compat user to decide. Provide a type alias for the existing `&str` variant (`SignatureRef`), and use it where we previously used the non-generic `Signature` one. Add some tests to ensure it's possible to *use* `Signature` with both `String` and `SmolStr` (but only pull in `smol_str` as dev dependency for the tests). Also, add some more docstrings, these were a bit sparse. Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 r/8539 feat(tvix/nix-compat): add a basic listing deserializerRyan Lahfa4-0/+77
.ls files are useful to seek in a NAR without parsing it entirely. The responsibility of validating the files is on the caller. Change-Id: I5d1da28b5479c38f20ca5babe60e362a2217c9ea Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz> Reviewed-on: https://cl.tvl.fyi/c/depot/+/12196 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 r/8538 feat(tvix): drop usage of sparseTree in favor of lib.sourceByRegexIlan Joselevich6-41/+25
We can avoid depending on things outside //tvix by just using a similar util from nixpkgs. Change-Id: I9ea3e1f0a8a059ea10caaec173569ba9f316aec6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12247 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 r/8537 feat(tvix/boot/tests): use pkgs.nixos instead of nixosForIlan Joselevich1-4/+5
Using pkgs.nixos directly allows us to create a smaller nixos closure for the tests and also not depend on things in depot.ops which can be beneficial for extending the tvix josh workspace. Change-Id: Ic6ad2122733418114b43aa692d6e42ac1e308eeb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12251 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-19 r/8536 refactor(nix-compat/narinfo/signature): use ed25519::SignatureBytesFlorian Klink1-4/+5
It's a `[u8; SIGNATURE_LENGTH]` type alias, and conveys what we're accepting or returning a bit nicer. Change-Id: I974cd97d56d383e51417eb0f26e1431a05711922 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12252 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 r/8535 feat(tazjin/german-string): add constructor method from owned dataVincent Ambo1-12/+40
Makes it possible to construct a GermanString from an owned byte vector, without having to clone the data. This is done by "disowning" the vector using ManuallyDrop to access its internal pointer. For transient strings, this memory is then owned (and freed) by the GermanString instance. Small strings are copied out of the heap and stored inline as before, to avoid any dereferencing operations. Change-Id: I754736099f71d646d430aed73e558a5a7626c394 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12249 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 r/8534 feat(tazjin/german-string): add pointer equality check in EqVincent Ambo1-2/+3
This can short-circuit two large string comparisons. Change-Id: If45e7cf33921fe571482dc710c27ef8cc7c70885 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12245 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 r/8533 chore(tazjin/german-string): add Nix buildVincent Ambo1-0/+5
Change-Id: I1e8c500c9bb4ba365ba32f53f60bcf657f62ed62 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12243 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8532 feat(tazjin/german-string): add [Partial]Ord implementationVincent Ambo1-0/+62
Change-Id: I8002ec63be45a15160387c21ed1fa8721a7c3eb4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12242 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2024-08-19 r/8531 test(tazjin/german-string): add roundtrip proptestsVincent Ambo1-1/+21
Change-Id: Iee392368252d5c6e96d879b213aee34a301d13d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12241 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8530 feat(tazjin/german-string): add Eq impl & corresponding proptestsVincent Ambo3-1/+428
Change-Id: I66a258fad5d4e249268b9d2743d46b30c5ac1dac Reviewed-on: https://cl.tvl.fyi/c/depot/+/12240 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2024-08-19 r/8529 feat(tazjin/german-string): add Debug implementationVincent Ambo1-0/+7
Change-Id: I9a4a8a4815ac210c402cfc2feb1fe0606affb327 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12239 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8528 feat(tazjin/german-string): PartialEq implementation for GermanStringVincent Ambo1-0/+16
This is where one of the advantages of this string representation starts to shine: For small strings there's no derefencing any heap memory at all, and for the long representation we can compare the prefix before moving on. Change-Id: Iac333a52e8d7c9dd09e33dbcf51754e321c880e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12238 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8527 feat(tvix/boot): Use buildGoModule instead of buildGo for tvix-initIlan Joselevich1-5/+8
//tvix depending less on other parts of depot is prefered as it will help with extending the josh workspace of tvix. Change-Id: Ifcac3af1782dfd82e7543cb4c3ae57fbd186edff Reviewed-on: https://cl.tvl.fyi/c/depot/+/12250 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-19 r/8526 feat(third_party/gitignoreSource): Get gitignore-nix from sources (niv)Ilan Joselevich2-10/+14
It's easier to implement readTree/depot polyfills for gitignoreSource when it's imported from third_party.sources, rather than in a file at //third_party.gitignoreSource. Change-Id: I1323f932bd0feeb2c50ccc76397a80e035842992 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12248 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 r/8525 fix(tazjin/german-string): add maximum length checkVincent Ambo1-0/+4
Change-Id: I4fd9b2a31749d65632b662807c4e703a9df9ec9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12237 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8524 feat(tazjin/german-string): add data accessors for &str and &[u8]Vincent Ambo1-1/+67
Change-Id: I992e625861f79ef6d9993e8caee4e02d3fc5557e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12236 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 r/8523 feat(tazjin/german-string): add Drop impl for transient stringsVincent Ambo1-0/+17
All of these strings are currently transient (the storage class is not yet represented anywhere), and the ones that are heap allocated need to be deallocated when the transient string dies. Change-Id: Iba0ca926df5db7594f304c5d5318db9d69c6f42c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12235 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8522 feat(tazjin/german-string): init initial representationVincent Ambo4-0/+82
This adds an initial implementation of the so-called "German Strings" in Rust. https://cedardb.com/blog/german_strings/ This implementation is *far from* complete, the only thing that can be done right now is construct a string, and even that I'm not fully happy with. Change-Id: I2697932a0ef373be76ffd14d59677493a5783b58 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12234 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 r/8521 refactor(tvix/eval): remove use of imbl::OrdMapVincent Ambo8-451/+95
Removes imbl::OrdMap in favour of an Rc over the standard library's BTreeMap, which allows us to drop the imbl dependency completely. In my local tests this is actually slightly faster for `hello` and `firefox`. Change-Id: Ic9597ead4e98bf9530f290c6a94a3c5c3efd0acc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12201 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-08-19 r/8520 refactor(tvix/eval): remove use of imbl::VectorVincent Ambo4-47/+46
This vector type has served us well for now, but it contains internal refcounts which are incompatible with upcoming changes related to garbage collection. The performance impact of this change within all benchmarks I ran was within the margin of error: [nix-shell:/tmp/perf]$ hyperfine "./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings" Benchmark 1: ./u64 -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings Time (mean ± σ): 7.528 s ± 0.272 s [User: 6.578 s, System: 0.631 s] Range (min … max): 7.160 s … 8.012 s 10 runs nix-shell:/tmp/perf]$ hyperfine "./std-vec -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings" Benchmark 1: ./std-vec -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings Time (mean ± σ): 7.515 s ± 0.178 s [User: 6.508 s, System: 0.652 s] Range (min … max): 7.276 s … 7.861 s 10 runs Change-Id: Ib95f871956e336a1e5771f6293583854b1efb276 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12197 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-08-19 r/8519 refactor(tvix/eval): ensure VM operations fit in a single byteVincent Ambo14-611/+821
This replaces the OpCode enum with a new Op enum which is guaranteed to fit in a single byte. Instead of carrying enum variants with data, every variant that has runtime data encodes it into the `Vec<u8>` that a `Chunk` now carries. This has several advantages: * Less stack space is required at runtime, and fewer allocations are required while compiling. * The OpCode doesn't need to carry "weird" special-cased data variants anymore. * It is faster (albeit, not by much). On my laptop, results consistently look approximately like this: Benchmark 1: ./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings Time (mean ± σ): 8.224 s ± 0.272 s [User: 7.149 s, System: 0.688 s] Range (min … max): 7.759 s … 8.583 s 10 runs Benchmark 2: ./after -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings Time (mean ± σ): 8.000 s ± 0.198 s [User: 7.036 s, System: 0.633 s] Range (min … max): 7.718 s … 8.334 s 10 runs See notes below for why the performance impact might be less than expected. * It is faster while at the same time dropping some optimisations we previously performed. This has several disadvantages: * The code is closer to how one would write it in C or Go. * Bit shifting! * There is (for now) slightly more code than before. On performance I have the following thoughts at the moment: In order to prepare for adding GC, there's a couple of places in Tvix where I'd like to fence off certain kinds of complexity (such as mutating bytecode, which, for various reaons, also has to be part of data that is subject to GC). With this change, we can drop optimisations like retroactively modifying existing bytecode and *still* achieve better performance than before. I believe that this is currently worth it to pave the way for changes that are more significant for performance. In general this also opens other avenues of optimisation: For example, we can profile which argument sizes actually exist and remove the copy overhead of varint decoding (which does show up in profiles) by using more adequately sized types for, e.g., constant indices. Known regressions: * Op::Constant is no longer printing its values in disassembly (this can be fixed, I just didn't get around to it, will do separately). Change-Id: Id9b3a4254623a45de03069dbdb70b8349e976743 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12191 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 r/8518 docs(tvix/glue/tvix_io): update doc-commentsFlorian Klink1-4/+2
Since we switched from reference scanning to string context, this only handles the `__corepkgs__` hack. Update the docstrings. Change-Id: Ie857c8c99ae1cdb4697323ec738f88be0580df3e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12246 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-19 r/8517 feat(ops/pipelines): support buildkite retriesFlorian Klink1-4/+6
cl/12228 did enable automatic retries for some flaky tests, which generally did work, as can be seen in https://buildkite.com/tvl/depot/builds/35893 However, ":duck:" still reports as failing, because we check the number of steps to be nonzero, which is not the case if retries have happened. We cannot check for the overall status of the build, as it's still "RUNNING", but instead of counting all failed steps so far, we can query all failed jobs and then filter out the ones that were already retried. Change-Id: Ib9d27587c8a8ba7970850812c4302fecdc4482e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12233 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-18 r/8516 docs(views): update where `josh-filter` is coming fromFlorian Klink1-3/+2
We started using josh from nixpkgs since cl/11457, but forgot to update this documentation. Change-Id: I6e07227bcd3e955076d46146024edd89b69f08f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12244 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-18 r/8515 refactor(tvix/eval): Pull context out into its own moduleAspen Smith2-156/+165
I'm gonna be doing some poking around in the internals of Context, so in preparation this pulls it out into its own module. Change-Id: I72ea7df80b5f36f838934ee07bdba66874c334c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12189 Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-18 r/8514 refactor(tvix/castore): have SymlinkTarget-specific errorsFlorian Klink5-26/+172
Don't use ValidateNodeError, but SymlinkTargetError. Also, add checks for too long symlink targets. Change-Id: I4b533325d494232ff9d0b3f4f695f5a1a0a36199 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12230 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-18 r/8513 refactor(tvix/castore): have PathComponent-specific errorsFlorian Klink6-76/+246
Don't use DirectoryError, but PathComponentError. Also add checks for too long path components. Change-Id: Ia9deb9dd0351138baadb2e9c9454c3e019d5a45e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12229 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2024-08-18 r/8512 feat(tvix/castore/proto): add owned conv to castore::DirectoryFlorian Klink2-65/+84
Replace the hand-rolled code comparing names with a try_fold. Also, make it slightly stricter here, detecting duplicates in the same fields earlier. Change-Id: I9c560838ece88c3b8b339249a8ecbf3b05969538 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12226 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-08-18 r/8511 feat(tvix/castore): add Directory::try_from_iter()Florian Klink4-160/+201
This provides a batched variant to construct a Directory, which reuses the previously calculated size. Checking and inserting code is factored out into a check_insert_node function, taking the current size as a parameter and returning the new size. Change-Id: Ia6c2970a0c12181b7c40e63cf7ce8c93298ea37c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12225 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-08-17 r/8510 fix(tvix/boot): workaround test flakinessFlorian Klink1-0/+8
The boot tests are sometimes flaky, and we don't want them to periodically fail other build. Have Buildkite auto-retry them 2 times on failure. Logs for individual failures are still available, so it won't hinder flakiness debuggability. See https://buildkite.com/docs/pipelines/command-step#retry-attributes for a documentation of a parameter, and cl/8983 for the introduction of that feature to //nix/buildkite. Change-Id: I1c0d25fa1d0ca940b3bdcd145ede87154b0c28eb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12228 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-17 r/8509 chore(tvix/glue): drop some explicit allow(clippy::mutable_key_type)Florian Klink2-3/+0
This is covered by clippy.toml these days. Change-Id: I2330af5781844d5f9d975793d770efcea48d371b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12223 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8508 refactor(tvix/castore): add `name` back to edge weightsFlorian Klink1-11/+22
Make this a proper struct with named fields. We apparently never access B3Digest in there, so it can be removed. Change-Id: Ifc07310393e1afb0a835778eae137a19b54070b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12224 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-17 r/8507 refactor(tvix/castore): add into_nodes(), implement consuming proto convFlorian Klink5-29/+24
Provide a into_nodes() function on a Directory, which consumes self and returns owned PathComponent and Node. Use it to provide a proper conversion from Directory to the proto variant that doesn't clone. There's no need for the one taking only &Directory, we don't use it anywhere, and once someone needs that they might as well clone Directory before converting it. Update all other users of the `.nodes()` function to use `.into_nodes()` where applicable, and avoid some more cloning there. Change-Id: Id4577b9eb173c012e225337458898d3937112bcb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12218 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink25-165/+282
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8505 refactor(tvix/castore): drop {Directory,File,Symlink}NodeFlorian Klink27-463/+557
Add a `SymlinkTarget` type to represent validated symlink targets. With this, no invalid states are representable, so we can make `Node` be just an enum of all three kind of types, and allow access to these fields directly. Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink46-1002/+785
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-16 r/8503 feat(tvix/cli): Extend nixpkgs eval test to include NixOS installerIlan Joselevich1-23/+39
The tvix-cli tests now include evaluating the NixOS GNOME installer ISO image and making sure that its drvPath and outPath matches the one evaluated by Nix. (This required extending the helper function a bit and adding docs). NixOS docs generation is disabled for now, see comments in diff. Change-Id: Ia510f209b1ec3ef9a823f1e5ac0ef2f5f193976f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12177 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8502 feat(tvix/glue): Add refscanner pattern and AsyncReadBrian Olsen4-22/+249
This splits the existing ReferenceScanner into a ReferenceScanner and ReferencePattern as well as adds an AsyncRead implementation that can do a scan while you read from it. The reason to split the scanner in two is that generating the pattern is expensive and when ingesting build results with multiple outputs you want to do several independant scans that look for the same pattern. The reader is for scanning files without having to load the entire file into memory. Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052 Autosubmit: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-16 r/8501 docs(tvix/build/protos): clarify content-addressingFlorian Klink2-4/+4
This comment is a bit misleading - everything that's not in this field is referenced in a content-addressed fashion (by their digest). Change-Id: I5097131530fd188173393063643c057f588ea2c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12219 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-16 r/8500 chore(tvix): bump object_storeFlorian Klink2-14/+44
Fix for RUSTSEC-2024-0358. Change-Id: Id9e8c6ae67845d95a52c884035fcea02d2d71505 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12214 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8499 chore(tvix/tools/crunch-v2): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I11651ea56e52988ee7cef2e02d0cbf61062f5c7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12213 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-08-16 r/8498 chore(tvix/tools/narinfo2parquet): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: Iad02bcf312af8b6bf450ab3c4c6de31d9c02ab87 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12212 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8497 chore(tvix/tools/weave): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I414c1424941af1352f4d3b4e5908083ab0cf4e9d Reviewed-on: https://cl.tvl.fyi/c/depot/+/12211 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-16 r/8496 chore(tvix/nix-compat): bump curve25519-dalekFlorian Klink2-30/+238
Fix for RUSTSEC-2024-0344. Change-Id: I9e9cac5f0dac9e5b2e5a934c99762d982136c0e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12210 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-16 r/8495 chore(tvix/tools/crunch-v2): update Cargo.{lock,nix}Florian Klink2-46/+245
These were out of date. Change-Id: Ideaf888c2851bb9ec36ae01b11d93165b62cd7a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12209 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>