about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink46-1002/+785
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-16 r/8503 feat(tvix/cli): Extend nixpkgs eval test to include NixOS installerIlan Joselevich1-23/+39
The tvix-cli tests now include evaluating the NixOS GNOME installer ISO image and making sure that its drvPath and outPath matches the one evaluated by Nix. (This required extending the helper function a bit and adding docs). NixOS docs generation is disabled for now, see comments in diff. Change-Id: Ia510f209b1ec3ef9a823f1e5ac0ef2f5f193976f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12177 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8502 feat(tvix/glue): Add refscanner pattern and AsyncReadBrian Olsen4-22/+249
This splits the existing ReferenceScanner into a ReferenceScanner and ReferencePattern as well as adds an AsyncRead implementation that can do a scan while you read from it. The reason to split the scanner in two is that generating the pattern is expensive and when ingesting build results with multiple outputs you want to do several independant scans that look for the same pattern. The reader is for scanning files without having to load the entire file into memory. Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052 Autosubmit: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-16 r/8501 docs(tvix/build/protos): clarify content-addressingFlorian Klink2-4/+4
This comment is a bit misleading - everything that's not in this field is referenced in a content-addressed fashion (by their digest). Change-Id: I5097131530fd188173393063643c057f588ea2c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12219 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-16 r/8500 chore(tvix): bump object_storeFlorian Klink2-14/+44
Fix for RUSTSEC-2024-0358. Change-Id: Id9e8c6ae67845d95a52c884035fcea02d2d71505 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12214 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8499 chore(tvix/tools/crunch-v2): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I11651ea56e52988ee7cef2e02d0cbf61062f5c7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12213 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-08-16 r/8498 chore(tvix/tools/narinfo2parquet): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: Iad02bcf312af8b6bf450ab3c4c6de31d9c02ab87 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12212 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8497 chore(tvix/tools/weave): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I414c1424941af1352f4d3b4e5908083ab0cf4e9d Reviewed-on: https://cl.tvl.fyi/c/depot/+/12211 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-16 r/8496 chore(tvix/nix-compat): bump curve25519-dalekFlorian Klink2-30/+238
Fix for RUSTSEC-2024-0344. Change-Id: I9e9cac5f0dac9e5b2e5a934c99762d982136c0e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12210 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-16 r/8495 chore(tvix/tools/crunch-v2): update Cargo.{lock,nix}Florian Klink2-46/+245
These were out of date. Change-Id: Ideaf888c2851bb9ec36ae01b11d93165b62cd7a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12209 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-15 r/8494 chore(tvix): Retain original passthru in Rust buildsBrian Olsen8-13/+13
When using the runTests feature of crate2nix the derivation that runs the tests is put into passthru.test but all default.nix files for Rust crates in Tvix threw that away. This commit retains passthru so that you can get access to the test derivation. Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-15 r/8493 chore(tvix/tools/narinfo2parquet): update Cargo.{lock,nix}Florian Klink2-39/+209
These were out of date. Change-Id: I89df37f088ad6c53b676d965f0dc1023f40481f4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12208 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-15 r/8492 chore(tvix/tools/turbofetch): update Cargo.nixFlorian Klink1-39/+100
This was out of date. Change-Id: Icea6ee865d389d56bc46941f6049721db293ef7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12207 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-15 r/8491 chore(tvix/tools/wave): update Cargo.lock and Cargo.nixFlorian Klink2-39/+209
This apparently was out of date. Change-Id: I9768d5e62f200169d7815ad85aa0f12eadc56a8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12206 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-14 r/8489 chore(tvix/verify-lang-tests): update latest Nix versionsterni1-1/+1
Change-Id: I656c183d970806565d45ffd31f99f73c804dda1d Reviewed-on: https://cl.tvl.fyi/c/depot/+/12203 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-13 r/8487 docs(tvix/TODO): castore Directory struct is doneFlorian Klink1-16/+8
There's some followup work on making tvix-build use the stricter structs natively, but that should probably be combined with the overall trait changes there. Add an item there so we won't forget, but this isn't really castore territory anymore. Change-Id: I6b1f9fa02d5c87c821d4ddc4b1dcc6a98c4eeaa5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12200 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink42-622/+620
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8485 docs(tvix/TODO): DirectoryService with bucketed directories is doneFlorian Klink1-6/+0
We have this (`ObjectStoreDirectoryService`), as well as some store composition. Change-Id: I876f85da1d5b183d80b148c378f825033ec080e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12198 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka53-1365/+1417
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 r/8483 docs(tvix/store/utils): add docstrings for ServiceUrls*Florian Klink1-2/+15
Describe what these structs are used for, and for each of it, explain which usecases it's used for. Change-Id: I8b7857bc68ec2b37df9f5163e06d028a64a12c79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-12 r/8482 feat(tvix/cli): Add derivation file dumping functionalityIlan Joselevich4-2/+49
Provides a derivation file dumping functionality for tvix-cli that can be used when passing the --drv-dumpdir CLI arg to tvix-cli. This will dump all the known derivation files into the specified directory, making it easier to debug derivation divergences between Tvix generated drvs and the drvs generated by Nix. Supersedes: https://cl.tvl.fyi/c/depot/+/11265 Change-Id: I0e10b26eba22032b84ac543af0d4150ad87aed3e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12192 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 r/8481 fix(tvix/cli): add toplevel docstring to Args structFlorian Klink1-0/+8
Otherwise the flattened `ServiceUrlsMemory` docstring becomes the main binary description, which doesn't make an awful lot of sense. The help now describes what this CLI does currently (an interface to use the different Tvix component. Also, describe what it does not, so people running it without looking too much other documentation might not get confused about why this doesn't do anything in `/nix/store`, or doesn't provide the same CLI surface as Nix. Change-Id: Ia4838b444f03a10821801a6171d3e956b3cdfdaf Reviewed-on: https://cl.tvl.fyi/c/depot/+/12194 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-11 r/8480 chore(tvix/eval): Update module comment for value::stringAspen Smith1-3/+2
This sentence is a little stale; let's just link to NixString directly for the authoritative source of truth. Change-Id: I64e065c4148d29702b09820a0e7724a65fae7c67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12181 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-11 r/8479 feat(tvix/value): Print derivation values speciallyAspen Smith1-0/+12
Just like tvix-repl does (except we don't force values when printing them, so... not entirely like tvix-repl does). But it's something. Change-Id: I2e69b08d7d82b0b2d337f1d4c5d87ed28475fa84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12180 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: aspen <root@gws.fyi>
2024-08-11 r/8476 fix(tvix/cli): always configure nix pathAspen Smith2-2/+15
Configure the nix path even if globals is already set. Change-Id: I6598c92ab40ff952f73da04d9e7d3aeb13c16b53 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12176 Tested-by: BuildkiteCI Autosubmit: aspen <root@gws.fyi> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-10 r/8475 chore(tvix/eval): Drop obsolete todoAspen Smith1-1/+0
the answer is at https://cl.tvl.fyi/c/depot/+/10798 Change-Id: I5f0ed51a3954c7241ef15a8268e0e51695e994c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12175 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-10 r/8474 feat(tvix/eval): Store hash in key of internerAspen Smith4-7/+45
Rather than storing the leaked allocation for the string as the key in the interner, store the hash (using NoHashHashBuilder). I thought this would improve performance, but it doesn't: hello outpath time: [736.85 ms 748.42 ms 760.42 ms] change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05) No change in performance detected. but it at least doesn't *hurt* performance, and it *does* avoid an `unsafe`, so it's probably net good. Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-08-09 r/8468 test(tvix/nix-compat/nar/wire): cover TOK_PAD_PARedef1-0/+2
Change-Id: Ib4da612a4144d062f55a22b437c725b8566bee70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12172 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-09 r/8467 feat(tvix/eval): Re-enable testing for eval-okay-readDirIlan Joselevich6-2/+0
When we moved to building tvix with crate2nix we had to disable the readDir tests for the same reasons mentioned in https://cl.tvl.fyi/c/depot/+/12131. But now with the 12131 CL, we can re-enable the readDir tests. In this change I re-enabled the readDir tests in nix_tests by moving them out of notyetpassing and I also deleted them from tvix_tests because we don't need them in both places. Change-Id: I82ac39605299a8b22d80f8b51fc8ec2476d21dc9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12133 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-09 r/8466 feat(tvix/eval): Implement builtins.readFileTypeIlan Joselevich10-3/+47
builtins.readFileType was added to Nix back in version 2.14. The tests were also moved out of notyetpassing in addition to the readDir fixtures they depend on. I caught a bug where we previously used std::fs::metadata (via the .metadata() method on File) which follows symlinks so it would always return false for is_symlink(). Instead we now use std::fs::symlink_metadata directly which does not follow symlinks, so tests now pass. This wasn't an issue for builtins.readDir as it uses walkdir and walkdir doesn't follow symlinks either. Change-Id: I58eb97bdb5ec95df4f6882f495f8c572fe7c6793 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12130 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-09 r/8465 fix(third_party/overlays): Patch crate2nix to use mkDerivation for testsIlan Joselevich1-39/+28
The problem with using runCommand and recreating the src directory with lndir is that it changes the file types of individual files, they will now be a symlink instead of a regular file. If you have a crate that tests that a file is of regular type then it will fail inside the crate2nix derivation. Also regenerate Cargo.nix for //tvix as it will be needed in the next commit. Change-Id: I9275602cc17a428f9fdf0e55daf12cd673bbc030 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12131 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-09 r/8464 feat(tvix/verify-lang-tests): Test nixVersions.nix_{2_3,2_23}Ilan Joselevich1-29/+38
With this change the latest Nix version being checked is 2.23 (previously was 2.18). A warning will now be printed when the latest verified Nix version is out of date. I added a test to the skip-list because the behavior of builtins.dirOf has changed in Nix 2.22 Change-Id: Iaf7fa98dbfa599a2d2723df0e42bb459711e2413 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12105 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-09 r/8463 feat(tvix/eval): Implement Display for io::FileTypeIlan Joselevich2-7/+14
In newer versions of Nix there's a builtins.readFileType builtin, we should try to avoid duplicating the enum -> string conversion by implementating Display before we implement builtins.readFileType. Change-Id: I579e95949a76eb33d2e7bda0000ed84859df765e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12129 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-09 r/8462 docs(tvix/nix-compat/nar/wire): fix TOK_EXE grammaredef1-1/+1
Change-Id: I12b640fe1cbe7a4c69efe4660040486e510deadc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12171 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich18-126/+153
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-08 r/8455 feat(tvix/eval): Put interner in a thread-local RefCellAspen Smith1-9/+8
Rather than making the interner be a global lazy_static mutex, put it in a thread-local RefCell. This doesn't change anything in terms of sharing (since we're currently actually just single threaded), but avoids the overhead of a mutex, for a nice performance boost (compared to the mutex version): hello outpath time: [726.71 ms 729.79 ms 735.69 ms] change: [-5.7277% -3.9733% -2.1144%] (p = 0.00 < 0.05) Performance has improved. Change-Id: I240b238dcbaf854ebafc3017b4425fb7d7b91b03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12048 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-08-08 r/8454 feat(tvix/eval): Intern (and leak) small strings, behind a mutexAspen Smith1-5/+82
This is the most naive version of string interning possible - we store a map from the string itself to the pointer behind a global mutex, and memoize the allocation of all strings below a threshold length (16 bytes, for now) into that map. This requires leaking /all/ strings, since it's not easy to know just from the pointer that a string has been interned - so interning is disabled if string leaking is also disabled. In the case where we're leaking strings (the default), even the naive version of this gets us a pretty nice perfomance boost: hello outpath time: [742.54 ms 745.89 ms 749.14 ms] change: [-2.8722% -2.0135% -1.0654%] (p = 0.00 < 0.05) Performance has improved. However, in the case where we're not leaking strings, we have to keep track of which strings have and haven't been interned, which makes this a little worse: hello outpath time: [779.30 ms 792.82 ms 808.74 ms] change: [+2.5258% +4.0884% +5.8931%] (p = 0.00 < 0.05) Performance has regressed. Hopefully we can close the gap here a bit with some clever tricks (coming next). Change-Id: If08cb48ede703c7fe3bdd8d617443f8a561ad09b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12047 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-08-07 r/8453 feat(tvix/eval): Forbid Hash{Map,Set}, use Fx insteadAspen Smith15-43/+88
Per https://nnethercote.github.io/perf-book/hashing.html, we have basically no reason to use the default hasher over a faster, non-DoS-resistant hasher. This gives a nice perf boost basically for free: hello outpath time: [704.76 ms 714.91 ms 725.63 ms] change: [-7.2391% -6.1018% -4.9189%] (p = 0.00 < 0.05) Performance has improved. Change-Id: If5587f444ed3af69f8af4eead6af3ea303b4ae68 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12046 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: aspen <root@gws.fyi>
2024-08-07 r/8452 chore(tvix): change deprecated default_features to default-featuresYureka1-2/+2
Change-Id: Ida31fe9a1205cdfc852d2ecbcbda32933b634562 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12143 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-08-07 r/8451 docs(tvix/TODO): document blob prefetching dependencyFlorian Klink1-0/+2
This depends on the ChunkReader work. Change-Id: I38878d0f822c312151131e55baee4db6ef1c3650 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12142 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-07 r/8450 docs(tvix/TODO): add Correctness > Performance sectionFlorian Klink1-1/+31
Explain the current caveats as far as performance tuning is concerned. Change-Id: I1a9c11c81de09350240fb61e3c130fc401ef6618 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12141 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-08-05 r/8443 feat(tvix/eval): Leak strings (with flag to disable)Aspen Smith3-4/+21
Default to always leaking strings, and copying strings by copying pointers rather than cloning the underlying allocation. This (somewhat bafflingly) doesn't seem to affect any benchmarks, but paves the way for some tricks around string allocation that do. Unfortunately, we can't do this (yet?) for contextful strings, for reasons I don't currently understand but which I will address later, when I address contextful strings more holistically. I've left a flag in here to disable this, both to test the cloning logic for strings for when/if we decide to bring this back, and to allow people who care more about memory usage than perf to disable leaking. Change-Id: Iec44bcbfe9b3d20389d2450b9a551792a79b9b26 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12045 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-04 r/8442 feat(tvix/store): Sled{PathInfo,Directory}Service -> Redb…Ilan Joselevich3-5/+4
Use redb instead of sled for the default filesystem implementation of PathInfoService and DirectoryService. In the future we'll also drop sled support completely. Change-Id: I513ff0c2ff953d59714aa50b9aa1301b02f53d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12085 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-04 r/8441 fix(tvix/crate2nix-check): Revert use of depotfmt.check wrapperIlan Joselevich1-2/+5
This is a partial revert of https://cl.tvl.fyi/c/depot/+/12068 where I changed tvix/crate2nix-check to use depotfmt.check. It turns out that we don't actually want to be using it in this case as the wrapper sets `--fail-on-change` which would always fail because the Cargo.nix generated by crate2nix will always need to be changed (It's not formatted properly). Change-Id: Ife35c812ca69c90459ce4445f4252d0a24c218fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/12132 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8437 feat(tvix/castore): add RedbDirectoryServiceIlan Joselevich6-0/+341
This provides a DirectoryService implementation which uses redb (https://github.com/cberner/redb) as the database. It provides both in-memory and persistent on-filesystem implementations. Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8435 chore(3p/sources): Bump channels & overlays (2024-07-28)Aspen Smith12-55/+61
* Treewide: re-run depotfmt * //third_party/nixpkgs:html5validator: build with Python 3.11, dependency openstackdocstheme doesn't support 3.12 * //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap module API change: https://github.com/NixOS/nixpkgs/pull/318599 * //tvix/*-go: regenerate protobuf files * //third_party/nixpkgs:treefmt: Remove patch for merged pull request * //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-31 r/8434 fix(tvix/glue): coerce path argument to builtins.path to a pathAspen Smith1-3/+8
The "path" key in the arguments to builtins.path supports any path-coercible type (a string, a path...). Coerce it to a path in the argument rather than just requiring it already be one and throwing an error if it's not. This is... annoying to test, since it requires a file with known contents that's available in the build sandbox. But it works! Trust me! Fixes: b/412 Change-Id: I3c8e339bf344a208d5ed5990193942651f318745 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12053 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-07-31 r/8433 fix(tvix): Reuse now exposed depotfmt wrapper in crate2nix-checkIlan Joselevich1-5/+2
This change exposes the already existing wrapper for treefmt/depotfmt that supports running it inside a sandbox. We now reuse it inside //tvix/crate2nix-check, where we previously duplicated the code. The check is now stricter and will also fail on changes, so I had to set the rust edition in the treefmt config. Change-Id: I000e52421258979c038ba6b1f1ff2db14e391b0c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12068 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-07-30 r/8432 chore(tvix): drop nar-bridge-goFlorian Klink30-2348/+2
This functionality exists in nar-bridge(-rs) now. Change-Id: I717484ef64aafad9466c982d2aa8195fe8b98bb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12065 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-30 r/8431 fix(tvix/nar-bridge): filter src in utils.nixIlan Joselevich1-0/+4
Change-Id: I3c8956a50ecdcc53f882fb46ffdc221dd1515b1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12066 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>