about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2023-03-03 r/5868 refactor(tvix/eval): enhance debug output for bytecode dumpsVincent Ambo2-1/+25
This adds addresses of thunk and closure chunks to the debug output displayed when dumping bytecode. This makes it possible to see in the dump which thunks are referenced by constants in other thunks. Change-Id: I2c98de5227e7cb415666cd3134c947a56979dc80 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8137 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-03 r/5867 feat(ops/terraform): add module for deploying NixOS system closuresVincent Ambo5-0/+187
This module makes it fairly easy to deploy NixOS system closures using Terraform, while properly separating the evaluation of a derivation (to determine whether a deploy is needed) from the building and copying of the closure itself. This has been on my stack for a while. It was originally developed for Resoptima, who agreed to open-sourcing it in depot back when we completed our work with them. Their contribution has been acknowledged in the README. Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: Ica4c170658cd25f1fb7072c9a45735fcc4351474 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7950 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-02 r/5866 chore(3p/sources): Bump channels & overlayssterni7-194/+36
* //3p/sources: temporarily switch to nixos-unstable-small, since it includes: - evans update we are interested in, allowing us to drop our evans patches. - awscli2 update that unbreaks //users/grfn * //3p/overlays/tvl: - drop evans patches - update tdlib to 1.8.11 to make tazjin's emacs happy - drop obsolete mullvad workaround * //users/grfn/keyboard: disable -Werror for array-bounds warnings. Seems like a non-trivial job to resolve the warning properly, hopefully GCC 12 still generates the same working code as GCC 11 used to. * //users/grfn/system/home: remove yubikey-manager-qt. Yubico can't seem to keep that on pace with yubikey-manager. It requires a <5 version of the latter which is incompatible with the recently released cryptography >= 39. * //3p/gerrit: update changed FOD hash for the fetch step Change-Id: I590ab996247e69b0ab5059cd173840ef4ebfe939 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8133 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi>
2023-03-01 r/5865 chore(users/Profpatsch): remove failed aerc experimentProfpatsch3-205/+0
I didn’t ever get aerc to work, so let’s get rid of this. Change-Id: Ie59b23fe7d5af70ca82c1c624c1e11d21da735f4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8189 Autosubmit: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-01 r/5864 feat(sterni/edwin/code.sterni.lv): mirror vuizvuisterni1-0/+4
Change-Id: I8b163614b588d1cfb8d758cafb8fb407397a1ef7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8188 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-01 r/5863 chore(sterni/external/likely-music): update to 2023-02-07sterni1-3/+3
Includes a frontend bug fix and a closure size reduction of the server application. Change-Id: I5713a5348281acb7126c1fd85a637a6fff969c98 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8187 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-02-28 r/5862 feat(nix/readTree): implement .skip-tree marker for subtree ignoringVincent Ambo6-18/+76
With this change, readTree gains the ability to notice a `.skip-tree` marker in addition to the `.skip-subtree` marker. The behaviour of the new marker will completely ignore the folder that the marker is located in (i.e. no node will be present for it in the parent at all). To make this work, the recursive function in readTree had to be modified to return a sentinel value (noting that a tree has requested to be skipped) which is then filtered out when constructing the list of children. The actual `readTree` function is now a wrapper around this inner, sentinel-yielding implementation which unwraps the result set. For obvious reasons, `.skip-tree` is not allowed at the top-level and readTree will throw an error if it encounters it there. Fixes: b/244 Change-Id: Ica731bc1af356e881fd3d31c7109f62ffd2762ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/8185 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-27 r/5861 feat(grfn/emacs): Add capture template for work todoGriffin Smith1-0/+4
Change-Id: If6e667a28619cd43d1135d6b0552bc09c5c038e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8163 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi>
2023-02-27 r/5860 feat(grfn/system): Install some rust profiling toolsGriffin Smith1-0/+2
Change-Id: I0d68c0c20b29ecd48b7c63c5a5498ccb7cc685fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8162 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2023-02-27 r/5859 chore(grfn/system): Upgrade emacs packagesGriffin Smith4-16/+18
Some interfaces changed in magit's transient, and doom switched more emphatically to rustic mode Change-Id: Ifb945f7fc693b0b7fb1739e25c6da70b06df96ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8161 Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2023-02-16 r/5858 chore(tvix/cli): iter instead of into_iter for referencesAaqa Ishtyaq1-1/+1
This CL removes calling into_iter on a reference, as it will not move out it's content into resulting iterator. Change-Id: Ifcc10b7cf33b98453570cbcec3eb82ffaba2ffcb Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/8126 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-16 r/5857 refactor(tvix/eval): remove redundant cloneAaqa Ishtyaq3-6/+6
This CL removes redundant clone from value which is going to be dropped without further use. Change-Id: Ibd2a724853c5cfbf8ca40bf0b3adf0fab89b9be5 Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/8125 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-16 r/5856 docs(tvix/eval): add proposal for VM loop restructuringVincent Ambo1-0/+76
Change-Id: Ib991d68724a73886a8343d7f785b5b3aedd637ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/8103 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2023-02-16 r/5855 feat(tazjin/generator-example): add an example for genawaiter crateVincent Ambo5-0/+260
This is an experiment for tvix-eval. Change-Id: Ic752b5b125cefefeb1343e38a70beb364478e6eb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8131 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 r/5854 refactor(tvix/store): remove needless borrow in build.rsFlorian Klink1-1/+1
Change-Id: I2c2a70e5ffbb080702f57218aa9dcfe53d08870f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8110 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-02-15 r/5853 refactor(tvix/store/proto): use .cloned()Florian Klink1-10/+4
Instead of using an explicit closure to clone elements, use .cloned(). Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 r/5852 refactor(tvix/store): avoid casting to the same typeFlorian Klink1-1/+1
The size field already is u32, we don't need to convert here. Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-14 r/5851 fix(tvix/eval): correctly print lambda address in observerVincent Ambo1-1/+1
We want the address that the Rc is pointing to, not the address of the Rc. Change-Id: I8eba21677f242bbe4166c74d4aa4269c316076e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8045 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-13 r/5850 chore(tvix/eval): use writeln for newline stringAaqa Ishtyaq1-2/+2
This CL address clippy warning which expects to use `writeln` instead of `write` for strings with new line. Change-Id: Ia72a07502c60cfd489ecf1e3833b9d42d44a8b17 Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/8030 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 r/5849 chore(tvix/eval): clippy warn is length zeroAaqa Ishtyaq1-1/+1
This CL address clippy warning about finding the zero length of something using `is_empty()` instead of `len() == 0`. Change-Id: I2b36c7c7b65b733609fc0dcd33be06f9d772bc9b Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/8029 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 r/5848 fix(tvix/eval): skip runtime completely on compiler errorsVincent Ambo2-0/+18
This branch was missing, and an assumption elsewhere just executed the returned (broken) bytecode. This fixes b/253. Change-Id: I015023ba921bc08ea03882167f1f560feca25e50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8090 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-02-13 r/5847 fix(tvix/eval): make fields of eval's Error type publicVincent Ambo1-3/+3
These should be inspectable by callers. Change-Id: Ia9ef871aa63958d06066aaea61b2aecbd217369b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8089 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-11 r/5846 chore(whitby): enable zram swapAlyssa Ross1-0/+2
Whitby has a lot of memory, but I've still been fighting with the OOM Killer trying to build a few big packages at the same time. Besides, it's generally a good idea to always have swap available even if there's lots of memory for caching optimisation reasons[1], and zram swap is efficient enough to basically provide bonus memory for free. [1]: https://haydenjames.io/linux-performance-almost-always-add-swap-space/ Change-Id: I1fbe60f7975ebfa38e341e0de76848ec79b6fcf0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8065 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-10 r/5845 docs(REVIEWS): update scp commandAlyssa Ross1-1/+1
cl.tvl.fyi only supports the legacy scp protocol, not the sftp protocol that recent versions of OpenSSH now use by default. So we need to pass -O to tell it to use scp. Cc: Vincent Ambo <mail@tazj.in> Change-Id: I2dcae77fef9126f4f8eebfa574c5a345a5c56615 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8039 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-09 r/5844 chore(ops/modules): add a GECOS for my userAlyssa Ross1-0/+1
This way, I won't have to teach my name one at a time to every program that wants to know my it (e.g. git). Change-Id: I45ddd9c2343a10cd4c13bacd9a97b7470db95c14 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8038 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-08 r/5843 feat(tvix/nix-compat/nixhash): support BASE64_NOPAD SRIFlorian Klink1-19/+61
Nix accepts SRI hashes that are missing their padding characters in base64, as seen in https://github.com/NixOS/nixpkgs/blob/7e49471316373c471a3bf4b78c130ebc907ae2d2/pkgs/development/libraries/kerberos/krb5.nix . It only seems to work in the SRI case, not with `sha256` being set to a (nopad) base64 string. Add regression tests for this, and document why we don't want to support *additional* characters afterwards. Reported in https://b.tvl.fyi/issues/252 Change-Id: I9ffc2b417501b426ced1894a9cbf95ff5f0e5159 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8037 Reviewed-by: Alyssa Ross <hi@alyssa.is> Tested-by: BuildkiteCI
2023-02-08 r/5842 refactor(tvix/nix-compat/nixhash): fix test function nameFlorian Klink1-1/+1
Change-Id: I3b089758ce8c01df544064422e56a97a8402513d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8036 Tested-by: BuildkiteCI Reviewed-by: Alyssa Ross <hi@alyssa.is>
2023-02-07 r/5841 fix(tvix): fix InvalidEncodedDigestLength messageAlyssa Ross1-1/+1
Before: error[E997]: invalid output hash: invalid encoded digest length '43' for algo 43 After: error[E997]: invalid output hash: invalid encoded digest length '43' for algo sha256 Change-Id: Icb533839586b47f90b4e8d22083cce397f9f3655 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8035 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-07 r/5840 docs(store/protos): update comment on blobstore Read and PutFlorian Klink3-8/+56
Further emphasize Read() can be used to ask for blobs OR chunks, and that clients usually want to stat and then request (smaller) chunks, rather than reading whole blobs. Also clarify that the chunking used to send BlobChunks over has nothing to do with the chunk sizes communicated in a Stat() request. Change-Id: Ia615d190aae570611de2655b11342a14d0b75976 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-07 r/5839 chore(3p/sources): bump nixpkgs & overlays (2023-02-07)Vincent Ambo6-59/+40
Included fixes: * //3p/overlays: tdlib override no longer needed (bump has landed upstream) * //corp/{predlozhnik,tvixbolt}: bump wasm-bindgen to match nixpkgs Home-manager has not been bumped as it has introduced an incompatibility with Nix 2.3 Change-Id: I96ac3462b82c73db1ba23be03d7968f10abc9b53 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8033 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-02-06 r/5838 fix(nixery): fix link to nixery logoVincent Ambo1-1/+1
Change-Id: Ib78659b971696feaff579bc0a31df7d8ee24e459 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8034 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-04 r/5837 fix(tvix/eval): fix the default case for path parsingVincent Ambo1-10/+4
Plain paths like `foo/bar.nix` are also allowed, so we can not determine this based on the prefix. The upstream PR that is referenced in a comment here has a significantly different interface than we expected, so I'm not touching that comment yet in this CL before I've had more time to digest it. Change-Id: Iea33bbb35de9c00a7d7fedf64d02253c75c1cc9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8032 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: Alyssa Ross <hi@alyssa.is> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-04 r/5836 fix(tvix/cli): use tvlfyi/wu-manber fork for refscannerVincent Ambo4-43/+443
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-03 r/5835 feat(corp/tvixbolt): link to tvl.su tooFlorian Klink1-0/+11
Make it clear that Tvixbolt is a project of TVL LLC, and link to the community website too. See https://b.tvl.fyi/issues/248 Change-Id: Iefefe0263fa5ef01587d49c5a130a38b78ca7981 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8019 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-02-03 r/5834 docs(corp/website): distinguish TVL LLC and TVLFlorian Klink1-2/+2
See https://b.tvl.fyi/issues/248 Change-Id: Iacc10841bdcdf1dbc8eab659d82e076fa2151979 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8020 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-03 r/5833 chore(tvix/eval): only use Rc with impure featureFlorian Klink1-2/+3
We only use Rc in `impl EvalIO for StdIO`, which is only included when building with the "impure" feature. Change-Id: Id29d647c899cbfcdda11abfb9fabd5aa7e24299f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8025 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-03 r/5832 refactor(tvix/eval): wrap `Builtin` type in a BoxVincent Ambo1-20/+30
This reduces the size of `Builtin` from 88 (!) bytes to 8, and as the largest variant of `Value`, the size of that type from 96 to 64. The next largest type is NixList, clocking in at 64 bytes. This has noticeable performance impact. In an implementation without disk I/O, evaluating nixpkgs.stdenv looks like this: Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath' Time (mean ± σ): 1.151 s ± 0.003 s [User: 1.041 s, System: 0.109 s] Range (min … max): 1.147 s … 1.155 s 10 runs After this change, it looks like this: Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath' Time (mean ± σ): 1.046 s ± 0.004 s [User: 0.954 s, System: 0.092 s] Range (min … max): 1.041 s … 1.053 s 10 runs Change-Id: I5ab7cc02a9a450c0227daf1f1f72966358311ebb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8027 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 r/5831 refactor(tvix/eval): statically resolve select from constant attrsVincent Ambo3-50/+94
When resolving a select expression (`attrs.name` or `attrs.name or default`), if the set compiles to a constant attribute set (as is most notably the case with `builtins`) we can backtrack and replace that attribute set directly with the compiled value. For something like `builtins.length`, this will directly emit an `OpConstant` that leaves the `length` builtin on the stack. Change-Id: I639654e065a06e8cfcbcacb528c6da7ec9e513ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/7957 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 r/5830 docs(tvix): move mg shell command into one lineFlorian Klink1-2/+2
This was reflowed in a funny way. Move the whole command into its own line, to prevent it from happening. Change-Id: Ifba4daf418487ca4c32586820071930d29020f42 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8026 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-03 r/5829 test(tvix/cli): eval nixpkgs.stdenv in CI and assert that it matchesVincent Ambo1-2/+26
Change-Id: If80194b5fdbf69512217bd4780e416e678045323 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8023 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-03 r/5828 fix(tvix/eval): ensure all evaluated thunks are correctly memoizedVincent Ambo5-51/+196
This fixes a very complicated bug (b/246). Evaluation progresses *much* further after this, leading to several less complicated bugs likely being uncovered by this What was the problem? ===================== Previously, when evaluating a thunk, we had a code path that looked like this: match *thunk { ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => { let inner_repr = inner_thunk.0.borrow().clone(); drop(thunk); self.0.replace(inner_repr); } /* ... */ } This code path created a copy of the inner `ThunkRepr` of a nested thunk, and moved that copy into the `ThunkRepr` of the parent. The effect of this was that the original `ThunkRepr` (unforced!) lived on in the original thunk, without the memoization of the subsequent forcing applying to it. This had the result that Tvix would repeatedly evaluate these thunks without ever memoizing them, if they occured repeatedly as shared inner thunks. Most notably, this would *always* occur when builtins.import was used. What's the solution? ==================== I have completely rewritten `Thunk::force_trampoline_self` to make all flows that can occur in it explicit. I have also removed the outer loop inside of that function, and resorted to more use of trampolining instead. The function is now well-commented and it should be possible to read it from top-to-bottom and get a general sense of what is going on, though the trampolining itself (which is implemented in the VM) needs to be at least partially understood for this. What's the new problem(s)? ========================== One new (known) problem is that we have to construct `Error` instances in all error types here, but we do not have spans available in some thunk-related situations. Due to b/238 we cannot ask the VM for an arbitrary span from the callsite leading to the force. This means that there are now code paths where, under certain conditions, causing an evaluation error during thunk forcing will panic. To fix this we will need to investigate and fix b/238, and/or add a span tracking mechanism to thunks themselves. What other impacts does this have? ================================== With this commit, eval of nixpkgs mostly succeeds (things like stdenv evaluate to the same hashes for us and C++ Nix, meaning we now construct identical derivations without eval breaking). Due to this we progress much further into nixpkgs, which lets us uncover more additional bugs. For example, after this commit we can quickly see that cl/7949 introduces some kind of behavioural issue and should not be merged as-is (this was not apparent before). Additionally, tvix-eval is now seemingly very fast. When doing performance analysis of a nixpkgs eval, we now mostly see the code path for shelling out to C++ Nix to add things to the store in there. We still need those code paths, so we can not (yet) do a performance analysis beyond that. Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5827 fix(tvix/cli): keep tracking full paths in known_pathsVincent Ambo3-42/+84
We need to distinguish explicitly between the paths used for the scanner, and the paths that populate the derivation inputs. The full paths must be accessible from the result of the refscanner to populate drv fields correctly. This was previously hidden by debug changes that masked actual IO operations with no-ops. Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-02 r/5826 feat(tvix/cli): cache imported paths in NixCompatIOVincent Ambo1-2/+20
Creates a cache of imported literal files (e.g. `./default-builder.sh`) which avoids shelling out to Nix for each instance of the same file. Note that a better way to tackle this is to create memoizable thunks for these expressions in the compiler, but we are lacking a little bit of infrastructure for that at the moment. Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5825 refactor(tvix/cli): use Wu-Manber string scanning for drv referencesVincent Ambo7-61/+64
Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5824 docs(tvix): add more information to READMEVincent Ambo1-8/+48
The README was very sparse before and we've actually had people email us (as it says to contact us) just to ask what Tvix *is*. This should answer some questions! Change-Id: I0f248cb060eccfe086468afed1d648652b35dfd1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8018 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5823 fix(tvix/eval): unsafeDiscardStringContext is a no-opVincent Ambo1-4/+2
... not just a TODO. Most use-cases of unsafeDiscardStringContext are for cases where a string is processed in some ways and no longer contains a "physical" reference, but still has its context attached in C++ Nix. We don't need to do this. This does diverge in behaviour in use-cases related to build scheduling, but that whole behaviour will be different in Tvix. Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-02 r/5822 refactor(tvix/eval): import_cache can be a HashMapVincent Ambo1-2/+2
... instead of a BTreeMap, as we do not need ordering guarantees here. HashMaps are noticeably faster here (especially as we've been sorting essentially random data!). Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 r/5821 chore(tvix/eval): elaborate on internal types in Value::type_ofVincent Ambo1-6/+8
This aids in debugging quite substantially. Change-Id: Ic43232aa6165ae1c3db7ac2701938e1dfeeb418c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8013 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-02-01 r/5820 chore(corp/tvixbolt): update Cargo.lockFlorian Klink1-52/+52
Change-Id: Ie6bb2df16f79f7d977a7e95187a6b81e3ac0108f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8011 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-01 r/5819 chore(tvix): update Cargo.lockFlorian Klink2-477/+502
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>