about summary refs log tree commit diff
path: root/tvix/cli
AgeCommit message (Collapse)AuthorFilesLines
2023-04-09 r/6088 refactor(nix-compat): Properly encapsulate store path constructionJohn Ericson1-10/+8
Before there was code scattered about (e.g. text hashing module and derivation output computation) constructing store paths from low level building blocks --- there was some duplication and it was easy to make nonsense store paths. Now, we have roughly the same "safe-ish" ways of constructing them as C++ Nix, and only those are exposed: - Make text hashed content-addressed store paths - Make other content-addressed store paths - Make input-addressed fixed output hashes Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-30 r/6060 refactor(tvix/nix-compat): move build_store_path out of derivationFlorian Klink1-6/+13
This doesn't have anything to do with ATerms, we just happen to be using the aterm representation of a Derivation as contents. Moving this into store_path/utils.rs makes these things much cleaner - Have a build_store_path_from_references function, and a build_store_path_from_fingerprint helper function that makes use of it. build_store_path_from_references is invoked from the derivation module which can be used to calculate the derivation path. In the derivation module, we also invoke build_store_path_from_fingerprint during the output path calculation. Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-22 r/6038 feat(tvix/cli): add `--strict` flag for evaluationVincent Ambo1-0/+9
This toggles whether tvix will evaluate the top-level value and deep-force it, or return it potentially still containing thunks. Change-Id: Ie910941e3b6a0f16c5c0cb896d73947626335f4b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8326 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-03-15 r/6010 fix(tvix/cli): populate output configuration after walking argsVincent Ambo1-15/+14
Walking the arguments might encounter an `outputs` output, which might explicitly (for whatever reason) specify the `out` output. To prevent dropping FOD settings in this case, we have to populate that part of the configuration after walking the other attributes. Change-Id: Iee6a7f0a71e9c9699e79d35e6cb19e1ddb49395d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8312 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-15 r/6009 refactor(tvix/nix-compat): -derivation::Hash, +NixHashFlorian Klink1-11/+10
This stops using our own custom Hash structure, which was mostly only used because we had to parse the JSON representation somehow. Since cl/8217, there's a `NixHash` struct, which is better suited to hold this data. Converting the format requires a bit of serde labor though, but that only really matters when interacting with JSON representations (which we mostly don't). Change-Id: Idc5ee511e36e6726c71f66face8300a441b0bf4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8304 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 r/5996 chore(tvix/cli): don't set COREPKGS anymoreFlorian Klink1-4/+0
Since cl/8213, tvix bundles corepkgs. Change-Id: I17fa4452a6a1d554ae67b4aed9d9b1e77cb495f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8305 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-14 r/5995 refactor(tvix/nix-compat): replace calculate_drv_replacement_strFlorian Klink2-20/+30
Call this function derivation_or_fod_hash, and return a NixHash. This is more in line with how cppnix calls this, and allows using to_nix_hash_string() in some places. Change-Id: Iebf5355f08ed5c9a044844739350f829f874f0ce Reviewed-on: https://cl.tvl.fyi/c/depot/+/8293 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-13 r/5986 chore(tvix/cli): silence useless warningVincent Ambo1-0/+1
Change-Id: I18f567d5dcbbeb934428bf9111eb5fae5fef0342 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8261 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-03-13 r/5981 feat(tvix/cli): bundle corepkgs/fetchurl.nix with tvix-cliVincent Ambo3-1/+76
This file which ships with C++ Nix is required for evaluating nixpkgs. Like C++ Nix, we now inject a pseudo path in EvalIO from which this will resolve as <nix/fetchurl.nix> Change-Id: Ic948c476a2cfc6381d5655d308bc2d5fa25b7123 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8213 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5964 refactor(tvix/eval): flatten call stack of VM using generatorsVincent Ambo2-291/+301
Warning: This is probably the biggest refactor in tvix-eval history, so far. This replaces all instances of trampolines and recursion during evaluation of the VM loop with generators. A generator is an asynchronous function that can be suspended to yield a message (in our case, vm::generators::GeneratorRequest) and receive a response (vm::generators::GeneratorResponsee). The `genawaiter` crate provides an interpreter for generators that can drive their execution and lets us move control flow between the VM and suspended generators. To do this, massive changes have occured basically everywhere in the code. On a high-level: 1. The VM is now organised around a frame stack. A frame is either a call frame (execution of Tvix bytecode) or a generator frame (a running or suspended generator). The VM has an outer loop that pops a frame off the frame stack, and then enters an inner loop either driving the execution of the bytecode or the execution of a generator. Both types of frames have several branches that can result in the frame re-enqueuing itself, and enqueuing some other work (in the form of a different frame) on top of itself. The VM will eventually resume the frame when everything "above" it has been suspended. In this way, the VM's new frame stack takes over much of the work that was previously achieved by recursion. 2. All methods previously taking a VM have been refactored into async functions that instead emit/receive generator messages for communication with the VM. Notably, this includes *all* builtins. This has had some other effects: - Some test have been removed or commented out, either because they tested code that was mostly already dead (nix_eq) or because they now require generator scaffolding which we do not have in place for tests (yet). - Because generator functions are technically async (though no async IO is involved), we lose the ability to use much of the Rust standard library e.g. in builtins. This has led to many algorithms being unrolled into iterative versions instead of iterator combinations, and things like sorting had to be implemented from scratch. - Many call sites that previously saw a `Result<..., ErrorKind>` bubble up now only see the result value, as the error handling is encapsulated within the generator loop. This reduces number of places inside of builtin implementations where error context can be attached to calls that can fail. Currently what we gain in this tradeoff is significantly more detailed span information (which we still need to bubble up, this commit does not change the error display). We'll need to do some analysis later of how useful the errors turn out to be and potentially introduce some methods for attaching context to a generator frame again. This change is very difficult to do in stages, as it is very much an "all or nothing" change that affects huge parts of the codebase. I've tried to isolate changes that can be isolated into the parent CLs of this one, but this change is still quite difficult to wrap one's mind and I'm available to discuss it and explain things to any reviewer. Fixes: b/238, b/237, b/251 and potentially others. Change-Id: I39244163ff5bbecd169fe7b274df19262b515699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-02-16 r/5858 chore(tvix/cli): iter instead of into_iter for referencesAaqa Ishtyaq1-1/+1
This CL removes calling into_iter on a reference, as it will not move out it's content into resulting iterator. Change-Id: Ifcc10b7cf33b98453570cbcec3eb82ffaba2ffcb Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/8126 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-04 r/5836 fix(tvix/cli): use tvlfyi/wu-manber fork for refscannerVincent Ambo1-1/+1
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-03 r/5829 test(tvix/cli): eval nixpkgs.stdenv in CI and assert that it matchesVincent Ambo1-2/+26
Change-Id: If80194b5fdbf69512217bd4780e416e678045323 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8023 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-02 r/5827 fix(tvix/cli): keep tracking full paths in known_pathsVincent Ambo3-42/+84
We need to distinguish explicitly between the paths used for the scanner, and the paths that populate the derivation inputs. The full paths must be accessible from the result of the refscanner to populate drv fields correctly. This was previously hidden by debug changes that masked actual IO operations with no-ops. Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-02 r/5826 feat(tvix/cli): cache imported paths in NixCompatIOVincent Ambo1-2/+20
Creates a cache of imported literal files (e.g. `./default-builder.sh`) which avoids shelling out to Nix for each instance of the same file. Note that a better way to tackle this is to create memoizable thunks for these expressions in the compiler, but we are lacking a little bit of infrastructure for that at the moment. Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5825 refactor(tvix/cli): use Wu-Manber string scanning for drv referencesVincent Ambo4-59/+37
Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 r/5807 refactor(tvix/cli): use nixhash module for output hash calculationFlorian Klink2-73/+20
This covers all the weird corner cases. Change-Id: I85637e82e8929828064ab562dc8a1c8bf161fffa Reviewed-on: https://cl.tvl.fyi/c/depot/+/7991 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 r/5799 feat(tvix/cli): implement builtins.placeholderVincent Ambo1-0/+26
This doesn't require any other corresponding handling *yet*, as the actual replacements happen in the builder logic (which we delegate to cppnix at the moment). Change-Id: I034147c933f05ae427c7a8794647132d108d0ede Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 r/5791 refactor(tvix/nix-compat): absorb //tvix/derivationFlorian Klink3-6/+7
Put this in its src/derivation. Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-31 r/5788 refactor(tvix): introduce nix-compat crateFlorian Klink1-1/+0
Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5787 refactor(tvix/cli/derivation): use `if let` to destructureFlorian Klink1-5/+5
We only do logic here if algo and hash_mode are Some(_) (and there's an `out` output). The fact we don't do anything in all in other cases is a bit hidden at the bottom. Use if let for the destructuring, and drop the other case. Change-Id: Icc0e38e62947d52d48ef610f754749737977fca9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7966 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5786 refactor(tvix/cli): remove unneeded cloneFlorian Klink1-2/+2
Change-Id: I6f4cb24bdd636af8918a2ade44075af92161c97d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7965 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5783 refactor(tvix/cli): absorb construct_output_hashFlorian Klink2-93/+116
This helper function only was created because populate_output_configuration was hard to test before cl/7939. With that out of the way, we can pull it in. Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5782 refactor(tvix/cli): force outside of output configuration helperVincent Ambo1-54/+36
Change-Id: I28357fe131cefedcef9761b08a72f675f4a10789 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7939 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-29 r/5776 refactor(tvix/cli): add helper method for strong string coercionVincent Ambo1-29/+29
This is repetitive and error prone (e.g. switching around to_string/as_str has drastic consequences) due to the ToString overloads. Change-Id: I9b16a2e0e05e4c21e83f43e9f603746eb42e53f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7947 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-01-27 r/5773 refactor(tvix/cli): describe errors with thiserrorFlorian Klink2-64/+16
This is much less code, and makes it much easier to read. Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-27 r/5772 fix(tvix/cli): handle SRI hashes in outputHashFlorian Klink3-18/+146
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`, `fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash` attribute, being an SRI hash. In that case, `builtin.derivation` is called with `outputHashAlgo` being an empty string, and `outputHash` being an SRI hash string. In other cases, an SRI hash is passed as outputHash, but outputHashAlgo is set too. Nix does modify these values in (single, fixed) output specification it serializes to ATerm, but keeps it unharmed in `env`. Move this into a construct_output_hash helper function, that can be tested better in isolation. Change-Id: Id9d716a119664c44ea7747540399966752e20187 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-27 r/5770 feat(tvix/cli): faux implementation of builtins.toFileVincent Ambo1-0/+31
This adds an implementation of this builtin which correctly calculates paths, but does not actually write anything to the store or verify references. Change-Id: Ie9764cbc1d13a73d8dc9350910304e2b7cad3fe8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7910 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 r/5769 feat(tvix/cli): implement `builtins.derivation`Vincent Ambo3-0/+41
This uses the actual upstream Nix code for `builtins.derivation` (which is not a primop in C++ Nix) to implement `builtins.derivation` as a wrapper around `builtins.derivationStrict`. We're doing it this way to ensure that our thunking logic is correct. An initial Rust-native rewrite (see e.g. cl/7363) is pretty difficult to debug while there are still other issues to root out, but eventually we might want to turn this into native code. Change-Id: I5845e18073e103b8670e40648bd7fd9b511058e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7902 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 r/5768 feat(tvix/cli): implement builtins.derivationStrictVincent Ambo2-4/+154
Implements the logic for converting an evaluator value supplied as arguments to builtins.derivationStrict into an actual, fully-functional derivation struct. This skips the implementation of structuredAttrs, which are left for a subsequent commit. Note: We will need to port some eval tests over to CLI to test this correct, which will be done in a separate commit later on. Change-Id: I0db69dcf12716180de0eb0b126e3da4683712966 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7756 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-27 r/5767 feat(tvix/cli): add helper for handling special drv parametersVincent Ambo1-0/+106
Adds a helper function which handles special parameters to `builtins.derivation` that are not just blindly passed through to the builder environment, but populate other specific fields of the derivation (outside of the ones handled by other, more complex helpers from previous commits). Change-Id: I82d1edf9af714fc4591e9071c0b83ece83be7eee Reviewed-on: https://cl.tvl.fyi/c/depot/+/7901 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-27 r/5766 feat(tvix/cli): add helper for populating drv output configurationVincent Ambo2-2/+125
This threads through the fields that control whether a derivation is a fixed-output derivation or not. Change-Id: I49739de178fed9f258291174ca1a2c15a7cf5c2a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7900 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 r/5765 feat(tvix/cli): add helper for populating derivation inputsVincent Ambo1-0/+89
This adds a helper function which takes the output of the reference scanner used on derivation inputs and populates the `input_sources` and `input_derivations` field of the derivation accordingly. Note that we have a divergence from C++ Nix here, as we do not populate the entire FS closure of a literally referred derivation (and our standing theory is that this is unnecessary for nixpkgs). Change-Id: Id0f605dd8c0a82973c56605c2b8f478fc17777d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7899 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 r/5764 feat(tvix/cli): add helper for populating derivation outputsVincent Ambo2-0/+89
Adds a small helper function which uses a Nix value supplied to `builtins.derivation{Strict}` to populate the `outputs` field of the `Derivation` struct. Change-Id: Iccc7a4f293b3d913140aed576a573a8992241e46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7898 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-26 r/5760 fix(tvix/cli): correctly trim cppnix output in NixCompatIOVincent Ambo1-3/+4
We only stripped one of the two uses of this string, leading to extraneous newlines in the refscanner. Change-Id: I25d9119be082c487352f0cf66b97ecdcc3e1de06 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7932 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-20 r/5709 feat(tvix/cli): add `errors` module with drv construction errorsVincent Ambo3-0/+51
These will be threaded through to eval through the new `TvixError` variant. Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-17 r/5677 refactor(tvix/cli): consistently assert type unity in known_pathsVincent Ambo1-46/+41
No situation should be allowed in which a path is inserted into known_paths with different types twice, which we previously enforced only for some path types. Change-Id: I8cb47d4b29c0aab3c58694f8b590e131deba7043 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7843 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-17 r/5673 feat(tvix/cli): add replacement strings tracking to KnownPathsVincent Ambo1-0/+29
Replacement strings are some weird internal feature of Nix that is required for calculating derivation hashes. We need to track these like other paths, as they need to be re-used on builds with dependencies on values from previous builds. Change-Id: Ie955b3fb5ae3685cfadfbe4d06ea6b5e219590c7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7828 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-17 r/5672 feat(tvix/cli): track known plain paths in NixCompatIOVincent Ambo3-27/+45
When adding things to a C++ Nix store, ensure that the path is tracked in the tracker. Since the mechanism for propagating the tracker instance isn't finalised yet, I've opted to take an Rc<RefCell> parameter for it. How exactly that ends up there is going to become clear in the next commits, but for now it's just instantiated in main with Default::default. Change-Id: I90f0b44f2d4f292dedc98ff1aa39041d279b61fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/7833 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 r/5671 refactor(tvix/cli): reference scanner owns all the stringsVincent Ambo2-50/+29
This gets very complex very quickly otherwise, as all the construction paths for a reference scanner and all the access patterns for the KnownPaths structure are not yet fully understood. Change-Id: Ibadf1f18b476695f3c286fc6896ae557760edf63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7827 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-17 r/5669 feat(tvix/cli): add known_paths moduleVincent Ambo3-5/+143
This module implements types used to track the set of known paths in the context of an evaluation. These are used to determine the build references of a derivation. Change-Id: I81e15ae33632784e699128916485751613b231a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7816 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-11 r/5644 feat(tvix/refscan): implement reference scanning over data streamsVincent Ambo1-0/+40
Using yet more machinery from the pretty comprehensive aho_corasick crate, this makes it possible to pass anything implementing `io::Read` to the `ReferenceScanner` to accumulate matches. Change-Id: I5b0e28eb44ea4df24010f40831e29f2cbb8c1f80 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7810 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-11 r/5643 feat(tvix/cli): implement initial refscan moduleVincent Ambo3-0/+99
This module implements a ReferenceScanner struct which uses the aho_corasick crate to scan string inputs for known, non-overlapping candidates (store paths, in our case). I experimented with several different APIs, and landed on this version with an initial accumulator in the scanner. The scanner is instantiated from the candidates and "fed" all the strings, then consumed by the caller to retrieve the result. Right now only things that look vaguely like bytestrings can be fed to the scanner, there is no streaming support in the API yet. Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-06 r/5600 feat(tvix/cli): add `--compile-only` "linter" functionalityVincent Ambo1-1/+52
With this, tvix/cli can be run on files and only produce compiler errors and warnings. Change-Id: I5dd9229fc065647787daafd17d7c1540579a1d98 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7764 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-04 r/5581 refactor(tvix/eval): streamline construction of globals/builtinsVincent Ambo1-1/+1
Previously the construction of globals (a compiler-only concept) and builtins (a (now) user-facing API) was intermingled between multiple different modules, and kind of difficult to understand. The complexity of this had grown in large part due to the implementation of `builtins.import`, which required the notorious "knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing the set of globals. As part of the new `Evaluation` API users should have the ability to bring their own builtins, and control explicitly whether or not impure builtins are available (regardless of whether they're compiled in or not). To streamline the construction and allow the new API features to work, this commit restructures things by making these changes: 1. The `tvix_eval::builtins` module is now only responsible for exporting sets of builtins. It no longer has any knowledge of whether or not certain sets (e.g. only pure, or pure+impure) are enabled, and it has no control over which builtins are globally available (this is now handled in the compiler). 2. The compiler module is now responsible for both constructing the final attribute set of builtins from the set of builtins supplied by a user, as well as for populating its globals (that is identifiers which are available at the top-level scope). 3. The `Evaluation` API now carries a `builtins` field which is populated with the pure builtins by default, and can be extended by users. 4. The `import` feature has been moved into the compiler, as a special case. In general, builtins no longer have the ability to reference the "fix point" of the globals set. This should not change any functionality, and in fact preserves minor differences between Tvix/Nix that we already had (such as `builtins.builtins` not existing). Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-22 r/5476 feat(tvix/cli): add `:d` REPL-prefix to print explanations of valuesVincent Ambo1-5/+14
Change-Id: I1dd8de8d996e07840f9b0aaebf932b812103a43a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7593 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-22 r/5474 feat(tvix/cli): implement `NixCompatIO` helper typeVincent Ambo3-1/+86
This type allows for temporarily compatibility with the C++ Nix store, specifically (for now) it gives us the store directory used by Nix and imports files the same way. Change-Id: I4767794ef2863eba49661315c63c4e17de946d60 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5461 fix(tvix/cli): use tvix_eval::StdIO in CLIVincent Ambo1-0/+1
... until we have a store-I/O layer, or something that intercepts the store-related stuff appropriately. Change-Id: I22f63435b3f9e118e3faeb2924fda8373a23ea7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7568 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5456 chore(tvix/cli): rename binary to just 'tvix'Vincent Ambo1-0/+4
Change-Id: If735c81a04f60dcd77f1c40b595b52ed88bbbe95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7580 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 r/5449 chore(tvix/cli): re-add observer flagsVincent Ambo1-2/+34
Users can again pass flags for dumping the AST, bytecode, and runtime trace. With this commit the CLI is at feature-parity with what we had before, but entirely through the new API. Change-Id: I30fe26f243224b25d1e4f828fec607325ef88306 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7550 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>