depot - monorepo for the virus lounge

Age	Commit message (Collapse)	Author	Files	Lines
2023-03-13	r/5964 refactor(tvix/eval): flatten call stack of VM using generators	Vincent Ambo	1	-53/+97
	Warning: This is probably the biggest refactor in tvix-eval history, so far. This replaces all instances of trampolines and recursion during evaluation of the VM loop with generators. A generator is an asynchronous function that can be suspended to yield a message (in our case, vm::generators::GeneratorRequest) and receive a response (vm::generators::GeneratorResponsee). The `genawaiter` crate provides an interpreter for generators that can drive their execution and lets us move control flow between the VM and suspended generators. To do this, massive changes have occured basically everywhere in the code. On a high-level: 1. The VM is now organised around a frame stack. A frame is either a call frame (execution of Tvix bytecode) or a generator frame (a running or suspended generator). The VM has an outer loop that pops a frame off the frame stack, and then enters an inner loop either driving the execution of the bytecode or the execution of a generator. Both types of frames have several branches that can result in the frame re-enqueuing itself, and enqueuing some other work (in the form of a different frame) on top of itself. The VM will eventually resume the frame when everything "above" it has been suspended. In this way, the VM's new frame stack takes over much of the work that was previously achieved by recursion. 2. All methods previously taking a VM have been refactored into async functions that instead emit/receive generator messages for communication with the VM. Notably, this includes all builtins. This has had some other effects: - Some test have been removed or commented out, either because they tested code that was mostly already dead (nix_eq) or because they now require generator scaffolding which we do not have in place for tests (yet). - Because generator functions are technically async (though no async IO is involved), we lose the ability to use much of the Rust standard library e.g. in builtins. This has led to many algorithms being unrolled into iterative versions instead of iterator combinations, and things like sorting had to be implemented from scratch. - Many call sites that previously saw a `Result<..., ErrorKind>` bubble up now only see the result value, as the error handling is encapsulated within the generator loop. This reduces number of places inside of builtin implementations where error context can be attached to calls that can fail. Currently what we gain in this tradeoff is significantly more detailed span information (which we still need to bubble up, this commit does not change the error display). We'll need to do some analysis later of how useful the errors turn out to be and potentially introduce some methods for attaching context to a generator frame again. This change is very difficult to do in stages, as it is very much an "all or nothing" change that affects huge parts of the codebase. I've tried to isolate changes that can be isolated into the parent CLs of this one, but this change is still quite difficult to wrap one's mind and I'm available to discuss it and explain things to any reviewer. Fixes: b/238, b/237, b/251 and potentially others. Change-Id: I39244163ff5bbecd169fe7b274df19262b515699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-01-20	r/5708 feat(tvix/eval): let builtin macro capture external state	Vincent Ambo	1	-26/+98
	This adds a feature to the `#[builtins]` macro which lets users specify an additional state type to (optionally) thread through to builtins when constructing them. This makes it possible for builtins-macro users to pass external state handles (specifically, in our case, known path tracking) into a set of builtins. Change-Id: I3ade20d333fc3ba90a80822cdfa5f87a9cfada75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7840 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-20	r/5707 refactor(tvix/eval): directly return builtin tuples from macro	Vincent Ambo	1	-2/+2
	All invocations of the builtin macro had to previously filter through the `builtin_tuple` function, but it's more sensible to directly return these from the macro. Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-06	r/5597 fix(tvix/eval): VM & Builtin* types have to be public	Vincent Ambo	1	-4/+4
	... without them, using the new Builtins API is basically impossible for library consumers. Change-Id: Ice0557a2e55e12d812f51bf5a99e6b8c91ad1b91 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7755 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-21	r/5468 fix(tvix/builtin-macros): parse multi-line docstrings correctly	Vincent Ambo	1	-3/+17
	Having a multi-line docstring yields multiple doc-attributes in order, however we were previously discarding all but the first one. This reduces them into a single string instead, which can then be displayed as multi-line documentation. Change-Id: I1f237956cdea2e4c746d3f13744e0373c1c645a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7594 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-08	r/5269 feat(tvix/eval): Add docstrings as documentation for builtins	Griffin Smith	1	-2/+38
	Add a new `documentation: Option<&'static str>` field to Builtin, and populate it in the `#[builtins]` macro with the docstring of the builtin function, if any. Change-Id: Ic68fdf9b314d15a780731974234e2ae43f6a44b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7205 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-08	r/5268 feat(tvix/eval): Give names to builtin arguments	Griffin Smith	1	-20/+45
	Refactor the arguments of a Builtin to be a vec of a new BuiltinArgument struct, which contains the old strictness boolean and also a static `name` str - this is automatically determined via the ident for the corresponding function argument in the proc-macro case, and passed in in the cases where we're still manually calling Builtin::new. Currently this name is unused, but in the future this can be used as part of a documentation system for builtins. Change-Id: Ib9dadb15b69bf8c9ea1983a4f4f197294a2394a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7204 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-08	r/5267 refactor(tvix/eval): Be clearer about public interface	Griffin Smith	1	-3/+3
	Some new top-level re-exports (specifically VM, Builtin, and ErrorKind) were added to lib.rs in tvix/eval to allow the builtin-macros tests to work - we should be clear which of these are part of the public interface (I think it's reasonable for ErrorKind to be) and which aren't (specifically I'm not sure VM and Builtin necessarily should be, at least yet). Change-Id: I3bbeaa63cdda9227224cd3bc298a9bb8da4deb7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7203 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-08	r/5263 feat(tvix/eval): Add a proc-macro for defining builtins	Griffin Smith	1	-0/+143
	Add a single new proc macro to a new proc-macro crate, `tvix-eval-proc-macros` for defining an inline module containing nix builtins, and automatically generating a function within that module which returns a list of those builtins as `tvix_eval::value::Builtin`. Change-Id: Ie4afae438914d2af93d15637151a49b4c68aa352 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7198 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI