about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2022-12-28 r/5527 feat(tvix/store): make blobstore stream chunksFlorian Klink3-172/+171
This changes the RPC methods to return/consume a stream of chunks, instead of a very big message containing the whole blob, to keep message sizes in manageable sizes (less than 4MiB). Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5526 feat(tvix): add evans to shellFlorian Klink1-0/+1
Change-Id: I7ec8bae236330ec1e38e82da67b47aef2815d4b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7437 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5525 chore(tvix/store/protos): add PathInfoService::CalculateNAR()Florian Klink3-39/+222
Expose the NAR calculation to a separate `CalculateNAR` method, which responds with the NAR size and sha256 hash. Contrary to what cl/7618 and cl/7620 initially did, don't add different other request types. In the CalculateNARResponse message, there's now some duplication in the (optional) `narinfo` field of a PathInfo, but I'm not entirely sure if we want to drop the fields from there yet. Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5524 chore(tvix/store/protos): drop non-sha256 NAR hashesFlorian Klink2-196/+41
Nix actually doesn't support anything else than sha256 NAR hashes, so there's little reason to communicate anything else in here. Co-Authored-By: edef <edef@unfathomable.blue> Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5523 chore(tvix/store/protos): move (Root)Node into separate messageFlorian Klink2-124/+191
This is gonna get used in another place in a second. Co-Authored-By: edef <edef@unfathomable.blue> Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5520 feat(tvix/store): implement Directory::validate()Florian Klink4-13/+351
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5519 chore(tvix/store): use DUMMY_DIGEST, tooFlorian Klink1-1/+1
Missed from cl/7649. Change-Id: Id2d382567f91cba9cdd792fe44bc0a83d040ec69 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7656 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5518 chore(tvix/store): use Directory::default in some more placesFlorian Klink1-18/+6
Change-Id: Iebad5e0b38f9506933fa855e1bfb2611d27ff572 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7655 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 r/5510 chore(tvix/store): use lazy_static and dummy digestsFlorian Klink4-2/+16
While there's currently nothing in here checking the size of the digest, we should use something that passes the to-be-introduced validate() function. Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5509 feat(tvix/store/protos): implement Directory.Validate()Florian Klink2-0/+297
Validate thecks the Directory message for invalid data, such as: - violations of name restrictions - invalid digest lengths - not properly sorted lists - duplicate names in the three lists Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5508 feat(tvix/store/protos): implement Size() and Digest() for DirectoryFlorian Klink4-0/+251
This adds Size() and Digest() functions for the golang version. Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5496 feat(txix/store/protos): init go.mod, generate .pb.go filesFlorian Klink9-0/+2390
This allows importing the generated .pb.go files into other go projects. I initially looked at buildGo.protos, but it doesn't work for multi-.proto files, and actually having LSP support for the generated structs is nice, too. Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5495 feat(tvix/store/protos): add go_package optionFlorian Klink5-0/+10
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321 Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2022-12-26 r/5494 docs(tvix/store): add README, document services and store modelFlorian Klink4-2/+391
These are intended to help digest the protocol definitions for tvix- store, and how they tie into the whole concept. Co-Authored-By: Vincent Ambo <mail@tazj.in> Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5492 feat(tvix/store): use tonic-build for protosFlorian Klink5-20/+2334
While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-25 r/5486 fix(tvix/eval): fix current clippy warningsVincent Ambo17-108/+69
It's been a while since the last time, so quite a lot of stuff has accumulated here. Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-25 r/5485 refactor(tvix/eval): non-recursive thunk forcingAdam Joseph2-67/+243
Introduces continuation-passing-based trampolining of thunk forcing to avoid recursing when forcing deeply nested expressions. This is required for evaluating large expressions. This change was extracted out of cl/7362. Co-authored-by: Vincent Ambo <tazjin@tvl.su> Co-authored-by: Griffin Smith <grfn@gws.fyi> Change-Id: Ifc1747e712663684b2fff53095de62b8459a47f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7551 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-25 r/5484 feat(tvix/eval): implement From<f64> for ValueRyan Lahfa1-0/+6
Change-Id: I287282a195d6f752260242739332b2357791974a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7625 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-22 r/5477 feat(tvix/eval): display function names in documentationVincent Ambo1-1/+8
... if they are known. We currently do not propagate names correctly for curried functions. Change-Id: I19d57fb30a5c0000ccdf690b91076f6b2191de23 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7596 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-22 r/5476 feat(tvix/cli): add `:d` REPL-prefix to print explanations of valuesVincent Ambo1-5/+14
Change-Id: I1dd8de8d996e07840f9b0aaebf932b812103a43a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7593 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-22 r/5475 feat(tvix/eval): add Value::explain methodVincent Ambo1-0/+33
This value creates a human-readable explanation of a value. This can be used to implement documentation related functionality. For some values, the amount of information displayed can be expanded quite a bit. Change-Id: Ie8c400feae909e7680af163596f99060262e4241 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7592 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-22 r/5474 feat(tvix/cli): implement `NixCompatIO` helper typeVincent Ambo6-2/+92
This type allows for temporarily compatibility with the C++ Nix store, specifically (for now) it gives us the store directory used by Nix and imports files the same way. Change-Id: I4767794ef2863eba49661315c63c4e17de946d60 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-22 r/5472 feat(tvix): add crate2nixGenerate targetFlorian Klink2-9/+11
`mg run //tvix:crate2nixGenerate --` runs crate2nix generate and depotfmt's it afterwards. This removes a frequent point of friction, because the Cargo.nix emitted by crate2nix needs another formatter. Change-Id: I649495980718cd3847d4cff77c9d4bfcb599387c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7612 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-22 r/5471 feat(tvix/proto): add rpc_directory.protoFlorian Klink3-0/+48
This provides an interface to retrieve and upload single Directory messages, or a DAG of them. Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 r/5470 feat(tvix/proto): add rpc_pathinfoFlorian Klink3-0/+44
This defines a service that can be used to upload and retrieve metadata of nix paths. Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-21 r/5469 chore(tvix/eval): fix a broken commentVincent Ambo1-1/+0
Change-Id: I0a6edb51685f94d4712089ae805170da3fb7faae Reviewed-on: https://cl.tvl.fyi/c/depot/+/7609 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5468 fix(tvix/builtin-macros): parse multi-line docstrings correctlyVincent Ambo2-5/+28
Having a multi-line docstring yields multiple doc-attributes in order, however we were previously discarding all but the first one. This reduces them into a single string instead, which can then be displayed as multi-line documentation. Change-Id: I1f237956cdea2e4c746d3f13744e0373c1c645a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7594 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5467 feat(tvix/eval): use `EvalIO::import_path` when coercing pathsVincent Ambo3-3/+28
This "ties the knot" of importing files into a store when referring to them through path literals, e.g. inside of strings. I'm not yet sure if this interface is sufficient for builtins.path (which we haven't implemented at all yet), but it's enough to wire up eval & store initially. In the default implementations nothing interesting happens in this function at all. Change-Id: Ie01ff4161617d1e743a68dbd1a5e54c1b40c0990 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7582 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5466 refactor(tvix/eval): use light spans in builtins.importVincent Ambo2-12/+8
Change-Id: I05732073155b430575babb6f076bf465aef98857 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7581 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5465 feat(tvix/eval): builtins.storeDirAdam Joseph2-1/+17
Returns the store directory through EvalIO::store_dir. Note that this is _optional_ in Tvix, as an evaluation can occur in a context where there simply is no store directory. In those contexts, `builtins.storeDir` returns `null` in Tvix. This would only happen in contexts like Tvixbolt (or completely unrelated use-cases) in practice. Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Change-Id: I5a752c7e89b2f75bd7efb082dbfa5b25e3b1ff3b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7452 Autosubmit: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5464 refactor(tvix/eval): use `EvalIO::read_dir` for equivalent builtinVincent Ambo2-30/+77
Change-Id: I6d782c07166f51587d2f1d06607823268debb5d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7574 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5463 refactor(tvix/eval): use `EvalIO::path_exists` for the builtinVincent Ambo2-2/+18
Change-Id: I49822ce30137777865e7370ee86666636e277b35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7573 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5462 chore(tvix/eval): gate tvix_eval::StdIO behind the `impure` featureVincent Ambo2-1/+6
This shouldn't be available if we've built a "pure" crate. Change-Id: I7c85827ee212890252ff7e0b6242e2c52618cba5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7572 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5461 fix(tvix/cli): use tvix_eval::StdIO in CLIVincent Ambo1-0/+1
... until we have a store-I/O layer, or something that intercepts the store-related stuff appropriately. Change-Id: I22f63435b3f9e118e3faeb2924fda8373a23ea7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7568 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5460 refactor(tvix/eval): use EvalIO::read_to_string in impure builtinsVincent Ambo3-38/+21
With this change, the behaviour of reading a string from a file path is controlled by the provided `EvalIO` structure. This is a huge step towards abstracting away I/O behaviour correctly. Change-Id: Ifde8e46cd863b16e0301dca45a434ad27560399f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7567 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5459 feat(tvix/eval): add EvalIO to public crate APIVincent Ambo6-14/+42
This lets users set the `io_handle` field on an `Evaluation`, which is then propagated to the VM. Change-Id: I616d7140724fb2b4db47c2ebf95451d5303a487a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7566 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5458 feat(tvix/eval): introduce initial EvalIO traitVincent Ambo2-0/+53
This trait is going to be used to abstract filesystem interactions in Tvix. For now, it only contains a `read_to_string` method that closely mirrors `std::fs::read_to_string`. As a first step, to see how this works in practice, we will thread through only this function to the various relevant parts. Two implementations are provided in tvix-eval itself: A dummy implementation (which just returns ErrorKind::NotImplemented for all operations), and a std implementation which delegates to `std` functions. Change-Id: Ied3e3bf4bd0e874dd84e166190e3873a0f923ddb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7565 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5457 refactor(tvix/eval): add a LightSpan type for lighter span trackingVincent Ambo5-12/+61
This type carries the information required for calculating a span (i.e. the chunk and offset), instead of the span itself. The span is then only calculated in cases where it is required (when throwing errors). This reduces the eval time for `builtins.length (builtins.attrNames (import <nixpkgs> {}))` by *one third*! The data structure in chunks that carries span information reduces in-memory size by trading off the speed of retrieving span information. This is because the span information is only actually required when throwing errors (or emitting warnings). However, somewhere along the way we grew a dependency on carrying span information in thunks (for correctly reporting error chains). Hitting the code paths for span retrieval was expensive, and carrying the spans in a different way would still be less cache-efficient. This change is the best tradeoff I could come up with. Refs: b/229. Change-Id: I27d4c4b5c5f9be90ac47f2db61941e123a78a77b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7558 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5456 chore(tvix/cli): rename binary to just 'tvix'Vincent Ambo2-1/+5
Change-Id: If735c81a04f60dcd77f1c40b595b52ed88bbbe95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7580 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 r/5455 feat(tvix/eval): add thunks with suspended native Rust codeAdam Joseph1-1/+45
Having thunks which, when forced, execute native Rust code rather than interpreted opcodes lets us avoid having to bundle `src/libexpr/primops/derivation.nix` like cppnix does by implementing it in Rust instead. Change-Id: If91d77a6736234321eee87ba4b4777eed5a3fe1c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7450 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5454 refactor(tvix/eval): add name-based index over compiler's localsVincent Ambo2-34/+108
Instead of finding locals by doing 2x O(n) walks over the compiler's locals list, use a secondary name-based index for resolving locals by name. Previously, almost 60% (!!) of eval time on some expressions over nixpkgs was spent in `Local::has_name`. This function doesn't even exist anymore now, and eval speed about doubles as a result. Note that this doesn't exactly make the locals code easier to read, but I'm also not sure what we can simplify in there in general. This fixes b/227. Change-Id: I29ce5eb9452b02d3b358c673e1f5cf8082e2fef9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7560 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5453 feat(tvix/eval): remove `derive(Copy)` from UpvaluesAdam Joseph3-12/+26
Change-Id: I0fa069fbeff6718a765ece948c2c1bce285496f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7449 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5452 feat(tvix/eval): wrap Closure in Rc<> to match cppnix semanticsAdam Joseph6-61/+34
Change-Id: I595087eff943d38a9fc78a83d37e207bb2ab79bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7443 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5451 docs(tvix/nar): document how to use NAR writerVincent Ambo1-0/+49
This is done in the form of some comments on the functions, as well as a functional doctest that writes a single file to a NAR. Change-Id: Ic97ebd439e91d6b076685807fe70de098ec02575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7599 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 r/5450 chore(tvix/nar): add CI build targetVincent Ambo6-8/+33
Change-Id: Ic73a391da9a733cade26114ab1127907c8d62a57 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7598 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5449 chore(tvix/cli): re-add observer flagsVincent Ambo1-2/+34
Users can again pass flags for dumping the AST, bytecode, and runtime trace. With this commit the CLI is at feature-parity with what we had before, but entirely through the new API. Change-Id: I30fe26f243224b25d1e4f828fec607325ef88306 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7550 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5448 chore(tvix/cli): re-add NIX_PATH handlingVincent Ambo1-1/+7
Change-Id: I5595d6f5141ed4a533ca44a46264dc604fca6be1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7549 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5447 chore(tvix/cli): re-add `--raw` argumentVincent Ambo1-14/+15
Change-Id: I82f469f7f186c754c40bd941587b30aa3fc48045 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7548 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5445 refactor(tvix/eval): consume `self` in Evaluation::evaluateVincent Ambo2-9/+8
This simplifies lifetime management for observers in callers of tvix_eval. Change-Id: I2f47c8d89f22b1c766526e5d1483c0d026b500ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/7546 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5444 feat(tvix/eval): add observer configuration to public APIVincent Ambo1-11/+21
With this change, it should be possible to have both existing use-cases (CLI & Tvixbolt) use the same API. Change-Id: I2195264f08cc892177b559a28660dc5f98e48e41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7545 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-28T04·52+0000