about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-12-11 r/7158 chore(tvix): drop sled compression supportFlorian Klink9-16/+14
It's been a while since the last sled release, and that one binds to a pretty old version of zstd, requiring workarounds like cl/10090. Upstream sled main branch currently has zstd halfway patched out (it's a no-op, but the feature flag and options are still there), and it's in that state for a year. Rather than maintaining our own fork of sled, let's just stop using the compression feature in sled, dropping the version pin to zstd that way, removing the need for cl/10090. This doesn't mean we won't reintroduce per-blob compression - but we probably just won't let sled take care of the compression, but do it ourselves - which is necessary for more chunked blob storage anyways. Even though we do drop the feature flag, we still need to explicitly use use_compression(false). Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-11 r/7157 chore(tvix/proto): drop evaluator.protoFlorian Klink2-165/+0
This only contained the (unused) evaluator.proto file. Considering we're less likely to have the CLI talk to a long-running evaluator, but instead embed the evaluator inside the CLI, remove this. If we add a RPC to speak to an evaluator, we can resurrect this from git history. Change-Id: I2196aade55221660330dfd32dc3e52c39ec6ed43 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10241 Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-11 r/7156 refactor(tvix): use granular proto filesFlorian Klink2-23/+17
Only pass in the proto files that are actually needed to build that crate. They are already constructed in depot.tvix.$crate.protos.protos. Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-11 r/7155 refactor(tvix/*/protos): separate lint targetFlorian Klink3-46/+107
Break up the go-bindings derivation. Keep "protos" containing all proto files (well, and the buf config), and use it for a check phase running linter and formatter, as well as the existing "go-bindings" attribute Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-11 r/7154 feat(tvix/build): add derivation_to_build_requestFlorian Klink7-0/+272
This function converts from a nix_compat::derivation::Derivation to a BuildRequest. In addition to the Derivation itself, it needs two lookup functions to map input paths to their castore nodes. Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 r/7153 docs(tvix/*store-go): fix README invocationFlorian Klink2-3/+3
The command is called `regenerate`, not `generate`. Change-Id: I18075042ebd461e4dd0718a936e6bbe738a144d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10259 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-11 r/7152 feat(tvix/build-go): initFlorian Klink8-0/+869
This adds the generated golang bindings for tvix-build. Change-Id: I2eb0d1cc38bc2fa34afd7c904eea05c5ee192cce Reviewed-on: https://cl.tvl.fyi/c/depot/+/10242 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 r/7151 feat(tvix/build): initFlorian Klink14-0/+330
This adds the tvix-build crate, currently only containing a `tvix_build::proto` module, exposing the data structures defined in tvix/build/protos. Change-Id: I75f5d9196969ed0877b1fe640cacfecba0fb2e03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10235 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-11 r/7150 chore(tvix/store/protof): buf formatFlorian Klink6-168/+168
Change-Id: I29b0fb62aa849ee01860dc7fe3dbba9b941ca172 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10238 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 r/7149 chore(tvix/castore/protof): buf formatFlorian Klink5-97/+97
Change-Id: Idf11de78b0d6eca69fda34a89f2c57a00ed89ad5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10237 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 r/7147 fix(tvix): Fix cwd in benchmark scriptAspen Smith1-1/+4
Change-Id: I6d9435c6710ce086aaedb2c1dc634b73a19f476c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10249 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 r/7146 chore(tvix): bump criterion to 0.5Florian Klink4-159/+26
Change-Id: I28904ca23437b4bb745c0eb1f4eb9ae33e09eb5a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10244 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-11 r/7145 docs(tvix): fix crate2nix-generate invocationFlorian Klink1-1/+1
This got renamed recently, but we forgot to update it here. Change-Id: I7d713c8a0e6ccca57fe67985d9cb4e7f1eeef3b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10243 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 r/7144 feat(tvix): Benchmark nixpkgs eval with hyperfineAspen Smith2-2/+11
Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for evaluating the outPath of hello and a cross-compiled hello. Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi>
2023-12-10 r/7142 fix(tvix/cli): panic on root cause of the fetchurl(non-boot) bugAdam Joseph1-1/+27
Currently we produce wrong drvPaths for a large number of packages that use fetchurl (but not fetchurlBoot, which is what stdenv uses). A simple reproducer is `pkgs.perl538`. I debugged this down to the root cause, which is the fact that tvix doesn't realize that the mapping from FOD-paths to outputHash is *NOT* a 1:1 mapping. It is a many-to-one mapping. You can have lots of different FODs with the same outputHash or even the same outPath. For example, perl538.src and perldevel.src use the same source tarball but a different `version`. Anyways, I have found the root cause but have run out of time for a while, so I've added a panic!() to in the spot where we have a logic bug in order to call it out. Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 r/7141 feat(tvix): make clippy stricterFlorian Klink1-1/+1
Also run clippy on tests, and enable all features. Change-Id: Ide9f1bc9f565333072afb918c391c7930b658f41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10234 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 r/7140 refactor(tvix/castore): address clippyFlorian Klink1-10/+7
We match to destructure a single pattern. Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-09 r/7139 refactor(nix-compat/aterm): address clippyFlorian Klink1-6/+1
This pushes to a Vec immediately after creation. Change-Id: I2360b45810475d98ededc1d097fb4cbdeabc576b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10232 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-09 r/7138 refactor(store/fs): address clippyFlorian Klink1-1/+1
Change-Id: I28d60263ef672942940656caa8e0f9d20cc6b7d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10231 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 r/7137 refactor(nix-compat/nixbase32): address clippyFlorian Klink1-1/+1
Change-Id: I07b53ead10b0efb31e88c6ae93c124c5f35261d3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10230 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 r/7136 refactor(nix-compat/narinfo): address clippy lintsFlorian Klink3-3/+3
Change-Id: I00f28624ec0469525e52f408f00caa20d3b701b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10229 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 r/7135 refactor(nix-compat/derivation): address clippyFlorian Klink1-3/+3
Change-Id: I5d7057347838f733d7dc1331c7b7ef8d27244412 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10228 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-12-09 r/7134 refactor(tvix/eval): address clippy lintsFlorian Klink2-2/+2
Change-Id: Ic2bd4e8291b30ceac9fa0e88a4f56e61ae99b603 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10227 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-09 r/7133 chore(tvix/nix_cli): drop crateFlorian Klink7-186/+0
The only thing this was doing was invoking nix-store --add, which is not gonna help us populate the tvix-store at all (and we now have `tvix-store import`). This is also (rightfully) causing clippy warnings, because of some fields being unused. It's more of a skeleton, and rather than shelling out to Nix for some usecases, we might introduce a "compatible" Nix CLI frontend for a subset of commands. Drop this for now, to decrease the noise and confusion. Change-Id: I2fd399e9320260f08893b685561755af9c7c961c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10226 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 r/7132 refactor(nix-compat/store_path): from_absolute_path to StorePathRefFlorian Klink3-19/+20
The only non-test usage was only checking for the error case, and we can still convert this to an owned StorePath by calling to_owned() on StorePathRef. Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-09 r/7131 refactor(nix-compat/derivation): rename input derivation componentsFlorian Klink1-3/+3
Match the naming in parse_input_derivations, call the keys "input_derivation", and the values "output_names". Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-09 r/7130 feat(tvix/eval): impl DoubleEndedIter for OwnedAttrsIteratorAdam Joseph1-0/+10
Change-Id: I4bd85dbe9c27047f4abbdeff4e2b796e9bcab3a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10211 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-12-09 r/7129 chore(store/src/tests): make clippy shut upAdam Joseph1-12/+6
Change-Id: I6dfceaa32a8c01f8395b7889ae19847fabf95ed3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10214 Tested-by: BuildkiteCI Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 r/7128 fix(tvix/glue): do not panic if PathInfoService returns ErrAdam Joseph1-1/+1
Nixpkgs tries to `import` the value returned by `builtins.unsafeGetAttrPos`, which in our case is the file `/deep/thought`. Since that doesn't exist, tvix-glue panics, but there's no interpreter backtrace to follow. Let's return an Err instead of panicking. ------------------------------------------------------------------------------ Before: thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27: called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'main' panicked at src/tvix_store_io.rs:276:58: called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...) Command exited with non-zero status 101 ------------------------------------------------------------------------------ After: thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27: called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace note: while evaluating this Nix code --> [code]:1:1 | 1 | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78 | 6 | runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } '' | ______________________________________________________________________________^ 7 | | mkdir -p $out/bin 8 | | 9 | | for prog in ${lld}/bin/*; do ... | 27 | | ln -s ${lld}/bin/lld $out/bin/ld 28 | | '' | |__^ note: while evaluating this as native code (coerce_to_string) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9 | 27 | ln -s ${lld}/bin/lld $out/bin/ld | ^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23 | 249 | outPath = assert condition; drv.${outputName}.outPath; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (force) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30 | 249 | outPath = assert condition; drv.${outputName}.outPath; | ^^^^^^^^^ note: while evaluating this as native code (throw) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8 | 262 | in handler msg; | ^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14 | 254 | else '' | ______________^ 255 | | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. 256 | | 257 | | '' + (builtins.getAttr reason remediation) attrs; | |________________________________________________________^ note: while evaluating this as native code (force) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14 | 254 | else '' | ______________^ 255 | | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. 256 | | 257 | | '' + (builtins.getAttr reason remediation) attrs; | |__________^ error[E029]: I/O error: /deep/thought:42: task panicked --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41 | 255 | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. | ^^^^^^^^^^^^^^^ Command exited with non-zero status 1 Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}} Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223 Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-09 r/7127 chore(nix_cli): make clippy shut upAdam Joseph1-0/+1
Change-Id: I0c243baf4deecfd81bd939da86d685e2cbc59d69 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10215 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 r/7126 fix(tvix/nar-bridge): fix FOD hashAdam Joseph1-1/+1
The golang mothership seems to be monkeying with hashes again. Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209 Tested-by: BuildkiteCI Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: flokli <flokli@flokli.de>
2023-12-07 r/7125 feat(tvix): run crate2nix generate in CIAdam Joseph1-5/+57
This runs `crate2nix generate` in CI and then runs `depotfmt` on the result to ensure that our machine-generated code is really, really readable and pretty. Then it checks that the result of all that is identical to the committed Cargo.nix. A self-hashing FOD is used to allow network access. No magic hashes are involved. Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: I68ec5003dbc6a40894a5a4d6e902f138c99f6719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10194 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-12-06 r/7123 refactor(tvix): Make benchmark script directory-agnosticAspen Smith1-1/+1
Allow running the benchmark script from any directory - primarily so Windtunnel can point to the /tvix josh workspace rather than the depot overall Change-Id: Ie5fc3ef995bf8114277298ae5c5010e6a0bf13ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/10205 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2023-12-06 r/7120 feat(tvix/eval): rewrite Thunk::force() in nonrecursive formAdam Joseph3-64/+97
This commit rewrites Thunk::force() so that it is not (directly) self-recursive. It maintains a Vec of all the previously-encountered thunks which point to the one it is currently forcing, rather than recursively calling itself. Benefits: - Short term: This commit saves the cost of a round-trip through the generator machinery for the generators::request_force() which is removed by this commit. - Medium term: Once a similar transformation has been applied to nix_cmp(), nix_add(), nix_eq(), and coerce_to_string(), those four functions, along with Thunk::force(), will make non-tail calls only to each other. They can then be merged into a single tail-recursive function which does not use the generator machinery at all: enum Task { Cmp, Add, Eq, CoerceToString, Force}; fn Value::walk(task:Task, v1:Value, v2:Value) { // ... - Long term: The long-term goal here is to use generators **only for builtins** and [Marionette]-style remote control of the VM. In other words: use `async` for things that actually involve concurrency. Calls from the VM to builtins can then be blocking calls, because even cppnix will overflow the stack if you make a MAX_STACK_DEPTH-deep recursive call which passes through a builtin at every stack frame (e.g. `{ func = builtins.sort (a: b: ... func ...) ...}`). This way the inner "tight loop" of the interpreter doesn't pay the costs of `async` and generators. These costs manifest in terms of: performance, complex nonlocal control flow, and language impediments (async Rust is a restricted subset of real Rust, and is missing things like traits). [Marionette]: https://firefox-source-docs.mozilla.org/testing/marionette/Intro.html Change-Id: I6179b8abb2ea0492180fcb347f37595a14665777 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10039 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-12-05 r/7119 fix(tvix/eval): Return error rather than panicking on bad substringAspen Smith2-1/+10
If builtins.substring is invoked with (byte!!) offsets that aren't at codepoint boundaries, return an error rather than panicking. This is still incorrect (see b/337) but pushes the incorrectness forward a step. Change-Id: I5a4261f2ff250874cd36489ef598dcf886669d04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10199 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-05 r/7118 refactor(tvix): move src into let bindingFlorian Klink1-4/+6
Change-Id: Ida2a3ac722fb2445745759323975884dfeef3e87 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10193 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-05 r/7115 feat(tvix): Add script for running benchmarks in WindtunnelAspen Smith1-0/+10
Currently this just uses a Docker container, which is gross but works fine for now since we don't have the ability to build benchmarks in nix as of cl/7538 Change-Id: I48e317f44bc2c73533d7663403786a3a37c7952f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10189 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi>
2023-11-28 r/7087 fix(tvix/nar-bridge): drop pathinfoserviceFlorian Klink3-356/+0
This now exists in tvix-store directly, as NixHTTPPathInfoService, and contrary to this version, also validates signatures. Change-Id: Ib6ca161e40d627b7d9741839fc849f2392f422da Reviewed-on: https://cl.tvl.fyi/c/depot/+/10155 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 r/7086 feat(tvix/store/pathinfosvc/nix_http): allow configuring pubkeysFlorian Klink1-6/+28
This allows setitng the trusted-public-keys URL parameter to a (whitespace-separated) list of public keys. NARInfo files retrieved need to contain a valid signature. Change-Id: Ifd6580b723cbae3182e9cadfa54f1ca2b41d6599 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10153 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-28 r/7085 feat(tvix/store/pathinfosvc): add signature verificationFlorian Klink1-1/+33
Introduce an Option<Vec<narinfo::PubKey>>, configurable with a `set_public_keys` method. If set, this configures NixHTTPPathInfoService to validate signatures. Change-Id: I157c5e13c41fc9bfd40b0655381fb4cf33900868 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10152 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-28 r/7084 feat(nix-compat/narinfo): add PubKey::verify(fp, sig)Florian Klink1-0/+32
This makes it easy for each PubKey to check if a given Signature is correct for a given fingerprint. Change-Id: I56e6211d133f74f390fd1ae3ae799eef12221904 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10151 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-28 r/7083 feat(nix-compat/narinfo): add PubKeyFlorian Klink2-0/+119
This represents a ed25519 public key and "name". These are normally passed in the `trusted-public-keys` Nix config option, and consist of a name and base64-encoded ed25519 pubkey, separated by a `:`. Change-Id: I9ab4b3e0e5821805ea6faf2499626630fc5a3f0a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10150 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-28 r/7082 refactor(nix-compat/narinfo/signature): rename Error typeFlorian Klink2-9/+7
Just call this Error, we can infer from the package what error this is. Change-Id: I5df25d2873ec739c49c08804f35562c84c222e06 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10149 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-28 r/7081 feat(tvix/store/pathinfosvc/nix_http): check Nar{Size,Hash} matchesFlorian Klink1-13/+84
Ensure the initially communicated NarHash/NarSize from the NarInfo matches what we read, and don't return a PathInfo message if there's a mismatch. Also move the buffering layer around a bit. Change-Id: I68c60ecfaf0f9cd5edacea648437ecb0c9729251 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10148 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-28 r/7080 fix(tvix/store/fs): don't panic on PathInfoService errorFlorian Klink1-11/+9
An error in the PathInfoService request can appear in case the underlying request returns an error. We shouldn't panic and bork the fuse mount, but instead return an IO error. Change-Id: I2daeae629e1627d06adcd7b82ddb76c50c602212 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10154 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-27 r/7076 chore(3p/sources): bump channels & overlays (2023-11-27)Vincent Ambo1-2/+2
* picked avrdude from stable channel * removed override for texlive, as the upstream fix is merged * picked awscli2 from stable channel * bump tdlib to 1.8.21 (new minimum for telega.el) * tvix/turbofetch: switch to nixpkgs-native mechanism for CARGO_MANIFEST_LINKS (whatever that is) Change-Id: Ic695721b5ca750b89d21cab7a257e1db682b23c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10083 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-27 r/7075 fix(nix-compat/narinfo/signature): validate name fieldFlorian Klink1-2/+15
We should restrict this to alphanumeric mostly, and we definitely don't want newlines. Not entirely sure about the exact additionally allowed characters outside of alphanumeric, but this can always be extended further. Change-Id: I1357e79e553f2df2fa97792889f63f0f35d50ed5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10147 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-11-27 r/7074 fix(nix-compat/narinfo): don't panic trying to parse signaturesFlorian Klink1-6/+13
BASE64.decode_mut panics if we're passing data that has the wrong size. Do the size check first and error out there. Also update the error, and talk about b64-encoded sizes. Change-Id: I290f80a37d48526a30bf1df9d1d9fe34865008eb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10146 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
2023-11-27 r/7073 refactor(nix-compat): use ed25519_dalek::SIGNATURE_LENGTHFlorian Klink1-8/+9
No need to hardcode magic numbers here, we have a constant for that. Change-Id: I67b671c0c4bb7c3bfb001e9c36499f31873ee717 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10145 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-25 r/7065 refactor(tvix/eval): use `or_default` helper in entry APIVincent Ambo1-3/+1
This fixes a future clippy lint. Change-Id: Ic830e94ef23595580c1037f10878c76bbb546dd9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10110 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-25T00·35+0000