about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-03-11 r/7685 feat(tvix/castore/blobsvc/from_addr): support object_storeFlorian Klink1-1/+21
The object_store crate supports a ton of different stores, with different schemes. For now, use a objectstore+ scheme prefix to enable these. Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-03-11 r/7684 feat(tvix/castore/blobsvc): add object storage implementationFlorian Klink6-9/+1271
This uses the `object_store` crate to expose a tvix-castore BlobService backed by object storage. It's using FastCDC to chunk blobs into smaller chunks when writing to it. These are exposed at the .chunks() method. Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-03-11 r/7683 feat(tvix/store/bin): allow disabling OTLP at runtimeFlorian Klink1-32/+39
This was only possible by disabling without the otlp feature flag so far. Introduce the same --otlp=false mechanism that nar-bridge also supports to be able to turn it off at runtime. Change-Id: Ib22a364c35056ca9d8e327c0e2a79970a4cf4b2b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11135 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-11 r/7679 feat(tvix/glue): Add AsyncRead wrapper to decompress streamsAspen Smith8-1/+777
Add a new AsyncRead wrapper, DecompressedReader, that wraps an underlying AsyncRead, but sniffs the magic bytes at the start of the stream to determine which compression format is being used out of the three that are supported by builtins.fetchTarball, and switches to the correct decompression algorithm adapter dynamically. This will be used in the implementation of builtins.fetchTarball Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-11 r/7678 feat(tvix/glue): Implement builtins.fetchurlAspen Smith11-54/+454
Implement the fetchurl builtin, and lay the groundwork for implementing the fetchTarball builtin (which works very similarly, and is implemented using almost the same code in C++ nix). An overview of how this works: 1. First, we check if the store path that *would* result from the download already exists in the store - if it does, we just return that 2. If we need to download the URL, TvixStoreIO has an `http_client: reqwest::Client` field now which we use to make the request 3. As we're downloading the blob, we hash the data incrementally into a SHA256 hasher 4. We compare the hash against the expected hash (if any) and bail out if it doesn't match 5. Finally, we put the blob in the store and return the store path Since the logic is very similar, this commit also implements a *chunk* of `fetchTarball` (though the actual implementation will likely include a refactor to some of the code reuse here). The main thing that's missing here is caching of downloaded blobs when fetchurl is called without a hash - I've opened b/381 to track the TODO there. Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests' rustls-native-roots feature, 2024-03-03) for more info. Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-03-10 r/7662 docs(tvix/README): update IRC channelsFlorian Klink1-4/+16
`#tvl` is the general TVL community channel, `#tvix-dev` is dedicated to Tvix development discussion. Change-Id: If899d9ae70dffb4acf6a41ded54f80a1f0551c5d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11110 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-09 r/7661 fix(tvix/store/grpc/pathinfo): skip_all fields, handle errorsFlorian Klink1-15/+20
request only contains the outer metadata wrapping, and that's not too interesting: > Request { metadata: MetadataMap { headers: {"content-type": > "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers", > "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions: > Extensions } Drop these fields for now, and rely on the underlying implementations to add instrumentation for the application-specific fields. Also, ensure we handle all error cases properly, and log them. We don't use `err` from instrument, as that'd also log an error on `Status::not_found`. Change-Id: Id1b983cb8b059c148c8a376f8802a1d28c59ba97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11103 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-09 r/7660 fix(tvix/castore/grpc/directory): skip_all fields in instrumentFlorian Klink1-15/+21
This only contains the outer metadata wrapping, and that's not too interesting: > Request { metadata: MetadataMap { headers: {"content-type": > "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers", > "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions: > Extensions } Drop these fields for now, and rely on the underlying implementations to add instrumentation for the application-specific fields. Clean up the error logging a bit. Change-Id: Ife1090ed411766a61e1fa60fd4c9570f38de1e98 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11102 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-09 r/7659 fix(tvix/castore/grpc/blob): skip_all fields in instrumentFlorian Klink1-5/+11
This only contains the outer metadata wrapping, and that's not too interesting: > Request { metadata: MetadataMap { headers: {"content-type": > "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers", > "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions: > Extensions } Drop these fields for now, and rely on the underlying implementations to add instrumentation for the application-specific fields. Log errors in some places where we didn't so far. Change-Id: Ia68d6c526987d3716be62a0809195401cf28512b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11101 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-09 r/7658 chore(tvix/nix-compat): bump zstd dependencyFlorian Klink3-32/+30
Otherwise this causes conflicts when adding a more recent zstd version to another crate. Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-03 r/7650 fix(tvix/castore): also set SSL_CERT_FILE for tests thereFlorian Klink1-0/+3
For everything using reqwest here during test cases, we also need to set SSL_CERT_FILE. Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7649 refactor(tvix/store/protos): more granular nix-compat conversionsFlorian Klink1-17/+18
implement From<&nix_compat::nixhash::CAHash> for nar_info::ca::Hash individually, and make From<&nix_compat::nixhash::CAHash> for nar_info::Ca a small wrapper that uses it, as well as the .hash().digest_as_bytes() for the digest. Change-Id: I7e9b6edd1e3f149eb270faf2928cd846d74e77ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/11071 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 r/7648 refactor(tvix/*/from_addr): improve test debuggabilityFlorian Klink3-6/+18
If there's an unexpected test failure, print it out, rather than just saying something is false even though it should be true. Use .expect() for this, which displays the error if it failed. We can't use expect_err(), as our stores are not display'able, so use an assertion with a message there. Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 r/7647 docs(tvix): remove differences-drv-paths.mdFlorian Klink2-36/+0
This was true while we didn't do string contexts, but drvPath should not differ anymore. Change-Id: I58e6ed439519bd4d68259d1577946f3c65110dc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11074 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
2024-03-03 r/7646 feat(tvix/docs): ship custom highlight.js with Nix supportFlorian Klink1-0/+590
Selected languages: - bash - c - cpp - diff - go - ini - json - markdown - nix - protobuf - rust - shell - xml - yaml Created with https://highlightjs.org/download Change-Id: I76a87d2c1e9fa582d1474f8ffb9f3129e9a858bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11073 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-03-03 r/7645 feat(tvix/docs): switch to mdbookFlorian Klink11-96/+46
Some of the docs are still outdated (like architecture and drv path inconsistencies). Change-Id: I7a6afceb008ef4cd19a764dd6c637b39fa842a2e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11072 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-03-03 r/7644 fix(tvix/tools/weave): handle sliced arrays correctlyedef1-8/+12
The start/end offsets are not necessarily coterminous with the underlying values array, so even if the stride is fixed, we still we need to slice the chunks down to match the start/end offsets. This bug shouldn't affect the correctness of any existing code, since we're always working with unsliced arrays read directly from Parquet. Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 r/7642 feat(tvix/store): use reqwests' rustls-native-roots featureFlorian Klink4-23/+14
This makes reqwest honor `SSL_CERT_FILE` - previously it was using the chain bundled in webpki-roots. `object_store` pulls in `reqwest` with this feature, and the cargo solver will enable that feature globally as soon as we pull it in, as it assumes features are additive. This requires setting `SSL_CERT_FILE` when running tests, otherwise they'll fail with the unhelpful "NotFound" error. This was quite some fun to debug, why adding `object_store` to tvix-castore suddenly made tvix-store tests fail! Change-Id: I64fc82b4d994715480efdb1ffecb279716456ab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11090 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7641 feat(tvix/castore): add HashingReader, B3HashingReaderFlorian Klink2-0/+90
HashingReader wraps an existing AsyncRead, and allows querying for the digest of all data read "through" it. The hash function is configurable by type parameter, and we define B3HashingReader. Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7640 feat(tvix/castore/digests): impl From digest::Output<_> for B3DigestFlorian Klink4-6/+33
This allows calling .into() to get a B3Digest. Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 r/7634 feat(tvix/nix-compat): add HashMode and CAHash::mode()Florian Klink2-0/+17
This allows peeking at the mode without matching on all enum kinds of CAHash directly. Change-Id: Icca147a876f7f4cff0bbaa067ac4dae40d05bee9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11069 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-03 r/7633 fix(tvix/nix-compat): Make CAHash deserialize more formatsBrian Olsen3-67/+230
Currently CAHash only deserializes the hash in hex code while the serializer outputs a nixbase32 hash. This means that you can't currently deserialize what has been serialized. This change makes deserialize support any digest format (so hex, nixbase32 and base64) as well as flattens the deserialize code and error handling. It also implements serde methods of HashAlgo directly using Display and TryFrom implementations because otherwise these would get serialized as eg. Sha256 instead of sha256 which also broke CAHash serialize/deserialize. Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 r/7632 refactor(tvix/nix-compat): rename NixHash::{digest,hash}Florian Klink3-4/+4
This doesn't return the digest, but the internal NixHash. To get the digest, you use `ca_hash.hash().digest_as_bytes()` (as done in tvix/nix-compat/src/derivation/write.rs). Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-03 r/7631 refactor(tvix/castore/blobsvc/chunked_reader): refactor, documentFlorian Klink1-131/+139
The public-consumable thing here is ChunkedReader, not ChunkedBlob. ChunkedBlob is a helper that can be used to get a new AsyncRead, but not AsyncSeek. It is used internally by ChunkedReader whenever the client seeks. Make this more obvious, by extending the documentation, and putting ChunkedReader at the top of this file. Also make ChunkedBlob and its methods private, and give ChunkedReader a more useful constructor (from_chunks, instead of from_chunked_blob). Change-Id: I2399867591df923faa73927b924e7c116ad98dc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-02 r/7627 feat(tvix/castore/blobsvc): BlobReader for more trivial typesFlorian Klink1-0/+2
Change-Id: I80e4f26c41a504fa4c6a013c2a1e76de613ba294 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11078 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-02 r/7626 fix(tvix/castore/blobwriter): don't require Sync + 'staticFlorian Klink1-1/+1
There's no reason for these two. Change-Id: Ie6f238bbb0b17971c9877b11b61ea7ebca573c13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11075 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-02-29 r/7625 feat(tvix/eval): implement `builtins.hashFile`Padraic-O-Mhuiris6-25/+42
The primary addition is the `builtin_hashFile` function which attempts value-path coercion and on success reads the file to a nix_string. Similar to the `builtins_hashString` implementation, the hash is then derived using the `hash_nix_string` function in the introduced hash module which has extracted hashing functionality initially specified in cl/11005. Change-Id: I5e3ef0317223af99488ebb79efc2fb49b4cbc1b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11007 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 r/7617 feat(tvix/tools/weave): initedef9-0/+11471
Scalable tracing GC for the cache.nixos.org dataset. Change-Id: I6c7852796f28e1a1c7607384ffb55f44407e1185 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10765 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-24 r/7601 chore: update links to tvixboltFlorian Klink2-2/+2
Use bolt.tvix.dev everywhere. Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-23 r/7597 feat(tvix/eval): implement `builtins.hashString`Padraic-O-Mhuiris13-11/+167
Implements md5, sha1, sha256 and sha512 using the related crates from the RustCrypto hashes project (https://github.com/RustCrypto/hashes) Change-Id: I00730dea44ec9ef85309edc27addab0ae88814b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11005 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-02-23 r/7596 refactor(tvix/glue): Make a single errors.rs moduleAspen Smith2-10/+9
To pave the way for adding a new error type for builtins in this crate, move DerivationError to a new builtins::errors module. Change-Id: I65fcad63e43ed40ad39c2c6540a2ab80fdd90fd4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11016 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-02-22 r/7594 refactor(nix-compat/store_path): simplify build_ca_pathFlorian Klink1-14/+12
Move the the `fixed:out:[r:]{}:` generation to a helper function, use matches! for more clarity. Change-Id: I4e930c42aacbf5c7451d1f8c8c80ccb4c45389f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11006 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi> Autosubmit: flokli <flokli@flokli.de>
2024-02-21 r/7591 feat(tvix/eval): Store string context alongside dataAspen Smith9-79/+377
Previously, Nix strings were represented as a Box (within Value) pointing to a tuple of an optional context, and another Box pointing to the actual string allocation itself. This is pretty inefficient, both in terms of memory usage (we use 48 whole bytes for a None context!) and in terms of the extra indirection required to get at the actual data. It was necessary, however, because with native Rust DSTs if we had something like `struct NixString(Option<NixContext>, BStr)` we could only pass around *fat* pointers to that value (with the length in the pointer) and that'd make Value need to be bigger (which is a waste of both memory and cache space, since that memory would be unused for all other Values). Instead, this commit implements *manual* allocation of a packed string representation, with the length *in the allocation* as a field past the context. This requires a big old pile of unsafe Rust, but the payoff is clear: hello outpath time: [882.18 ms 897.16 ms 911.23 ms] change: [-15.143% -13.819% -12.500%] (p = 0.00 < 0.05) Performance has improved. Fortunately this change can be localized entirely within value/string.rs, since we were abstracting things out nicely. Change-Id: Ibf56dd16c9c503884f64facbb7f0ac596463efb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10852 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: aspen <root@gws.fyi>
2024-02-21 r/7588 revert(tvix): don't use Rc::clone explicitlyaspen4-9/+9
This reverts commit d9565a4d0af3bffd735a77aa6f1fd0ec0e03b14a. Reason for revert: this was intentional - putting Rc::clone instead of .clone is a common Rust idiom, and makes it explicit that we're cloning a shared reference, not an underlying resource Change-Id: I41a5f323ee35d7025dc7bb02f7d5d05d0051798d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10995 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 r/7587 fix(tvix/eval): allow reading non-UTF8 filesFlorian Klink4-14/+14
With our values using bstr now, we're not restricted to only reading files that contain valid UTF-8. Update our `read_to_string` function to `read_to_end` (named like `std::io::Read::read_to_end`), and have it return a Vec<u8>. Change-Id: I87f0291dc855a132689576559c891d66c30ddf2b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11003 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Pádraic Ó Mhuiris <patrick.morris.310@gmail.com> Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 r/7586 docs(tvix/eval): notes on the implementation of catchable errorssterni1-0/+131
Change-Id: I3b2f039d2c2d40ee5d13d071915ced6f3deb18f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11000 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-21 r/7585 feat(tvix/nix-compat): Use `StorePath` in `Output`Peter Kolloch11-82/+119
https: //b.tvl.fyi/issues/264 Change-Id: Icb09be9643245cc68d09f01d7723af2d44d6bd1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11001 Autosubmit: Peter Kolloch <info@eigenvalue.net> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-21 r/7584 feat(tvix/nix-compat): input_sources as StorePathPeter Kolloch6-31/+53
https: //b.tvl.fyi/issues/264 Change-Id: I7a235734dc1f8e93e387a04ba369f3b702c6d5b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10992 Autosubmit: Peter Kolloch <info@eigenvalue.net> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Peter Kolloch <info@eigenvalue.net> Tested-by: BuildkiteCI
2024-02-21 r/7583 feat(tvix/nix-compat): input_derivations with StorePathsPeter Kolloch9-73/+138
...in `Derivation`. This is more type-safe and should consume less memory. This also removes some allocations in the potentially hot path of output hash calculation. https: //b.tvl.fyi/issues/264 Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Peter Kolloch <info@eigenvalue.net> Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-21 r/7582 feat(tvix/nix-compat): generalize aterm writing for derivationPeter Kolloch3-9/+100
...so that we can also use `StorePath`s in derivation.input_derivations. Towards https://b.tvl.fyi/issues/264 Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973 Reviewed-by: Peter Kolloch <info@eigenvalue.net> Autosubmit: Peter Kolloch <info@eigenvalue.net> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 r/7580 fix(tvix/eval): Propagate catchables in builtins.groupByAspen Smith3-4/+6
One last place where we needed to wrap the forcing of the element of a list in `try_value!`. This fixes a previously `notyetpassing` test Change-Id: I8827a3e39630e6959013b70bdfa9cbcb93f4e91c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10789 Autosubmit: aspen <root@gws.fyi> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2024-02-20 r/7578 feat(tvix/eval): implement `builtins.filterSource`Ryan Lahfa9-69/+521
We add a new set of builtins called `import_builtins`, which will contain import-related builtins, such as `builtins.path` and `builtins.filterSource`. Both can import paths into the store, with various knobs to alter the result, e.g. filtering, renaming, expected hashes. We introduce `filtered_ingest` which will drive the filtered ingestion via the Nix function via the generator machinery, and then we register the root node to the path info service inside the store. `builtins.filterSource` is very simple, `builtins.path` is a more complicated model requiring the same logic albeit more sophisticated with name customization, file ingestion method and expected SHA-256. Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 r/7577 fix(tvix/eval): propagate catchable errors at the top of an evalVincent Ambo3-6/+46
(Re-)Adds an error variant that wraps a catchable error kind, which is used for returning the result of an evaluation. Previously this would return the internal catchable value, which would lead to panics if users tried to use these. Somehow this was missed; I think we need error output tests. Change-Id: Id6e24aa2ce4ea4358a29b2e1cf4a6749986baf8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10991 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 r/7576 refactor(tvix): don't use Rc::clone explicitlyFlorian Klink3-5/+5
All these functions have an Rc of Rc<TvixStoreIO> or Rc<dyn StoreIO>, so we can call io.clone() directly. Change-Id: I96a5bcb9af4aca93892cb72bcfaf14540da52381 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10989 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-20 r/7575 feat(tvix/serde): add an example using nixpkgs/libVincent Ambo1-0/+34
This displays how users can configure an impure evaluation for tvix-serde, which makes it possible to use e.g. `nixpkgs/lib`. We might want to add an example showing how the full Nix-glue compatibility stuff can be added here, too. Change-Id: I2224a3fc66e739969d4c723c3d9d8127a046b6fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10994 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2024-02-20 r/7574 fix(tvix/eval): make path resolution work by default in impure modeVincent Ambo1-0/+6
The previous behaviour (enabling `import`, but not allowing e.g. `<nixpkgs/lib>` to resolve) was very confusing. Now imports from NIX_PATH become enabled by default, unless the user already overrode that behaviour with something else by setting `Evaluation::nix_path` manually. Change-Id: Iad970beb633d9887be4b185b01e6f5858d81bea3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10993 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-02-20 r/7573 fix(tvix/eval): fix accidental recursion in printing errorsVincent Ambo1-1/+1
There's some code path where fancy_format_str()->Error::Display, which recurses forever and overflows the stack. This was introduced in a previous commit today. Change-Id: I87a59492099f6c138c752478901b9aa614bb57cc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10990 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-20 r/7572 refactor(tvix/eval): use internal SourceCode field in error printersVincent Ambo4-27/+21
Makes use of the SourceCode field now being stored directly in errors (see parent CL). With this change, the default `Display` implementation can now format errors correctly, and there is no need to keep a `SourceCode` around just for error formatting. Updates dependent crates (CLI, serde, tvixbolt) to use this correctly. Change-Id: Iddc5d7a6b4bab391f30a999e4c68aca34304c059 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10987 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 r/7571 refactor(tvix/eval): add SourceCode directly into error typesVincent Ambo6-40/+67
With this change it's no longer necessary to track the SourceCode struct separately from the evaluation for error reporting: It's just stored directly in the errors. This also ends up resolving an issue in compiler::bindings, where we cloned the Arc containing file references way too often. In fact those clones probably compensate for all additional SourceCode clones during error construction now. Change-Id: Ice93bf161e61f8ea3d48103435e20c53e6aa8c3a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10986 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-20 r/7570 docs(tvix/castore/directorysvc): K/V is not necessarily flatFlorian Klink3-0/+20
Some implementations of DirectoryService might not allow retrieval of intermediate Directory nodes, that are not at the "root". Think about an object store implementation. The client is doing a get_recursive anyways to reduce the number of roundtrips. By documenting the fact we don't need to support looking up intermediate Directory messages, we can just batch all directories into the same object, keyed by the root. Change-Id: I019d720186d03c4125cec9191e93d20586a20963 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI