depot/tvix/eval, branch refs/r/6458 monorepo for the virus lounge http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F6458 2023-07-22T09:03:02+00:00 feat(tvix/store/proto): use Bytes instead of Vec<u8> 2023-07-22T09:03:02+00:00 Florian Klink flokli@flokli.de 2023-07-19T15:52:50+00:00 urn:sha1:432222f098bfceb033e63e9a63687e35574457f9 Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> refactor(tvix/store): use bytes for node names and symlink targets 2023-07-21T19:01:49+00:00 Florian Klink flokli@flokli.de 2023-07-18T16:37:25+00:00 urn:sha1:72e82ffcb11b1aaf1f1cc8db4189ced5ec0aa42e Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI docs(tvix): document when pointer equality is preserved in C++ Nix 2023-07-11T16:17:42+00:00 sterni sternenseemann@systemli.org 2023-06-26T23:20:17+00:00 urn:sha1:8adc9c56f21d8418dc5f4b73a4327d04cd10ebd3 This explicitly documents behavior of C++ Nix that goes against the intuition you'd gather from this document: that e.g. a simple select from an attribute set causes a value to no longer be pointer equal to its former self. The point of documenting this is that we can show in a to be written section on the use of pointer equality in nixpkgs that pointer equality is only needed in a limited sense for evaluating it (C++ Nix's exterior pointer equality). Tvix's pointer equality is far more powerful since value identity preserving operations also preserve pointer equality, generally speaking (this is because we implement interior pointer equality in my made up terminology). This should eventually also be documented. Change-Id: I6ce7ef2d67b012f5ebc92f9e81bba33fb9dce7d0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8856 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> fix(tvix/eval): use byte, not codepoint index for slicing in escape 2023-07-11T16:11:40+00:00 sterni sternenseemann@systemli.org 2023-07-11T15:54:20+00:00 urn:sha1:4ba624efae2d63057c2bd5be23841be5017bd457 This fixes a subtle issue which would occasionally lead to a crash (e.g. when evaluating (pkgs.systemd.outPath with --trace-runtime): With each character in the string that has a multi byte representation in UTF-8, the actual byte position and what tvix thought it was would get out of sync. This could either lead to * Tvix swallowing characters or jumbling characters if multi byte characters would cause the tracked index to become out of sync with the byte position before the first character to be escaped, or * Tvix crashing if (in the same situation) the out of sync index would be within a UTF-8 byte sequence. Luckily, std's `char_indices()` iterator implements exactly what `nix_escape_char()`'s original author had in mind with `.chars().enumerate()`. Using `i + 1` for continuing is safe, since all characters that need (in fact, can) to be escaped in Nix are represented as a single byte in UTF-8. Change-Id: I1c836f70cde3d72db1c644e9112852f0d824715e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8952 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> chore(tvix/eval/vm): drop unused import 2023-06-30T12:42:59+00:00 Florian Klink flokli@flokli.de 2023-06-22T13:34:04+00:00 urn:sha1:5796563f1f398fd0ddd87ac526c1c0928dc00803 Change-Id: Ia04778391c198fde21da217bf697aa40157898b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8846 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI feat(tvix/eval): allow extending builtins outside of tvix_eval 2023-06-22T20:19:06+00:00 Evgeny Zemtsov eze@resoptima.com 2023-06-22T15:57:50+00:00 urn:sha1:c8fcdca4eb09709966ea25883cbe91f34c038236 The change allows applications that use tvix_serde for parsing nix-based configuration to extend the language with domain-specific set of features. Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI fix(tvix/eval): use realpaths for import cache 2023-06-21T07:48:52+00:00 sterni sternenseemann@systemli.org 2023-06-20T14:21:26+00:00 urn:sha1:66047063e02be3188a558958de8938c9015c0f89 I've noticed this behavior when writing the admittedly cursed test case included in this CL. Alternatively we could use some sort of machinery using `builtins.trace`, but I don't think we capture stderr anywhere. I've elected to put this into the eval cache itself while C++ Nix does it in builtins.import already, namely via `realisePath`. We don't have an equivalent for this yet, since we don't support any kind of IfD, but we could revise that later. In any case, it seems good to encapsulate `ImportCache` in this way, as it'll also allow using file hashes as identifiers, for example. C++ Nix also does our equivalent of canon_path in `builtins.import` which we still don't, but I suspect it hardly makes a difference. Change-Id: I05004737ca2458a4c67359d9e7d9a2f2154a0a0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8839 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI fix(tvix/eval): only finalise formal arguments if defaulting 2023-06-20T10:07:44+00:00 sterni sternenseemann@systemli.org 2023-06-03T00:10:31+00:00 urn:sha1:4516cd09c51b7a19707de0a5ba171c9592241a18 When dealing with a formal argument in a function argument pattern that has a default expression, there are two different things that can happen at runtime: Either we select its value from the passed attribute successfully or we need to use the default expression. Both of these may be thunks and both of these may need finalisers. However, in the former case this is taken care of elsewhere, the value will always be finalised already if necessary. In the latter case we may need to finalise the thunk resulting from the default expression. However, the thunk corresponding to the expression may never end up in the local's stack slot. Since finalisation goes by stack slot (and not constants), we need to prevent a case where we don't fall back to the default expression, but finalise anyways. Previously, we worked around this by making `OpFinalise` ignore non-thunks. Since finalisation of already evaluated thunks still crashed, the faulty compilation of function pattern arguments could still cause a crash. As a new approach, we reinstate the old behavior of `OpFinalise` to crash whenever encountering something that is either not a thunk or doesn't need finalisation. This can also help catching (similar) miscompilations in the future. To then prevent the crash, we need to track whether we have fallen back or not at runtime. This is done using an additional phantom on the stack that holds a new `FinaliseRequest` value. When it comes to finalisation we check this value and conditionally execute `OpFinalise` based on its value. Resolves b/261 and b/265 (partially). Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> chore(tvix/eval): fix markdown labeled link syntax 2023-06-15T19:40:08+00:00 sterni sternenseemann@systemli.org 2023-06-15T11:40:52+00:00 urn:sha1:77b0dddc3d915509a929cd5595047e17eab47cf7 Change-Id: I639dc0801090eaba56b61858e28204b5a0e631b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8784 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> test(tvix/eval): update nix_tests suite to C++ Nix master 2023-06-15T19:28:16+00:00 sterni sternenseemann@systemli.org 2023-06-14T15:36:10+00:00 urn:sha1:3b8c9ec9c871c50855bb6fe78859ff25590b616b Adds new tests for foldl', intersectAttrs as well as fills in missing .exp files. New test cases we don't pass: - fromTOML timestamp capabilities - path antiquotation - replaceStrings is lazier on C++ Nix master The C++ Nix revision used is 7066d21a0ddb421967980094222c4bc1f5a0f45a. Change-Id: Ic619c96e2d41e6c5ea6fa93f9402b12e564af3c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8778 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<feed xmlns="http://www.w3.org/2005/Atom">
<title>depot/tvix/eval, branch refs/r/6458</title>
<subtitle>monorepo for the virus lounge</subtitle>
<id>http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F6458</id>
<link rel="self" href="http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F6458"/>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/"/>
<updated>2023-07-22T09:03:02+00:00</updated>
<entry>
<title>feat(tvix/store/proto): use Bytes instead of Vec<u8></title>
<updated>2023-07-22T09:03:02+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-19T15:52:50+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=432222f098bfceb033e63e9a63687e35574457f9"/>
<id>urn:sha1:432222f098bfceb033e63e9a63687e35574457f9</id>
<content type="text"> Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> </content>
</entry>
<entry>
<title>refactor(tvix/store): use bytes for node names and symlink targets</title>
<updated>2023-07-21T19:01:49+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-07-18T16:37:25+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=72e82ffcb11b1aaf1f1cc8db4189ced5ec0aa42e"/>
<id>urn:sha1:72e82ffcb11b1aaf1f1cc8db4189ced5ec0aa42e</id>
<content type="text"> Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>docs(tvix): document when pointer equality is preserved in C++ Nix</title>
<updated>2023-07-11T16:17:42+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-06-26T23:20:17+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=8adc9c56f21d8418dc5f4b73a4327d04cd10ebd3"/>
<id>urn:sha1:8adc9c56f21d8418dc5f4b73a4327d04cd10ebd3</id>
<content type="text"> This explicitly documents behavior of C++ Nix that goes against the intuition you'd gather from this document: that e.g. a simple select from an attribute set causes a value to no longer be pointer equal to its former self. The point of documenting this is that we can show in a to be written section on the use of pointer equality in nixpkgs that pointer equality is only needed in a limited sense for evaluating it (C++ Nix's exterior pointer equality). Tvix's pointer equality is far more powerful since value identity preserving operations also preserve pointer equality, generally speaking (this is because we implement interior pointer equality in my made up terminology). This should eventually also be documented. Change-Id: I6ce7ef2d67b012f5ebc92f9e81bba33fb9dce7d0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8856 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> </content>
</entry>
<entry>
<title>fix(tvix/eval): use byte, not codepoint index for slicing in escape</title>
<updated>2023-07-11T16:11:40+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-07-11T15:54:20+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=4ba624efae2d63057c2bd5be23841be5017bd457"/>
<id>urn:sha1:4ba624efae2d63057c2bd5be23841be5017bd457</id>
<content type="text"> This fixes a subtle issue which would occasionally lead to a crash (e.g. when evaluating (pkgs.systemd.outPath with --trace-runtime): With each character in the string that has a multi byte representation in UTF-8, the actual byte position and what tvix thought it was would get out of sync. This could either lead to * Tvix swallowing characters or jumbling characters if multi byte characters would cause the tracked index to become out of sync with the byte position before the first character to be escaped, or * Tvix crashing if (in the same situation) the out of sync index would be within a UTF-8 byte sequence. Luckily, std's `char_indices()` iterator implements exactly what `nix_escape_char()`'s original author had in mind with `.chars().enumerate()`. Using `i + 1` for continuing is safe, since all characters that need (in fact, can) to be escaped in Nix are represented as a single byte in UTF-8. Change-Id: I1c836f70cde3d72db1c644e9112852f0d824715e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8952 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> </content>
</entry>
<entry>
<title>chore(tvix/eval/vm): drop unused import</title>
<updated>2023-06-30T12:42:59+00:00</updated>
<author>
<name>Florian Klink</name>
<email>flokli@flokli.de</email>
</author>
<published>2023-06-22T13:34:04+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=5796563f1f398fd0ddd87ac526c1c0928dc00803"/>
<id>urn:sha1:5796563f1f398fd0ddd87ac526c1c0928dc00803</id>
<content type="text"> Change-Id: Ia04778391c198fde21da217bf697aa40157898b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8846 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>feat(tvix/eval): allow extending builtins outside of tvix_eval</title>
<updated>2023-06-22T20:19:06+00:00</updated>
<author>
<name>Evgeny Zemtsov</name>
<email>eze@resoptima.com</email>
</author>
<published>2023-06-22T15:57:50+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=c8fcdca4eb09709966ea25883cbe91f34c038236"/>
<id>urn:sha1:c8fcdca4eb09709966ea25883cbe91f34c038236</id>
<content type="text"> The change allows applications that use tvix_serde for parsing nix-based configuration to extend the language with domain-specific set of features. Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>fix(tvix/eval): use realpaths for import cache</title>
<updated>2023-06-21T07:48:52+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-06-20T14:21:26+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=66047063e02be3188a558958de8938c9015c0f89"/>
<id>urn:sha1:66047063e02be3188a558958de8938c9015c0f89</id>
<content type="text"> I've noticed this behavior when writing the admittedly cursed test case included in this CL. Alternatively we could use some sort of machinery using `builtins.trace`, but I don't think we capture stderr anywhere. I've elected to put this into the eval cache itself while C++ Nix does it in builtins.import already, namely via `realisePath`. We don't have an equivalent for this yet, since we don't support any kind of IfD, but we could revise that later. In any case, it seems good to encapsulate `ImportCache` in this way, as it'll also allow using file hashes as identifiers, for example. C++ Nix also does our equivalent of canon_path in `builtins.import` which we still don't, but I suspect it hardly makes a difference. Change-Id: I05004737ca2458a4c67359d9e7d9a2f2154a0a0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8839 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>fix(tvix/eval): only finalise formal arguments if defaulting</title>
<updated>2023-06-20T10:07:44+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-06-03T00:10:31+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=4516cd09c51b7a19707de0a5ba171c9592241a18"/>
<id>urn:sha1:4516cd09c51b7a19707de0a5ba171c9592241a18</id>
<content type="text"> When dealing with a formal argument in a function argument pattern that has a default expression, there are two different things that can happen at runtime: Either we select its value from the passed attribute successfully or we need to use the default expression. Both of these may be thunks and both of these may need finalisers. However, in the former case this is taken care of elsewhere, the value will always be finalised already if necessary. In the latter case we may need to finalise the thunk resulting from the default expression. However, the thunk corresponding to the expression may never end up in the local's stack slot. Since finalisation goes by stack slot (and not constants), we need to prevent a case where we don't fall back to the default expression, but finalise anyways. Previously, we worked around this by making `OpFinalise` ignore non-thunks. Since finalisation of already evaluated thunks still crashed, the faulty compilation of function pattern arguments could still cause a crash. As a new approach, we reinstate the old behavior of `OpFinalise` to crash whenever encountering something that is either not a thunk or doesn't need finalisation. This can also help catching (similar) miscompilations in the future. To then prevent the crash, we need to track whether we have fallen back or not at runtime. This is done using an additional phantom on the stack that holds a new `FinaliseRequest` value. When it comes to finalisation we check this value and conditionally execute `OpFinalise` based on its value. Resolves b/261 and b/265 (partially). Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> </content>
</entry>
<entry>
<title>chore(tvix/eval): fix markdown labeled link syntax</title>
<updated>2023-06-15T19:40:08+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-06-15T11:40:52+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=77b0dddc3d915509a929cd5595047e17eab47cf7"/>
<id>urn:sha1:77b0dddc3d915509a929cd5595047e17eab47cf7</id>
<content type="text"> Change-Id: I639dc0801090eaba56b61858e28204b5a0e631b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8784 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> </content>
</entry>
<entry>
<title>test(tvix/eval): update nix_tests suite to C++ Nix master</title>
<updated>2023-06-15T19:28:16+00:00</updated>
<author>
<name>sterni</name>
<email>sternenseemann@systemli.org</email>
</author>
<published>2023-06-14T15:36:10+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=3b8c9ec9c871c50855bb6fe78859ff25590b616b"/>
<id>urn:sha1:3b8c9ec9c871c50855bb6fe78859ff25590b616b</id>
<content type="text"> Adds new tests for foldl', intersectAttrs as well as fills in missing .exp files. New test cases we don't pass: - fromTOML timestamp capabilities - path antiquotation - replaceStrings is lazier on C++ Nix master The C++ Nix revision used is 7066d21a0ddb421967980094222c4bc1f5a0f45a. Change-Id: Ic619c96e2d41e6c5ea6fa93f9402b12e564af3c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8778 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI </content>
</entry>
</feed>