depot/third_party/nix/src/nix-daemon, branch refs/r/5315 monorepo for the virus lounge http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F5315 2022-05-19T14:08:20+00:00 chore(3p/nix): unvendor tvix 0.1 2022-05-19T14:08:20+00:00 Vincent Ambo mail@tazj.in 2022-05-18T15:39:39+00:00 urn:sha1:d127f9bd0e7b9b2e0df2de8a2227f77c0907468d Nothing is using this now, and we'll likely never pick this up again, but we learned a lot in the process. Every now and then this breaks in some bizarre way on channel bumps and it's just a waste of time to maintain that. Change-Id: Idcf2f5acd4ca7070ce18d7149cbfc0d967dc0a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5632 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Autosubmit: tazjin <tazjin@tvl.su> chore(3p): Bump channels to 2021-04-09 2021-04-11T22:13:32+00:00 Vincent Ambo mail@tazj.in 2021-04-07T12:34:13+00:00 urn:sha1:7deabb8c8d6f4c7e58e2b16548b8a1895795963b Fixes included: * exposed gtest in the package set, required for protobuf * pinned SBCL to version 2.0.8: The channel moved it to >2.1, and a bunch of warnings seemed to be killing our builds - we should investigate this later. * removed kernel patches from //users/tazjin/frog: this machine is currently out of service anyways, not worth fixing while it's offline * removed steam & lutris from frog (they're currently broken) * removed Haskell overrides for hedgehog-classes & hgeometry-combinatorial * use gRPC sources from upstream and inject Abseil via Nix instead * fix for renamed grpc import in //third_party/nix * use libfprint-tod from upstream nixpkgs in glittershark/yeren and delete glittershark/pkgs/fprintd entirely, since all of the patches used there are available and working from upstream now (and stopped working here after the bump) Change-Id: Ia90e6f774f7b88bc9e60d28351b900ca43ee2695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2901 Reviewed-by: glittershark <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Tested-by: BuildkiteCI feat(tvix): Thread a log sink through calls to buildDerivation 2020-11-28T00:47:48+00:00 Griffin Smith grfn@gws.fyi 2020-11-27T23:12:44+00:00 urn:sha1:49024be05644d4fac252e2191e9de74e0ffd4fe3 Similarly to how we did for buildPaths, add a std::ostream& log_sink parameter to the build_derivation method on Store, and pass it std::cerr when called at the top level by nix commands - most notably, the build-remote hook binary, so that we get build logs when using tvix as a remote builder. Change-Id: I0f8f729ba8429d4838a0a135a5c2ac1e1a95d575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2176 Tested-by: BuildkiteCI Reviewed-by: andi <andi@notmuch.email> Reviewed-by: kanepyork <rikingcoding@gmail.com> fix(tvix): Don't call assertStorePath on BasicDerivation.builder 2020-11-27T21:28:40+00:00 Griffin Smith grfn@gws.fyi 2020-11-27T21:24:08+00:00 urn:sha1:1d22874ab29f696c38146750bdd26645d4d75ccc Upstream doesn't, and for good reason - this might not be a store path, for example if it's /bin/sh Fixes: b/73 Change-Id: If9aa96de2cd8ab941c098a9f122b8b302a92ec38 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2175 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI fix(txix): Override overflow in BuildLogStreambuf 2020-11-27T20:08:45+00:00 Griffin Smith grfn@gws.fyi 2020-11-27T19:49:57+00:00 urn:sha1:39f96c4d9f5799aad1c064ef80c6fb1aed45f4e3 Subclasses of std::streambuf are expected to override overflow in addition to xsputn, as it's called in certain cases by the non-virtual methods. In our case, this was preventing endlines from getting sent over the log stream. Change-Id: I70d00f0c7cb8f8cf2f744f58974c21e7a70a715b Reviewed-on: https://cl.tvl.fyi/c/depot/+/2172 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> chore(tvix): Use StatusOr API available in Abseil's version 2020-11-21T14:26:02+00:00 Vincent Ambo mail@tazj.in 2020-11-21T14:26:02+00:00 urn:sha1:cc27324d0226953943f408ce3c69ad7d648e005e The Abseil version of `StatusOr` does not come with the status macros or the `Consume*` family of functions. This change modifies the existing code to use the common denominator of the API that is available between Abseil's own implementation of `StatusOr` and the one from Tensorflow that we are currently using. Change-Id: I5c37f68636a1fd54d153f95d7303ab8644abb774 feat(tvix): Implement EnsurePath RPC handler 2020-09-06T21:41:39+00:00 Griffin Smith grfn@gws.fyi 2020-09-06T21:34:29+00:00 urn:sha1:8f4d89e429b58a78a3052d03715dd9fc867474d2 Missed this one Fixes: b/62 Change-Id: I6f3bbcd3e2a7db5786283761b57466da33a497f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1936 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI feat(tvix): Support systemd socket activation 2020-09-01T01:07:11+00:00 Griffin Smith grfn@gws.fyi 2020-08-29T03:17:28+00:00 urn:sha1:c5f3b12f0484cd1a5152b6c64a336e9852d7c484 When the nix daemon starts up, first check (using sd_listen_fds) whether we have been systemd socket-activated. If so, instead of passing the nix daemon socket path to grpc, start a manual accept(2) loop, passing the client file descriptors to grpc via AddInsecureChannelFromFd. There's an open grpc issue at https://github.com/grpc/grpc/issues/19133 for building support into grpc to do this automatically, but as of right now this appears to be the only way to make this happen. Making this happen, by the way, was a bit of a journey - at one point I attempted to ServerBuilder's experimental AddExternalConnectionAcceptor API, and that didn't work either - it appears that the final missing piece to getting this working was explicitly fcntl(2)ing the client file descriptors to set O_NONBLOCK before passing them into gRPC. With that set, this all works inside of the test vm. Fixes: b/56 Change-Id: I5d2ab2b5b02eb570249b30a9674e115c61b0ab0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1882 Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI fix(tvix): Chunk the AddTextToStore request 2020-08-29T14:29:45+00:00 Griffin Smith grfn@gws.fyi 2020-08-21T22:58:58+00:00 urn:sha1:74a8c3d3591801eea4ad00c74b98f0043f20d4cc Rather than sending the entire AddTextToStore request along in a single message, send it in a stream of chunks using the same metadata-first approach we've been using for the other store gRPC requests. This fixes a bug where certain builds could send more data than the maximum gRPC request size (4194304 bytes, it would appear), resulting in a RESOURCE_EXHAUSTED error. The initial chunk size, which is currently constant but should be made dynamic at some point in the future, has been chosen based on the IPC bandwidth delay product for tazjin's desktop, rounded up. Change-Id: I6f0232cdbc98653484816b39855126873fc59a03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1835 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: kanepyork <rikingcoding@gmail.com> chore(tvix): Thread a std::ostream through Store::buildPaths 2020-08-20T11:48:40+00:00 Vincent Ambo mail@tazj.in 2020-08-20T02:28:35+00:00 urn:sha1:33e8b0f975cd8934405c568cfa1d7e2a1edfa425 This part of the store API needs to carry a handle to the log sink from now on, so that it can be passed in as appropriate from the gRPC handlers. In all places where there is no such handler available at the moment, the discarding log sink has been inserted. This can be used as a convenient grep target in the future. Change-Id: I26628e30b4c6437dccdf8f722ca2e8ed827dfc19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1797 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi>
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<feed xmlns="http://www.w3.org/2005/Atom">
<title>depot/third_party/nix/src/nix-daemon, branch refs/r/5315</title>
<subtitle>monorepo for the virus lounge</subtitle>
<id>http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F5315</id>
<link rel="self" href="http://code.tvl.fyi/depot/atom?h=refs%2Fr%2F5315"/>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/"/>
<updated>2022-05-19T14:08:20+00:00</updated>
<entry>
<title>chore(3p/nix): unvendor tvix 0.1</title>
<updated>2022-05-19T14:08:20+00:00</updated>
<author>
<name>Vincent Ambo</name>
<email>mail@tazj.in</email>
</author>
<published>2022-05-18T15:39:39+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=d127f9bd0e7b9b2e0df2de8a2227f77c0907468d"/>
<id>urn:sha1:d127f9bd0e7b9b2e0df2de8a2227f77c0907468d</id>
<content type="text"> Nothing is using this now, and we'll likely never pick this up again, but we learned a lot in the process. Every now and then this breaks in some bizarre way on channel bumps and it's just a waste of time to maintain that. Change-Id: Idcf2f5acd4ca7070ce18d7149cbfc0d967dc0a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5632 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Autosubmit: tazjin <tazjin@tvl.su> </content>
</entry>
<entry>
<title>chore(3p): Bump channels to 2021-04-09</title>
<updated>2021-04-11T22:13:32+00:00</updated>
<author>
<name>Vincent Ambo</name>
<email>mail@tazj.in</email>
</author>
<published>2021-04-07T12:34:13+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=7deabb8c8d6f4c7e58e2b16548b8a1895795963b"/>
<id>urn:sha1:7deabb8c8d6f4c7e58e2b16548b8a1895795963b</id>
<content type="text"> Fixes included: * exposed gtest in the package set, required for protobuf * pinned SBCL to version 2.0.8: The channel moved it to >2.1, and a bunch of warnings seemed to be killing our builds - we should investigate this later. * removed kernel patches from //users/tazjin/frog: this machine is currently out of service anyways, not worth fixing while it's offline * removed steam & lutris from frog (they're currently broken) * removed Haskell overrides for hedgehog-classes & hgeometry-combinatorial * use gRPC sources from upstream and inject Abseil via Nix instead * fix for renamed grpc import in //third_party/nix * use libfprint-tod from upstream nixpkgs in glittershark/yeren and delete glittershark/pkgs/fprintd entirely, since all of the patches used there are available and working from upstream now (and stopped working here after the bump) Change-Id: Ia90e6f774f7b88bc9e60d28351b900ca43ee2695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2901 Reviewed-by: glittershark <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>feat(tvix): Thread a log sink through calls to buildDerivation</title>
<updated>2020-11-28T00:47:48+00:00</updated>
<author>
<name>Griffin Smith</name>
<email>grfn@gws.fyi</email>
</author>
<published>2020-11-27T23:12:44+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=49024be05644d4fac252e2191e9de74e0ffd4fe3"/>
<id>urn:sha1:49024be05644d4fac252e2191e9de74e0ffd4fe3</id>
<content type="text"> Similarly to how we did for buildPaths, add a std::ostream& log_sink parameter to the build_derivation method on Store, and pass it std::cerr when called at the top level by nix commands - most notably, the build-remote hook binary, so that we get build logs when using tvix as a remote builder. Change-Id: I0f8f729ba8429d4838a0a135a5c2ac1e1a95d575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2176 Tested-by: BuildkiteCI Reviewed-by: andi <andi@notmuch.email> Reviewed-by: kanepyork <rikingcoding@gmail.com> </content>
</entry>
<entry>
<title>fix(tvix): Don't call assertStorePath on BasicDerivation.builder</title>
<updated>2020-11-27T21:28:40+00:00</updated>
<author>
<name>Griffin Smith</name>
<email>grfn@gws.fyi</email>
</author>
<published>2020-11-27T21:24:08+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=1d22874ab29f696c38146750bdd26645d4d75ccc"/>
<id>urn:sha1:1d22874ab29f696c38146750bdd26645d4d75ccc</id>
<content type="text"> Upstream doesn't, and for good reason - this might not be a store path, for example if it's /bin/sh Fixes: b/73 Change-Id: If9aa96de2cd8ab941c098a9f122b8b302a92ec38 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2175 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>fix(txix): Override overflow in BuildLogStreambuf</title>
<updated>2020-11-27T20:08:45+00:00</updated>
<author>
<name>Griffin Smith</name>
<email>grfn@gws.fyi</email>
</author>
<published>2020-11-27T19:49:57+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=39f96c4d9f5799aad1c064ef80c6fb1aed45f4e3"/>
<id>urn:sha1:39f96c4d9f5799aad1c064ef80c6fb1aed45f4e3</id>
<content type="text"> Subclasses of std::streambuf are expected to override overflow in addition to xsputn, as it's called in certain cases by the non-virtual methods. In our case, this was preventing endlines from getting sent over the log stream. Change-Id: I70d00f0c7cb8f8cf2f744f58974c21e7a70a715b Reviewed-on: https://cl.tvl.fyi/c/depot/+/2172 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> </content>
</entry>
<entry>
<title>chore(tvix): Use StatusOr API available in Abseil's version</title>
<updated>2020-11-21T14:26:02+00:00</updated>
<author>
<name>Vincent Ambo</name>
<email>mail@tazj.in</email>
</author>
<published>2020-11-21T14:26:02+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=cc27324d0226953943f408ce3c69ad7d648e005e"/>
<id>urn:sha1:cc27324d0226953943f408ce3c69ad7d648e005e</id>
<content type="text"> The Abseil version of `StatusOr` does not come with the status macros or the `Consume*` family of functions. This change modifies the existing code to use the common denominator of the API that is available between Abseil's own implementation of `StatusOr` and the one from Tensorflow that we are currently using. Change-Id: I5c37f68636a1fd54d153f95d7303ab8644abb774 </content>
</entry>
<entry>
<title>feat(tvix): Implement EnsurePath RPC handler</title>
<updated>2020-09-06T21:41:39+00:00</updated>
<author>
<name>Griffin Smith</name>
<email>grfn@gws.fyi</email>
</author>
<published>2020-09-06T21:34:29+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=8f4d89e429b58a78a3052d03715dd9fc867474d2"/>
<id>urn:sha1:8f4d89e429b58a78a3052d03715dd9fc867474d2</id>
<content type="text"> Missed this one Fixes: b/62 Change-Id: I6f3bbcd3e2a7db5786283761b57466da33a497f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1936 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>feat(tvix): Support systemd socket activation</title>
<updated>2020-09-01T01:07:11+00:00</updated>
<author>
<name>Griffin Smith</name>
<email>grfn@gws.fyi</email>
</author>
<published>2020-08-29T03:17:28+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=c5f3b12f0484cd1a5152b6c64a336e9852d7c484"/>
<id>urn:sha1:c5f3b12f0484cd1a5152b6c64a336e9852d7c484</id>
<content type="text"> When the nix daemon starts up, first check (using sd_listen_fds) whether we have been systemd socket-activated. If so, instead of passing the nix daemon socket path to grpc, start a manual accept(2) loop, passing the client file descriptors to grpc via AddInsecureChannelFromFd. There's an open grpc issue at https://github.com/grpc/grpc/issues/19133 for building support into grpc to do this automatically, but as of right now this appears to be the only way to make this happen. Making this happen, by the way, was a bit of a journey - at one point I attempted to ServerBuilder's experimental AddExternalConnectionAcceptor API, and that didn't work either - it appears that the final missing piece to getting this working was explicitly fcntl(2)ing the client file descriptors to set O_NONBLOCK before passing them into gRPC. With that set, this all works inside of the test vm. Fixes: b/56 Change-Id: I5d2ab2b5b02eb570249b30a9674e115c61b0ab0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1882 Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI </content>
</entry>
<entry>
<title>fix(tvix): Chunk the AddTextToStore request</title>
<updated>2020-08-29T14:29:45+00:00</updated>
<author>
<name>Griffin Smith</name>
<email>grfn@gws.fyi</email>
</author>
<published>2020-08-21T22:58:58+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=74a8c3d3591801eea4ad00c74b98f0043f20d4cc"/>
<id>urn:sha1:74a8c3d3591801eea4ad00c74b98f0043f20d4cc</id>
<content type="text"> Rather than sending the entire AddTextToStore request along in a single message, send it in a stream of chunks using the same metadata-first approach we've been using for the other store gRPC requests. This fixes a bug where certain builds could send more data than the maximum gRPC request size (4194304 bytes, it would appear), resulting in a RESOURCE_EXHAUSTED error. The initial chunk size, which is currently constant but should be made dynamic at some point in the future, has been chosen based on the IPC bandwidth delay product for tazjin's desktop, rounded up. Change-Id: I6f0232cdbc98653484816b39855126873fc59a03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1835 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: kanepyork <rikingcoding@gmail.com> </content>
</entry>
<entry>
<title>chore(tvix): Thread a std::ostream through Store::buildPaths</title>
<updated>2020-08-20T11:48:40+00:00</updated>
<author>
<name>Vincent Ambo</name>
<email>mail@tazj.in</email>
</author>
<published>2020-08-20T02:28:35+00:00</published>
<link rel="alternate" type="text/html" href="http://code.tvl.fyi/commit/?id=33e8b0f975cd8934405c568cfa1d7e2a1edfa425"/>
<id>urn:sha1:33e8b0f975cd8934405c568cfa1d7e2a1edfa425</id>
<content type="text"> This part of the store API needs to carry a handle to the log sink from now on, so that it can be passed in as appropriate from the gRPC handlers. In all places where there is no such handler available at the moment, the discarding log sink has been inserted. This can be used as a convenient grep target in the future. Change-Id: I26628e30b4c6437dccdf8f722ca2e8ed827dfc19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1797 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi> </content>
</entry>
</feed>