about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-10-17 r/6844 chore(tvix): move store golang bindings to tvix/store-goFlorian Klink18-7/+94
Similar to the castore-go CL before, this also updates the store-go bindings to the new layout. Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 r/6843 chore(tvix): move castore golang bindings to tvix/castore-goFlorian Klink15-7/+93
Have `tvix/castore/protos` only contain the protos, no go noise. Make the `.pb.go` file generation a pure Nix build at `//tvix/castore/protos:go-bindings`, and have a script at `//tvix:castore-go-generate` (TBD) that copies the results to `tvix/castore-go`. `//tvix:castore-go`, with sources in `tvix/castore-go` now contains the tooling around the generated bindings, and the generated bindings themselves (So go mod replace workflows still work). An additional CI step is added from there to ensure idempotenty of the .pb.go files. The code.tvl.fyi webserver config is updated to the new source code path. I'm still unsure if we want to also update the go.mod name. While being a backwards-incompatible change, it'll probbaly make it easier where to find these files, and the amount of external consumers is still low enough. Part of b/323. Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6842 fix(tvix/castore): Fix race when ingesting into castoreConnor Brewster1-0/+4
After finishing the ingestion, the directory putter was not being closed. This caused a race where the root directory node was accessed before the directory node had been flushed to the server. This patch makes it so we close the putter before returning the root node which should ensure that the root node exists on the directory service server before the `ingest_path` function returns. Fixes b/326 Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-16 r/6841 refactor(tvix/nix-compat/derivation): remove some clonesFlorian Klink1-3/+9
Change-Id: Ie464467edfafd2050aa9e5fd0e07ff745cb15df8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9753 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 r/6840 feat(tvix): add go to shellFlorian Klink1-0/+1
That's needed to build nar-bridge, as well as things in tvix/*store/ protos. Change-Id: If5f71b6c528a27689faa3c5ab3489cc528f9e6e5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9759 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-16 r/6839 fix(tvix/cli): don't use protobuf from pkgs, but pkgs.buildPackagesFlorian Klink1-1/+1
We need to invoke protoc at build time (only). Change-Id: Ib4c101d2ccdbdbb078725fc98c30d6c00086667d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9754 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-16 r/6838 fix(tvix): fix binary building on DarwinFlorian Klink1-0/+8
On Darwin, some crates producing binaries need to be able to link against security. Change-Id: I5bdd69247c12729b9efd5c4f18527d361ef99e87 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9758 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-16 r/6837 refactor(tvix): push rarely-used darwin-specific hacks downFlorian Klink1-8/+10
Also, make some of them buildInputs, not nativeBuildInputs. We likely need to link against libiconv, so it shouldn't be a native build input. Change-Id: I757bc7f5d8c864ecaa3db26988d00b496c085c8f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9757 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-16 r/6836 fix(tvix/shell): fix cargo build on MacOSFlorian Klink1-4/+6
It looks like we need Security Framework in the shell to be able to imperatively run `cargo build`. Change-Id: Ia5df9052bafca3610d4ed235b180b812d33d0d15 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9756 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-16 r/6835 feat(nix-compat): add bench for aterm parsingFlorian Klink4-1/+43
Change-Id: Ie9416f0046b6ee0a02096a7712cdba9cc84db4f6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9733 Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Gabriel Doriath Döhler <gabriel.doriath.dohler@ens.fr> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-16 r/6834 refactor(nix-compat/derivation): drop non_unicode testFlorian Klink1-81/+1
The previous CLs did already absorb all the logic into the common tests, no need to write this here again. Change-Id: I7ba84ba86d5445ed247e5d11d5e59b7fa815670e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9732 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-16 r/6833 refactor(nix-compat/derivation): use ATerm for output_paths testFlorian Klink1-5/+13
Due to the lack of a ATerm parser, we were previously loading the JSON fixtures to construct our Derivation structs to run the output path calculations with. However, as we now have a ATerm parser, we can load the ATerm representation directly. This also means we can test the output path calculation for non-UTF8 Derivations. Change-Id: I0e53f41a23566b5ad5f0fed12724e02a10b02707 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9731 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 r/6832 fix(nix-compat/derivation): handle dupsFlorian Klink2-23/+173
This now properly checks for duplicate output names in input derivation output names, and duplicate input sources. Change-Id: I0053854bfbf504f4f511fb3fe1a77a82b3aa61dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9738 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-16 r/6831 feat(nix-compat/{aterm,derivation}): init parserFlorian Klink14-3/+679
This provides a nom-based parser for Nix derivations in ATerm format, which can be reached via `Derivation::from_aterm_bytes`. Some of the lower-level ATerm primitives are moved into a (new) aterm module, and some more higher-level ones that construct derivation- specific types. Also, move the escape_bytes function into there, this is a generic ATerm thing. Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-16 r/6828 refactor(nix-compat/derivation): fix function nameFlorian Klink2-2/+2
typo: write_enviroment -> write_environment. Change-Id: I1a2dc826c0bf71f6ba97fe41c4d44619b5e6beff Reviewed-on: https://cl.tvl.fyi/c/depot/+/9739 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-16 r/6827 refactor(tvix/cli): clippyFlorian Klink1-1/+1
Change-Id: I35a28a320c890f62e6a76641dfdf99e33e839c02 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9749 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-10-16 r/6826 refactor(tvix/store): clippyFlorian Klink1-1/+1
Change-Id: I31ba268bda21c1f07c63520a269dd62f7b309e99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9748 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-16 r/6825 refactor(tvix/nix-compat): clippyFlorian Klink1-2/+2
Change-Id: I56282d9754b42e0e6cdbb08e9ae71f5700d28dd1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9747 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-16 r/6824 refactor(tvix/castore): clippyFlorian Klink1-1/+1
Change-Id: I8a792d6b238190a78390c5d8f638669a2681f70f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9746 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-16 r/6823 refactor(tvix/store/fs): clippyFlorian Klink2-7/+14
Change-Id: I11247e24475f0e49dc22f2d10bf3e1e276737899 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9709 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-15 r/6815 refactor(nix-compat/derivation): make write_environment more genericFlorian Klink1-5/+7
We don't actually care if it's a BTreeMap of strings, bstrings or any of that sort. The only thing we want to be able to do is get a reference to the bytes from the keys and values. Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-15 r/6814 refactor(nix-compat/derivation): move test fixtures to subdirFlorian Klink21-10/+10
We use test_resources and globbing for some of the test cases, so adding additional files in there will also create new test cases, which we don't always want. Move it down one level to make some more space. Change-Id: I619867dc80a4ced59d45096d0703678663b559cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-15 r/6813 feat(tvix/nix-compat/nixhash/NixHashWithMode): from_algo_mode_hashFlorian Klink5-2/+54
Change-Id: Ieae628fab1926a498ae5c3eb27df1b722e6151d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9728 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-15 r/6812 refactor(nix-compat/nixhash): add Result type aliasFlorian Klink1-6/+9
Change-Id: Id0248047e9642d38afc106629957a2e7608f8c78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9727 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-15 r/6811 refactor(nix-compat/nixhash): box sha512Florian Klink1-4/+4
Limit the amount of memory consumed on the stack for NixHash. Sha512 isn't used that often, so it's fine if we heap-allocate it. Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6810 refactor(tvix/store): Upgrade tokio-listener to get tonic supportConnor Brewster6-141/+29
Tonic support was added to tokio-listener upstream which removes the need for use to have tonic compatibility wrapper types around it. See: https://github.com/vi/tokio-listener/pull/2 Fixes b/311 Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-14 r/6808 refactor(tvix/nix-compat/derivation): fod_digest return [u8; 32]Florian Klink1-4/+4
This is of known length. Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6807 refactor(tvix/nix-compat): make NixHash an enum with fixed-len bytesFlorian Klink8-221/+227
Less Vec<u8> passed around. Change-Id: Ie153a6bfaa084d7490ffa38634efdf5f3c31a768 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9722 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-14 r/6806 feat(tvix/store/protos): use Validate() function on root nodeFlorian Klink4-37/+22
This updates the code to make use of the new Validate() function defined on a Node. Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-14 r/6805 feat(tvix/store/protos): add Deriver field to PathInfoFlorian Klink10-39/+161
This uses the newly introduced StorePath message type to add a Deriver field to the PathInfo message. Support for validation is added to both the golang and rust implementation. This includes extending unit tests. Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6804 feat(tvix/store/protos): add StorePath messageFlorian Klink3-48/+137
This encodes a store path a bit more concise, which is used in the Deriver field. Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6803 refactor(tvix/store/fs): make fetch_directory_inode_data asyncFlorian Klink1-30/+32
To make this easier, move it outside of TvixStoreFs, and accept the DirectoryService as a function argument, so we don't need to worry about the lifetime of self. This also aligns with how we spawn async tasks inside the rest of TvixStoreFs. Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink6-129/+88
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink3-36/+27
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink3-27/+25
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6799 feat(tvix/castore/protos): add more granular validation methodsFlorian Klink2-29/+104
Similar to cl/9715, this makes the validation checks more granular, introducing a Validate on all *Node. A check for symlink targets is added too. Once merged, it can also be used from tvix/store/protos. Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6798 refactor(tvix/castore): simplify node validation checksFlorian Klink1-9/+8
We can just check the digest length to be correct, and move the symlink target checks to a single line. Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-13 r/6796 docs(tvix/castore): point out use of contents_firstLinus Heckemann1-0/+5
Change-Id: I7620d2abe01675ea7028a478d4f8447e36d5768b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9605 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-13 r/6795 refactor(tvix/store/protos): remove unused mustBlobDigest functionFlorian Klink1-10/+0
Change-Id: I79a73de0cb06abcf16a6579ae298876004ab0ba8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9719 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-12 r/6794 refactor(tvix/castore): factor out node checksFlorian Klink4-67/+108
Implement `validate()` on `node::Node`, and call it from PathInfo's validate() too. Node-related errors are moved to a ValidateNodeError error type. This additionally adds some more validations for symlink targets (they must not be empty, and not contain null bytes). Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-12 r/6793 refactor(tvix/store/fs): use while let statementFlorian Klink1-22/+16
Suggested by clippy. Change-Id: I686b1abcb40fab42017972a3f814fae29fa70c25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9712 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-12 r/6792 refactor(tvix/store/proto): merge two match statements into oneFlorian Klink1-29/+23
Change-Id: I3daca008dff5527169f5916f4845234e8f3263cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9711 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-12 r/6791 refactor(tvix/store): clippy lintFlorian Klink4-4/+4
Change-Id: I03aa5c6cb0167ff6cb7312d81d1e43116fcb6d70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9710 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-12 r/6790 refactor(tvix/castore/utils): clippyFlorian Klink1-8/+4
Change-Id: Ib0fa1e1fb90d7f315f943a3f28266c0fb3714a31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9708 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-12 r/6789 feat(tvix/store): enable tls[-roots] feature in tonicFlorian Klink3-5/+564
This teaches tonic to speak https, so `grpc+https` becomes a working connection string. Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-12 r/6788 fix(tvix/store/fs): fix typoedef1-3/+3
Change-Id: I132987dda1073dcc0b30e50e26682d6c8724d988 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9689 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-12 r/6787 refactor(tvix/nar-bridge): panic on inval PathInfo.NarInfo.NarSha256Florian Klink4-9/+6
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate() already ensures the NarSha256 has the correct size. Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-11 r/6786 feat(tvix/store): validate nar_sha256Florian Klink2-2/+26
Change-Id: I4c4dcdb75ea7748f2ab01a0bab218596b90b7b58 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9648 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-11 r/6785 feat(tvix/store/protos): validate NarSha256Florian Klink2-1/+17
Change-Id: I016d1b020b04850f0dca68c2ea96643230dada10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9649 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-11 r/6784 docs(tvix/nar-bridge): remove redundant commentsFlorian Klink1-2/+0
These really only describe what the code is doing, remove. Pointed out in https://cl.tvl.fyi/c/depot/+/9652/comment/987b3b7f_c7b5e47b/ Change-Id: I512a274295244f96744a489871861fed9523bb37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9654 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-28T06·10+0000