about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2023-10-15 r/6819 feat(users/Profpatsch/whatcd-resolver): start checking musicbrainzProfpatsch3-22/+232
Ideally we can figure out how to search for single songs by grepping through musicbrainz. For this we kinda need the jsonld results, so this is a first step which visualizes the structure and makes it easy-ish to lazily traverse it. Change-Id: Ieca21674dee8e8c2dacbab4f2f15ccbe067da647 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9743 Reviewed-by: Profpatsch <mail@profpatsch.de> Autosubmit: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI
2023-10-15 r/6818 refactor(users/Profpatsch/jbovlaste): factor Parse into own moduleProfpatsch7-167/+176
We want to use this quite generic parser type for other things as well. Change-Id: I890b43c58e479bdf2d179a724280ef1d8748fafa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9742 Autosubmit: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-10-15 r/6817 chore(3p/sources): bump channels & overlays (2023-10-15)Vincent Ambo1-18/+18
Change-Id: I36cc5e45085ff7891df92730b3438c5a9a5094d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9741 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-15 r/6816 chore(third_party/haskell): update pa packagesProfpatsch6-21/+28
Change-Id: I8abcb479b0f5c0bd6ed1abc3c9618c2362ff835a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9740 Autosubmit: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-10-15 r/6815 refactor(nix-compat/derivation): make write_environment more genericFlorian Klink1-5/+7
We don't actually care if it's a BTreeMap of strings, bstrings or any of that sort. The only thing we want to be able to do is get a reference to the bytes from the keys and values. Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-15 r/6814 refactor(nix-compat/derivation): move test fixtures to subdirFlorian Klink21-10/+10
We use test_resources and globbing for some of the test cases, so adding additional files in there will also create new test cases, which we don't always want. Move it down one level to make some more space. Change-Id: I619867dc80a4ced59d45096d0703678663b559cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-15 r/6813 feat(tvix/nix-compat/nixhash/NixHashWithMode): from_algo_mode_hashFlorian Klink5-2/+54
Change-Id: Ieae628fab1926a498ae5c3eb27df1b722e6151d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9728 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-15 r/6812 refactor(nix-compat/nixhash): add Result type aliasFlorian Klink1-6/+9
Change-Id: Id0248047e9642d38afc106629957a2e7608f8c78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9727 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-15 r/6811 refactor(nix-compat/nixhash): box sha512Florian Klink1-4/+4
Limit the amount of memory consumed on the stack for NixHash. Sha512 isn't used that often, so it's fine if we heap-allocate it. Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6810 refactor(tvix/store): Upgrade tokio-listener to get tonic supportConnor Brewster6-141/+29
Tonic support was added to tokio-listener upstream which removes the need for use to have tonic compatibility wrapper types around it. See: https://github.com/vi/tokio-listener/pull/2 Fixes b/311 Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-14 r/6809 feat(users/Profpatsch/whatcd-resolver): more otel tracesProfpatsch3-117/+273
Change-Id: I5094b64f202eeedb57510a25850bba2edd9ec36f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9725 Reviewed-by: Profpatsch <mail@profpatsch.de> Autosubmit: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI
2023-10-14 r/6808 refactor(tvix/nix-compat/derivation): fod_digest return [u8; 32]Florian Klink1-4/+4
This is of known length. Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6807 refactor(tvix/nix-compat): make NixHash an enum with fixed-len bytesFlorian Klink8-221/+227
Less Vec<u8> passed around. Change-Id: Ie153a6bfaa084d7490ffa38634efdf5f3c31a768 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9722 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-14 r/6806 feat(tvix/store/protos): use Validate() function on root nodeFlorian Klink4-37/+22
This updates the code to make use of the new Validate() function defined on a Node. Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-14 r/6805 feat(tvix/store/protos): add Deriver field to PathInfoFlorian Klink10-39/+161
This uses the newly introduced StorePath message type to add a Deriver field to the PathInfo message. Support for validation is added to both the golang and rust implementation. This includes extending unit tests. Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6804 feat(tvix/store/protos): add StorePath messageFlorian Klink3-48/+137
This encodes a store path a bit more concise, which is used in the Deriver field. Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6803 refactor(tvix/store/fs): make fetch_directory_inode_data asyncFlorian Klink1-30/+32
To make this easier, move it outside of TvixStoreFs, and accept the DirectoryService as a function argument, so we don't need to worry about the lifetime of self. This also aligns with how we spawn async tasks inside the rest of TvixStoreFs. Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink6-129/+88
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink3-36/+27
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink3-27/+25
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6799 feat(tvix/castore/protos): add more granular validation methodsFlorian Klink2-29/+104
Similar to cl/9715, this makes the validation checks more granular, introducing a Validate on all *Node. A check for symlink targets is added too. Once merged, it can also be used from tvix/store/protos. Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6798 refactor(tvix/castore): simplify node validation checksFlorian Klink1-9/+8
We can just check the digest length to be correct, and move the symlink target checks to a single line. Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-13 r/6797 chore(3p/sources): bump channels & overlays (2023-10-13)Vincent Ambo1-15/+15
Change-Id: If93c7ab4e459fe5a4e586da3d340cbf454aed7ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/9720 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-13 r/6796 docs(tvix/castore): point out use of contents_firstLinus Heckemann1-0/+5
Change-Id: I7620d2abe01675ea7028a478d4f8447e36d5768b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9605 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-13 r/6795 refactor(tvix/store/protos): remove unused mustBlobDigest functionFlorian Klink1-10/+0
Change-Id: I79a73de0cb06abcf16a6579ae298876004ab0ba8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9719 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-12 r/6794 refactor(tvix/castore): factor out node checksFlorian Klink4-67/+108
Implement `validate()` on `node::Node`, and call it from PathInfo's validate() too. Node-related errors are moved to a ValidateNodeError error type. This additionally adds some more validations for symlink targets (they must not be empty, and not contain null bytes). Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-12 r/6793 refactor(tvix/store/fs): use while let statementFlorian Klink1-22/+16
Suggested by clippy. Change-Id: I686b1abcb40fab42017972a3f814fae29fa70c25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9712 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-12 r/6792 refactor(tvix/store/proto): merge two match statements into oneFlorian Klink1-29/+23
Change-Id: I3daca008dff5527169f5916f4845234e8f3263cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9711 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-12 r/6791 refactor(tvix/store): clippy lintFlorian Klink4-4/+4
Change-Id: I03aa5c6cb0167ff6cb7312d81d1e43116fcb6d70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9710 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-12 r/6790 refactor(tvix/castore/utils): clippyFlorian Klink1-8/+4
Change-Id: Ib0fa1e1fb90d7f315f943a3f28266c0fb3714a31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9708 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-12 r/6789 feat(tvix/store): enable tls[-roots] feature in tonicFlorian Klink3-5/+564
This teaches tonic to speak https, so `grpc+https` becomes a working connection string. Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-12 r/6788 fix(tvix/store/fs): fix typoedef1-3/+3
Change-Id: I132987dda1073dcc0b30e50e26682d6c8724d988 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9689 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-12 r/6787 refactor(tvix/nar-bridge): panic on inval PathInfo.NarInfo.NarSha256Florian Klink4-9/+6
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate() already ensures the NarSha256 has the correct size. Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-11 r/6786 feat(tvix/store): validate nar_sha256Florian Klink2-2/+26
Change-Id: I4c4dcdb75ea7748f2ab01a0bab218596b90b7b58 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9648 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-11 r/6785 feat(tvix/store/protos): validate NarSha256Florian Klink2-1/+17
Change-Id: I016d1b020b04850f0dca68c2ea96643230dada10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9649 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-11 r/6784 docs(tvix/nar-bridge): remove redundant commentsFlorian Klink1-2/+0
These really only describe what the code is doing, remove. Pointed out in https://cl.tvl.fyi/c/depot/+/9652/comment/987b3b7f_c7b5e47b/ Change-Id: I512a274295244f96744a489871861fed9523bb37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9654 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-11 r/6783 refactor(tvix/nar-bridge): use mh.SHA2_256Florian Klink2-2/+5
As correctly mentioned in https://cl.tvl.fyi/c/depot/+/9652/comment/03b9b96e_bbb337fd/, we shouldn't be using these magic constants, but pull them from where they're defined. This already is a dependency of go-nix, and pkg/pathinfosvc/server.go, so no changes in go.mod. Change-Id: I0cc41ce040fcbddf4b6171417bc9b0de55af4991 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9653 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6782 docs(tvix/store/protos): explain a bit more what we're doingFlorian Klink1-1/+3
Explain a bit better what we do in the codeblock below, and add a newline to make it clear we don't just document the first branch. Change-Id: Ifb142d9984e6d2cbca648525c10298f2fcbdfedf Reviewed-on: https://cl.tvl.fyi/c/depot/+/9645 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de>
2023-10-11 r/6781 feat(tvix/nar-bridge): stop parsing nixbase32 manually, validateFlorian Klink2-10/+24
We have nixhash.FromHashTypeAndDigest now. Also, run Validate() on the PathInfo received from the remote PathInfoService. Change-Id: I14db0d9356c539c084afc9dd712314b56da2587e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9652 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6780 refactor(tvix/nar-bridge): have Export return root nodeFlorian Klink10-297/+211
… and nar size / sha256 digest. Instead of producing sparse PathInfo messages when NARs are sent to nar-bridge, the nar-bridge http server now keeps a lookup table (narsha256) -> (rootNode, narSize) This removes a whole bunch of noise, because we don't need to keep sparse fields around. A convenience function `GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is added, which is used to produce PathInfo messages, either when receiving a NAR file over http and uploading it to a remote PathInfoService, or to synthesize the PathInfoMessage to return to the client, if nar-bridge is acting as a PathInfoService for a remove Nix HTTP Binary cache. Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6779 refactor(tvix/nar-bridge): pass root node to storev1pb.ExportFlorian Klink5-6/+6
Bumps the go module past cl/9604 and update the consumer side. Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6778 fix(tvix/castore): use bstr for formatting names in errorsedef4-4/+10
Much friendlier than either Base64 or raw byte slices. Change-Id: I9b4cdd57c83ddc76c0be8103da4320207657a72b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9622 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-10 r/6777 fix(tvix/castore): handle Directory::size overflow explicitlyedef2-9/+90
We use checked arithmetic for computing the total size, and verify that size is in-bounds in Directory::validate. If an out-of-bounds size makes it to the "unchecked" size method, we either panic (in debug mode), or silently saturate to u32::MAX. No new panic sites are added, since overflows in debug mode already panic at the language level. Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-10 r/6776 test(tvix/castore): demonstrate that Directory::size may panicedef1-0/+31
Change-Id: Iebfa12df51c8c9d39b4d9eb2fccc6ffa312639cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9620 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 r/6775 fix(third_party/crate2nix): run tests in debug modeedef3-0/+18
Behave like `cargo test` rather than `cargo test --release`. Change-Id: Ie013d04ac68d7dec2a3b870fa9f0060a70a9635d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9621 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-10 r/6774 refactor(tvix/store/protos): have Export accept root nodeFlorian Klink2-43/+34
We don't need the full PathInfo message, only the root node. Change-Id: I667045ed766875dfbf8ac126a50b02baa2df67a4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9604 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-10 r/6773 chore(tvix/nar-bridge): move to Export from storev1pbFlorian Klink8-335/+49
This removes the Export method in nar-bridge, and updates all users to the version now in storev1pb. It moves the roundtrip test to the importer crate, and some of the utility functions into a separate util_test.go file. Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6772 feat(tvix/store/protos): add ExportFlorian Klink10-79/+387
Export will traverse a given PathInfo structure, and write the contents in NAR format to the passed Writer. It uses directoryLookupFn and blobLookupFn to resolve references. This is being moved over from nar-bridge. We need to keep the code there around until we can bump go.mod to storev1 with this merged, but the tests can already be moved entirely. Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de>
2023-10-10 r/6771 refactor(tvix/nar-bridge): use castorev1pb.RenamedNodeFlorian Klink4-100/+35
We can use the helper to rename the node. Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6770 chore(tvix/nar-bridge): bump google.golang.org/protobufFlorian Klink3-4/+4
… to 1.28.1 as well. Change-Id: Ieaaf9363a8dd63c2437c34d78a0ab369e7492f19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9600 Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-21T13·39+0000