about summary refs log tree commit diff
path: root/tvix/nar-bridge/pkg/importer
AgeCommit message (Collapse)AuthorFilesLines
2023-10-17 r/6849 chore(tvix/nar-bridge): update to latest go mod renamesFlorian Klink7-9/+9
- code.tvl.fyi/tvix/castore/protos -> code.tvl.fyi/tvix/castore-go - code.tvl.fyi/tvix/store/protos -> code.tvl.fyi/tvix/store-go See cl/9791, cl/9792 for context. Change-Id: I44614c6ed40b9f52d9dcdea8e61fe2c3c830ce78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9793 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-11 r/6780 refactor(tvix/nar-bridge): have Export return root nodeFlorian Klink4-132/+145
… and nar size / sha256 digest. Instead of producing sparse PathInfo messages when NARs are sent to nar-bridge, the nar-bridge http server now keeps a lookup table (narsha256) -> (rootNode, narSize) This removes a whole bunch of noise, because we don't need to keep sparse fields around. A convenience function `GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is added, which is used to produce PathInfo messages, either when receiving a NAR file over http and uploading it to a remote PathInfoService, or to synthesize the PathInfoMessage to return to the client, if nar-bridge is acting as a PathInfoService for a remove Nix HTTP Binary cache. Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6779 refactor(tvix/nar-bridge): pass root node to storev1pb.ExportFlorian Klink1-1/+1
Bumps the go module past cl/9604 and update the consumer side. Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6773 chore(tvix/nar-bridge): move to Export from storev1pbFlorian Klink3-26/+115
This removes the Export method in nar-bridge, and updates all users to the version now in storev1pb. It moves the roundtrip test to the importer crate, and some of the utility functions into a separate util_test.go file. Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-05 r/6709 refactor(tvix/nar-bridge): do root directory dgst check in uploaderFlorian Klink1-0/+18
This check makes more sense there, and gives stronger semantics - Done() only succeeds if the other side successfully received everything, *and* came up with the same hashes as we did. Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6699 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importerFlorian Klink1-0/+70
This is useful outside a HTTP server scenario. Change-Id: I0042a6e773906a15a254d850520e6f841035bf20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6698 refactor(tvix/nar-bridge): move blob cb function to pkg/importerFlorian Klink1-0/+71
This is useful outside a HTTP server scenario. Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6695 refactor(tvix/nar-bridge): combine writers/readersFlorian Klink2-14/+37
We can drop most of Hasher if we use a MultiWriter writing to the hash function and a minimal CountingWriter. This should make things a bit more understandable. Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6694 refactor(tvix/nar-bridge): let callbaks return calculated digestsFlorian Klink2-62/+63
This aligns behaviour more with how it should be - it's the responsibility of the callback functions to return digests of the things they consume(d). It allows further cleaning up the hasher struct. Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6693 refactor(tvix/nar-bridge): drop reader packageFlorian Klink2-0/+845
Make the import function usable on any reader. Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>