about summary refs log tree commit diff
path: root/tvix/nar-bridge/pkg/pathinfosvc/server.go
AgeCommit message (Collapse)AuthorFilesLines
2023-10-11 r/6780 refactor(tvix/nar-bridge): have Export return root nodeFlorian Klink1-48/+6
… and nar size / sha256 digest. Instead of producing sparse PathInfo messages when NARs are sent to nar-bridge, the nar-bridge http server now keeps a lookup table (narsha256) -> (rootNode, narSize) This removes a whole bunch of noise, because we don't need to keep sparse fields around. A convenience function `GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is added, which is used to produce PathInfo messages, either when receiving a NAR file over http and uploading it to a remote PathInfoService, or to synthesize the PathInfoMessage to return to the client, if nar-bridge is acting as a PathInfoService for a remove Nix HTTP Binary cache. Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-10 r/6771 refactor(tvix/nar-bridge): use castorev1pb.RenamedNodeFlorian Klink1-10/+1
We can use the helper to rename the node. Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-09 r/6759 refactor(tvix/nar-bridge): use storepath for store pathsFlorian Klink1-19/+13
Remove the handwritten parsing and formatting of store paths. Change-Id: Ia4ba486b4363c33b98937bcbf6f5f7bcda289b82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9588 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-07 r/6719 feat(tvix/nar-bridge): do pathInfo.Validate() as additional checkFlorian Klink1-0/+14
This should make it quite quick to spot writing code breaking some of the assumptions we have on PathInfo messages ourselves. Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6714 docs(tvix/nar-bridge): fix commentFlorian Klink1-1/+1
We run narInfo.Check to ensure this parses to a StorePath, not nixpath.Check. Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-05 r/6713 chore(tvix/nar-bridge): bump go-nix dependencyFlorian Klink1-2/+2
nixpath.FromString -> storepath.FromAbsolutePath. See https://github.com/nix-community/go-nix/pull/113 for details. Closes: https://b.tvl.fyi/issues/314 Change-Id: I25277fb6006cbbb2a323ffb5809a1be500822a97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9551 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6709 refactor(tvix/nar-bridge): do root directory dgst check in uploaderFlorian Klink1-20/+1
This check makes more sense there, and gives stronger semantics - Done() only succeeds if the other side successfully received everything, *and* came up with the same hashes as we did. Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6708 feat(tvix/nar-bridge): add nar-bridge-pathinfo commandFlorian Klink1-0/+300
This adds an additional nar-bridge-pathinfo command. It exposes a PathInfoService for a HTTP Binary Cache, ingesting data into a BlobService/DirectoryService as it goes through the NAR file. It does this whenever it receives a Get request for a specific output path, and waits returning with the PathInfo response until it ingested the data. It does not do any sort of caching - this means it re-downloads NAR files again whenever the PathInfo is requested again, so you most likely do not want to use this currently. It's one building component as soon as we have store composition (which we currently don't, so don't use this). It can be used as an alternative mechanism to ingest data (Blobs and Directories) of a given store path from a binary cache into tvix-store. ``` ❯ nix-build -A third_party.nixpkgs.hello /nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1 ❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6 sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs= ❯ out=$(mg build //tvix/nar-bridge) $out/bin/nar-bridge-pathinfo --log-level debug & INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001 ❯ mg run //tvix:store -- daemon & [mg] building target //tvix:store [mg] running target //tvix:store 2023-10-03T16:21:57.433739Z INFO tvix_store: tvix-store listening on [::]:8000 at src/bin/tvix-store.rs:229 ❯ evans --host localhost --port 8001 -r repl […] tvix.store.v1.PathInfoService@localhost:8001> call Get ✔ by_output_hash by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs= { "narinfo": { "narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=", "narSize": "226552", "referenceNames": [ "aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8", "mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1" ], "signatures": [ { "data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==", "name": "cache.nixos.org-1" } ] }, "node": { "directory": { "digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=", "name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x", "size": 141 } }, "references": [ "ptgFMIhdl2nJxMDdlDkITyXuBFc=", "Rs/INeK+7IGbG/u7fHoVNm96Yqs=" ] } ``` Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Brian Olsen <me@griff.name>