about summary refs log tree commit diff
path: root/tvix/nar-bridge
AgeCommit message (Collapse)AuthorFilesLines
2024-07-21 r/8387 fix(tvix/nar-bridge): remove leftover debug statementFlorian Klink1-3/+1
Change-Id: Ie6fb822e1a4b76f2f6aef15bbe846e0a89c644fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11999 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev>
2024-07-21 r/8386 feat(tvix/nar-bridge): change default portFlorian Klink1-1/+1
Having `tvix-store daemon` and `nar-bridge` listen on the same port by default is silly. Use the same port that nar-bridge-go was using previously. Change-Id: Id374a12e52ba9dac4318e43d0ca0853866eadabc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11998 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-4/+4
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8384 feat(tvix/tracing): http propagation for axumSimon Hauser2-2/+18
It introduces a new accept_trace function for axum0.7 which can be used to accept a header trace from a received request. This function can be used for tonic 0.12 once that version is released, and the specific `accept_trace` function within `tvix_tracing::propagate::tonic` can then be removed. This also integrates http propagation into the nar_bridge crate. Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8382 fix(tvix/nar-bridge): fix root node decoding and validationFlorian Klink1-2/+7
This got broken while moving things around. We need to parse the b64-decoded bytes. Since we're now validating the root node, we also need to rename the root node to get past the node name validation. There probably should be some tests for this. Co-Authored-By: sinavir@sinavir.fr Change-Id: I8f24a4a0ac107b1ea5b94c0e0ed872a34eb7b587 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11996 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka1-1/+1
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-20 r/8379 feat(tvix/nar-bridge): implement PUT $outhash.narinfoFlorian Klink2-3/+79
This adds support to upload NARInfo files. We lookup the root node from the LRU cache, rename it appropriately and then put it into the PathInfoService. Change-Id: I5479032b51cd855363bc016dee63cf84b3304a36 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11988 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 r/8378 feat(tvix/nar-bridge): add 404 handler for GET/HEAD `/nar/…`Florian Klink1-0/+9
We currently send 405, and that confuses `nix copy`. Send a 404 for now, and add a futurework, as we can actually at least do something more meaningful in case we still have that nar hash in our LRU, which would avoid some unnecessary uploads in some cases. Change-Id: If625e9bd0fd6506cb73b88962d889aa08315fcea Reviewed-on: https://cl.tvl.fyi/c/depot/+/11987 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 r/8377 feat(tvix/nar-bridge): support uploading NAR filesFlorian Klink3-2/+132
This ingests NAR files into the {Blob,Directory}Service, which are already part of the AppState. As we then need to correlate the root node to the uploaded PathInfo, we need to keep a (short-lived) lookup table from NARHash to root node around. We insert it into a `LruCache` after the NAR is uploaded, and use `peek()` to do the lookup, which doesn't update the LRU list. Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 r/8376 feat(tvix/nar-bridge): initFlorian Klink6-0/+393
This adds an implementation of nar-bridge in Rust. Currently, only the GET parts are implemented. Contrary to the Go variant, this doesn't try to keep a mapping from nar hashes to root node in memory, it simply encodes the root node itself (stripped by its basename) into the URL. This pulls in a more recent version of axum than what we use in tonic, causing two versions of http and hyper, however dealing with `Body::from_stream` in axum 0.6 is much more annoying, and https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed soon. Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-05-15 r/8147 chore(tvix/nar-bridge): move to nar-bridge-goFlorian Klink28-2346/+0
Make some space for the rust implementation. Change-Id: I924dc1657be10abe5a11951c3b9de50bae06db19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11662 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: yuka <yuka@yuka.dev>
2024-05-01 r/8054 feat(tvix/nar-bridge): support HTTP HEAD for narinfoEdwin Mackenzie-Owen1-28/+33
Change-Id: I4ec8e9108f52d0a84bc1af01a27bd51de4ab493f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11456 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-18 r/7721 feat(tvix/nar-bridge): support listening on unix socketsFlorian Klink1-2/+16
This simply checks for the address to contain slashes, and if so, opens a unix socket, rather than a tcp one. We'll use this in //tvix/boot tests to simplify waiting for nar-bridge to be up. Change-Id: I7184f548d57142b1c5f698a1f0c30343489373a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11184 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-01-23 r/7443 feat(tvix/store/nar-info): Support async blob uploadConnor Brewster5-11/+85
Right now all blob uploads are performed synchronously, this means if a NAR contains many small files, the import time is dominated by round trip time to the blob service. For small files, we can buffer them in memory and upload them asynchronously to the blob service. Before returning we make sure to join all the uploads to make sure they complete successfully before responding OK. This reduces time to import a bash-interactive closure on my machine from 1m19s to 7s. Change-Id: Ica3695c159e6c8ad8769281ac20d037e3143e856 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10679 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 r/7442 feat(tvix/store/nar-bridge): Setup OpenTelemetryConnor Brewster6-34/+200
Sets up OpenTelemetry integration for nar-bridge. Right now it will export spans for HTTP server requests and all gRPC client requests. Having the spans available will make performance work significantly easier as it provides a high level overview of where time is being spent. In the future we can add application-specifc metrics and integrate logrus. Change-Id: Ie3860675d7ffc626a95673ba062c3c798d8bb2a7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10678 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-12-09 r/7126 fix(tvix/nar-bridge): fix FOD hashAdam Joseph1-1/+1
The golang mothership seems to be monkeying with hashes again. Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209 Tested-by: BuildkiteCI Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: flokli <flokli@flokli.de>
2023-11-28 r/7087 fix(tvix/nar-bridge): drop pathinfoserviceFlorian Klink2-355/+0
This now exists in tvix-store directly, as NixHTTPPathInfoService, and contrary to this version, also validates signatures. Change-Id: Ib6ca161e40d627b7d9741839fc849f2392f422da Reviewed-on: https://cl.tvl.fyi/c/depot/+/10155 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-06 r/6962 chore(tvix/nar-bridge): bump castore-goFlorian Klink4-23/+23
Change-Id: I81814ae326fb90ab50201cd06e3d9c0a3b15aec5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9958 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-17 r/6849 chore(tvix/nar-bridge): update to latest go mod renamesFlorian Klink18-29/+29
- code.tvl.fyi/tvix/castore/protos -> code.tvl.fyi/tvix/castore-go - code.tvl.fyi/tvix/store/protos -> code.tvl.fyi/tvix/store-go See cl/9791, cl/9792 for context. Change-Id: I44614c6ed40b9f52d9dcdea8e61fe2c3c830ce78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9793 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-12 r/6787 refactor(tvix/nar-bridge): panic on inval PathInfo.NarInfo.NarSha256Florian Klink4-9/+6
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate() already ensures the NarSha256 has the correct size. Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-11 r/6784 docs(tvix/nar-bridge): remove redundant commentsFlorian Klink1-2/+0
These really only describe what the code is doing, remove. Pointed out in https://cl.tvl.fyi/c/depot/+/9652/comment/987b3b7f_c7b5e47b/ Change-Id: I512a274295244f96744a489871861fed9523bb37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9654 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-11 r/6783 refactor(tvix/nar-bridge): use mh.SHA2_256Florian Klink2-2/+5
As correctly mentioned in https://cl.tvl.fyi/c/depot/+/9652/comment/03b9b96e_bbb337fd/, we shouldn't be using these magic constants, but pull them from where they're defined. This already is a dependency of go-nix, and pkg/pathinfosvc/server.go, so no changes in go.mod. Change-Id: I0cc41ce040fcbddf4b6171417bc9b0de55af4991 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9653 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6781 feat(tvix/nar-bridge): stop parsing nixbase32 manually, validateFlorian Klink2-10/+24
We have nixhash.FromHashTypeAndDigest now. Also, run Validate() on the PathInfo received from the remote PathInfoService. Change-Id: I14db0d9356c539c084afc9dd712314b56da2587e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9652 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6780 refactor(tvix/nar-bridge): have Export return root nodeFlorian Klink10-297/+211
… and nar size / sha256 digest. Instead of producing sparse PathInfo messages when NARs are sent to nar-bridge, the nar-bridge http server now keeps a lookup table (narsha256) -> (rootNode, narSize) This removes a whole bunch of noise, because we don't need to keep sparse fields around. A convenience function `GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is added, which is used to produce PathInfo messages, either when receiving a NAR file over http and uploading it to a remote PathInfoService, or to synthesize the PathInfoMessage to return to the client, if nar-bridge is acting as a PathInfoService for a remove Nix HTTP Binary cache. Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 r/6779 refactor(tvix/nar-bridge): pass root node to storev1pb.ExportFlorian Klink5-6/+6
Bumps the go module past cl/9604 and update the consumer side. Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6773 chore(tvix/nar-bridge): move to Export from storev1pbFlorian Klink8-335/+49
This removes the Export method in nar-bridge, and updates all users to the version now in storev1pb. It moves the roundtrip test to the importer crate, and some of the utility functions into a separate util_test.go file. Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6772 feat(tvix/store/protos): add ExportFlorian Klink3-271/+98
Export will traverse a given PathInfo structure, and write the contents in NAR format to the passed Writer. It uses directoryLookupFn and blobLookupFn to resolve references. This is being moved over from nar-bridge. We need to keep the code there around until we can bump go.mod to storev1 with this merged, but the tests can already be moved entirely. Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de>
2023-10-10 r/6771 refactor(tvix/nar-bridge): use castorev1pb.RenamedNodeFlorian Klink4-100/+35
We can use the helper to rename the node. Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-10 r/6770 chore(tvix/nar-bridge): bump google.golang.org/protobufFlorian Klink3-4/+4
… to 1.28.1 as well. Change-Id: Ieaaf9363a8dd63c2437c34d78a0ab369e7492f19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9600 Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-09 r/6759 refactor(tvix/nar-bridge): use storepath for store pathsFlorian Klink1-19/+13
Remove the handwritten parsing and formatting of store paths. Change-Id: Ia4ba486b4363c33b98937bcbf6f5f7bcda289b82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9588 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-09 r/6758 feat(tvix/nar-bridge): add ToNixNarInfo()Florian Klink2-35/+52
Convenience function, moves all code converting from a PathInfo struct to to go-nix's NarInfo. Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 r/6757 test(tvix/store/protos): add go tests for Validate()Florian Klink1-3/+0
We already have validation tests for Rust, let's add the missing ones for golang too. Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595 Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-09 r/6755 chore(tvix/nar-bridge): bump go-nixFlorian Klink3-4/+4
Change-Id: I72d139c47ede56d63d896bb5d5ed385e22a32264 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9592 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-07 r/6719 feat(tvix/nar-bridge): do pathInfo.Validate() as additional checkFlorian Klink4-9/+20
This should make it quite quick to spot writing code breaking some of the assumptions we have on PathInfo messages ourselves. Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6716 fix(tvix/nar-bridge): properly check error during HTTP shutdownFlorian Klink1-1/+1
We didn't bind err := here. Change-Id: I779d7ba5660ba4b126a4a0db7fe044bf94425855 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9547 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-05 r/6714 docs(tvix/nar-bridge): fix commentFlorian Klink1-1/+1
We run narInfo.Check to ensure this parses to a StorePath, not nixpath.Check. Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-05 r/6713 chore(tvix/nar-bridge): bump go-nix dependencyFlorian Klink6-9/+11
nixpath.FromString -> storepath.FromAbsolutePath. See https://github.com/nix-community/go-nix/pull/113 for details. Closes: https://b.tvl.fyi/issues/314 Change-Id: I25277fb6006cbbb2a323ffb5809a1be500822a97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9551 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6709 refactor(tvix/nar-bridge): do root directory dgst check in uploaderFlorian Klink2-20/+19
This check makes more sense there, and gives stronger semantics - Done() only succeeds if the other side successfully received everything, *and* came up with the same hashes as we did. Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6708 feat(tvix/nar-bridge): add nar-bridge-pathinfo commandFlorian Klink6-1/+423
This adds an additional nar-bridge-pathinfo command. It exposes a PathInfoService for a HTTP Binary Cache, ingesting data into a BlobService/DirectoryService as it goes through the NAR file. It does this whenever it receives a Get request for a specific output path, and waits returning with the PathInfo response until it ingested the data. It does not do any sort of caching - this means it re-downloads NAR files again whenever the PathInfo is requested again, so you most likely do not want to use this currently. It's one building component as soon as we have store composition (which we currently don't, so don't use this). It can be used as an alternative mechanism to ingest data (Blobs and Directories) of a given store path from a binary cache into tvix-store. ``` ❯ nix-build -A third_party.nixpkgs.hello /nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1 ❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6 sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs= ❯ out=$(mg build //tvix/nar-bridge) $out/bin/nar-bridge-pathinfo --log-level debug & INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001 ❯ mg run //tvix:store -- daemon & [mg] building target //tvix:store [mg] running target //tvix:store 2023-10-03T16:21:57.433739Z INFO tvix_store: tvix-store listening on [::]:8000 at src/bin/tvix-store.rs:229 ❯ evans --host localhost --port 8001 -r repl […] tvix.store.v1.PathInfoService@localhost:8001> call Get ✔ by_output_hash by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs= { "narinfo": { "narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=", "narSize": "226552", "referenceNames": [ "aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8", "mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1" ], "signatures": [ { "data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==", "name": "cache.nixos.org-1" } ] }, "node": { "directory": { "digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=", "name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x", "size": 141 } }, "references": [ "ptgFMIhdl2nJxMDdlDkITyXuBFc=", "Rs/INeK+7IGbG/u7fHoVNm96Yqs=" ] } ``` Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Brian Olsen <me@griff.name>
2023-10-05 r/6703 refactor(tvix/nar-bridge): mv nar-bridge{,-http}Florian Klink2-2/+2
Rename the nar-bridge CLI to nar-bridge-http, because it's the one spinning up an http server. Change-Id: I0fb75c50e4299272a128dd5ecaa4be8f06fa3dbe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9538 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6702 refactor(tvix/nar-bridge): deduplicate NAR HEAD and GETFlorian Klink1-49/+26
Use a genNarHandler() function accepting a boolean to construct the HTTP handler. Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6701 docs(tvix/nar-bridge): fix typo in commentFlorian Klink1-1/+1
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6700 refactor(tvix/nar-bridge): move pkg/server to pkg/httpFlorian Klink7-8/+8
This is only dealing with the HTTP interface. Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6699 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importerFlorian Klink2-2/+2
This is useful outside a HTTP server scenario. Change-Id: I0042a6e773906a15a254d850520e6f841035bf20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6698 refactor(tvix/nar-bridge): move blob cb function to pkg/importerFlorian Klink2-3/+3
This is useful outside a HTTP server scenario. Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6697 docs(tvix/nar-bridge): add more comments for DirectoriesUploader.Florian Klink1-1/+5
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-05 r/6696 refactor(tvix/nar-bridge): rename writer package to exporterFlorian Klink4-11/+11
Change-Id: I7cf6a3ed0dca4c450fadd9228dc0154ad6254f51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9530 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6695 refactor(tvix/nar-bridge): combine writers/readersFlorian Klink3-80/+37
We can drop most of Hasher if we use a MultiWriter writing to the hash function and a minimal CountingWriter. This should make things a bit more understandable. Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6694 refactor(tvix/nar-bridge): let callbaks return calculated digestsFlorian Klink6-100/+109
This aligns behaviour more with how it should be - it's the responsibility of the callback functions to return digests of the things they consume(d). It allows further cleaning up the hasher struct. Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6693 refactor(tvix/nar-bridge): drop reader packageFlorian Klink6-56/+41
Make the import function usable on any reader. Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>