about summary refs log tree commit diff
path: root/tvix/nar-bridge/pkg
AgeCommit message (Collapse)AuthorFilesLines
2023-10-05 r/6702 refactor(tvix/nar-bridge): deduplicate NAR HEAD and GETFlorian Klink1-49/+26
Use a genNarHandler() function accepting a boolean to construct the HTTP handler. Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6701 docs(tvix/nar-bridge): fix typo in commentFlorian Klink1-1/+1
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6700 refactor(tvix/nar-bridge): move pkg/server to pkg/httpFlorian Klink6-6/+6
This is only dealing with the HTTP interface. Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6699 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importerFlorian Klink2-2/+2
This is useful outside a HTTP server scenario. Change-Id: I0042a6e773906a15a254d850520e6f841035bf20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6698 refactor(tvix/nar-bridge): move blob cb function to pkg/importerFlorian Klink2-3/+3
This is useful outside a HTTP server scenario. Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6697 docs(tvix/nar-bridge): add more comments for DirectoriesUploader.Florian Klink1-1/+5
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-05 r/6696 refactor(tvix/nar-bridge): rename writer package to exporterFlorian Klink4-11/+11
Change-Id: I7cf6a3ed0dca4c450fadd9228dc0154ad6254f51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9530 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6695 refactor(tvix/nar-bridge): combine writers/readersFlorian Klink3-80/+37
We can drop most of Hasher if we use a MultiWriter writing to the hash function and a minimal CountingWriter. This should make things a bit more understandable. Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-05 r/6694 refactor(tvix/nar-bridge): let callbaks return calculated digestsFlorian Klink6-100/+109
This aligns behaviour more with how it should be - it's the responsibility of the callback functions to return digests of the things they consume(d). It allows further cleaning up the hasher struct. Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6693 refactor(tvix/nar-bridge): drop reader packageFlorian Klink6-56/+41
Make the import function usable on any reader. Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-05 r/6692 refactor(tvix/nar-bridge): rename item to stackItemFlorian Klink1-3/+3
Change-Id: I76c8db3b04d9f97e51e91f7d36a4dd8c6500d1a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9526 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 r/6691 refactor(tvix/nar-bridge): inline assemblePathInfoFlorian Klink1-17/+14
This is only called once. Change-Id: I342443b8d04050929733fc84d5f36cd64060afe3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9525 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-22 r/6633 refactor(tvix/nar-bridge): update to new separated protosFlorian Klink12-152/+160
Change-Id: I5eaadc837a4d3a7f635574437127a22de88f556b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9407 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-09-18 r/6616 feat(tvix/nar-bridge): graceful shutdownFlorian Klink1-2/+10
This gives existing clients 30s to finish their requests after receiving an interrupt. Change-Id: Ia9b0e662fd1ffbbb6c2d03f3dd6548b13cf3d241 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9365 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 r/6615 refactor(tvix/nar-bridge): rename contentReader to blobReaderFlorian Klink1-5/+5
More aligned with how it's called in other places Change-Id: I759ac7ca3b5b69c1101d2d51a569d76c183a6330 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9362 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-18 r/6614 refactor(tvix/nar-bridge): don't buffer blob in memoryFlorian Klink1-16/+25
Create a pipe, pass the read end, and have a goroutine write to the write end. Change-Id: I301c273355705e60113b018e7e84b76972200e8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9361 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 r/6613 refactor(tvix/nar-bridge): simplify CLI interfaceFlorian Klink5-10/+7
Only keep the `serve` subcommand, and make it appear at the root. Introduce a --log-level argument, and be a bit less noisy in normal operation. Change-Id: I86b8abde1869a5c0c947508bcc29f845222aac09 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9360 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-18 r/6612 fix(tvix/nar-bridge): don't log error on simple 404sFlorian Klink2-2/+2
`nix copy` checks if NARs and NARInfo files are present, before uploading them. That's not an error, but normal behaviour, so no need to log with level info for these cases. We only want to log if the error is not a 404, and log with Warn level. Change-Id: I762de3b862d070a0f18bc62e324e94ca5c7c3693 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9359 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-18 r/6611 feat(tvix/nar-bridge): buffer request body a bit moreFlorian Klink1-1/+2
Let's make sure we don't end up blocking a client too much when inserting very small blobs. Change-Id: I640dda92efae538c70d32a40e6e85a23e9749e20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9358 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 r/6610 fix(tvix/nar-bridge): chunk blobsFlorian Klink2-25/+48
Instead of creating one big BlobChunk containing all data, and creating way too large proto messages, chunk blobs up to a reasonable (1MiB) chunk size, and send them to the server like that. Change-Id: Ia45a53956a6d7c0599cc59ac516ba37e9fb1b30e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9357 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-17 r/6603 refactor(tvix/nar-bridge): Clean up directory popping loopConnor Brewster1-9/+4
This change got lost in the rebases in cl/9348. There's unnecessary `break`/`continues` that can be replaced by moving the conditional into the for loop condition. Change-Id: I559e21087630b05e483f768ab59f8067961a2eae Reviewed-on: https://cl.tvl.fyi/c/depot/+/9352 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-17 r/6602 fix(tvix/nar-bridge): Fix directory stack trackingConnor Brewster2-5/+39
Previously, nar-bridge, had a couple of bugs with tracking the current directory when traversing a NAR file. The included test case looks like: ``` / (dir) /test (dir) /test/tested (file) /tested (file) ``` Previously, we would do a string prefix match between the current node and the top of the directory stack to determine if the node is in the directory. In this case `/test` is a substring of `/tested`; however, `/tested` is not in the `/test` directory. The fix is to append a `/` to the directory name when doing the prefix match, so `/test/` is not a prefix of `/tested`. Additionally, when popping the stack, we need to continuously pop the stack until the new node is in the directory at the top of the stack (stopping before we pop the root directory) Example: ``` / (dir) /a (dir) /a/b (dir) /a/b/c (file) /z (file) ``` Previously, `z` would end up in directory `/a` because we only the pop the stack once. The included test case requires both of these issues to be fixed for it to pass, so I think it is sufficient. Change-Id: I22f601babf04d39d85535ba7ad585d3970757211 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9348 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-17 r/6601 feat(tvix/nar-bridge): increase timeoutsFlorian Klink1-3/+3
In some cases, Nix is not able to stream the NAR file fast enough. Bump the timeouts for now. We might want to get a better understanding in what's happening here long-term, and/or make the timeouts configurable. Change-Id: Ieaa9c8f04bc73c6ce0679a058d07eaf87126634e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9340 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-09-17 r/6600 feat(tvix/nar-bridge): initFlorian Klink14-0/+2334
This provides a Nix HTTP Binary Cache interface in front of a tvix-store that's reachable via gRPC. TODOs: - remove import command, move serve up to toplevel. We have nix-copy- closure and tvix-store commands. - loop into CI. We should be able to fetch the protos as a third-party dependency. - Check if we can test nar-bridge slightly easier in an integration test. - Ensure we support connecting to unix sockets and grpc+http at least, using the same syntax as tvix-store. - Don't buffer the entire blob when rendering NAR Co-Authored-By: Connor Brewster <cbrewster@hey.com> Co-Authored-By: Márton Boros <martonboros@gmail.com> Co-Authored-By: Vo Minh Thu <noteed@gmail.com> Change-Id: I6064474e49dfe78cea67676957462d9f28658d4a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9339 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-26T09·35+0000