Age | Commit message (Collapse) | Author | Files | Lines |
|
This check makes more sense there, and gives stronger semantics - Done()
only succeeds if the other side successfully received everything, *and*
came up with the same hashes as we did.
Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This adds an additional nar-bridge-pathinfo command.
It exposes a PathInfoService for a HTTP Binary Cache, ingesting data
into a BlobService/DirectoryService as it goes through the NAR file.
It does this whenever it receives a Get request for a specific output
path, and waits returning with the PathInfo response until it ingested
the data.
It does not do any sort of caching - this means it re-downloads NAR
files again whenever the PathInfo is requested again, so you most likely
do not want to use this currently.
It's one building component as soon as we have store composition (which
we currently don't, so don't use this).
It can be used as an alternative mechanism to ingest data (Blobs and
Directories) of a given store path from a binary cache into tvix-store.
```
❯ nix-build -A third_party.nixpkgs.hello
/nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1
❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6
sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs=
❯ out=$(mg build //tvix/nar-bridge)
$out/bin/nar-bridge-pathinfo --log-level debug &
INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001
❯ mg run //tvix:store -- daemon &
[mg] building target //tvix:store
[mg] running target //tvix:store
2023-10-03T16:21:57.433739Z INFO tvix_store: tvix-store listening on [::]:8000
at src/bin/tvix-store.rs:229
❯ evans --host localhost --port 8001 -r repl
[…]
tvix.store.v1.PathInfoService@localhost:8001> call Get
✔ by_output_hash
by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs=
{
"narinfo": {
"narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=",
"narSize": "226552",
"referenceNames": [
"aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8",
"mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1"
],
"signatures": [
{
"data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==",
"name": "cache.nixos.org-1"
}
]
},
"node": {
"directory": {
"digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=",
"name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x",
"size": 141
}
},
"references": [
"ptgFMIhdl2nJxMDdlDkITyXuBFc=",
"Rs/INeK+7IGbG/u7fHoVNm96Yqs="
]
}
```
Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Brian Olsen <me@griff.name>
|
|
This should cover all error cases produced.
Change-Id: If31816d9b087551d86d7913df55df8f9f44bb554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9546
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
This validates the size of reference digests in the PathInfo message,
as well as inside the narinfo submessage. If narinfo is set, they need
to parse to StorePath, and have the same digest there as in the PathInfo
message.
`proto::tests::pathinfo::validate_references` needed to be updated,
because we actually did not populate the proper references before.
Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
There's other digests in the PathInfo structure, that also might have
wrong digest lengths. Rename this to give some room for them, and update
the error message a bit as we go.
Change-Id: I06562664721156e658f2ed14ba1de907377d284b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
… and export it.
Change-Id: I47d2dc2f5a8174da65c614b43801d648506e2d73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9544
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
|
|
Rename the nar-bridge CLI to nar-bridge-http, because it's the one
spinning up an http server.
Change-Id: I0fb75c50e4299272a128dd5ecaa4be8f06fa3dbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9538
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Use a genNarHandler() function accepting a boolean to construct the
HTTP handler.
Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This is only dealing with the HTTP interface.
Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This is useful outside a HTTP server scenario.
Change-Id: I0042a6e773906a15a254d850520e6f841035bf20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This is useful outside a HTTP server scenario.
Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I7cf6a3ed0dca4c450fadd9228dc0154ad6254f51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9530
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
We can drop most of Hasher if we use a MultiWriter writing to the hash
function and a minimal CountingWriter.
This should make things a bit more understandable.
Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This aligns behaviour more with how it should be - it's the
responsibility of the callback functions to return digests of the things
they consume(d). It allows further cleaning up the hasher struct.
Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Make the import function usable on any reader.
Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I76c8db3b04d9f97e51e91f7d36a4dd8c6500d1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9526
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This is only called once.
Change-Id: I342443b8d04050929733fc84d5f36cd64060afe3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9525
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Nix has historically rejected these. The current behaviour was
accidentally introduced in Nix 2.4, and is considered a bug.
Link: https://github.com/NixOS/nix/pull/9095
Change-Id: I38ffa911f0a413086479bd972de09671dbe85121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
|
|
Otherwise this produces absolutely inscrutable errors:
note: hidden type `[async fn body@castore/src/import.rs:63:1: 63:94]` captures lifetime '_#24r
Change-Id: If5d9626c9edf400de5bcec038bcaa5a3117561f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9506
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Ib5744ea240895fe84454ad271ee5640b29cab838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9505
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This partially fixes b/312 and gets FUSE to work again on MacOS.
It is mostly small type changes and an update to fuse-backend-rs because
upstream currently doesn't work with MacFuse. It also sets the default
FUSE thread count on MacOS to 1 because otherwise the mount command will
hang when shutting down as only one thread gets ENODEV and all the others
just keep blocking.
Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
|
|
The unmount method in FuseDaemon calls join on a bunch of threads and that is
a blocking call but it is called from an async context in the tvix-store
binary.
This change wraps the call to unmount in a spawn_blocking.
Change-Id: If89183b4a3f890874e75f5faf90cd24cb18da1e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9489
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
I think this may be representative of reality, but I forgot :D
Change-Id: I6aaeb959a44b3a5cdbbaea1d99b3034a7db05ba6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9498
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Without this line, accessing external APIs from Gonic (e.g. scrobbling
to Listenbrainz) does not work (unless a local resolver exists). This
is because in the pure Go DNS implementation used in gonic, only
/etc/resolv.conf can provide resolvers.
Change-Id: I26dd9a845b0a70c4cfb983c68da2b76b7e57dfec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9502
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
* zramSwap: during reindexing, geesefs can be pushed to the boundaries
of what the new overaggressive OOM killer accepts, and I don't want
to bother configuring that thing instead.
* umount: geesefs dying unexpectedly leaves broken mounts around,
clean these up before proceeding
This bakes in the assumption that there's only ever one geesefs
service, but that assumption is baked in anyways.
Change-Id: Id85c2f5bc2312a7246ea20229eb36d2cc1bd82c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9500
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: I9f0dec6491906319550c97d4cc15b2cdda4a5ebe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9499
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
|
|
I found some links to nix manual are not working (they redirect me to
manual homepage and one github link led to 404 page), so I fixed all
the links for which I was able to find appropriate target.
Imported from https://github.com/tazjin/nix-1p/pull/15
Change-Id: Ie933951e9743649d7794a6bd2a82c3a0ed328336
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9501
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: Id281142e59647d9db02f64a352cba768d4d83237
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9497
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
... this will make sense soon!
Change-Id: I1f8f32d655afdf868fff4bd09e1fea2943fd7558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9496
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I833b3cc3571d77729b83743f7e37c789337f55a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9495
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: Ib2111bf10babf42ad7984c39a25acd84cecd0d76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9494
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Can't be bothered to make something more automated for now.
Change-Id: Ie245af90c1a62a5988b97d16f86b6567e1ffafd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9493
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: I93d0ccd35eb16005779a2784d1fabc3fd9ef1b83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9492
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Experiment of how to instrument a lib I’m using to trace instead of
log.
Now that we added MonadTracer to Transaction, we can drop the unlifted `inSpanT`.
Change-Id: Iea891a58cfb33a0837978611456c33aefcccf0d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9491
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
For it to work, you need otel (e.g. jaeger) to run on port 4317.
Change-Id: I36f0493b9be26af256769ae5af8916029036a76e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9488
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
This is horrible, but I don’t think there’s a more pragmatic way to
set this up right now.
Change-Id: I166550292b4eb979864f7d69ea56a2827bbf21bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9487
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Instead of inserting torrents and every tour group seperately, insert
the tour groups and then the torrents in one go (unzipped).
I finally found a good use for list comprehensions, flattening nested
lists.
Change-Id: I7dfc765ad058dff3afb3b03887141b334a4b1988
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9486
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
|
|
Still n+1, but now we got the IO where we want it to make the database
inserts efficient.
Change-Id: I2ee36ea41bf186cbeb5581b0df802bb3611769fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9485
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Now we can move the I/O into a where block.
Change-Id: Ib5334948f3d11ca120ce0b7a46c67f8500fdab3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9484
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
|
|
Returning an I/O action was a good first approximation, but leads to a
n+1 query problem, making the whole shebang pretty slow after doing a
search.
Thus we need to split data & I/O, so we can be more clever in the next
commit.
Change-Id: Ieb2f8d5445f1258047da9b121b977c0b8d2dd7f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9483
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
This used to be in tvix/test but was moved to tvix/boot, but not all the
command were updated so they no longer work as documented.
This updates the commands to use the new tvix/boot directory.
Change-Id: I8658d928a152140df182c159bf98ea9d9ecd1302
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9482
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Puts this under the prefix command for #'telega itself.
Change-Id: I7dc394840ded3f31bf54848e581a0add36da7f3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9479
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
|
|
Telega tries to use some random completing-read functions, but I just
want it to use the default one, which I've configured the way I want.
Change-Id: I33da24f257e8ac0cae4a7d25ba6c5e4e656fcad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9478
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
|
|
Change-Id: I8e38445457fe2663572ed9d15cdb1942336247dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9477
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This call was renamed to `Read` a while ago, but I forgot to update
the README.
Change-Id: Ibcb97fe2520a5e9bb8444ec1adade6890dda4a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9476
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This commit adds a test case for b/281.
Change-Id: I8dfbfc0ff636184d7882530d8aefb329a3af9e5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9288
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Curently both fuse and virtiofs features fail to build on Darwin.
Disable them for now until fixed.
Change-Id: Ic99fa5e3ebec8a7072ba884101a5c18652ed409c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9473
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
- rename the feature to `tonic-reflection` in both tvix-store and tvix-
castore.
- set it to disabled in tvix-castore by default
- enable it in tvix-store, and pull in tvix-castore with the feature
enabled.
Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|