| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
We wanted to enable this earlier but the log level of many of the
messages in redb were too high, they've now been downgraded, so we can
enable logs from redb.
Context on the fix and release:
- https://github.com/cberner/redb/pull/828
- https://github.com/cberner/redb/releases/tag/v2.1.2
Change-Id: I8635e8a0bcb01a7d0b580387ac9134ccdd0205f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12568
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Over the past couple of months we've been using redb instead of sled as
the default filesystem-based database in PS and DS. I am confident that
we can get rid of sled completely now, and just keep redb.
Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
This is just patchset 10 of CL10855, before the color_eyre changes,
rebased to the tvix_castore api.
Change-Id: If4b42412ff8568058908cda971ad7d6f2d9f9b7b
---
This provides a build service invoking runc. It can be used by using the
`oci://$path_to_some_tempdir` builder URL for now.
For now, it can be tested as such:
```
BUILD_SERVICE_ADDR=oci://$PWD/bundles target/debug/tvix
let pkgs = (import <nixpkgs> {}); in builtins.readDir pkgs.perl
```
readDir is to actually trigger IO into the store path (which triggers
the builds).
For now it fails due to missing reference scanning (see followup CLs).
Change-Id: I09b40e410114ce69966a41a0e3c33281b859e443
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12526
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
With an implementation of AsyncRead + AsyncSeek, axum-range can answer
range requests.
We only use it if a range has been requested, as it uses more memory
than the linear variant.
Change-Id: I0072b0a09b328f3e932f14567a2caa3a49abcbf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12509
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: yuka <yuka@yuka.dev>
|
|
This release includes support for overlay filesystems
Change-Id: I946cbf346df045209afaea2d720bb57fc2f2659f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12538
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
|
|
This is required to add the optional refscanner parameter to the
ingest functions.
Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
- Add a new PathInfoService implementation that wraps transparently
around another except that it dynamically signs all the incoming
path-infos with the provided signer.
- Add a ServiceBuilder for this PathInfoService that provides a
SigningPathInfoService with a keyfile signer
Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12032
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sinavir <tvix@sinavir.fr>
Tested-by: BuildkiteCI
|
|
thiserror is much more easier to maintain than manually implementing Error and Display.
Change-Id: Ibf13e2d8a96fba69c8acb362b7515274a593dfd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12452
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
|
|
From now on we will add the dependencies and their version in the root
Cargo.toml and in order to enable the dependency for a workspace member
we set `workspace = true` in the member's Cargo.toml.
Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This updates all the dependencies and their "minimum" versions in
Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s
`cargo upgrade` command that will eventually be merged into `cargo
update`.
Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
|
|
This bumps bigtable_rs to
https://github.com/liufuyang/bigtable_rs/pull/86, allowing us to drop
our second set of prost/tonic/http/axum crates.
Change-Id: I70f9150289c3e8611ebe8a7d99490e3dfd085a6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12384
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Ia5eaf7f4614701ad4dd06114476a73e1a041d830
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12383
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: If7d1ae4491d3c70f323cf872d1a70afe6238f78a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12382
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection
protocol.
However, most clients, like Postman, Kreya and evans don't support that
one yet.
Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support
alongside the v1 version of it, registering both services.
This fixes the example documented in tvix/store/README.md, it was
previously failing as evans couldn't find the v1alpha reflection
service.
See https://github.com/hyperium/tonic/pull/1888 for details.
Change-Id: I55438877317f82dc39face13afeb9594cda07a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
This adds a nix-compat-derive derive crate that implements a deriver
for NixDeserialize implementations. This is to reduce the amount of
code needed to implement deserialization for all the types used by
the Nix daemon protocol.
Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.
The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.
Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This moves the URL component parsing code we had in nar-bridge to
nix-compat.
We change the function signature to return an Option, not a
Result<_, StatusCode>.
This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".
This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.
Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.
Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.
Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.
We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.
Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.
Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).
Also, add some more docstrings, these were a bit sparse.
Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Removes imbl::OrdMap in favour of an Rc over the standard library's BTreeMap,
which allows us to drop the imbl dependency completely.
In my local tests this is actually slightly faster for `hello` and `firefox`.
Change-Id: Ic9597ead4e98bf9530f290c6a94a3c5c3efd0acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12201
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
|
|
This replaces the OpCode enum with a new Op enum which is guaranteed to fit in a
single byte. Instead of carrying enum variants with data, every variant that has
runtime data encodes it into the `Vec<u8>` that a `Chunk` now carries.
This has several advantages:
* Less stack space is required at runtime, and fewer allocations are required
while compiling.
* The OpCode doesn't need to carry "weird" special-cased data variants anymore.
* It is faster (albeit, not by much). On my laptop, results consistently look
approximately like this:
Benchmark 1: ./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
Time (mean ± σ): 8.224 s ± 0.272 s [User: 7.149 s, System: 0.688 s]
Range (min … max): 7.759 s … 8.583 s 10 runs
Benchmark 2: ./after -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
Time (mean ± σ): 8.000 s ± 0.198 s [User: 7.036 s, System: 0.633 s]
Range (min … max): 7.718 s … 8.334 s 10 runs
See notes below for why the performance impact might be less than expected.
* It is faster while at the same time dropping some optimisations we previously
performed.
This has several disadvantages:
* The code is closer to how one would write it in C or Go.
* Bit shifting!
* There is (for now) slightly more code than before.
On performance I have the following thoughts at the moment:
In order to prepare for adding GC, there's a couple of places in Tvix where I'd
like to fence off certain kinds of complexity (such as mutating bytecode, which,
for various reaons, also has to be part of data that is subject to GC). With
this change, we can drop optimisations like retroactively modifying existing
bytecode and *still* achieve better performance than before.
I believe that this is currently worth it to pave the way for changes that are
more significant for performance.
In general this also opens other avenues of optimisation: For example, we can
profile which argument sizes actually exist and remove the copy overhead of
varint decoding (which does show up in profiles) by using more adequately sized
types for, e.g., constant indices.
Known regressions:
* Op::Constant is no longer printing its values in disassembly (this can be
fixed, I just didn't get around to it, will do separately).
Change-Id: Id9b3a4254623a45de03069dbdb70b8349e976743
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12191
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This splits the existing ReferenceScanner into a ReferenceScanner and
ReferencePattern as well as adds an AsyncRead implementation that can
do a scan while you read from it.
The reason to split the scanner in two is that generating the pattern
is expensive and when ingesting build results with multiple outputs you
want to do several independant scans that look for the same pattern.
The reader is for scanning files without having to load the entire file
into memory.
Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Fix for RUSTSEC-2024-0358.
Change-Id: Id9e8c6ae67845d95a52c884035fcea02d2d71505
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12214
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Fix for RUSTSEC-2024-0344.
Change-Id: I9e9cac5f0dac9e5b2e5a934c99762d982136c0e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12210
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
|
|
Rather than storing the leaked allocation for the string as the key in
the interner, store the hash (using NoHashHashBuilder). I thought this
would improve performance, but it doesn't:
hello outpath time: [736.85 ms 748.42 ms 760.42 ms]
change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05)
No change in performance detected.
but it at least doesn't *hurt* performance, and it *does* avoid an
`unsafe`, so it's probably net good.
Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
|
|
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.
Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
|
|
Per https://nnethercote.github.io/perf-book/hashing.html, we have
basically no reason to use the default hasher over a faster,
non-DoS-resistant hasher. This gives a nice perf boost basically for
free:
hello outpath time: [704.76 ms 714.91 ms 725.63 ms]
change: [-7.2391% -6.1018% -4.9189%] (p = 0.00 < 0.05)
Performance has improved.
Change-Id: If5587f444ed3af69f8af4eead6af3ea303b4ae68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12046
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: aspen <root@gws.fyi>
|
|
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.
Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.
Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
|
|
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
|
|
Change-Id: Ib830c8b642496a6cdf3d4b9093f0343f5fb80622
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12018
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
It introduces a new accept_trace function for axum0.7 which can be used
to accept a header trace from a received request. This function can be
used for tonic 0.12 once that version is released, and the specific
`accept_trace` function within `tvix_tracing::propagate::tonic` can then
be removed.
This also integrates http propagation into the nar_bridge crate.
Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
|
|
This ingests NAR files into the {Blob,Directory}Service, which are
already part of the AppState.
As we then need to correlate the root node to the uploaded PathInfo, we
need to keep a (short-lived) lookup table from NARHash to root node
around. We insert it into a `LruCache` after the NAR is uploaded, and
use `peek()` to do the lookup, which doesn't update the LRU list.
Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
|
|
This adds an implementation of nar-bridge in Rust.
Currently, only the GET parts are implemented.
Contrary to the Go variant, this doesn't try to keep a mapping from nar
hashes to root node in memory, it simply encodes the root node itself
(stripped by its basename) into the URL.
This pulls in a more recent version of axum than what we use in
tonic, causing two versions of http and hyper, however dealing with
`Body::from_stream` in axum 0.6 is much more annoying, and
https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed
soon.
Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
|
|
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748
Signed-off-by: Yureka <tvl@yuka.dev>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I88053fa2df35dc486c51d015f4b2156541c07af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11962
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Juggle around the internals of the tvix-cli crate so that we expose the
Repl as a public type with a `send` method, that sends a string to the
repl and *captures all output* so that it can be subsequently asserted
on in tests. Then, demonstrate that this works with a single (for now)
REPL test using expect-test to assert on the output of a single command
sent to the REPL.
As the REPL gets more complicated, this will allow us to make tests that
cover that complex behavior.
Change-Id: I88175bd72d8760c79faade95ebb1d956f08a7b83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11958
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Allow binding variables at the REPL's toplevel in the same way the Nix
REPL does, using the syntax <ident> = <expr>. This fully, strictly
evaluates the value and sets it in the repl's "env", which gets passed
in at the toplevel when evaluating expressions.
The laziness behavior differs from Nix's, but I think this is good:
❯ nix repl
Welcome to Nix version 2.3.18. Type :? for help.
nix-repl> x = builtins.trace "x" 1
nix-repl> x
trace: x
1
nix-repl> x
1
vs tvix:
tvix-repl> x = builtins.trace "x" 1
trace: "x" :: string
tvix-repl> x
=> 1 :: int
tvix-repl> x
=> 1 :: int
Bug: https://b.tvl.fyi/issues/371
Change-Id: Ieb2d626b7195fa87be638c9a4dae2eee45eb9ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11954
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
|
|
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.
It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.
Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.
Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This allows binding on unix sockets, as well as systemd socket
activation.
Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a
while, no need to use our own checkout here.
Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.
The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.
This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.
Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.
Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.
Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.
Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
|
|
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.
So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.
Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.
Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.
Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.
Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
|
|
This introduces another feature flag, "tracy" to the `tvix-tracing` crate.
If enabled (not enabled by default), it'll add an additional layer
emitting packets in a format that https://github.com/wolfpld/tracy can
display.
I had to be a bit tricky with the combinatorial complexity when adding
this, but the resulting code still seems manageable.
Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Provide a new interface for forcing a flush of otlp traces and use this
interface to shutdown otlp prior to exiting tvix-store, either if the
tool was stopped with a SIGTERM or ended regularly.
This also fixes an issue where traces were not even exported if for
example we just imported 10 paths and never even emitted more than 256
traces. The implementation uses a mpsc channel so a flush can be done
without having to wait for it to complete. If you want to wait for a
flush to complete you can provide a oneshot channel which will receive a
message once flushing is complete.
Because of a otlp bug `force_flush` as well as
`shutdown_tracer_provider` need to be executed using `spawn_blocking`
otherwise the function will deadlock. See
https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335
Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
|
|
This gives some better feedback something is going on, gives a (named)
root for all the progress children we're drawing, and also counts the
time we are in eval.
Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
|
|
Change-Id: I9fbd32596fbffe09a590a50044a4f1d074a424c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11794
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
