about summary refs log tree commit diff
path: root/tvix/Cargo.nix
AgeCommit message (Collapse)AuthorFilesLines
2024-08-16 r/8500 chore(tvix): bump object_storeFlorian Klink1-7/+28
Fix for RUSTSEC-2024-0358. Change-Id: Id9e8c6ae67845d95a52c884035fcea02d2d71505 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12214 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8496 chore(tvix/nix-compat): bump curve25519-dalekFlorian Klink1-21/+236
Fix for RUSTSEC-2024-0344. Change-Id: I9e9cac5f0dac9e5b2e5a934c99762d982136c0e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12210 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-10 r/8474 feat(tvix/eval): Store hash in key of internerAspen Smith1-0/+17
Rather than storing the leaked allocation for the string as the key in the interner, store the hash (using NoHashHashBuilder). I thought this would improve performance, but it doesn't: hello outpath time: [736.85 ms 748.42 ms 760.42 ms] change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05) No change in performance detected. but it at least doesn't *hurt* performance, and it *does* avoid an `unsafe`, so it's probably net good. Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-08-09 r/8465 fix(third_party/overlays): Patch crate2nix to use mkDerivation for testsIlan Joselevich1-39/+28
The problem with using runCommand and recreating the src directory with lndir is that it changes the file types of individual files, they will now be a symlink instead of a regular file. If you have a crate that tests that a file is of regular type then it will fail inside the crate2nix derivation. Also regenerate Cargo.nix for //tvix as it will be needed in the next commit. Change-Id: I9275602cc17a428f9fdf0e55daf12cd673bbc030 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12131 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich1-81/+86
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-07 r/8453 feat(tvix/eval): Forbid Hash{Map,Set}, use Fx insteadAspen Smith1-4/+26
Per https://nnethercote.github.io/perf-book/hashing.html, we have basically no reason to use the default hasher over a faster, non-DoS-resistant hasher. This gives a nice perf boost basically for free: hello outpath time: [704.76 ms 714.91 ms 725.63 ms] change: [-7.2391% -6.1018% -4.9189%] (p = 0.00 < 0.05) Performance has improved. Change-Id: If5587f444ed3af69f8af4eead6af3ea303b4ae68 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12046 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: aspen <root@gws.fyi>
2024-08-05 r/8443 feat(tvix/eval): Leak strings (with flag to disable)Aspen Smith1-1/+1
Default to always leaking strings, and copying strings by copying pointers rather than cloning the underlying allocation. This (somewhat bafflingly) doesn't seem to affect any benchmarks, but paves the way for some tricks around string allocation that do. Unfortunately, we can't do this (yet?) for contextful strings, for reasons I don't currently understand but which I will address later, when I address contextful strings more holistically. I've left a flag in here to disable this, both to test the cloning logic for strings for when/if we decide to bring this back, and to allow people who care more about memory usage than perf to disable leaking. Change-Id: Iec44bcbfe9b3d20389d2450b9a551792a79b9b26 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12045 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich1-0/+39
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8403 feat(tvix/store): add grpc healthcheck service to daemonYureka1-0/+53
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8402 fix(store): add toml dependency for feature xp-store-compositionYureka1-0/+1
Fixes 'use of undeclared crate or module ' with --features xp-store-composition Change-Id: I44dce86e656094d180b91a00f385f685d21f3fbd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12021 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8400 feat(tvix): add experimental-store-composition optionYureka1-10/+149
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 r/8399 refactor(tvix): move service addrs into shared clap structYureka1-0/+5
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8398 chore(tvix/tracing): switch tracing-opentelemetry from git to releaseYureka1-6/+1
Change-Id: Ib830c8b642496a6cdf3d4b9093f0343f5fb80622 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12018 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-495/+1380
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8384 feat(tvix/tracing): http propagation for axumSimon Hauser1-4/+112
It introduces a new accept_trace function for axum0.7 which can be used to accept a header trace from a received request. This function can be used for tonic 0.12 once that version is released, and the specific `accept_trace` function within `tvix_tracing::propagate::tonic` can then be removed. This also integrates http propagation into the nar_bridge crate. Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Reviewed-by: flokli <flokli@flokli.de>
2024-07-20 r/8377 feat(tvix/nar-bridge): support uploading NAR filesFlorian Klink1-6/+18
This ingests NAR files into the {Blob,Directory}Service, which are already part of the AppState. As we then need to correlate the root node to the uploaded PathInfo, we need to keep a (short-lived) lookup table from NARHash to root node around. We insert it into a `LruCache` after the NAR is uploaded, and use `peek()` to do the lookup, which doesn't update the LRU list. Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 r/8376 feat(tvix/nar-bridge): initFlorian Klink1-50/+835
This adds an implementation of nar-bridge in Rust. Currently, only the GET parts are implemented. Contrary to the Go variant, this doesn't try to keep a mapping from nar hashes to root node in memory, it simply encodes the root node itself (stripped by its basename) into the URL. This pulls in a more recent version of axum than what we use in tonic, causing two versions of http and hyper, however dealing with `Body::from_stream` in axum 0.6 is much more annoying, and https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed soon. Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-07-18 r/8365 feat(tvix/castore): add composition moduleYureka1-3/+56
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748 Signed-off-by: Yureka <tvl@yuka.dev> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-07 r/8356 feat(tvix/eval): Bump imbl to 3.0.0Aspen Smith1-2/+3
Change-Id: I88053fa2df35dc486c51d015f4b2156541c07af3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11962 Tested-by: BuildkiteCI Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 r/8353 test(tvix/cli): Make the REPL testableAspen Smith1-0/+36
Juggle around the internals of the tvix-cli crate so that we expose the Repl as a public type with a `send` method, that sends a string to the repl and *captures all output* so that it can be subsequently asserted on in tests. Then, demonstrate that this works with a single (for now) REPL test using expect-test to assert on the output of a single command sent to the REPL. As the REPL gets more complicated, this will allow us to make tests that cover that complex behavior. Change-Id: I88175bd72d8760c79faade95ebb1d956f08a7b83 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11958 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 r/8346 feat(tvix/repl): Allow binding variables at the top-levelAspen Smith1-0/+8
Allow binding variables at the REPL's toplevel in the same way the Nix REPL does, using the syntax <ident> = <expr>. This fully, strictly evaluates the value and sets it in the repl's "env", which gets passed in at the toplevel when evaluating expressions. The laziness behavior differs from Nix's, but I think this is good: ❯ nix repl Welcome to Nix version 2.3.18. Type :? for help. nix-repl> x = builtins.trace "x" 1 nix-repl> x trace: x 1 nix-repl> x 1 vs tvix: tvix-repl> x = builtins.trace "x" 1 trace: "x" :: string tvix-repl> x => 1 :: int tvix-repl> x => 1 :: int Bug: https://b.tvl.fyi/issues/371 Change-Id: Ieb2d626b7195fa87be638c9a4dae2eee45eb9ab1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11954 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: aspen <root@gws.fyi>
2024-07-02 r/8339 feat(tvix/tracing): http trace propagationSimon Hauser1-5/+213
Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 r/8337 chore(3p/sources): bump to OpenSSH vulnerability hotfixVincent Ambo1-78/+27
See https://github.com/NixOS/nixpkgs/pull/323753 for details. Changes: * git: temporarily comment out dottime patch (it doesn't apply, but it's not critical) * third-party/cgit: use an older git version where dottime patch still applies * 3p/crate2nix: remove crate2nix patches included in latest release * tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs) * tvix: regenerate Cargo.nix * tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function * tvix/nix-compat: remove redundant trait bounds * tvix/glue: use clone_into() to set drv.{builder,system} * tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327 * toold/depotfmt: expose treefmt config as passthru * tools/crate2nix: undo some more hacks in the crate2nix-check drv Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c Co-Authored-By: Florian Klink <flokli@flokli.de> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 r/8335 feat(tvix/store): use tokio-listener for tvix-store daemon commandFlorian Klink1-2/+14
This allows binding on unix sockets, as well as systemd socket activation. Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 r/8334 chore(tvix): bump bigtable_rs to latest releaseFlorian Klink1-7/+2
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a while, no need to use our own checkout here. Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-07-01 r/8331 chore(tvix): bump to data-encoding 2.6.0Florian Klink1-2/+2
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser1-2/+165
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-18 r/8294 chore(tvix): patch crate2nix to drop darwin dontStripIlan Joselevich1-2/+0
This patch has already been applied on upstream but there hasn't been an official release yet so we patch it ourselves. Upstreamed patch: https://github.com/nix-community/crate2nix/commit/0209f258cda8a9972a785e26d92fb477ce4d1b0e Follow-up of: https://cl.tvl.fyi/c/depot/+/11856 Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8289 feat(tvix/tvix-store): improve progress barsFlorian Klink1-0/+4
Don't show an empty spinner for daemon commands. Move the bar to the right, so the text is better aligned between spinner progress and bar progress styles. Generally, push progress bars a bit more down to the place where we can track progress. This includes adding one in the upload_blob span. Introduce another progress style template for transfers, which interprets the counter as bytes (not just a plain integer), and also a data rate. Use it for here and in the fetching code, and also make the progress bar itself a bit less wide. Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8285 fix(tvix/store/bin): fix shutdown behaviour for FUSEFlorian Klink1-1/+24
Both umounts happening from another process, as well as tvix-store itself calling umount() on FuseDaemon will cause the FUSE worker threads to terminate. So far there was no nice way to wait on these threads to be terminated from multiple places, causing the `tvix-store mount` command to only be terminated if interrupted via ctrl-c, not via an external umount. Update FuseDaemon to use a ThreadPool, which gives us a join primitive over all threads, that can also be called from multiple places. Await on a join() from there to end the program, not the ctrl-c signal handler as it was before. Using FuseDaemon from multiple tasks requires Arc<>-ing both the ThreadPool as well as the inner FuseSession (which also needs to be inside a Mutex if we want to unmount), but now we can clone FuseDaemon around and use it in two places. We could probably also have used an Option and drop the FuseSession after the first umount, but this looks cleaner. Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-15 r/8280 feat(tvix/cli,store): add tracy featureFlorian Klink1-2/+6
If compiled with this features, this emits packets compatible with the [Tracy](https://github.com/wolfpld/tracy) format. Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-14 r/8275 feat(tvix/tracing): add tracing-tracy supportFlorian Klink1-39/+1019
This introduces another feature flag, "tracy" to the `tvix-tracing` crate. If enabled (not enabled by default), it'll add an additional layer emitting packets in a format that https://github.com/wolfpld/tracy can display. I had to be a bit tricky with the combinatorial complexity when adding this, but the resulting code still seems manageable. Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Reviewed-by: flokli <flokli@flokli.de>
2024-06-14 r/8271 feat(tvix/tracing): correctly close otlp on exitSimon Hauser1-0/+9
Provide a new interface for forcing a flush of otlp traces and use this interface to shutdown otlp prior to exiting tvix-store, either if the tool was stopped with a SIGTERM or ended regularly. This also fixes an issue where traces were not even exported if for example we just imported 10 paths and never even emitted more than 256 traces. The implementation uses a mpsc channel so a flush can be done without having to wait for it to complete. If you want to wait for a flush to complete you can provide a oneshot channel which will receive a message once flushing is complete. Because of a otlp bug `force_flush` as well as `shutdown_tracer_provider` need to be executed using `spawn_blocking` otherwise the function will deadlock. See https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335 Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-13 r/8262 feat(tvix/tracing): set release_max_level_debug for tracingFlorian Klink1-2/+2
This allows explicitly opting in to get DEBUG-level log lines, by setting RUST_LOG. It currently also causes traces to be emitted in all cases, so we might do some runtime filtering there too, as discussed in cl/11791. Change-Id: I2865bb06a62465836d63196422f5f734f7165386 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11801 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi> Autosubmit: flokli <flokli@flokli.de>
2024-06-13 r/8261 feat(tvix/cli): add toplevel progress spanFlorian Klink1-0/+4
This gives some better feedback something is going on, gives a (named) root for all the progress children we're drawing, and also counts the time we are in eval. Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-06-12 r/8256 chore(tvix/castore): drop unused indicatif dependencyFlorian Klink1-4/+0
Change-Id: I9fbd32596fbffe09a590a50044a4f1d074a424c3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11794 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-06-12 r/8255 feat(tvix/glue/tvix_store_io): show progress infoFlorian Klink1-0/+8
In `store_path_to_node`, in case we need to build or fetch something, render a progress bar, using the spinner for now. We can upgrade this to a progress *bar* later. Change-Id: I4a7cf5ef8f639076f176af9b39d276be3f37c8ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11793 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser1-39/+83
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-06 r/8222 feat(tvix): Switch to jemallocAspen Smith1-0/+81
Switch tvix to using jemalloc as the default global allocator on supported (eg, non-msvc) platforms. This gives a pretty reasonable performance boost basically for free: int time: [24.361 µs 24.386 µs 24.418 µs] change: [-19.355% -18.859% -18.527%] (p = 0.00 < 0.05) Performance has improved. merge small attrs time: [37.201 µs 37.328 µs 37.442 µs] change: [-24.609% -24.266% -23.982%] (p = 0.00 < 0.05) Performance has improved. merge large attrs with small attrs time: [20.030 ms 20.135 ms 20.251 ms] change: [-4.2811% -3.2549% -2.3807%] (p = 0.00 < 0.05) Performance has improved. hello outpath time: [967.91 ms 974.07 ms 983.33 ms] change: [-5.5008% -4.4552% -3.4099%] (p = 0.00 < 0.05) Performance has improved. Change-Id: I6c6e6e3295ec2fca01ea28dc37bcb201cd811767 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10851 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de>
2024-06-06 r/8221 fix(tvix/glue): drop tracing-indicatif from depsFlorian Klink1-4/+0
We don't use this in tvix-glue currently. A previous version of cl/11747 did, and I forgot to remove it afterwards. Change-Id: I6191ffc60450f3e79f22a3aca55246e3956ee9cc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11756 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de>
2024-06-06 r/8220 feat(tvix/store/bin): add progress bar infrastructureFlorian Klink1-1/+269
This adds the tracing-indicatif crate, and configures it as a layer in our tracing_subscriber pipeline to emit progress for every span that's configured so. It also moves from using std::io::stderr to write logs to using their writer, to avoid clobbering output. Progress bar styles are defined in a lazy_static, moving this into a general tracing is left for later. This adds some usage of this to the `imports` and `copy` commands. The output can still be improved a bit - we should probably split each task up into a smaller (instrumented) helper functions, so we can create a progress bar for each task. Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-06-05 r/8215 fix(tvix): update Cargo.nix as it misses a default featureIlan Joselevich1-0/+1
Change-Id: I30d91a9aca9f5546ae5e1fa5c55ee389695ae085 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11752 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-01 r/8186 feat(tvix/repl): Support multiline inputAspen Smith1-0/+4
Transparently support multiline input in the Tvix REPL, by handling the UnexpectedEOF error returned by the parser and using it to progressively build up an input expr over multiple iterations of the REPL's outer loop. This works quite nicely: ❯ cargo r --bin tvix Compiling tvix-cli v0.1.0 (/home/aspen/code/depot/tvix/cli) Finished dev [unoptimized + debuginfo] target(s) in 1.72s Running `target/debug/tvix` tvix-repl> { foo > = > 1; > } => { foo = 1; } :: set <press up arrow> tvix-repl> { foo = 1; } => { foo = 1; } :: set Change-Id: Ib0ed4766b13e8231d696cdc27281ac158e20a777 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11732 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-26 r/8173 fix(tvix/build): tonic-reflection feature needs castore with it tooFlorian Klink1-1/+1
If building tvix-build with the tonic-reflection feature, it needs to import `tvix_castore::proto::FILE_DESCRIPTOR_SET`, which is only available if tvix-castore is built with the `tonic-reflection` feature. Change-Id: I355b4c5b4c1333d5cc56335de47ad5d2f1db6337 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11716 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2024-05-22 r/8160 refactor(tvix/eval): rewrite xml emitter to be simple-stupidProfpatsch1-16/+0
In order to be compatible with the nix XML generator, it’s easier to generate the XML directly, instead of going through a library which we have to bend to do what we need. Removes dependency on `xml-rs`, which came with a full XML parser that we didn’t use. Only takes a tiny bit of code for the XML escaping, somewhat simplified. I add a little escaping value, to make sure we have the same behaviour as nix proper. Interestingly enough, we never need to escape XML attribute names, because the `builtins.toXML` format encodes user-defined values as attribute keys only. So we only escape attribute values. Fixes: https://b.tvl.fyi/issues/399 Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Profpatsch <mail@profpatsch.de>
2024-05-16 r/8152 chore(tvix): drop json feature for tracing-subscriberFlorian Klink1-42/+2
Drop the (unused) feature flags here, and get rid of some crate dependencies. Change-Id: Id64852b498725467d56abb12b548301cfba6a760 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11678 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-16 r/8151 feat(tvix/build): drop json loggingFlorian Klink1-1/+0
This has already been dropped for tvix-store, drop it here as well. Change-Id: Ib3aa37bbe9fd4c498b5ba1213f7d922d0c64ffc7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11677 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-16 r/8150 feat(tvix/store): add ObjectStoreDirectoryServiceYureka1-1/+6
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-15 r/8149 chore(tvix): bump to tokio-listener 0.4.1Florian Klink1-654/+43
This gives multi-listener support, as well as listening on named socket FDs. Change-Id: I92b441f854e2faf80074463d0ca6bdc23cbd890a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11663 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de>
2024-05-15 r/8148 chore(tvix/store): bump otlp stackFlorian Klink1-265/+60
This aligns the tonic version we use directly in Tvix to the one pulled in by the OTLP stack. Change-Id: I658528c8dabb7cd6948f1207ddcdef1984e82037 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11666 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI