about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-06-28 r/8318 feat(nix-compat/narinfo/pubkey): derive Clone, [Partial]EqFlorian Klink1-1/+1
We want to use this in another struct that we want to compare and clone. Change-Id: I5c11d589a4922d85e190e1a9199468411016ff1e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11887 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-27 r/8317 docs(tvix): create Welcome + Contributing sectionstoastal5-0/+288
A lot of this content needs to be duplicated from the monorepo’s root directory. However, some of the content was not relevant or could have been more condensed in descriptions or split up for consumption that also matched the styles of some other library documentation I had been looking at for examples. I wanted to add the new access to Admonish callouts but also aimed at the readablitiy in one’s text editor for the user that needs to `cat` the docs. In the future, a “Introduction” page would be nice to add to “Welcome”, but I don’t think that is as clear. Change-Id: Ibfc169c5fc3501d7eeb9a76423c6f64cd68897aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11869 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-06-27 r/8316 docs: prompts in shell sessions shouldn’t be selectabletoastal2-1/+8
these just indicate the start of the prompt by convention & if the user is root or not with `#` & `$` respectively Change-Id: Id65f5f879b067ee96715b3e7dc63f68c7ad791b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11868 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-26 r/8315 fix(tvix/glue): reject unknown attrset args for fetch builtinsFlorian Klink3-1/+18
This now uses UnexpectedArgumentBuiltin in case builtins.fetchurl or builtins.fetchTarball are called with the wrong arguments: ``` note: while evaluating this Nix code --> [code]:1:1 | 1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";}) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error[E040]: Unexpected agrument `"hash"` passed to builtin --> [code]:1:19 | 1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";}) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ while calling this builtin ``` Change-Id: I51124255a46b78d3cf4dc89a1eca9e68750858d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11878 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-26 r/8314 refactor(tvix/eval): prefix ctx iterators in NixStringFlorian Klink2-12/+27
NixString::iter_plain() didn't make much sense, especially without a docstring. Rename it to iter_ctx_plain(), and copy the docstring from NixContext. Do the same for the two other context element types too. Change-Id: I1bbfcb967d8d9b14487d069bfe3a1f762253ef4d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11882 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-06-26 r/8313 feat(tvix/glue): implement builtins.toFileFlorian Klink4-9/+94
We currently only had a dummy implementation that didn't actually persist the files in the store(s). Copy the contents to the BlobService, and do the output path calculation as part of the upload. Use the plain context elements to construct the references. Change-Id: Ibdaf7a645ddc31e847faa4b87a79f2f95116a7ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/11879 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-26 r/8312 docs(tvix): use `#` for toplevel headingsFlorian Klink7-14/+7
See discussion in cl/11869. Let's use `#` consistently, we also use `##` for deeper nested subheadings. Change-Id: Ie25ebda708639fb617d456c275ae5a264fc4ce85 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11881 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: toastal <toastal@posteo.net>
2024-06-26 r/8311 feat(tvix/glue): add eval-fail-* support to glue testsFlorian Klink3-0/+10
Add some eval-fail-fetcht-*.nix test cases. Change-Id: Ic02beabf120d1de8df6f9c098d5a074127069f81 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11880 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 r/8310 fix(tvix/tracing): make cargo check and clippy happyFlorian Klink1-10/+8
In case the otlp feature is not enabled, these generate warnings during `cargo check`. Fix by moving some imports into their functions, or using the fully-qualified name (and one #[allow(unused_mut)]) Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 r/8309 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltinFlorian Klink1-0/+9
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8308 refactor(tvix/eval): rename UnexpectedArgument{,Formals}Florian Klink2-6/+6
There's other places where unexpected arguments can be provided, like in builtins. Make space for that error type. Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-26 r/8307 refactor(tvix/glue): return a parsed Url in NixFetchArgsFlorian Klink2-44/+29
The only two consumers (fetchurl, fetchtarball) of these do try to parse it as URL, so do it in the helper. Update url_basename to take a &url::URL, not a &str. Also update the test to use rstest for the fixtures to reduce some boilerplate there. Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8306 refactor(tvix/glue): take &CAHash, not CAHashFlorian Klink3-14/+16
We use a bit less cloning that way. Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 r/8305 feat(tvix/glue): handle regular file at builtins.path importFlorian Klink3-72/+114
If builtins.path is passed a regular file, no filtering is applied. We use the just-introduced file_type function in the EvalIO trait for that. This means, we don't need to pass through filtered_ingest, and can assemble the FileNode directly in that specific match case. This also means, we can explicitly calculate the sha256 flat digest, and avoid having to pipe through the file contents again (via blob_to_sha256_hash) to construct the sha256 digest. Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8304 feat(tvix/eval): add file_type methodFlorian Klink3-0/+51
This allows peeking of the type at a given path. It's necessary, as an open() might not fail until you try to read() from it, and generally, stat'ing can be faster in some cases. Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-22 r/8303 refactor(tvix/eval): drop placeholder impls of builtins in glueIlan Joselevich1-16/+0
The builtins `placeholder` and `filterSource` are now implemented in tvix-glue and so we no longer need their placeholder "implementations" in tvix-eval. Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-21 r/8302 docs: use admonish for calloutstoastal5-9/+21
Change-Id: Icec7321a49c40e7f7bcb1e4b307f13b8db4eac25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11862 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8301 docs(tvix): drop done TODOs in the O11Y sectionFlorian Klink1-8/+0
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in cl/10952 and cl/11815. Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink3-24/+6
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser18-48/+399
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8298 feat(tvix/castore): instrument tokio task with current spanSimon Hauser3-18/+27
By default tokio::spawn does not instrument the spawned task with the current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do this manually for all tokio::spawn functions in functions that are instrumented. Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8297 feat(tvix/tracing): optional progressbarSimon Hauser4-6/+19
Disable the progressbar on default and provide a interface for optionally enabling the progressbar. Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8296 docs(tvix/docs): document nar renderer optimization potentialFlorian Klink1-0/+12
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8295 feat(tvix/store/utils): detect gRPC NAR calculation serviceFlorian Klink1-5/+26
We were currently always using SimpleRenderer, which would mean the client would download every blob locally to calculate the checksum, which of course is very slow. Detect this special case and create a second instance (and client) for now. Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-18 r/8294 chore(tvix): patch crate2nix to drop darwin dontStripIlan Joselevich1-2/+0
This patch has already been applied on upstream but there hasn't been an official release yet so we patch it ourselves. Upstreamed patch: https://github.com/nix-community/crate2nix/commit/0209f258cda8a9972a785e26d92fb477ce4d1b0e Follow-up of: https://cl.tvl.fyi/c/depot/+/11856 Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8293 chore(tvix): move filterRustCrateSrc to utils.nixIlan Joselevich2-29/+29
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8292 chore(tvix): Remove no longer needed libiconv addition for darwinIlan Joselevich1-7/+1
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now automatically added for Darwin. Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 r/8291 chore(tvix): Use callPackage instead of import on Cargo.nixIlan Joselevich1-4/+1
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is with callPackage instead of import. Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 r/8290 chore(tvix): Remove no longer needed rlib stripping fix for darwinIlan Joselevich1-12/+0
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900 Change-Id: I4f569099e62c4284bdea18a512387735f139e702 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 r/8289 feat(tvix/tvix-store): improve progress barsFlorian Klink8-40/+52
Don't show an empty spinner for daemon commands. Move the bar to the right, so the text is better aligned between spinner progress and bar progress styles. Generally, push progress bars a bit more down to the place where we can track progress. This includes adding one in the upload_blob span. Introduce another progress style template for transfers, which interprets the counter as bytes (not just a plain integer), and also a data rate. Use it for here and in the fetching code, and also make the progress bar itself a bit less wide. Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 r/8288 feat(tvix/store): display progress for NAR calculationFlorian Klink2-2/+14
This is currently still taking a noticeable amount of time, so make sure we show it is happening. Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8287 docs(tvix/store/import): update import_path_as_nar_ca docstrFlorian Klink1-4/+5
Change-Id: I697172220efe73b2a215185027df09f815d32627 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8286 refactor(tvix/store): drop FUSE multithread workaround for DarwinFlorian Klink2-8/+14
The underlying issue in macFUSE has been fixed in https://github.com/osxfuse/osxfuse/issues/974. Bump our `macfuse` in nixpkgs to a version containing the fix. This can be removed while our nixpkgs pin is bumped past https://github.com/NixOS/nixpkgs/pull/320197. Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 r/8285 fix(tvix/store/bin): fix shutdown behaviour for FUSEFlorian Klink6-56/+113
Both umounts happening from another process, as well as tvix-store itself calling umount() on FuseDaemon will cause the FUSE worker threads to terminate. So far there was no nice way to wait on these threads to be terminated from multiple places, causing the `tvix-store mount` command to only be terminated if interrupted via ctrl-c, not via an external umount. Update FuseDaemon to use a ThreadPool, which gives us a join primitive over all threads, that can also be called from multiple places. Await on a join() from there to end the program, not the ctrl-c signal handler as it was before. Using FuseDaemon from multiple tasks requires Arc<>-ing both the ThreadPool as well as the inner FuseSession (which also needs to be inside a Mutex if we want to unmount), but now we can clone FuseDaemon around and use it in two places. We could probably also have used an Option and drop the FuseSession after the first umount, but this looks cleaner. Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-16 r/8284 feat(castore/directory): add cache combinatorYureka2-0/+144
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 r/8283 refactor(castore/directory/objectstore): use new order validation logicYureka1-14/+10
Use the new helper introduced in CL 11708 instead of rolling our own. Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 r/8282 refactor(castore/directory): separate order logic from ClosureValidatorYureka8-328/+640
ClosureValidator was previously only suitable for a very narrow use case: Validating incoming uploads, which are in leaves-to-root order. This is because the ordering validation was hard-wired into the add() function. This - Re-name ClosureValidator to DirectoryGraph, which is more suitable since it actually stores the Directory structs and is drained in the end. - Move the ordering-related logic to a separate OrderValidator, which can be used independently. - re-write DirectoryGraph to be a general purpose validator which can accept the input in both orders and can be drained in both orders as well. This means the DirectoryGraph and OrderValidator can now serve multiple new purposes: - Validating the incoming closure on the client while downloading. - Validating the incoming closure downloaded in a caching layer from the `far` cache, and re-order it for insertion into the `near` cache. Change-Id: I2b4b226348416912d7a31935bec050e53d911b70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-06-15 r/8280 feat(tvix/cli,store): add tracy featureFlorian Klink3-2/+11
If compiled with this features, this emits packets compatible with the [Tracy](https://github.com/wolfpld/tracy) format. Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 r/8279 docs(tvix/TODO): dibs on crate2nix WASMIlan Joselevich1-1/+1
Change-Id: Idde9aaa31ecaa9570559ffe3271bc23b7a71133c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11820 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 r/8278 docs(tvix/TODO): add builtins.path roundtrip for flatFlorian Klink1-0/+9
Change-Id: I8291d2fdfa2ea31ec24c82dcfb447fa9eaa3a961 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11819 Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-06-15 r/8277 docs(tvix/*store): add some internal linksFlorian Klink4-5/+5
We can now cross-link between pages, and it actually works. Change-Id: Id8bd85111672c687118db9adae7f5066f22441f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11817 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 r/8276 docs(tvix/TODO): drop builtins:fetchurl itemFlorian Klink1-4/+0
This has been implemented in cl/11786, cl/11790 and cl/10975. This includes support for the `executable` and `unpack` variants of it. Change-Id: I5bcad5b8e86b9fd341219a59ba86660b692e66e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11818 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-14 r/8275 feat(tvix/tracing): add tracing-tracy supportFlorian Klink5-69/+1175
This introduces another feature flag, "tracy" to the `tvix-tracing` crate. If enabled (not enabled by default), it'll add an additional layer emitting packets in a format that https://github.com/wolfpld/tracy can display. I had to be a bit tricky with the combinatorial complexity when adding this, but the resulting code still seems manageable. Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Reviewed-by: flokli <flokli@flokli.de>
2024-06-14 r/8274 refactor(tvix/tracing): move otlp setup into helper functionFlorian Klink1-67/+78
Having all this in the main control flow makes it a bit hard to read. Moving it into a helper function makes it a bit cleaner. Change-Id: Ibdb739dbd1e013b4f8c4aaf9b036a6bd556a1871 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11814 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-14 r/8273 fix(tvix/tracing): reduce the error logs of otlp if collector is offlineSimon Hauser2-12/+15
The problem is that opentelemetry_otlp tonic batch exporter tries to exports if either the `scheduled_delay` or if the `max_export_batch_size` is reached. Per default the `max_export_batch_size` is set to 512 spans, which means that we try to export these spans once that counter is reached. Each export will then try to connect to the exporter (if that not already happening) and will result in a `tcp connect error`. Increasing the max_export_batch_size to 4096 will then ensure that the export only happens if the `scheduled_delay` is met after the 10 seconds. `max_queue_size` is also increased, because `max_export_batch_size` should not be greater than `max_queue_size`, so similar to the default config its set to `max_export_batch_size * 4`. This will reduce the amount of tries to otlp if the collector is not available and otlp enabled. Change-Id: Ic3430006e8a104fa3b34d274678cae55b3620ce9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11791 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 r/8272 feat(tvix/tracing): apply EnvFilter to all layersSimon Hauser1-10/+12
Currently we apply the EnvFilter only to the stderr output writer. This didn't affect any other layer, like the otlp layer, causing spans from `h2`, `tokio_util` or other third party crate dependencies to be always sent out via OTLP. This changes that behaviour, applying EnvFilter to all exports, leading to a lot less spans being exported. Change-Id: I9f3a7233e9d0aeaa81fe08914579f0b3c80d134e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11813 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 r/8271 feat(tvix/tracing): correctly close otlp on exitSimon Hauser7-65/+242
Provide a new interface for forcing a flush of otlp traces and use this interface to shutdown otlp prior to exiting tvix-store, either if the tool was stopped with a SIGTERM or ended regularly. This also fixes an issue where traces were not even exported if for example we just imported 10 paths and never even emitted more than 256 traces. The implementation uses a mpsc channel so a flush can be done without having to wait for it to complete. If you want to wait for a flush to complete you can provide a oneshot channel which will receive a message once flushing is complete. Because of a otlp bug `force_flush` as well as `shutdown_tracer_provider` need to be executed using `spawn_blocking` otherwise the function will deadlock. See https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335 Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 r/8270 chore(tvix/eval): move eval docs to tvix/docsFlorian Klink12-0/+16
Change-Id: I75b33c43456389de6e521b4f0ad46d68bc9e98f6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11809 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-06-14 r/8269 chore(tvix/docs): move [ca]store docs to tvix/docsFlorian Klink7-1/+8
Change-Id: Idd78ffae34b6ea7b93d13de73b98c61a348869fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11808 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2024-06-14 r/8268 feat(tvix/docs): add d2 supportFlorian Klink3-0/+8
This provides a nicer syntax to draw some diagrams. Change-Id: I2ab580e61399a45f01ed422ee29382e860753ffa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11807 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev>