about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2024-07-01 r/8331 chore(tvix): bump to data-encoding 2.6.0Florian Klink9-12/+12
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-29 r/8330 feat(tvix/boot/tests): add closure-nixos-s3Florian Klink1-0/+20
This uses the `preStart` mechanism to boot up a minio instance, which we point blob/directoryservice to. Change-Id: I9e67b19006cb08ce82fe63c885cb0178e97389e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11896 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 r/8329 fix(tvix/castore/object_store): make query pairs object_store optsFlorian Klink2-4/+10
We previously called ObjectStoreBlobService::parse_url, which passes an empty list of options when constructing the ObjectStore. This is most likely not what we want. The more reasonable thing to do is pass along the query string (pairs) as options to `object_store::parse_url_opts`, and remove them from the plain URL we pass to object_store itself. Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-29 r/8328 refactor(tvix/castore/blobsvc): inline things a bitFlorian Klink1-5/+5
Get rid of the `let grpc_client` and `let resp` in some cases. Change-Id: Idc1c0f566a3b1b48da62e6f1977b07620656b16c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11884 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 r/8327 docs(tvix/castore): drop wrong commentFlorian Klink1-1/+0
This doesnt panic anymore. Change-Id: I1987f8b0640b1e5fe33abcbf46164449c766fd0d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11883 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-06-29 r/8326 feat(tvix/boot/tests): closure-nixos-bigtableFlorian Klink1-0/+21
This uses the `preStart` mechanism to boot up a bigtable emulator, which we point directory/pathinfoservice to. Change-Id: Icb383403cb2d1f7e19c71b6298e82b391f9c8d22 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11895 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-29 r/8325 fix(tvix/boot/tests): escape addrsFlorian Klink1-9/+9
Otherwise some URL parameters (containing &) might break the shell. We cannot use `$PWD` anymore, but that's statically known anyways. Change-Id: I739249111461ecdb33d42a48a4352f7612fce90f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11894 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-29 r/8324 feat(tvix/boot/tests): add preStart parameterFlorian Klink1-0/+6
This allows running some code before tvix-daemon is started up, which is helpful to spin up some mock services. Change-Id: Ie7b6eaaf76c40def493f50879ee5255fd1ebd2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11893 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-06-29 r/8323 fix(tvix/boot/tests): set SSL_CERT_FILEFlorian Klink1-0/+2
This is necessary for http clients to successfully construct. Change-Id: If0d968e964f5c58d9956b92ee0ab3d97c39391d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11892 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-29 r/8322 feat(tvix/boot): introduce useNarBridge parameterFlorian Klink1-2/+50
This is still useful to test the ingestion part of nar-bridge. Reintroduce the logic removed in cl/11398, but put it behind a `useNarBridge` parameter, defaulting to false. Add an additional target to tvix.boot.tests, `closure-nixos-nar-bridge`, which does the same as `closure-nixos`, except seeding the contents through nar-bridge. Change-Id: Iee87d5d9c7b7d02df11ee47fbc3f1fbe951935e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11891 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-06-29 r/8321 refactor(tvix/boot/tests): only use exportReferencesGraph when neededFlorian Klink1-5/+5
These fields don't need to be set if isClosure is false. Change-Id: Id29ed034e0757c61cef09885962dfc5507ca106c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11890 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-28 r/8320 feat(nix-compat/nixcpp/conf): make fields pubFlorian Klink1-16/+16
Otherwise these are not accessible after parsing if parsed from another crate. Change-Id: Iba7b442722a2eb1d37c2c496a45aa559aa49c1f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11889 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-28 r/8319 feat(nix-compat/nixcpp): init nix.conf config parsingFlorian Klink5-0/+250
This allows parsing files like `/etc/nix/nix.conf` into the `NixConfig` struct. Change-Id: I90b25f43c429dd56127500ff5068e83852adee13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11888 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-06-28 r/8318 feat(nix-compat/narinfo/pubkey): derive Clone, [Partial]EqFlorian Klink1-1/+1
We want to use this in another struct that we want to compare and clone. Change-Id: I5c11d589a4922d85e190e1a9199468411016ff1e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11887 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-27 r/8317 docs(tvix): create Welcome + Contributing sectionstoastal5-0/+288
A lot of this content needs to be duplicated from the monorepo’s root directory. However, some of the content was not relevant or could have been more condensed in descriptions or split up for consumption that also matched the styles of some other library documentation I had been looking at for examples. I wanted to add the new access to Admonish callouts but also aimed at the readablitiy in one’s text editor for the user that needs to `cat` the docs. In the future, a “Introduction” page would be nice to add to “Welcome”, but I don’t think that is as clear. Change-Id: Ibfc169c5fc3501d7eeb9a76423c6f64cd68897aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11869 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-06-27 r/8316 docs: prompts in shell sessions shouldn’t be selectabletoastal2-1/+8
these just indicate the start of the prompt by convention & if the user is root or not with `#` & `$` respectively Change-Id: Id65f5f879b067ee96715b3e7dc63f68c7ad791b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11868 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-26 r/8315 fix(tvix/glue): reject unknown attrset args for fetch builtinsFlorian Klink3-1/+18
This now uses UnexpectedArgumentBuiltin in case builtins.fetchurl or builtins.fetchTarball are called with the wrong arguments: ``` note: while evaluating this Nix code --> [code]:1:1 | 1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";}) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error[E040]: Unexpected agrument `"hash"` passed to builtin --> [code]:1:19 | 1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";}) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ while calling this builtin ``` Change-Id: I51124255a46b78d3cf4dc89a1eca9e68750858d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11878 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-26 r/8314 refactor(tvix/eval): prefix ctx iterators in NixStringFlorian Klink2-12/+27
NixString::iter_plain() didn't make much sense, especially without a docstring. Rename it to iter_ctx_plain(), and copy the docstring from NixContext. Do the same for the two other context element types too. Change-Id: I1bbfcb967d8d9b14487d069bfe3a1f762253ef4d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11882 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-06-26 r/8313 feat(tvix/glue): implement builtins.toFileFlorian Klink4-9/+94
We currently only had a dummy implementation that didn't actually persist the files in the store(s). Copy the contents to the BlobService, and do the output path calculation as part of the upload. Use the plain context elements to construct the references. Change-Id: Ibdaf7a645ddc31e847faa4b87a79f2f95116a7ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/11879 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-26 r/8312 docs(tvix): use `#` for toplevel headingsFlorian Klink7-14/+7
See discussion in cl/11869. Let's use `#` consistently, we also use `##` for deeper nested subheadings. Change-Id: Ie25ebda708639fb617d456c275ae5a264fc4ce85 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11881 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: toastal <toastal@posteo.net>
2024-06-26 r/8311 feat(tvix/glue): add eval-fail-* support to glue testsFlorian Klink3-0/+10
Add some eval-fail-fetcht-*.nix test cases. Change-Id: Ic02beabf120d1de8df6f9c098d5a074127069f81 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11880 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 r/8310 fix(tvix/tracing): make cargo check and clippy happyFlorian Klink1-10/+8
In case the otlp feature is not enabled, these generate warnings during `cargo check`. Fix by moving some imports into their functions, or using the fully-qualified name (and one #[allow(unused_mut)]) Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 r/8309 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltinFlorian Klink1-0/+9
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8308 refactor(tvix/eval): rename UnexpectedArgument{,Formals}Florian Klink2-6/+6
There's other places where unexpected arguments can be provided, like in builtins. Make space for that error type. Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-26 r/8307 refactor(tvix/glue): return a parsed Url in NixFetchArgsFlorian Klink2-44/+29
The only two consumers (fetchurl, fetchtarball) of these do try to parse it as URL, so do it in the helper. Update url_basename to take a &url::URL, not a &str. Also update the test to use rstest for the fixtures to reduce some boilerplate there. Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8306 refactor(tvix/glue): take &CAHash, not CAHashFlorian Klink3-14/+16
We use a bit less cloning that way. Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 r/8305 feat(tvix/glue): handle regular file at builtins.path importFlorian Klink3-72/+114
If builtins.path is passed a regular file, no filtering is applied. We use the just-introduced file_type function in the EvalIO trait for that. This means, we don't need to pass through filtered_ingest, and can assemble the FileNode directly in that specific match case. This also means, we can explicitly calculate the sha256 flat digest, and avoid having to pipe through the file contents again (via blob_to_sha256_hash) to construct the sha256 digest. Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 r/8304 feat(tvix/eval): add file_type methodFlorian Klink3-0/+51
This allows peeking of the type at a given path. It's necessary, as an open() might not fail until you try to read() from it, and generally, stat'ing can be faster in some cases. Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-22 r/8303 refactor(tvix/eval): drop placeholder impls of builtins in glueIlan Joselevich1-16/+0
The builtins `placeholder` and `filterSource` are now implemented in tvix-glue and so we no longer need their placeholder "implementations" in tvix-eval. Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-21 r/8302 docs: use admonish for calloutstoastal5-9/+21
Change-Id: Icec7321a49c40e7f7bcb1e4b307f13b8db4eac25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11862 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8301 docs(tvix): drop done TODOs in the O11Y sectionFlorian Klink1-8/+0
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in cl/10952 and cl/11815. Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink3-24/+6
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser18-48/+399
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8298 feat(tvix/castore): instrument tokio task with current spanSimon Hauser3-18/+27
By default tokio::spawn does not instrument the spawned task with the current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do this manually for all tokio::spawn functions in functions that are instrumented. Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8297 feat(tvix/tracing): optional progressbarSimon Hauser4-6/+19
Disable the progressbar on default and provide a interface for optionally enabling the progressbar. Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8296 docs(tvix/docs): document nar renderer optimization potentialFlorian Klink1-0/+12
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 r/8295 feat(tvix/store/utils): detect gRPC NAR calculation serviceFlorian Klink1-5/+26
We were currently always using SimpleRenderer, which would mean the client would download every blob locally to calculate the checksum, which of course is very slow. Detect this special case and create a second instance (and client) for now. Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-18 r/8294 chore(tvix): patch crate2nix to drop darwin dontStripIlan Joselevich3-2/+24
This patch has already been applied on upstream but there hasn't been an official release yet so we patch it ourselves. Upstreamed patch: https://github.com/nix-community/crate2nix/commit/0209f258cda8a9972a785e26d92fb477ce4d1b0e Follow-up of: https://cl.tvl.fyi/c/depot/+/11856 Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8293 chore(tvix): move filterRustCrateSrc to utils.nixIlan Joselevich2-29/+29
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8292 chore(tvix): Remove no longer needed libiconv addition for darwinIlan Joselevich1-7/+1
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now automatically added for Darwin. Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 r/8291 chore(tvix): Use callPackage instead of import on Cargo.nixIlan Joselevich1-4/+1
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is with callPackage instead of import. Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 r/8290 chore(tvix): Remove no longer needed rlib stripping fix for darwinIlan Joselevich1-12/+0
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900 Change-Id: I4f569099e62c4284bdea18a512387735f139e702 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 r/8289 feat(tvix/tvix-store): improve progress barsFlorian Klink8-40/+52
Don't show an empty spinner for daemon commands. Move the bar to the right, so the text is better aligned between spinner progress and bar progress styles. Generally, push progress bars a bit more down to the place where we can track progress. This includes adding one in the upload_blob span. Introduce another progress style template for transfers, which interprets the counter as bytes (not just a plain integer), and also a data rate. Use it for here and in the fetching code, and also make the progress bar itself a bit less wide. Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 r/8288 feat(tvix/store): display progress for NAR calculationFlorian Klink2-2/+14
This is currently still taking a noticeable amount of time, so make sure we show it is happening. Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8287 docs(tvix/store/import): update import_path_as_nar_ca docstrFlorian Klink1-4/+5
Change-Id: I697172220efe73b2a215185027df09f815d32627 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8286 refactor(tvix/store): drop FUSE multithread workaround for DarwinFlorian Klink3-8/+27
The underlying issue in macFUSE has been fixed in https://github.com/osxfuse/osxfuse/issues/974. Bump our `macfuse` in nixpkgs to a version containing the fix. This can be removed while our nixpkgs pin is bumped past https://github.com/NixOS/nixpkgs/pull/320197. Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 r/8285 fix(tvix/store/bin): fix shutdown behaviour for FUSEFlorian Klink6-56/+113
Both umounts happening from another process, as well as tvix-store itself calling umount() on FuseDaemon will cause the FUSE worker threads to terminate. So far there was no nice way to wait on these threads to be terminated from multiple places, causing the `tvix-store mount` command to only be terminated if interrupted via ctrl-c, not via an external umount. Update FuseDaemon to use a ThreadPool, which gives us a join primitive over all threads, that can also be called from multiple places. Await on a join() from there to end the program, not the ctrl-c signal handler as it was before. Using FuseDaemon from multiple tasks requires Arc<>-ing both the ThreadPool as well as the inner FuseSession (which also needs to be inside a Mutex if we want to unmount), but now we can clone FuseDaemon around and use it in two places. We could probably also have used an Option and drop the FuseSession after the first umount, but this looks cleaner. Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-16 r/8284 feat(castore/directory): add cache combinatorYureka2-0/+144
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 r/8283 refactor(castore/directory/objectstore): use new order validation logicYureka1-14/+10
Use the new helper introduced in CL 11708 instead of rolling our own. Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 r/8282 refactor(castore/directory): separate order logic from ClosureValidatorYureka8-328/+640
ClosureValidator was previously only suitable for a very narrow use case: Validating incoming uploads, which are in leaves-to-root order. This is because the ordering validation was hard-wired into the add() function. This - Re-name ClosureValidator to DirectoryGraph, which is more suitable since it actually stores the Directory structs and is drained in the end. - Move the ordering-related logic to a separate OrderValidator, which can be used independently. - re-write DirectoryGraph to be a general purpose validator which can accept the input in both orders and can be drained in both orders as well. This means the DirectoryGraph and OrderValidator can now serve multiple new purposes: - Validating the incoming closure on the client while downloading. - Validating the incoming closure downloaded in a caching layer from the `far` cache, and re-order it for insertion into the `near` cache. Change-Id: I2b4b226348416912d7a31935bec050e53d911b70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-27T10·23+0000