about summary refs log tree commit diff
path: root/tvix/docs/src/TODO.md
AgeCommit message (Collapse)AuthorFilesLines
2024-10-11 r/8789 refactor(tvix/glue/register_in_path_info_service): return only PathInfoFlorian Klink1-0/+14
The store path is already contained in the PathInfo, and the ca bits is already passed into the function, so known to the caller - there's no need to duplicate this. We can also avoid having two separate block_on in our import builtin - we already know the content hash before constructing, as we pass it in via ca_hash. There's still some room to unclutter some more of the code around importing - we still do NAR calculation twice in some cases, and some of the code might be share-able from other places producing PathInfo too. Log a TODO for this cleanup. Change-Id: I6a5fc427d15bc9293a396310143c7694dd2996c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12592 Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-11 r/8787 refactor(tvix/store): use strictly typed PathInfo structMarijan Petričević1-16/+0
This switches the PathInfoService trait from using the proto-derived PathInfo struct to a more restrictive struct, and updates all implementations to use it. It removes a lot of the previous conversion and checks, as invalid states became nonrepresentable, and validations are expressed on the type level. PathInfoService implementations consuming protobuf need to convert and do the verification internally, and can only return the strongly typed variant. The nix_compat::narinfo::NarInfo conversions for the proto PathInfo are removed, we only keep a version showing a NarInfo representation for the strong struct. Converting back to a PathInfo requires the root node now, but is otherwise trivial, so left to the users. Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-09 r/8782 refactor(tvix/cli/args): remove log-level argumentMarijan Petričević1-3/+0
RUST_LOG allows for a more granular tracing configuration Change-Id: Iea29fc2b154345eb7479dfd8bbea6abfd4716341 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12585 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-01 r/8746 docs(tvix/TODO): add PathInfo data types and ca reference itemsFlorian Klink1-0/+43
With https://cl.tvl.fyi/12533 in, we still need to lookup references to properly populate `BuildRequest`. It currently fails as the reference to h9lc1dpi14z7is86ffhl3ld569138595-audit-tmpdir.sh is not propagated. We should prevent Frankenbuilds from the go, so let's update our PathInfo type to accomodate for that. Change-Id: I26f9215312c258bba222efd390bc135f1a3a3d6d Reviewed-on: https://cl.tvl.fyi/c/depot/+/12560 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8487 docs(tvix/TODO): castore Directory struct is doneFlorian Klink1-16/+8
There's some followup work on making tvix-build use the stricter structs natively, but that should probably be combined with the overall trait changes there. Add an item there so we won't forget, but this isn't really castore territory anymore. Change-Id: I6b1f9fa02d5c87c821d4ddc4b1dcc6a98c4eeaa5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12200 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-13 r/8485 docs(tvix/TODO): DirectoryService with bucketed directories is doneFlorian Klink1-6/+0
We have this (`ObjectStoreDirectoryService`), as well as some store composition. Change-Id: I876f85da1d5b183d80b148c378f825033ec080e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12198 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-07 r/8451 docs(tvix/TODO): document blob prefetching dependencyFlorian Klink1-0/+2
This depends on the ChunkReader work. Change-Id: I38878d0f822c312151131e55baee4db6ef1c3650 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12142 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-07 r/8450 docs(tvix/TODO): add Correctness > Performance sectionFlorian Klink1-1/+31
Explain the current caveats as far as performance tuning is concerned. Change-Id: I1a9c11c81de09350240fb61e3c130fc401ef6618 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12141 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-08-04 r/8442 feat(tvix/store): Sled{PathInfo,Directory}Service -> Redb…Ilan Joselevich1-1/+0
Use redb instead of sled for the default filesystem implementation of PathInfoService and DirectoryService. In the future we'll also drop sled support completely. Change-Id: I513ff0c2ff953d59714aa50b9aa1301b02f53d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12085 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-29 r/8423 docs(tvix/docs): Remove redb path info service from TODOIlan Joselevich1-1/+0
It is now implemented and tested since https://cl.tvl.fyi/c/depot/+/11995 Change-Id: Ie08f511edf10bba4f46efc2c13f8db17eb7182a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12059 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-23 r/8406 docs(tvix/docs): add TODO for protobufFlorian Klink1-0/+14
Change-Id: Iacf4ef517885bdd6d6366446124863d9a138b12b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12024 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-07-06 r/8350 chore(tvix/docs): Cross off a TODOAspen Smith1-6/+0
This is done now, as of fc6359463 (feat(tvix/repl): Allow binding variables at the top-level, 2024-07-04) Change-Id: I2126cbadeb66625cda93a7cd7ddb31409ec60fd9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11955 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-07-05 r/8349 refactor(web/tvixbolt): buildRustPackage -> crate2nixIlan Joselevich1-12/+0
With the recent changes to crate2nix and buildRustCrate in nixpkgs it is now possible to build tvixbolt via crate2nix like we do for other tvix crates. We can reuse a lot of the customizations done in //tvix in tvixbolt to avoid repeating ourselves. A script for serving tvixbolt locally for testing purposes is also available now through the .serve attribute of tvixbolt. This change supersedes https://cl.tvl.fyi/c/depot/+/11821. Change-Id: I4864df8b75aec73cf5fee2428924ed4cfbb32902 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11952 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 r/8344 docs(tvix): update WASM statusFlorian Klink1-20/+6
crate2nix can now build WASM, and cl/11859 showcases this now. Even if it's not in the same cargo workspace, we should still migrate tvixbolt to a crate2nix build. Remove the part about Build/Store frontends, it was more of an explanation why we want WASM builds to be nice, it can be tracked in separate TODOs once more concrete. Change-Id: If4f5e0994b55520ba70cabefb4fcef9dc17bc394 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11945 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de>
2024-07-02 r/8339 feat(tvix/tracing): http trace propagationSimon Hauser1-4/+3
Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-06-26 r/8305 feat(tvix/glue): handle regular file at builtins.path importFlorian Klink1-9/+0
If builtins.path is passed a regular file, no filtering is applied. We use the just-introduced file_type function in the EvalIO trait for that. This means, we don't need to pass through filtered_ingest, and can assemble the FileNode directly in that specific match case. This also means, we can explicitly calculate the sha256 flat digest, and avoid having to pipe through the file contents again (via blob_to_sha256_hash) to construct the sha256 digest. Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-20 r/8301 docs(tvix): drop done TODOs in the O11Y sectionFlorian Klink1-8/+0
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in cl/10952 and cl/11815. Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser1-3/+0
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8296 docs(tvix/docs): document nar renderer optimization potentialFlorian Klink1-0/+12
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849 Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-15 r/8279 docs(tvix/TODO): dibs on crate2nix WASMIlan Joselevich1-1/+1
Change-Id: Idde9aaa31ecaa9570559ffe3271bc23b7a71133c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11820 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 r/8278 docs(tvix/TODO): add builtins.path roundtrip for flatFlorian Klink1-0/+9
Change-Id: I8291d2fdfa2ea31ec24c82dcfb447fa9eaa3a961 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11819 Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-06-15 r/8276 docs(tvix/TODO): drop builtins:fetchurl itemFlorian Klink1-4/+0
This has been implemented in cl/11786, cl/11790 and cl/10975. This includes support for the `executable` and `unpack` variants of it. Change-Id: I5bcad5b8e86b9fd341219a59ba86660b692e66e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11818 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-14 r/8273 fix(tvix/tracing): reduce the error logs of otlp if collector is offlineSimon Hauser1-10/+0
The problem is that opentelemetry_otlp tonic batch exporter tries to exports if either the `scheduled_delay` or if the `max_export_batch_size` is reached. Per default the `max_export_batch_size` is set to 512 spans, which means that we try to export these spans once that counter is reached. Each export will then try to connect to the exporter (if that not already happening) and will result in a `tcp connect error`. Increasing the max_export_batch_size to 4096 will then ensure that the export only happens if the `scheduled_delay` is met after the 10 seconds. `max_queue_size` is also increased, because `max_export_batch_size` should not be greater than `max_queue_size`, so similar to the default config its set to `max_export_batch_size * 4`. This will reduce the amount of tries to otlp if the collector is not available and otlp enabled. Change-Id: Ic3430006e8a104fa3b34d274678cae55b3620ce9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11791 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-11 r/8246 docs(tvix/docs/TODO): document ChunkService split ideaFlorian Klink1-1/+16
Change-Id: Ie9c88b0d14902c642e2d3d6603265688eef0e10d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11755 Reviewed-by: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser1-8/+3
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-10 r/8241 docs(tvix/docs/TODO): document attempt to add more granular errorsFlorian Klink1-0/+8
Change-Id: I46f99d15a4ce9c98dcdad822d3032bda3b09c12f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11754 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-10 r/8239 docs(tvix/docs/TODO): document crate2nix for WASM attemptsFlorian Klink1-0/+26
I tried moving web/tvixbolt to tvix/tvixbolt, and adding it to the cargo workspace. I then made `crates =` in `default.nix` a function accepting `pkgs` (so we can pass in another nixpkgs for some invocations), and then constructed another nixpkgs and crates instance like this: ``` pkgs-wasm = (import pkgs.path { localSystem = localSystem; crossSystem = { system = localSystem; rustc.config = "wasm32-unknown-unknown"; }; } crates-wasm = (crates pkgs-wasm); tvixbolt-test = crates-wasm.workspaceMembers.tvixbolt.build; ``` … leading to the architecture build failures described in the TODO. Change-Id: I32112d75f8c098d9810ca52b2d07cd76fae8d8d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11777 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-05 r/8216 docs(tvix/docs/TODO): expand Store Composition config sectionFlorian Klink1-3/+27
Write down some of the thoughts after brainstorming with yuka. Change-Id: I01c94474dc643b8c4993db80e50d3ec65f5c17f4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11749 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev>
2024-06-05 r/8214 docs(tvix/docs/TODO): extend O11Y sectionFlorian Klink1-6/+33
Expand on tvix-tracing crate strategy, add some more context regarding OTLP and span propagation. Change-Id: Ice55c116c20aaf60531100465192ce11969551ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/11750 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Reviewed-by: flokli <flokli@flokli.de>
2024-05-23 r/8166 docs(tvix/docs/TODO): add item for string context reworkFlorian Klink1-0/+9
Change-Id: I4592490a75fb05093b3fd8649db3b57bff748c0d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11707 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-05-23 r/8165 docs(tvix/TODO): toXML string context and self-closing tags is doneFlorian Klink1-3/+0
Change-Id: I83f6b1863a023ea0bf8518c67eb51f740c3ea89a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11706 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Alyssa Ross <hi@alyssa.is>
2024-05-11 r/8111 docs(tvix/docs): async nar reader is doneFlorian Klink1-4/+1
Change-Id: I4e8fd03d521b7d30904f84a6afcf842f1505ec7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11628 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8110 docs(tvix/todo): add section on [ca]store errorsFlorian Klink1-0/+7
Change-Id: I832eed8abb62b7e13972ed58649a96e31b17f033 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11610 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8109 docs(tvix/todo): update test suite sectionFlorian Klink1-4/+13
nix_oracle is quite misleading, and fooled me into thinking this validates all .exp files in the repo to match Nix output (which it doesn't, that's done by verify-lang-tests). Also, the whole test suite structure thing can use a bit more explanation. Change-Id: I2fadcc871843143270ad2ed9ac98de8287280e6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11609 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-25 r/8005 docs(tvix): add a list of TODOs/ideasFlorian Klink1-0/+129
This contains a rough collection of ideas on the TODO list, trying to keep track of it somewhere. Change-Id: Ifc5b0cf9f7ac38f7a8e56515882bdf70e349544b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11512 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-24T22·18+0000