about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-11-02 r/6921 docs(tvix/castore): add notes on verified streamingFlorian Klink1-0/+121
These are still a bit WIP, but should already explain the idea better than not having any documentation about how we intend to implement this at all. Let's put this into the repo, as a working document. Co-Authored-By: edef <edef@edef.eu> Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6920 chore(tvix/castore): move data model docs to hereFlorian Klink3-15/+20
These describe the castore data model, so it should live in the castore crate. Also, some minor edits to //tvix/store/docs/api.md, to honor the move of the castore bits to tvix-castore. Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-31 r/6919 chore(tvix): regenerate Cargo.nixFlorian Klink1-6/+22
Change-Id: I96e01f938a46d12a94da85968caaf190d041b9ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9887 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-28 r/6902 fix(nix-compat/narinfo): don't allocate errors earlyedef1-11/+12
We also switch the MissingField error to &'static str, since we only parse a fixed set of fields. Together, this makes the performance impact of error handling negligible in batch happy-path parsing. Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-28 r/6901 fix(nix-compat/narinfo): unwrap in the benchmarkedef1-2/+6
We primarily want to measure the speed of the happy path. Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu>
2023-10-28 r/6900 refactor(tvix/nix-compat/narinfo): return errorsFlorian Klink1-45/+127
This provides more info about where a NARInfo failed to parse, rather than just returning None and leaving a library user to manually debug. Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-28 r/6899 refactor(nix-compat/nixbase32): clean up encodeedef1-16/+15
Change-Id: I63da5bd47cd9033e1cc13fbe4b4864514a5343cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9865 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6898 fix(nix-compat/nar/writer): `use ByteSlice` closer to call siteedef2-2/+2
Otherwise, it'll generate unused import warnings in release mode. Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6897 feat(tvix/nix-compat): add narinfo parsing and serialisationedef8-76/+492
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6896 refactor(nix-compat/store_path): use nixbase32::decode_fixededef1-4/+2
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6895 feat(nix-compat/store_path): add StorePathRefedef1-22/+72
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6894 feat(tvix/castore): carry name in ValidateNodeError::InvalidNameedef2-14/+14
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef6-30/+30
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6892 refactor(nix-compat/store_path): speed up validate_name fast pathedef1-12/+39
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6891 feat(nix-compat/store_path): validate_name takes AsRef<[u8]>edef2-3/+4
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6890 feat(nix-compat/store_path): validate_name over borrowed dataedef2-6/+10
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6889 refactor(nix-compat/store_path): don't materialise fingerprintedef1-8/+9
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef6-21/+27
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6887 fix(tvix/nix-compat): validate store path name lengthedef1-2/+13
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6886 fix(tvix/nix-compat): don't box CAHash::Textedef3-14/+9
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6885 feat(tvix/nix-compat): add fixed-length base32 decodingedef1-1/+34
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6884 feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode inputedef1-3/+5
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6883 refactor(tvix): condense long bytestringsedef11-93/+67
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6882 feat(tvix/nix-compat): decode base32 with a lookup tableedef1-22/+42
This also takes input validation out of the loop, leaving the loop backedge as the sole branch in the hot path. Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6881 refactor(tvix/nix-compat): clean up base32 decodingedef1-23/+18
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6880 refactor(tvix/nix-compat): simplify encode_lenedef1-6/+5
Change-Id: Idfefd3b5b0d1ffbd4e9208e634a2d9989965a291 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9845 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-25 r/6878 chore(tvix): move buildkite.yml from views/tvix to tvix/Florian Klink1-0/+9
This should remove the ambiguity on where buildkite.yml should show up at least. We might still want to move workspace.josh into tvix/ too at some point, but that's blocked on https://github.com/josh-project/josh/issues/1287 This commit doesn't cause any change in views/tvix, with a previous version of this commit added to my HEAD, I ran josh-filter to verify it still points to ce0df88561fee8a4d6052ca9f2a1792f899a202a. Change-Id: Ic03e92260dc7725bc2fcf01b7c72a097d92c028e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9828 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-23 r/6876 fix(tvix/cli/derivation): fix populate_output_configurationFlorian Klink1-302/+81
The `if let` wasn't matching `outputHashAlgo` being unset, and didn't populate it in that case. Port the remaining commented-out testcases over to nix-lang based tests. Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-23 r/6875 refactor(tvix/cli): rename `a` to `algo_str`Florian Klink1-2/+3
Change-Id: I65d307126647841b50db008c21c155441a6b6cb5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9818 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-23 r/6874 test(tvix/cli/derivation): add test cases for output path calcFlorian Klink4-0/+49
Get some test coverage with some calls to builtins.derivation. The expected output paths were calculated with Nix. Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 r/6873 refactor(tvix/cli): move evaluator instantiation to helperFlorian Klink1-30/+29
Have a private `eval` function in the test module that returns an EvaluationResult, and migrate the existing tests over to use it, rather than repeating itself. Change-Id: I879987700c8507248c644ef03b62a8cb8e308139 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9816 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 r/6872 fix(nix-compat/src/derivation/validate): remove breakFlorian Klink1-2/+31
If the output was fixed, we broke out of the for loop too early, before actually validating individual outputs. Change-Id: I2259697dfa2a157764358f6d326a1f7f6610647c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9815 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 r/6871 refactor(tvix/nix-compat): rename NixHashWithMode -> CAHashFlorian Klink10-209/+222
This specific struct is only used to represent content-addressed paths (in case a Derivation has a fixed-output hash, for example). Rename `Output`'s `hash_with_mode` to `ca_hash`. We now also include `CAHash::Text`, and update the `validate` function of the `Output` struct to reject text hashes there. This allows cleaning up the various output path calculation functions inside nix-compat/src/store_path/utils.rs, as they can now match on the type. `make_type` is renamed to `make_references_string`, `build_regular_ca_path` is renamed to `build_ca_path`, and `build_text_path` has a disclaimer added, because you might not actually want to use it. Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-22 r/6870 feat(tvix/store/protos): add CA fieldFlorian Klink4-3/+67
This adds support to represent the `CA` field found in some .narinfo files. As `deriver`, it's also a subfield of the `narinfo` field. Extending nix-compat with a more accessible data structure that can take care of formatting, as well as writing validation functions in Rust + Golang, and integrating it into nar-bridge is something for a followup CL. Change-Id: I71e9c30957bcd03051a491aa54d7baac25b6dd2d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9795 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-22 r/6869 refactor(tvix/store): move deriver field into narinfo submessageFlorian Klink11-90/+92
This information is present in the .narinfo files, it should have gone there. Change-Id: Ib43d0cf30c2795bf1fe77c46646174353ade0458 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9794 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-18 r/6857 test(tvix/nix-compat/nar/writer/sync): add testsFlorian Klink3-0/+138
Change-Id: I04c556e7b57f20dd33f71b0379e6104e1702a8ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/9785 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-18 r/6856 feat(tvix/nix-compat/nar/reader): add testsFlorian Klink5-0/+122
Change-Id: If0abaf74a3b7d201af8972e54e3e34b20b8c0bc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9764 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-18 r/6855 fix(tvix/nix-compat/nar/wire): correct Node::Some doc commentedef1-1/+2
It is *eventually* followed by a Node, but there is some stuff in between. Change-Id: Ie7c7b462828bd3e066f4a7e774895f30b82763ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/9768 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6854 refactor(nix-compat/nar/writer): use clone_intoedef2-4/+2
Equivalent to the existing code, but a little less cryptic. Change-Id: Ib9b2f9aedddc84d0e79840bba4cce01f92d9bc56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9766 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6853 feat(tvix/nix-compat): NAR readeredef8-4/+530
Change-Id: I50d51baf62c0419eaf17f0dc262f728aaff9794d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9688 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6852 docs(tvix/nix-compat/nar): document the wire formatedef1-2/+66
Change-Id: I6c8e23bad27fa6ada1b8973482b4d99190cf050d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9767 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-18 r/6851 refactor(tvix/nix-compat): use hash_with_mode's nix_hash_stringFlorian Klink1-7/+1
HashWithMode already provides a to_nix_hash_string() method, giving us exactly the string we need here. Change-Id: Id2635bf3ea6c2514faf3c26b297866d774f4ff4a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9799 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-17 r/6850 docs(tvix/castore/directorysvc): document drop without put and closeFlorian Klink1-0/+4
As seen in cl/9761, we only use put() and close() in some cases. Make sure this is documented, so we don't end up having implementations not able to deal with that. Change-Id: I406ad23f2f4c9604eca731d898ae41a4663fb846 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9786 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6849 chore(tvix/nar-bridge): update to latest go mod renamesFlorian Klink18-29/+29
- code.tvl.fyi/tvix/castore/protos -> code.tvl.fyi/tvix/castore-go - code.tvl.fyi/tvix/store/protos -> code.tvl.fyi/tvix/store-go See cl/9791, cl/9792 for context. Change-Id: I44614c6ed40b9f52d9dcdea8e61fe2c3c830ce78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9793 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-17 r/6848 chore(tvix/store-go): rename go moduleFlorian Klink12-16/+16
`code.tvl.fyi/tvix/store/protos` now points to a directory that only contains the `.proto` files, while all golang tooling and .pb.go files live in tvix/store-go. As discussed in https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount of people currently using this is still small, so rename the go.mod now, while it doesn't yet hurt. Also, use code.tvl.fyi/tvix/castore-go instead of code.tvl.fyi/tvix/ castore/protos, to make use of cl/9791. Change-Id: I9ea89957d7c29dfae4c893b9aae8ac8a0bad2d8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9792 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 r/6847 chore(tvix/castore-go): rename go moduleFlorian Klink5-5/+5
`code.tvl.fyi/tvix/castore/protos` now points to a directory that only contains the `.proto` files, while all golang tooling and .pb.go files live in tvix/castore-go. As discussed in https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount of people currently using this is still small, so rename the go.mod now, while it doesn't yet hurt. Change-Id: Ib3c6a2dac2923b3806ebb05be00af66d0da9f698 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9791 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-17 r/6846 chore(tvix): move regenerate scripts to passthruFlorian Klink3-14/+16
This keeps things isolated a bit more. Change-Id: I437f2f63ee1567c0cbc02298514ad4d89f1dce85 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9790 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6844 chore(tvix): move store golang bindings to tvix/store-goFlorian Klink18-7/+94
Similar to the castore-go CL before, this also updates the store-go bindings to the new layout. Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 r/6843 chore(tvix): move castore golang bindings to tvix/castore-goFlorian Klink15-7/+93
Have `tvix/castore/protos` only contain the protos, no go noise. Make the `.pb.go` file generation a pure Nix build at `//tvix/castore/protos:go-bindings`, and have a script at `//tvix:castore-go-generate` (TBD) that copies the results to `tvix/castore-go`. `//tvix:castore-go`, with sources in `tvix/castore-go` now contains the tooling around the generated bindings, and the generated bindings themselves (So go mod replace workflows still work). An additional CI step is added from there to ensure idempotenty of the .pb.go files. The code.tvl.fyi webserver config is updated to the new source code path. I'm still unsure if we want to also update the go.mod name. While being a backwards-incompatible change, it'll probbaly make it easier where to find these files, and the amount of external consumers is still low enough. Part of b/323. Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6842 fix(tvix/castore): Fix race when ingesting into castoreConnor Brewster1-0/+4
After finishing the ingestion, the directory putter was not being closed. This caused a race where the root directory node was accessed before the directory node had been flushed to the server. This patch makes it so we close the putter before returning the root node which should ensure that the root node exists on the directory service server before the `ingest_path` function returns. Fixes b/326 Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-24T10·07+0000