about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-03-13 r/5967 refactor(tvix/eval): simplify NixString representation(s)Vincent Ambo3-65/+37
Instead of the two different representations (which we don't really use much), use a `Box<str>` (which potentially shaves another 8 bytes off `Value`). NixString values themselves are immutable anyways (which was a guarantee we already had with `SmolStr`), so this doesn't change anything else. Change-Id: I1d8454c056c21ecb0aebc473cfb3ae06cd70dbb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8151 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-13 r/5966 refactor(tvix/eval): wrap NixList in RcVincent Ambo2-14/+16
The size of a `Vector<Value>` is 64 *bytes*, which is quite large, and it bloated the entire Value type to this size. This change adds an indirection for the inner vector through Rc. Initially I tried to use a Box, but this breaks pointer equality guarantees for the Vector when it is small enough to be inlined. This reduces the size of Value from 64 to 32 bytes. Change-Id: Ic3211e861b1966c78b2c3d536ba291fea92647fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8150 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5965 test(tvix/eval): add test for infinite recursion detectionVincent Ambo1-0/+1
Change-Id: Ief20544a44c3542fe40a5c09f81d0f064a346f44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8149 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 r/5964 refactor(tvix/eval): flatten call stack of VM using generatorsVincent Ambo24-2677/+2500
Warning: This is probably the biggest refactor in tvix-eval history, so far. This replaces all instances of trampolines and recursion during evaluation of the VM loop with generators. A generator is an asynchronous function that can be suspended to yield a message (in our case, vm::generators::GeneratorRequest) and receive a response (vm::generators::GeneratorResponsee). The `genawaiter` crate provides an interpreter for generators that can drive their execution and lets us move control flow between the VM and suspended generators. To do this, massive changes have occured basically everywhere in the code. On a high-level: 1. The VM is now organised around a frame stack. A frame is either a call frame (execution of Tvix bytecode) or a generator frame (a running or suspended generator). The VM has an outer loop that pops a frame off the frame stack, and then enters an inner loop either driving the execution of the bytecode or the execution of a generator. Both types of frames have several branches that can result in the frame re-enqueuing itself, and enqueuing some other work (in the form of a different frame) on top of itself. The VM will eventually resume the frame when everything "above" it has been suspended. In this way, the VM's new frame stack takes over much of the work that was previously achieved by recursion. 2. All methods previously taking a VM have been refactored into async functions that instead emit/receive generator messages for communication with the VM. Notably, this includes *all* builtins. This has had some other effects: - Some test have been removed or commented out, either because they tested code that was mostly already dead (nix_eq) or because they now require generator scaffolding which we do not have in place for tests (yet). - Because generator functions are technically async (though no async IO is involved), we lose the ability to use much of the Rust standard library e.g. in builtins. This has led to many algorithms being unrolled into iterative versions instead of iterator combinations, and things like sorting had to be implemented from scratch. - Many call sites that previously saw a `Result<..., ErrorKind>` bubble up now only see the result value, as the error handling is encapsulated within the generator loop. This reduces number of places inside of builtin implementations where error context can be attached to calls that can fail. Currently what we gain in this tradeoff is significantly more detailed span information (which we still need to bubble up, this commit does not change the error display). We'll need to do some analysis later of how useful the errors turn out to be and potentially introduce some methods for attaching context to a generator frame again. This change is very difficult to do in stages, as it is very much an "all or nothing" change that affects huge parts of the codebase. I've tried to isolate changes that can be isolated into the parent CLs of this one, but this change is still quite difficult to wrap one's mind and I'm available to discuss it and explain things to any reviewer. Fixes: b/238, b/237, b/251 and potentially others. Change-Id: I39244163ff5bbecd169fe7b274df19262b515699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-03-13 r/5963 feat(tvix/eval): implement generator-based Nix equality logicVincent Ambo2-6/+182
Adds a `Value::neo_nix_eq` method (the `neo_` prefix will be dropped when we flip over to the generator implementation of the VM) which implements Nix equality semantics using async, generator-based comparisons. Instead of tracking the "kind" of equality that is being compared (see the pointer-equality doc) through a pair of booleans, I've introduced an enum that explicitly lists the possible comparisons. Change-Id: I3354cc1470eeccb3000a5ae24f2418db1a7a2edc Reviewed-on: https://cl.tvl.fyi/c/depot/+/8241 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-03-13 r/5962 feat(tvix/eval): add generator-related functions to RuntimeObserverVincent Ambo2-7/+89
These functions will be used by the changes in the VM to observe the runtime execution of generator frames, and provide a more linear view of the execution of the Tvix VM. Change-Id: I10b1b1933dedc065e7c61d5d6062f0aaeee0097e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8240 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-03-13 r/5961 feat(tvix/eval): implement asynchronous list sorting algorithmVincent Ambo1-0/+47
In order to implement an asynchronous builtins.sort (required for moving builtins to generators), we need an `async` sorting algorithm as our comparators involve invoking a Nix function. This commit implements a fairly simple, optimised bubble sort as the sorting algorithm used in our `async fn sort_by`. There don't seem to be any crates providing async versions of things like this, and they might actually be pretty hard to implement generically due to some constraints about how `async` works. Note that this algorithm is less efficient than the hybrid "timsort/mergesort/insert sort" used in the Rust standard library. I tried to write a merge sort implementation, but ran into isuses with the sort becoming unstable because our comparators can not yield equality. This is the simplest implementation which I know to be correct. Note that as of this commit this is *not* covered by the Tvix test suite, but it will be as soon as the rest of the generator code lands. Change-Id: Ia9a604f7dd941d6acc9212c902e0e637ed75bebc Reviewed-on: https://cl.tvl.fyi/c/depot/+/8239 Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-03-13 r/5960 feat(tvix/store): drop BlobWriterFlorian Klink2-142/+0
All code initially using this has been replaced by the simpler and more performant implementation with StreamCDC and read_all_and_chunk. Change-Id: I08889e9a6984de91c5debcf2b612cb68ae5072d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8265 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5959 refactor(tvix/store): use read_all_and_chunk in gRPC blobserviceFlorian Klink4-40/+50
This was the last piece of code using BlobWriter. We can also use `read_all_and_chunk`, it's just requires a bit more plumbing: - The data coming from the client (stream) needs to be mapped (we extract the .data field). - The stream needs to be turned into an (async) reader - The reader needs to be made sync, and that code using the sync reader needs to be in a `task::spawn_blocking`. Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 r/5958 refactor(tvix/store): add read_all_and_chunk methodFlorian Klink3-43/+46
This moves the logic from src/import.rs that - reads over the contents of a file - chunks them up and uploads individual chunks - keeps track of the uploaded chunks in a BlobMeta structure - returns the hash of the blob and the BlobMeta structure … into a generic read_all_and_chunk function in src/chunkservice/util.rs. It will work on anything implementing io::Read, not just files, which will help us in a bit. Change-Id: I53bf628114b73ee2e515bdae29974571ea2b6f6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8259 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-03-13 r/5957 refactor(tvix/store): use update_hasher in blobwriterFlorian Klink1-7/+3
Make use of the helper function here as well. Change-Id: Ia0afd84eb3903bb897ee6aee884dc291f3e4371c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8258 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-12 r/5956 chore(tvix/store): allow(unused_mut)Florian Klink1-0/+1
When building tvix-store without default features, this variable doesn't need to be mutable. Silence the warning. Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-11 r/5954 feat(tvix/eval): don't warn twice about dead codeFlorian Klink1-1/+3
We currently send two warnings in case of detecting dead code - W008 inside compile_dead_code, and a more detailed warning in all places that invoke compile_dead_code: ``` warning[W007]: useless operation on boolean: this expression is always false --> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:15 | 12079 | doCheck = false && !stdenv.isDarwin; | ^^^^^^^^^^^^^^^^^^^^^^^^^ warning[W008]: this code will never be executed --> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:24 | 12079 | doCheck = false && !stdenv.isDarwin; | ^^^^^^^^^^^^^^^^ ``` The place invoking `compile_dead_code` has more context to why the code is unused, so it's error message is much more useful. Stop emitting the less informative warning inside compile_dead_code (W008), and update the comment that we expect the caller to emit a warning. I kept W008 itself still around, in case we end up having places this will get used again. Change-Id: I2c5d84fc0cb4035872cd4b71cc3e9e34e120eb37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8024 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 r/5953 refactor(tvix/store): bump fastcdc depFlorian Klink4-6/+6
This removes the use of Box::new, switching fastcdc to version 3.0.2 with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed. Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-11 r/5952 refactor(tvix/store): factor out hash update into functionFlorian Klink4-19/+35
We're using this in a bunch of places. Let's move it into a helper function. Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 r/5951 docs(tvix/store): make upload_chunk docs a docstringFlorian Klink1-1/+1
Change-Id: I6794e2a69b4907ab8d9ed2d55335c11db2c30c02 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8250 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 r/5950 feat(tvix/store/import): use StreamCDC instead of blobwriterFlorian Klink1-9/+35
This seems to be way faster. Change-Id: Ica7cee95d108c51fe67365f07366634ddbbfa060 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8246 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-10 r/5949 refactor(tvix/store): rename C to DS, client to directory_serviceFlorian Klink1-9/+11
This matches the semantics in other gRPC wrappers. Change-Id: I505c901aa77a58deab216316d08542877b0b2c14 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5948 refactor(tvix/store): move upload_chunk out of blobwriterFlorian Klink3-27/+35
This is useful not only in blobwriter contexts. Change-Id: I4c584b5264ff7b4bb3b1a9671affc39e18bf4ccf Reviewed-on: https://cl.tvl.fyi/c/depot/+/8245 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5946 feat(tvix/store): clippyFlorian Klink1-2/+2
warning: the following explicit lifetimes could be elided: 'set --> store/src/proto/mod.rs:201:1 | 201 | / fn update_if_lt_prev<'set, 'n>( 202 | | prev_name: &'set mut &'n str, 203 | | name: &'n str, 204 | | ) -> Result<(), ValidateDirectoryError> { | |_______________________________________^ | = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes = note: `#[warn(clippy::needless_lifetimes)]` on by default Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5945 refactor(tvix/store/blobwriter): don't cloneFlorian Klink1-2/+1
This already is a &CS, so we don't need to clone(). Change-Id: I5397a5948ae7fe4781f18df760a79047f83dca01 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8243 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 r/5944 refactor(tvix/store): clippyFlorian Klink1-1/+1
Change-Id: Ib41c766a45e7d5467a4e5e90e1c00b009121959a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8238 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5943 refactor(tvix/store): impl Default for MemoryBlobServiceFlorian Klink2-10/+2
Change-Id: I78a4cd86cb364b970e99393579808e773db5ceb2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8237 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5942 refactor(tvix/store): impl Default for MemoryChunkServiceFlorian Klink2-10/+2
Change-Id: If1ceb870e9e5d03b651923a3a786350a8663e990 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8236 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 r/5941 refactor(tvix/store): impl Default for MemoryDirectoryServiceFlorian Klink2-10/+2
Change-Id: Ie75a1e67c8c0499ff10f60d90251b768937dad3a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8235 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5940 refactor(tvix/store): impl Default for MemoryPathInfoServiceFlorian Klink2-9/+2
Change-Id: I07f83f8aaa6312cc37ce7e7ee2b713e60df23782 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8234 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5939 feat(tvix/store): use rayon to upload chunks concurrentlyFlorian Klink5-39/+66
Look at the data that's written to us, and upload all chunks but the rest in parallel, using rayon. This required moving `upload_chunk` outside the struct, and accepting a ChunkService to use for upload (which it was previously getting from `self.chunk_service`). This doesn't speed up things too much for now, because things are still mostly linear. Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5938 feat(tvix/store/import): make sure entries are sortedFlorian Klink1-2/+5
The Directory service does already reject inserting invalid (wrongly sorted) Directory messages, but our test case didn't provoke it. Change-Id: I228e201925e8999186659a2d8da0118db184d9ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/8167 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5937 feat(tvix/store): move to daemon subcommand, add import subcommandFlorian Klink1-45/+106
This exposes the previous default behavior at the `tvix-store daemon` subcommand. It also adds a `tvix-store import` command, which will ingest a given path into the store. Change-Id: Ide14f1d409b9364e7f98090690c744326486e470 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5936 refactor(tvix/store): move entrypoint to src/bin/tvix-store.rsFlorian Klink2-3/+2
Change-Id: Ibb83be75a2be27debd9e85b43c1b824f59e54dab Reviewed-on: https://cl.tvl.fyi/c/depot/+/8165 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5935 feat(tvix/store): use Memory*Service instead of temporary sledFlorian Klink1-8/+8
Change-Id: I65d2030aa643b6937dd92baa85db033bdce38b02 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8208 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5934 feat(tvix/store): add new_temporary for all Sled servicesFlorian Klink11-118/+107
This provides a service using /dev/shm, that's deleted once the reference is dropped. Refactor all tests to use these, which allows getting rid of most TempDir usage in the tests. The only place where we still use TempDir is in the importer tests, which work on a filesystem path. Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5933 fix(tvix/store/proto/grpc_blobservice_wrapper): buffer recv dataFlorian Klink1-7/+14
While we don't want to keep all of the data in memory, we want to feed a reasonably-enough buffer to the chunking function, to prevent unnecessarily trying to chunk over and over again. Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5932 feat(tvix/store): add import::import_pathFlorian Klink7-0/+392
This imports the contents at a given Path into the tvix store. It doesn't register the contents at a Path in the store itself, that's up to the PathInfoService. Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5931 refactor(tvix/store/tests/nar_renderer): only upload EMPTY_BLOBFlorian Klink1-16/+14
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not referred from anywhere in the fixture. Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5930 refactor(tvix/store/tests): move nar contents to fixturesFlorian Klink2-90/+88
Change-Id: I89a700240e7250a4d93eb8c4235e79c29ecb7f64 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8157 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5929 refactor(tvix/store/tests): move fixtures into separate moduleFlorian Klink7-95/+94
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5928 refactor(tvix/store/tests): move gen_*_service() into helperFlorian Klink7-71/+57
This allows hiding to tests what exact implementation we're using, when testing things that do something with a store, but don't care what's used for underlying storage. Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5927 refactor(tvix/store): move blob splitting into a BlobWriter structFlorian Klink4-76/+172
This will moves the chunking-as-we-receive logic that so far only lived in grpc_blobservice_wrapper.rs into a generic BlobWriter. Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5926 feat(tvix/store): do not buffer blob dataFlorian Klink2-73/+139
Use the FastCDC::cut function to ask fastcd for cutting points as we receive the data. Make sure to keep the last chunk in the temporary buffer, as we might not actually cut at the end. Also, use rayon to calculate the blake3 hash if the input data is > 128KiB. Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5925 feat(tvix/store): bump fastcdc, use v2020 versionFlorian Klink4-6/+6
This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5924 feat(tvix/store): add --json argFlorian Klink4-4/+82
This configures logging as JSON. Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5923 chore(tvix/store): remove old implementationsFlorian Klink8-784/+0
These were implementing against the (more complicated) gRPC interface, for which we now have a wrapper. Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5922 refactor(tvix/store): use GRPC Wrappers and new servicesFlorian Klink2-10/+30
This swaps out the implementation used in the main entrypoint. We now use the non-gRPC aware Sled*Service, and wrap it with GRPC*ServiceWrapper to spin up the gRPC server. Change-Id: I5a1236c0612eee85e49891062040127c8bd95058 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5921 feat(tvix/store): add tests for GRPCPathInfoServiceWrapperFlorian Klink2-0/+108
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5920 feat(tvix/store): add GRPCPathInfoServiceWrapperFlorian Klink2-0/+87
This exposes a proto::pathinfo_service_server::PathInfoService for a directoryservice::PathInfoService and a way to calculate NARs. Change-Id: I30cd058562f83d063c78b84976ec97190de49400 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5919 feat(tvix/store): add pathinfoserviceFlorian Klink4-0/+165
This adds a PathInfoService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5918 feat(tvix/store): add NARCalculationService trait and implFlorian Klink5-3/+97
This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5917 feat(tvix/store): validate blob size in NARRendererFlorian Klink3-0/+115
Make sure the blob size in the current proto node matches what we get back from the blob backend. Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5916 refactor(tvix/store): don't use anyhow::Result<()> in testsFlorian Klink1-11/+6
It's much easier to just unwrap the TempDir::new() result. Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-01T22·59+0000