| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Have fetcher builtins call queue_fetch() whenever they don't need to
fetch something immediately, and teach TvixStoreIO::store_path_to_node
on how to look up (and call ingest_and persist on our Fetcher).
Change-Id: Id4bd9d639fac9e4bee20c0b1c584148740b15c2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11501
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We actually want to delay fetching until we actually need the file. A
simple evaluation asking for `.outPath` or `.drvPath` should work even
in a pure offline environment.
Before this CL, the fetching logic was quite distributed between
tvix_store_io, and builtins/fetchers.rs.
Rather than having various functions and conversions between structs,
describe a Fetch as an enum type, with the fields describing the fetch.
Define a store_path() function on top of `Fetch` which can be used to
ask for the calculated store path (if the digest has been provided
upfront).
Have a `Fetcher` struct, and give it a `fetch_and_persist` function,
taking a `Fetch` as well as a desired name, and have it deal with all
the logic of persisting the PathInfos. It also returns a StorePathRef,
similar to the `.store_path()` method on a `Fetch` struct.
In a followup CL, we can extend KnownPaths to track fetches AND
derivations, and then use `Fetcher` when we need to do IO into that
store path.
Change-Id: Ib39a96baeb661750a8706b461f8ba4abb342e777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11500
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
See https://patchwork.kernel.org/project/qemu-devel/patch/20200706135650.438362-5-stefanha@redhat.com/
Change-Id: I3ba9a1c1f8b0be27f215da6b71cd3a0ded513a75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11502
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This caught me by accident in an earlier revision of cl/11500 -
I had a `NixString`, wanted to return it as a `String`, so I was naively
calling `s.into()`.
That unfortunately gave me the `Display` implementation of `NixString`,
which quotes strings, causing an annoying error further up the stack.
NixStrings are bytes, we can keep the impl From<NixString> for BString,
but having a `.into()` suddenly do quoting is more than unexpected.
Change-Id: I5434ba94bfe6c493d0a57e68225ecc22daa4b948
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11505
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
This makes the eval tests and benchmarks standalone Nix derivations and
readTree targets:
```
nix-build -A tvix.cli.eval-nixpkgs-cross-hello-outpath
nix-build -A tvix.cli.benchmark-nixpkgs-attrnames
```
Even without doing any fetches, We need to set `SSL_CERT_FILE`, so
reqwest is able to load its CA roots.
Change-Id: Ib45282d01044165c7816391adbeeb26334f8e924
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11493
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We can speedup uploads further by not uploading all store paths
sequentially, but in parallel.
We still don't respect the reference graph, so nothing changed here.
Change-Id: I40edb5725fe7a15170f5b9f8f4c8bec2130ecca3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11492
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.
Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.
(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.
We still keep the same order, to avoid shuffling things and violating
the stream order.
This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.
Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This function was originally called `read_to_string` but was changed to
`open` to make it so that file contents aren't always held in memory.
A comment and error message were not updated to reflect the new name of
this method.
Change-Id: I3d86e2f6d7006c2e1513121fc3c62efcb7e7b9bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11495
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Fixes some build warnings that only happen when building in release mode
which disables `debug_assertions`.
Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
|
|
`ingest_entries` requires that all directories referenced by entries in
the ingestion stream have an explicit entry in the stream.
For example, if the stream contains a file with path `foo/bar`, there
must be an entry that comes later in the stream for the directory `foo`.
Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Implement a first pass at the fetchTarball builtin.
This uses much of the same machinery as fetchUrl, but has the extra
complexity that tarballs have to be extracted and imported as store
paths (into the directory- and blob-services) before hashing. That's
reasonably involved due to the structure of those two services.
This is (unfortunately) not easy to test in an automated way, but I've
tested it manually for now and it seems to work:
tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath
=> "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string
Co-authored-by: Connor Brewster <cbrewster@hey.com>
Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Explicitly document and add a debug assertion for that.
It's up to callers to ensure this doesn't happen.
Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Move error types and filesystem-specific functions to a separate file,
and keep the fs:: namespace in public exports.
Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
This is not a stream of direntries anymore, but a stream of ingestion
entries.
Change-Id: I387f4497b6567066b24c58ca0262e710348180e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Previously the store ingestion code was coupled to `walkdir::DirEntry`s
produced by the `walkdir` crate which made it impossible to reuse
ingesting from other sources like tarballs or NARs.
This introduces a `IngestionEntry` which carries enough information for
store ingestion and a future for computing the Blake3 digest of files.
This allows the producer to perform file uploads in a way that makes
sense for the source, ie. the filesystem upload could concurrently
upload multiple files at the same time, while the NAR ingestor will need
to ingest the entire blob before yielding the next blob in the stream.
In the future we can buffer small blobs and upload them concurrently,
but the full blob still needs to be read from the NAR before advancing.
Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This adds `Directory::add` which is a convenience helper for adding
nodes into a `Directory` while preserving sorted order.
This implements `Ord` and `PartialOrd` for `FileNode`, `SymlinkNode`,
and `DirectoryNode` so `binary_search` can be used.
Change-Id: I94b86bdef5d0da55aa352e098988b9704cafca19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11481
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Id1bac94e7b95960dabd55f095f04bdabfeee10ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11476
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I6a6cabde22d28bb9afc391b9bcd896a999090366
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11475
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This is especially nice, as it allows us getting rid of the other
versions of proc-macro2 and quote.
Change-Id: I9fdd012ee6c0ded3e18ec30504b83ac2032d1390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11474
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Also, replace the useless read_file helper with std::fs::read.
Change-Id: I704ab42d1ffb9089716533125ff390b7441a8591
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11473
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I7c29e7d9e2b9cd66483ed44a9f2b3d291061f81d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11472
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I053f2c7fc9fdf59de5d8697eab32644b860e40be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11471
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: If05fa3d0f41d45e3574e0c5ddff406cb33dc52f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11470
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Ic466a27d61b95ca4d297abd6eb976c083e8b40af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11469
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I9ed5e728ff0706a37d0a238210a8dd23eb306033
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11468
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Change-Id: Ibd3cd1bc0d9deaf018a66748cea56af51451703c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11467
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I75d8d61f836c76e8765e0e3b49022c056de84850
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11466
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I62c3b93af438fd6397ded85dd4fe5acc01e0093a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11465
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Also, rename the DUMMY_NAME constant in the fixtures to DUMMY_PATH,
which aligns more with the ToString representation and from_bytes
conversions we have on StorePath[Ref].
Change-Id: I39763c9dfa84c5d86f2fd0171b3a4d36fd72f267
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11464
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Change-Id: Id673cd9c574b00a739de78aab2ee86f9d9b1ed40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11463
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This provides a nice wrapper struct to deal with versions.
Change-Id: I6acc03bc9f8d84a0583196073b52776c45d3fe92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11454
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
There's no point in not enabling these, and rust-analyzer is quite
useless too.
nix-compat users outside of here can still explicitly disable default
features if they don't want to pull in these features.
Change-Id: I8f2f24e7734ad152d1733bc3421098fc91af290a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11455
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This makes use of the vmCmdline and assertVMOutput to boot a NixOS VM
off of `tvix-store virtiofs` have it write a message on successful
bootup, then have it shut down again.
We check we see the message in the output.
Seeding the store with the closure it's what's currently taking most
of the time (as it's all linear still), so log a message before booting
the VM.
Change-Id: I6887d935f589eb0ad6507033c01337e1ec500a0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11191
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Userland likes to seek backwards, and until we have store composition
and can serve chunks from a local cache, we need to buffer the
individual chunks in memory.
Change-Id: I66978a0722d5f55ed4a9a49d116cecb64a01995d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11448
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This makes it easier to separate concurrent requests on the same inode.
Change-Id: I7637c1d889336beeb0d186182ce22fbf60fd16c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11447
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
The docs state we must fill all of the buffer, except on EOF.
Change-Id: Id977ba99c0b15132422474ebbf82bb92b79d55ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11446
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: Iecf4a82a7d84008a8620825570b34e9094e6d590
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11445
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: I2408707a7bc0e1c0cd8bd2933f8d68805b9e12c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11444
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
When (re)initializing a chunked reader, we were erroneously skipping the
first n bytes from all chunks, not just the first one.
Fix this, by passing in an enumerated list of chunks, and only calling
SeekFrom::Start() on the first chunk in the stream.
With this, I'm able to invoke b3sum on bin/bash successfully.
Change-Id: I52ea480569267e093b0ac9d6bcd5c2d1b4db25f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11443
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Increase the discard_buf to 4096 (as I've seen this size).
Use the ready! macro to propagate pendings.
Make it more clear what exactly should be skipped in total, and what
during the current iteration.
Also write down that poll_read call already takes care of updating
self.pos, as I ran into that trap earlier (and added it here).
Change-Id: I2d22e1c8a835c0f3dd0c648917009b2bad4fd57c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11442
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
If the resulting offset equals to our current position, there's no need
to recreate a reader.
Change-Id: I855f0c79c514c16ca48a78e12978af2835fbbd6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11441
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
State that this case applies if the blob is small enough to fit inside a
single chunk.
Change-Id: I0383514729e686799599b629cf1303b284147bb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11440
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
We previously returned Ok(None) when being asked for more granular
chunking info, signalling the blob does not exist at all.
This is however incorrect, we should return an empty Vec instead, as
documented in the trait.
Change-Id: I83ecc2027e0767134c7598792c2ee6d964853c66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11439
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Don't log with info! here, bug debug!.
Change-Id: I57bd5f2a45276090b893a4051fd175e3948ddfa4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11438
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
It's perfectly normal if we ask for more granular chunking info and the
backend responds it does not have it.
Change-Id: I593ab3e53b4f4e70c99f39b266546d2ac8eb10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11437
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We're receiving bytes over the wire, and encode them the same way
B3Digest does internally, but don't use it for formatting, as we're
discarding that string.
In case the sent bytes don't have the right length, the string will be
short, but it's better to still have it as a field, even if it's not a
valid b3 digest.
Change-Id: I6ef08275d51c8a0d98f5e46844b15dfd05d17cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11436
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
As suggested in cl/11426.
Change-Id: Ic2bb8cf2838bf0be09fb8bc62b8e598a3d153699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11434
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
These were copying unnecessarily. Instead, have a
InodeData::from_node(), which *consumes* the Node entirely, returns
`InodeData` and the split-off name (which is not part of InodeData).
Callers can then use the result in various helper functions, like:
- InodeData::as_fuse_type
- InodeData::as_fuse_file_attr
- InodeData::as_fuse_entry
… to prepare their replies to the kernel.
This removes not only a bunch of clones, but also a lot of copy-pasted
code.
Change-Id: Idbca5f25cc29e96c1f4c614b33dff2becb0a8738
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11435
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
We allow reading individual chunks via open_read(), it's inconsistent if
a has() would return Ok(false).
Change-Id: Ie713d968172ccd2687d2e6e0dfef89ee152ef511
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11420
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
We don't need to clone BlobService anymore.
Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
