| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
With `ingest_entries` being more generalized, we can now use it for
ingesting the directory entries generated from tarballs.
Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.
Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.
(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.
We still keep the same order, to avoid shuffling things and violating
the stream order.
This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.
Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Fixes some build warnings that only happen when building in release mode
which disables `debug_assertions`.
Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
|
|
`ingest_entries` requires that all directories referenced by entries in
the ingestion stream have an explicit entry in the stream.
For example, if the stream contains a file with path `foo/bar`, there
must be an entry that comes later in the stream for the directory `foo`.
Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Implement a first pass at the fetchTarball builtin.
This uses much of the same machinery as fetchUrl, but has the extra
complexity that tarballs have to be extracted and imported as store
paths (into the directory- and blob-services) before hashing. That's
reasonably involved due to the structure of those two services.
This is (unfortunately) not easy to test in an automated way, but I've
tested it manually for now and it seems to work:
tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath
=> "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string
Co-authored-by: Connor Brewster <cbrewster@hey.com>
Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Explicitly document and add a debug assertion for that.
It's up to callers to ensure this doesn't happen.
Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Move error types and filesystem-specific functions to a separate file,
and keep the fs:: namespace in public exports.
Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
