Age | Commit message (Collapse) | Author | Files | Lines |
|
This allows StorePathRef as keys in a hashmap, by deriving Hash. The same is already done for StorePath.
Change-Id: I3fc54c45787948116dcb27dfb5dc806b9b505835
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11269
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Allow constructing a StorePath with a fixed-size digest.
Change-Id: Id7d0b0152f6c55660a8973a02c84afa9188ce3ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11144
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
|
|
This is only used inside this function, in 2 of the match cases.
Change-Id: Ib361f5ee0e3b203802f7d05b9a7f332d14bbcf80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11143
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
|
|
The outer fingerprint used for store path calculation is always a sha256
digest. This includes both input and output-addressed store paths.
We used a NixHash here, which can also represent other hash types, and
that had a bunch of annoyances:
- Whenever we had the bytes, we had to wrap them in a NixHash::Sha256().
- Things like AtermWriteable had to be implemented on NixHash,
even though we then had an assertion it was only called in the
NixHash::Sha256 case.
Change-Id: Ic895503d9b071800d2e52ae057666f44bd0ab9d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11142
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Implement the fetchurl builtin, and lay the groundwork for implementing
the fetchTarball builtin (which works very similarly, and is implemented
using almost the same code in C++ nix).
An overview of how this works:
1. First, we check if the store path that *would* result from the
download already exists in the store - if it does, we just return
that
2. If we need to download the URL, TvixStoreIO has an `http_client:
reqwest::Client` field now which we use to make the request
3. As we're downloading the blob, we hash the data incrementally into a
SHA256 hasher
4. We compare the hash against the expected hash (if any) and bail out
if it doesn't match
5. Finally, we put the blob in the store and return the store path
Since the logic is very similar, this commit also implements a *chunk*
of `fetchTarball` (though the actual implementation will likely include
a refactor to some of the code reuse here).
The main thing that's missing here is caching of downloaded blobs when
fetchurl is called without a hash - I've opened b/381 to track the TODO
there.
Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to
load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests'
rustls-native-roots feature, 2024-03-03) for more info.
Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
|
|
Move the the `fixed:out:[r:]{}:` generation to a helper function,
use matches! for more clarity.
Change-Id: I4e930c42aacbf5c7451d1f8c8c80ccb4c45389f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11006
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
|
|
...in `Derivation`.
This is more type-safe and should consume less memory.
This also removes some allocations in the potentially hot path of output hash calculation.
https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
|
|
Necessary, if we want to use it inside of `Derivation` etc.
Change-Id: I8888060417b2ee83ac52d7ec3e7b27c393271d8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10947
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Ia74ee870f38b7966501458bace541092256c3213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10509
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
self_reference being set to true is only allowed for
`CAHash::Nar(NixHash::Sha256(_))`, so we can handle this in a check at
the front.
Change-Id: Ic363ade4789a7767cbe26a6959b143bb53e50e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10391
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
All match cases essentially construct `ty` and `hash`, which is then
passed to the `build_store_path_from_fingerprint_parts` function.
Change-Id: I01dfd219f9b0ac1afe8af7c6e361ea048117a0e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10390
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
In most case, we don't actually need an owned `StorePath` struct, but a
`StorePathRef<'_>` is sufficient. The lifetime is only due to it holding
onto the name, but that one is mostly left untouched.
`Derivation::calculate_derivation_path` still needs to return
`StorePath`, as its name has a `.drv` appended.
Change-Id: Ie0d52f369d785711bb0658ea2b0bd2617fd9f45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10389
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
Keep the method around in StorePath for convenience, but move the
implementation to StorePathRef.
Change-Id: Ie1844fa01ce6529dc1a58907563c95c3112c831d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10387
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
This allows comparing StorePathRef structs.
Change-Id: Ia69967ea9358052e2d6e76042a7e6d394f7f29a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10386
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
|
|
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.
Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This fixes a *future* clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#/incorrect_partial_ord_impl_on_ord_type
In essence, because the implementation of *both* Ord and PartialOrd
implies that ordering is not partial, all results of PartialOrd should
simply be those of Ord. This is to avoid subtle bugs in future
refactorings.
Change-Id: I8fc6694010208752dd47746a2aaaeca0c788d574
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10109
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
We can simply use .to_owned() on that thing afterwards if we want to
construct an owned StorePath.
Change-Id: I0f3e2e4434b99ee522f2a7dbfa391e13a987479c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10105
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
These were used to format to and parse from strings.
Move this to the CAHash and NixHash structs directly, and be explicit in
the name about which encoding for digests is used.
For output path calculation, nix encodes the nixpaths in hex, but for
writing out NARInfos, it's using nixbase32.
Change-Id: Ia585a76a3811b2609e7ce259fda66a29403b7e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10079
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We don't need to validate UTF-8 separately, since valid names are
a strict subset of ASCII, and therefore a strict subset of UTF-8.
Change-Id: I3261bf0efe3480b5b315074efafcf5e47a6c5a65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10087
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This suggests it's cheap to convert around, but name actually does
allocate.
Move to a `to_owned(&self) -> StorePath`, to better signal that this
does allocate.
Change-Id: Ifaf7c21599e2a467d06e2b4ae1364228370275db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10066
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: Id093a0131aa7e3ac532daffbf5a883ca213c83ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9996
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Rather than having our own error type, just make decoding errors use
the same common error type.
Change-Id: Ie2c86972f3745c695253adc3214444ac0ab8db6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I64523df2344233d9e424812d94b1c7ed2ecb1a74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9956
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Flagged by ``#[warn(clippy::match_like_matches_macro)]`.`
Change-Id: If07109e5ec01b05df898119f9a577196dfe11b37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9955
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This already has the right type.
Change-Id: I8f5850a41f9e97f1bc5f2a45ca05cf7439665c9d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9954
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: Ib688bbb37cd54cfcd01e5cb3a8c376414ee8311e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9926
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This specific struct is only used to represent content-addressed paths
(in case a Derivation has a fixed-output hash, for example).
Rename `Output`'s `hash_with_mode` to `ca_hash`.
We now also include `CAHash::Text`, and update the `validate` function
of the `Output` struct to reject text hashes there.
This allows cleaning up the various output path calculation functions
inside nix-compat/src/store_path/utils.rs, as they can now match on
the type.
`make_type` is renamed to `make_references_string`,
`build_regular_ca_path` is renamed to `build_ca_path`, and
`build_text_path` has a disclaimer added, because you might not actually
want to use it.
Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
HashWithMode already provides a to_nix_hash_string() method, giving us
exactly the string we need here.
Change-Id: Id2635bf3ea6c2514faf3c26b297866d774f4ff4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9799
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Less Vec<u8> passed around.
Change-Id: Ie153a6bfaa084d7490ffa38634efdf5f3c31a768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This uses the newly introduced StorePath message type to add a Deriver
field to the PathInfo message.
Support for validation is added to both the golang and rust
implementation. This includes extending unit tests.
Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Found by Clippy, which we should probably run in CI.
Change-Id: Id79c30b63f681021ab79358e02d29454d43c0aa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9614
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Nix has historically rejected these. The current behaviour was
accidentally introduced in Nix 2.4, and is considered a bug.
Link: https://github.com/NixOS/nix/pull/9095
Change-Id: I38ffa911f0a413086479bd972de09671dbe85121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
|
|
Suggested in https://cl.tvl.fyi/c/depot/+/9108/5, but this disallows
adding a .gitignore file to the store.
Remove the check, and add a testcase.
Change-Id: Ieb78c417934756b2dbeb493040fe79726d1b9079
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9447
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
|
|
Similar to cl/9350. Most of the times, this is still a regular string,
so let's print it like that if we can, and resort to base64 encoding if
we can't.
We were also wrongly always outputting the second character in the
string.
Change-Id: Id0e2a9d9f1ad3d2d7b554893ecd89a7e6383e9c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9445
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.
This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.
There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.
Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.
Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
|
|
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.
Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`, which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.
Fixes b/291.
Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Yes:
```
$ nix-build -E 'derivation { name = ""; builder = "/bin/sh"; system = "x86_64-linux"; }'
error: store path 'nr7i5pf18hw2zg487vkdyrbasdqylfcj-' has an empty name
```
Change-Id: I552f9ed1c1fe3bfceca18ca9b8e13d4b06dc6ff7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9108
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ie95ac265f8707e138c6a7b529760650d211f259d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9087
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Don't import thiserror::Error directly, so rustdoc knows what
`crate::store_path::Error` we're talking about.
Change-Id: I755c9377521a6833e9a77cb1a41b48edafb31fe0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|