Age | Commit message (Collapse) | Author | Files | Lines |
|
Currently CAHash only deserializes the hash in hex code while
the serializer outputs a nixbase32 hash. This means that you can't currently
deserialize what has been serialized.
This change makes deserialize support any digest format (so hex, nixbase32
and base64) as well as flattens the deserialize code and error handling.
It also implements serde methods of HashAlgo directly using Display and TryFrom
implementations because otherwise these would get serialized as eg. Sha256 instead
of sha256 which also broke CAHash serialize/deserialize.
Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This doesn't return the digest, but the internal NixHash.
To get the digest, you use `ca_hash.hash().digest_as_bytes()`
(as done in tvix/nix-compat/src/derivation/write.rs).
Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
https: //b.tvl.fyi/issues/264
Change-Id: Icb09be9643245cc68d09f01d7723af2d44d6bd1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11001
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
https: //b.tvl.fyi/issues/264
Change-Id: I7a235734dc1f8e93e387a04ba369f3b702c6d5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10992
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
|
|
...in `Derivation`.
This is more type-safe and should consume less memory.
This also removes some allocations in the potentially hot path of output hash calculation.
https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
|
|
...so that we can also use `StorePath`s in
derivation.input_derivations.
Towards https://b.tvl.fyi/issues/264
Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Towards https://b.tvl.fyi/issues/264
Change-Id: I323907cb842ed10ddd05b72c285c598b60d213e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10972
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
|
|
...so that we can parse (and at the same time
validate) store paths directly to `StorePath`s.
https: //b.tvl.fyi/issues/264
Change-Id: Ide4cb5403f8e3400f7d2a5838c652ac35dbd1a4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10968
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
|
|
Instead of constructing a completely new derivation for hashing,
allow to call hashing with substituted input_derivations.
This
1. reduces the number of allocations,
2. prepares substituting `String`s for store paths with proper `StorePath`s without needing a separate derivation struct that allows hashes without `/nix/store` in input_derivations, and
3. keeps the change local to the `Derivation` implementation.
Change-Id: I36732c78f98fc59f0925b65823773222782017b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10935
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
|
|
This will be used inside glue, to populate the CA field.
Change-Id: I52350d66ad09c90c4e9f3bc6ea44fba4427f1576
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10946
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: Ie78b1c49d7386c70833a3b751799993ad9121aa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10933
Tested-by: BuildkiteCI
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Populate the input_derivations struct field in-place, using
self.input_derivations.from_iter, which seems a bit more readable.
Change-Id: I4aa12cfd66e08486ef27aedab1bb731d94b63657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10934
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
- Adjust to ecl 23.9.9 release
- Regenerate go protos after protoc-gen-go update
- Drop dhall fork which hasn't kept up with 1.42.*
- Address new clippy warnings:
- Variant naming of Error::ValidationError
- Simplify .try_into().unwrap()
- Drop unnecessary identity function
- Test module must be last in file
- Drop unused `pub use`
- Update agenix to 0.15.0. Current master has a installCheckPhase that
doesn't work with C++ Nix 2.3.*:
https://github.com/ryantm/agenix/commit/a23aa271bec82d3e962bafb994595c1c4a62b133#commitcomment-137185861
Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
|
|
Rather than passing strings around, use a StorePathRef.
This makes things a bit more typesafe, and more aligned with what we
want to do in b/264.
Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I76200aa474b86cf655308f6710a2298b77fa4b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10404
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I0ebe256c62435901c12ca1895f30e380280e9ea8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10403
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: If6962931350edfbae206fa739a673e7ccbc3ee89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10393
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
In most case, we don't actually need an owned `StorePath` struct, but a
`StorePathRef<'_>` is sufficient. The lifetime is only due to it holding
onto the name, but that one is mostly left untouched.
`Derivation::calculate_derivation_path` still needs to return
`StorePath`, as its name has a `.drv` appended.
Change-Id: Ie0d52f369d785711bb0658ea2b0bd2617fd9f45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10389
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
Change-Id: I5d7057347838f733d7dc1331c7b7ef8d27244412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10228
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
|
|
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.
Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Match the naming in parse_input_derivations, call the keys
"input_derivation", and the values "output_names".
Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
These were used to format to and parse from strings.
Move this to the CAHash and NixHash structs directly, and be explicit in
the name about which encoding for digests is used.
For output path calculation, nix encodes the nixpaths in hex, but for
writing out NARInfos, it's using nixbase32.
Change-Id: Ia585a76a3811b2609e7ce259fda66a29403b7e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10079
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Walking a btree_map twice is more expensive than copying a string,
especially because the cloning only happens in the (non-hot) error
path.
This fixes a clippy lint, so it's related to b/321.
Change-Id: I2ccfd0bc46792a45d277f47564e595b87107d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Relates to b/321.
Change-Id: I37284f89b186e469eb432e2bbedb37aa125a6ad4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9961
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
|
|
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
If the output was fixed, we broke out of the for loop too early, before
actually validating individual outputs.
Change-Id: I2259697dfa2a157764358f6d326a1f7f6610647c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9815
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This specific struct is only used to represent content-addressed paths
(in case a Derivation has a fixed-output hash, for example).
Rename `Output`'s `hash_with_mode` to `ca_hash`.
We now also include `CAHash::Text`, and update the `validate` function
of the `Output` struct to reject text hashes there.
This allows cleaning up the various output path calculation functions
inside nix-compat/src/store_path/utils.rs, as they can now match on
the type.
`make_type` is renamed to `make_references_string`,
`build_regular_ca_path` is renamed to `build_ca_path`, and
`build_text_path` has a disclaimer added, because you might not actually
want to use it.
Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: Ie464467edfafd2050aa9e5fd0e07ff745cb15df8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9753
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
The previous CLs did already absorb all the logic into the common tests,
no need to write this here again.
Change-Id: I7ba84ba86d5445ed247e5d11d5e59b7fa815670e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9732
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Due to the lack of a ATerm parser, we were previously loading the JSON
fixtures to construct our Derivation structs to run the output path
calculations with.
However, as we now have a ATerm parser, we can load the ATerm
representation directly.
This also means we can test the output path calculation for non-UTF8
Derivations.
Change-Id: I0e53f41a23566b5ad5f0fed12724e02a10b02707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9731
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This now properly checks for duplicate output names in input derivation
output names, and duplicate input sources.
Change-Id: I0053854bfbf504f4f511fb3fe1a77a82b3aa61dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9738
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This provides a nom-based parser for Nix derivations in ATerm format,
which can be reached via `Derivation::from_aterm_bytes`.
Some of the lower-level ATerm primitives are moved into a (new) aterm
module, and some more higher-level ones that construct derivation-
specific types.
Also, move the escape_bytes function into there, this is a generic ATerm
thing.
Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
typo: write_enviroment -> write_environment.
Change-Id: I1a2dc826c0bf71f6ba97fe41c4d44619b5e6beff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9739
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
We don't actually care if it's a BTreeMap of strings, bstrings or any of
that sort.
The only thing we want to be able to do is get a reference to the bytes
from the keys and values.
Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
We use test_resources and globbing for some of the test cases, so adding
additional files in there will also create new test cases, which we
don't always want.
Move it down one level to make some more space.
Change-Id: I619867dc80a4ced59d45096d0703678663b559cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This is of known length.
Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Less Vec<u8> passed around.
Change-Id: Ie153a6bfaa084d7490ffa38634efdf5f3c31a768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.
Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`, which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.
Fixes b/291.
Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I1c1608a6e75e451940fe1c61dc5ace5f0e7d7752
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9111
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Previously, Output deserialization would silence validation errors and
provide `None` for `hash_with_mode` as soon as a validation error would
happen inside of the `NixHashWithMode` deserialization, e.g. invalid
hash length would not provide an validation error but a silent `None`
value.
This is problematic, we workaround a serde limitation here by writing
our own Deserializer.
As you can see, we write some boilerplate code unfortunately, as, for
example:
- `#[serde(fail_if_unparsed_as="Option::is_none")]` is not a thing,
otherwise, we could have been able to just bubble up errors in case of
"not fully parsed" (and not missing) values.
- `From<&serde_json::Value> for serde::de::Unexpected` is not a thing,
otherwise, we could just map invalid type errors and reuse the
existing types instead of doing extremely bizarre things with
`serde::de::Unexpected::Other`, note: this is a not problem for
expected, we know what we expect, we don't know what we received in
practice.
I decided to write a `NixHashWithMode::from_map` which will eat a map
deserialized via `serde_json`, so our serde magic is not totally "data
model" agnostic.
I wanted to go for data model agnosticity and enable maximal
performance, e.g. building the structure as the map values are streamed
in the Visitor, this is needlessly painful because `Output` and
`NixHashWithMode` are in different files and this really makes sense
only if we write the full implementation in one file, indeed, otherwise,
you end up duplicating the work or having strange coupling.
So, for now, we will allocate a full map of the fields inside the
`Output`, i.e. if any "unknown field" is in that map, we will
deserialize it for no reason.
Doing it properly, as I repeat it in the code and to flokli at C3Camp
2023, requires to patch serde upstream IMHO.
Change-Id: I46fe6ccb8c390c48d6934fd3e3f02a0dfe59557b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9107
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This `.into_iter()` call is equivalent to `.iter()` and will not consume
the `BTreeMap`.
Change-Id: Ie26637ebecb0bea5b09c447cc45ed207f8b50913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9088
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
We only need bstr::ByteSlice to be able to use replace, it doesn't need
to return a BString.
Change-Id: I811948436fb89652e880970c2c05356183f3e439
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9084
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I764232378f79f16c781870d467469db91a2f14d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9001
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: I43cc1d5b3419255f815490b2e10f05db19e3bffc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9000
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Make it more obvious if these are bytes pointing to JSON or ATerm, so we
don't get confused.
Change-Id: I2402c687b7ba9c05aac20ed63b0df54e4e96a9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8998
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I85e44e24e9ec3fe4284f02623a1ac7ca9935e554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8999
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This is more concise than a io::copy of a Cursor to bytes, and we have
everything to be written in memory.
Change-Id: I81f34666aa61aef4e16b33423ce4a69c3781efc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8997
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
write_input_derivations shouldn't need to write a comma to separate it
from the previous output from write_outputs.
This is better placed in the function calling all of these helper
functions.
Change-Id: I9ccc440e4665b52369ef39e75151b9a29469ce48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8995
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|