Age | Commit message (Collapse) | Author | Files | Lines |
|
This makes it easier to understand what the specific test is testing.
Change-Id: I34b2798841c6b9367849668451af2165dc78f997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12626
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
|
|
Change-Id: Ib87f41b3e0d634dd76e648acfe1b3026211f947a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12619
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Tested-by: BuildkiteCI
|
|
This didn't support store paths with a subpath joined to them, while
Nix does.
Use state.path_exists, which does. This also means we can drop the
`store_path_exists` helper, which was only used here.
Change-Id: I918ccb270f64acbdc41cb4d2a9c3c5871ce15002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12618
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
|
|
These are not necessarily strings, and making it paths allows us to stop
converting them to lossy strings.
Change-Id: I11366c721dc5da1778aafe89092a1966b5a43178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12617
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Make this generic on the StorePath<SP> that's being used, similar to the
other functions in there.
Change-Id: I453d1fd3749053d4e5aca156abc18da1f95ca264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: I14bee4068dc73c948321481b5a4e1fc922a89a27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12611
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ie2af74beda9fcf8aa19fca7d844bcbe732f05bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12610
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
Change-Id: Ibb3be8458b8a8912ea04c9360d64c5cf914254d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12609
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
I found at least one dead variable along the way, which I deleted.
Change-Id: I8600c87c49078fb5ff72671994c77b919259e67b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12608
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This is not a core Tvix tool, it's a tool that uses a Tvix component.
Change-Id: I81d2b2374da23489df0097dcabb8295c82652fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12606
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This is not a core Tvix tool, it's a tool that uses a Tvix component.
Change-Id: I705f2c4ab87f1512e005007c933e16b84ed4279f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12605
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This was introduced in cl/9925 without any commit message, but this is clearly
not relevant to Tvix itself (it even says so in a comment in Cargo.toml).
Change-Id: I84f12d5145c3f53c9df23863f887bad913856c50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12604
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
|
|
This is not a core Tvix tool, it's some sort of one-off analysis thing.
Change-Id: I05fcbed45abad27d6b5cfd49db1727249dad3971
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12603
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
Equivalent logic is now in the standard library, and this dependency is no
longer needed for eval.
Change-Id: Iaa4410d89fdaa5b84cbd9e6bc6ae479c659d92f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12602
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
Refactor the `strict` boolean passed into evaluation at the top-level to
be a (two-variant, so far) EvalMode enum of Lazy and Strict.
This is more explicit than a boolean, and if we ever add more EvalModes
it's a simple extension of the enum.
Change-Id: I3de50e74ec971011664f6cd0999d08b792118410
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12186
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
|
|
Change-Id: Ib67e0121cebf6b2f049490125c7ded9637955928
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12599
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I31e5f3d46321265d662d41aa79aa4178e121818f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12598
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
Change-Id: I81988571344829708881021527b83e7d99b598c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12597
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
tonic-build deprecated their `compile()` function, it's now called
`compile_protos()`.
Change-Id: I8cacd7f01a251c207401e4e226b0e880744e96e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12596
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Move things around a bit to make it easier to understand what's going on:
- We first validate our fixture invariants
- We then insert into the PathInfoService
- Do all comparisons and checks we can on the returned PathInfo struct
- Only convert to the NarInfo variant to calculate the fingerprint,
and don't keep intermediate let bindings for this
Before cl/12588, this was arguably much harder to do that way, as we
relied on some of the conversions done in the to_narinfo() function.
Change-Id: Iaddbf1079f73ce566ef6d56f69a823e080b2e006
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12595
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sinavir <tvix@sinavir.fr>
|
|
The store path is already contained in the PathInfo, and the ca bits is
already passed into the function, so known to the caller - there's no
need to duplicate this.
We can also avoid having two separate block_on in our import builtin -
we already know the content hash before constructing, as we pass it in
via ca_hash.
There's still some room to unclutter some more of the code around
importing - we still do NAR calculation twice in some cases, and some of
the code might be share-able from other places producing PathInfo too.
Log a TODO for this cleanup.
Change-Id: I6a5fc427d15bc9293a396310143c7694dd2996c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12592
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We also use S in other places in the same file, but that's for the
string-like references.
SP is now consistently used as the type parameter for StorePath<_> (and
build_output_path) gets support for it).
By being a bit more careful in the order of assignments in nix-compat/
src/derivation, we can nudge the compiler to use the type we want.
Change-Id: Ia7c298e110dff98d3b113d2388674ce9e22b80e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12590
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
|
|
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.
It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.
PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.
The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.
Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Our oci-spec was a bit oudated and there were some renamings in one of
the release, which made building tvix-build fail if it's a dependency.
I encountered this issue while working on tvix-eval-jobs.
Change-Id: I6d982965176b83170a07445e351d3f5e5679ed2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12586
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
|
|
This allows specifying an url in place of a named reference to another
composition entry, if the castore crate has been compiled with the
xp-store-composition feature.
Example: `--directory-service-addr cache://?near=memory://&far=memory://`
This would be equivalent to the instantiation via toml file:
```toml
[memory1]
type = "memory"
[memory2]
type = "memory"
[default]
type = "cache"
near = "memory1"
far = "memory2"
```
Note that each anonymous url causes a distinct instance to be created.
Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I80121319795319bb977427efeca3666c6b87a1b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12147
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This still defaults to the "default" services, but allows users to tell the
nix+http pathinfoservice to ingest the castore nodes into a non-default
blob-/directoryservice when used with the experimental store composition.
Change-Id: I5c0f683ce95d888eadf3f302520a47f42f1a481d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12148
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
RUST_LOG allows for a more granular tracing configuration
Change-Id: Iea29fc2b154345eb7479dfd8bbea6abfd4716341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12585
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
We wanted to enable this earlier but the log level of many of the
messages in redb were too high, they've now been downgraded, so we can
enable logs from redb.
Context on the fix and release:
- https://github.com/cberner/redb/pull/828
- https://github.com/cberner/redb/releases/tag/v2.1.2
Change-Id: I8635e8a0bcb01a7d0b580387ac9134ccdd0205f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12568
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Over the past couple of months we've been using redb instead of sled as
the default filesystem-based database in PS and DS. I am confident that
we can get rid of sled completely now, and just keep redb.
Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
Change-Id: I165261170edaabfc56f6ac0a6baae388332b3f73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12559
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
|
|
Previously, the overlap calculation would underflow when
the pattern is empty.
Change-Id: I1f6bf49fafc4b8183a3a5e5e491a5a5bfc41ca97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12558
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
|
|
With https://cl.tvl.fyi/12533 in, we still need to lookup references to
properly populate `BuildRequest`.
It currently fails as the reference to
h9lc1dpi14z7is86ffhl3ld569138595-audit-tmpdir.sh is not propagated.
We should prevent Frankenbuilds from the go, so let's update our
PathInfo type to accomodate for that.
Change-Id: I26f9215312c258bba222efd390bc135f1a3a3d6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12560
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I07d016f831dcc596b4627f1d8f33909e632be416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12533
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is just patchset 10 of CL10855, before the color_eyre changes,
rebased to the tvix_castore api.
Change-Id: If4b42412ff8568058908cda971ad7d6f2d9f9b7b
---
This provides a build service invoking runc. It can be used by using the
`oci://$path_to_some_tempdir` builder URL for now.
For now, it can be tested as such:
```
BUILD_SERVICE_ADDR=oci://$PWD/bundles target/debug/tvix
let pkgs = (import <nixpkgs> {}); in builtins.readDir pkgs.perl
```
readDir is to actually trigger IO into the store path (which triggers
the builds).
For now it fails due to missing reference scanning (see followup CLs).
Change-Id: I09b40e410114ce69966a41a0e3c33281b859e443
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12526
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
After this, attempting to build the nixpkgs still fails in the same way,
because the references are not yet properly used by the code at
`tvix/glue/src/tvix_store_io.rs`.
Change-Id: I8a59ef8ef3c9a6f6aa7b05106dd9eef2e9ac0d0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12532
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This provides a generic interface to let the builder search for needles
in the output, as described in the notes at `docs/src/build/index.md`.
Change-Id: Ic2c5bd563e9aa2e766c157f2b13cdb19aede12f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12531
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
|
|
Change-Id: Ieca06de4c2e2680d89fe05a380079fafa5454837
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12529
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This changes the only actual state the ReferenceScanner has to use atomic bools, so it no longer requires a mutable borrow for .scan(). This allows passing an immutable borrow of a reference scanner to multiple threads which might be ingesting blobs in parallel, and using
them in the ReferenceReader or calling .scan() there.
Change-Id: Id5c30bcebb06bf15eae8c4451d70eb806cab722e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12528
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We don't need to access castore for HEAD requests.
Change-Id: I9365d9520d5a9e52ed92897d3c4972ec5b6e11fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12547
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
With an implementation of AsyncRead + AsyncSeek, axum-range can answer
range requests.
We only use it if a range has been requested, as it uses more memory
than the linear variant.
Change-Id: I0072b0a09b328f3e932f14567a2caa3a49abcbf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12509
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: yuka <yuka@yuka.dev>
|
|
Allows using a TvixStoreFs in overlay filesystems
Change-Id: I10e63c92776b1c783947d92070f360865ef2883c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12539
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This release includes support for overlay filesystems
Change-Id: I946cbf346df045209afaea2d720bb57fc2f2659f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12538
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
|
|
This is required to add the optional refscanner parameter to the
ingest functions.
Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This otherwise spams the logs quite a bit, for example when uploading
to nar-bridge with the new check from cl/12497.
Change-Id: Idc2bcc513caea6fae38ae04489e4e3ee7be64bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12510
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
- Add a new PathInfoService implementation that wraps transparently
around another except that it dynamically signs all the incoming
path-infos with the provided signer.
- Add a ServiceBuilder for this PathInfoService that provides a
SigningPathInfoService with a keyfile signer
Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12032
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sinavir <tvix@sinavir.fr>
Tested-by: BuildkiteCI
|
|
Co-authored-by: edef <edef@edef.eu>
Change-Id: I233206e8aae35504ca0519ac88178dfc5596bedb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12439
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
|
|
Change-Id: I9ff43b29be68b9840c58286da96fa52927691804
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
thiserror is much more easier to maintain than manually implementing Error and Display.
Change-Id: Ibf13e2d8a96fba69c8acb362b7515274a593dfd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12452
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
|
|
Nar-bridge tried to parse the name of the protobuf node encoded in the
URL into a PathComponent but this name was empty, leading to an error
when the user tried to retrieve the nar file.
This was an oversight from the conversion to stricter types (some of the
CLs in the serious containing cl/12217).
We need a version converting a protobuf without a name to our stricter
types, but an empty PathComponent cannot be constructed.
So we need a into_name_and_node() version that returns the name as
Bytes, not PathComponent.
Change-Id: I2996cdd2e0107133e502748947298f512f1cc521
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12504
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|