| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Change-Id: I86480cf625a457c4aa8153262f829d34c230b084
Co-authored-by: edef <edef@edef.eu>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ia0370ca46cb1c6122a452b1d117160536b632c7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This is the only (remaining) occurence of it, and not really
more code than just calling store_path::build_ca_path with
`CAHash::Nar(NixHash::Sha256(…))`, especially considering we need the
CAHash in the PathInfo struct later anyways - so let's remove this
function.
Change-Id: Ia82212086062c366e0280ca0823d9e68a3f91d3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12632
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This became obsolete, since the introduction of a stricter `Directory`
struct invalid names cannot be represented anymore.
Change-Id: I9e4b1b6cca01831d0a9735f58d8a1f59ac18676b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12615
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This is not gonna work out as-is, as we still key PathInfos by
their store path digest, and how to handle thing if we encounter a
Frankenbuild.
For now, let's keep the PathInfoService data as it is, we can record
this information (and more) in the builder structures.
Change-Id: Ic38fc3ecd8096a5fe002e681bdc812a9dbeaa7d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
|
|
This moves the implementation from builtins.path into a helper function,
which we now call from both builtins.
Most of the Value plumbing stays inside this helper.
We also implemented handling of symlinks at the root, which was handled
in builtins.filterSource, but not builtins.path - by peeking at the
FileType using std::fs::metadata, instead of the EvalIO trait.
For now, this is fine, as our filtered_ingest also goes via the
filesystem directly. It ends up with the same semantics as before and in
Nix - symlinks at the root are followed, except if they point to an
invalid target.
In the future, we should revisit this, and then maybe get both stat and
lstat into EvalIO, though we will need to be very careful about the
semantics for following symlink inside store paths.
Change-Id: I6a941c0187db36165c2f7a338015e4e32d41b298
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12629
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
In a previous refactoring CL this into_bstring method was accidentally
kept, when we don't need it and can just to_str directly.
Change-Id: Idd531d508b8fd530611b213d0164e7aaf0e87d80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12631
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
Change-Id: If92ddaf3b469c4635c234b193f8d7716e11887f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12630
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
These nested ifs are a bit confusing, a match block makes this cleaner.
Change-Id: I256fd0bc921fbf2e60ad0f6e1ea51c2e0fb00317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12628
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This removes all the intermediate helper functions and reorganizes the
import code to only do the calculations where/when needed, and hopefully
makes things easier to understand as well.
Change-Id: I7e4c89c742bf8569b45e303523f7f801da7127ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12627
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Reviewed-by: edef <edef@edef.eu>
|
|
This makes it easier to understand what the specific test is testing.
Change-Id: I34b2798841c6b9367849668451af2165dc78f997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12626
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
|
|
Change-Id: Ib87f41b3e0d634dd76e648acfe1b3026211f947a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12619
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Tested-by: BuildkiteCI
|
|
This didn't support store paths with a subpath joined to them, while
Nix does.
Use state.path_exists, which does. This also means we can drop the
`store_path_exists` helper, which was only used here.
Change-Id: I918ccb270f64acbdc41cb4d2a9c3c5871ce15002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12618
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
|
|
These are not necessarily strings, and making it paths allows us to stop
converting them to lossy strings.
Change-Id: I11366c721dc5da1778aafe89092a1966b5a43178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12617
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Make this generic on the StorePath<SP> that's being used, similar to the
other functions in there.
Change-Id: I453d1fd3749053d4e5aca156abc18da1f95ca264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: I14bee4068dc73c948321481b5a4e1fc922a89a27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12611
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ie2af74beda9fcf8aa19fca7d844bcbe732f05bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12610
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
Change-Id: Ibb3be8458b8a8912ea04c9360d64c5cf914254d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12609
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
I found at least one dead variable along the way, which I deleted.
Change-Id: I8600c87c49078fb5ff72671994c77b919259e67b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12608
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This is not a core Tvix tool, it's a tool that uses a Tvix component.
Change-Id: I81d2b2374da23489df0097dcabb8295c82652fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12606
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This is not a core Tvix tool, it's a tool that uses a Tvix component.
Change-Id: I705f2c4ab87f1512e005007c933e16b84ed4279f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12605
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This was introduced in cl/9925 without any commit message, but this is clearly
not relevant to Tvix itself (it even says so in a comment in Cargo.toml).
Change-Id: I84f12d5145c3f53c9df23863f887bad913856c50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12604
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
|
|
This is not a core Tvix tool, it's some sort of one-off analysis thing.
Change-Id: I05fcbed45abad27d6b5cfd49db1727249dad3971
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12603
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
Equivalent logic is now in the standard library, and this dependency is no
longer needed for eval.
Change-Id: Iaa4410d89fdaa5b84cbd9e6bc6ae479c659d92f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12602
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
Refactor the `strict` boolean passed into evaluation at the top-level to
be a (two-variant, so far) EvalMode enum of Lazy and Strict.
This is more explicit than a boolean, and if we ever add more EvalModes
it's a simple extension of the enum.
Change-Id: I3de50e74ec971011664f6cd0999d08b792118410
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12186
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
|
|
Change-Id: Ib67e0121cebf6b2f049490125c7ded9637955928
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12599
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I31e5f3d46321265d662d41aa79aa4178e121818f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12598
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
Change-Id: I81988571344829708881021527b83e7d99b598c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12597
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
tonic-build deprecated their `compile()` function, it's now called
`compile_protos()`.
Change-Id: I8cacd7f01a251c207401e4e226b0e880744e96e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12596
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Move things around a bit to make it easier to understand what's going on:
- We first validate our fixture invariants
- We then insert into the PathInfoService
- Do all comparisons and checks we can on the returned PathInfo struct
- Only convert to the NarInfo variant to calculate the fingerprint,
and don't keep intermediate let bindings for this
Before cl/12588, this was arguably much harder to do that way, as we
relied on some of the conversions done in the to_narinfo() function.
Change-Id: Iaddbf1079f73ce566ef6d56f69a823e080b2e006
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12595
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sinavir <tvix@sinavir.fr>
|
|
The store path is already contained in the PathInfo, and the ca bits is
already passed into the function, so known to the caller - there's no
need to duplicate this.
We can also avoid having two separate block_on in our import builtin -
we already know the content hash before constructing, as we pass it in
via ca_hash.
There's still some room to unclutter some more of the code around
importing - we still do NAR calculation twice in some cases, and some of
the code might be share-able from other places producing PathInfo too.
Log a TODO for this cleanup.
Change-Id: I6a5fc427d15bc9293a396310143c7694dd2996c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12592
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We also use S in other places in the same file, but that's for the
string-like references.
SP is now consistently used as the type parameter for StorePath<_> (and
build_output_path) gets support for it).
By being a bit more careful in the order of assignments in nix-compat/
src/derivation, we can nudge the compiler to use the type we want.
Change-Id: Ia7c298e110dff98d3b113d2388674ce9e22b80e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12590
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
|
|
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.
It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.
PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.
The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.
Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Our oci-spec was a bit oudated and there were some renamings in one of
the release, which made building tvix-build fail if it's a dependency.
I encountered this issue while working on tvix-eval-jobs.
Change-Id: I6d982965176b83170a07445e351d3f5e5679ed2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12586
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
|
|
This allows specifying an url in place of a named reference to another
composition entry, if the castore crate has been compiled with the
xp-store-composition feature.
Example: `--directory-service-addr cache://?near=memory://&far=memory://`
This would be equivalent to the instantiation via toml file:
```toml
[memory1]
type = "memory"
[memory2]
type = "memory"
[default]
type = "cache"
near = "memory1"
far = "memory2"
```
Note that each anonymous url causes a distinct instance to be created.
Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I80121319795319bb977427efeca3666c6b87a1b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12147
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This still defaults to the "default" services, but allows users to tell the
nix+http pathinfoservice to ingest the castore nodes into a non-default
blob-/directoryservice when used with the experimental store composition.
Change-Id: I5c0f683ce95d888eadf3f302520a47f42f1a481d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12148
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
RUST_LOG allows for a more granular tracing configuration
Change-Id: Iea29fc2b154345eb7479dfd8bbea6abfd4716341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12585
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
We wanted to enable this earlier but the log level of many of the
messages in redb were too high, they've now been downgraded, so we can
enable logs from redb.
Context on the fix and release:
- https://github.com/cberner/redb/pull/828
- https://github.com/cberner/redb/releases/tag/v2.1.2
Change-Id: I8635e8a0bcb01a7d0b580387ac9134ccdd0205f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12568
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Over the past couple of months we've been using redb instead of sled as
the default filesystem-based database in PS and DS. I am confident that
we can get rid of sled completely now, and just keep redb.
Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
Change-Id: I165261170edaabfc56f6ac0a6baae388332b3f73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12559
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
|
|
Previously, the overlap calculation would underflow when
the pattern is empty.
Change-Id: I1f6bf49fafc4b8183a3a5e5e491a5a5bfc41ca97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12558
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
|
|
With https://cl.tvl.fyi/12533 in, we still need to lookup references to
properly populate `BuildRequest`.
It currently fails as the reference to
h9lc1dpi14z7is86ffhl3ld569138595-audit-tmpdir.sh is not propagated.
We should prevent Frankenbuilds from the go, so let's update our
PathInfo type to accomodate for that.
Change-Id: I26f9215312c258bba222efd390bc135f1a3a3d6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12560
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I07d016f831dcc596b4627f1d8f33909e632be416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12533
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is just patchset 10 of CL10855, before the color_eyre changes,
rebased to the tvix_castore api.
Change-Id: If4b42412ff8568058908cda971ad7d6f2d9f9b7b
---
This provides a build service invoking runc. It can be used by using the
`oci://$path_to_some_tempdir` builder URL for now.
For now, it can be tested as such:
```
BUILD_SERVICE_ADDR=oci://$PWD/bundles target/debug/tvix
let pkgs = (import <nixpkgs> {}); in builtins.readDir pkgs.perl
```
readDir is to actually trigger IO into the store path (which triggers
the builds).
For now it fails due to missing reference scanning (see followup CLs).
Change-Id: I09b40e410114ce69966a41a0e3c33281b859e443
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12526
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
After this, attempting to build the nixpkgs still fails in the same way,
because the references are not yet properly used by the code at
`tvix/glue/src/tvix_store_io.rs`.
Change-Id: I8a59ef8ef3c9a6f6aa7b05106dd9eef2e9ac0d0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12532
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This provides a generic interface to let the builder search for needles
in the output, as described in the notes at `docs/src/build/index.md`.
Change-Id: Ic2c5bd563e9aa2e766c157f2b13cdb19aede12f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12531
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
|
|
Change-Id: Ieca06de4c2e2680d89fe05a380079fafa5454837
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12529
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This changes the only actual state the ReferenceScanner has to use atomic bools, so it no longer requires a mutable borrow for .scan(). This allows passing an immutable borrow of a reference scanner to multiple threads which might be ingesting blobs in parallel, and using
them in the ReferenceReader or calling .scan() there.
Change-Id: Id5c30bcebb06bf15eae8c4451d70eb806cab722e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12528
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We don't need to access castore for HEAD requests.
Change-Id: I9365d9520d5a9e52ed92897d3c4972ec5b6e11fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12547
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
