| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
A significant fraction of all created attribute sets are empty; hence
this is an easy optimisation to make.
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0884194d04c1ee95b2b239a253515f2152bc0856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1179
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
|
|
This reverts parts of the CLs splitting the backing implementation for
Bindings and moves back to only the BTreeMap-backed implementation.
Our evaluation has indicated that the Vector-backed implementation
does not match the performance of the plain array used upstream, and
in my view the complexity introduced by it is not worth the relatively
small (single-digit percentage) performance increase with a
pivot-point close to the number of attributes yielded by
stdenv.mkDerivation.
Going forward we will trial implementations of attribute sets backed
by HAMTs, and investigate other mechanisms of speeding up the language.
Some changes from the previous CLs are retained, for example the
removal of insert_or_assign and the passing of capacity.
Change-Id: I6eb4b075b453949583360755055c21a29d7ff642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1172
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
|
|
Since one of the two implementations essentially uses the same shape
as the upstream Bindings, we backport their merge sort implementation
to ensure that we're doing the same thing semantically.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0d865897991eec0c4dd84d9bd0415cd1ca437792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1162
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
|
|
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Ief2e59d461452ce599abc63f6ebcfa07a7062491
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1161
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
|
|
We accidentally returned the incremented iterator in the
post-increment, this fixes it.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I36c79eb56359bb12a78ad3489e7d7d2eb2053510
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1140
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
|
|
This function in never called, so let's just remove it
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I79125866254d90dd0842bc86830d2103ac313cb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1125
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
|
|
To aid in making the decision of where to (currently just statically)
use a vector or btree as the backing implementation, add an extra
constructor argument to Bindings::NewGC for a capacity, and use
a (currently hardcoded at 32, for no good reason other than it felt like
a reasonable number) pivot to switch between our possible backing
implementations. Then, update all the call sites where it feels
reasonable that we know the capacity statically to *pass* that capacity
to the constructor.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I1858c161301a1cd0e83aeeb9a58839378869e71d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1124
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
|
|
Add an alternative impl of the now-abstract Bindings base class that is
backed by a std::vector, somewhat similar but stylistically a little
superior to the array-backed implementation in upstream nix. The
underlying iterator type in BindingsIterator is now backed by a
std::variant that we std::visit an overload over in order to implement
the various bits of the iterator interface.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I7fbd1f4d5c449e2f9b82102a701b0bacd5e80672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1123
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
|
|
To pave the way for the thing we want to do eventually which is use a
linear-time array for bindings (aka attribute sets) that are statically
known to be small enough to get a performance benefit from doing so,
make the Bindings class abstract, and define a BTreeBindings class that
inherits from it and is (currently always) returned from the static
initializer. The idea is that we'll have an ArrayBindings class as well
later that we can dispatch to conditionally based on an optional
"capacity" parameter or something like that.
There was some difficulty here in getting the iterator to work - the
approach we settled on ended up making a concrete BindingsIterator class
which will wrap a std::variant of either a btree iterator or something
else later, but right now just wraps a btree iterator.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Ie02ca5a1c55e8ebf99ab1e957110bd9284278907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1121
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
|
|
Change-Id: I1cfbb7e933da54198115b28ac509b0d04870fd8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1127
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
|
|
Having a default constructor for this causes a variety of annoying
situations across the codebase in which this is initialised to an
unexpected value, leading to constant guarding against those
conditions.
It turns out there's actually no intrinsic reason that this default
constructor needs to exist. The biggest one was addressed in CL/1138
and this commit cleans up the remaining bits.
Change-Id: I4a847f50bc90e72f028598196592a7d8730a4e01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1139
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
|
|
This has been providing a warning and it's been bothering me.
Change-Id: I0548059950ec4250d7cf0938f9deae09eafe593c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1141
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
|
|
nix:AttrName was one of the few classes that relied on the default
constructor of nix::Symbol (which I am trying to remove in a separate
change).
The class essentially represents the name of an attribute in a set,
which is either just a string expression or a dynamically evaluated
expression (e.g. string interpolation).
Previously it would be constructed by only setting one of the fields
and defaulting the other, now it is an explicit std::variant.
Note that there are several code paths where not all eventualities are
handled and this code is bug-for-bug compatible with those, except
that unknown conditions (which should never work) are now throwing
instead of silently doing ... something.
The language tests pass with this change, and the depot derivations
that I tested with evaluated successfully.
Change-Id: Icf1ee60a5f8308f4ab18a82749e00cf37a938a8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1138
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
|
|
Implements the fairly common lambda overload class used for std::visit
over variants and other things that require groups of callables.
Change-Id: Ia7448b7e1bd349b4909974758e6e6303a80d86d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1137
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: edef <edef@edef.eu>
|
|
Backport of https://github.com/NixOS/nix/commit/b244e65cdbc2949af70bd539bf8f3bd2fa952d07
Fixes https://github.com/NixOS/nix/issues/3175
------
Original fault description by puck:
I suspect I found the issue: The Nix Command structs are stored on the
heap (using ref<T>, a wrapper around std::shared_ptr<T>), which means
that any pointers that the NixRepl struct contains are eligible to be
reaped by the GC. This includes, but is not limited to, the Env
pointer, which seems to cause most of the random segfaults, or random
other values in the environment, which seems to be what @arianvp
experienced too.
Change-Id: I376d7cfd432daaa6f1fbbf77788ff048082f34e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1001
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
|
|
malformed
Previously the nix-daemon would crash if a user fed it invalid store
paths for drv files. The crash was due to the changed assertion
triggering. Whenever that assertion would hit the nix-daemon process
along with all it's current childs (running builds from all users) would
be interrupted.
Before this patch:
$ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
don't know how to build these paths:
/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
error: unexpected end-of-file
< nix-daemon terminates >
With this patch:
$ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
don't know how to build these paths:
/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
error: path '/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv' is not a valid store path
< nix-daemon does *NOT* terminate >
Change-Id: I01c5048c8a43a8b9154bdeb781d05b7744869ec0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/981
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
|
|
This takes us about halfway through worker-protocol.hh
I have left out the documentation strings for some of these items
because I don't feel that I can currently write an unambigous
description of them. For now I am just attempting to match the types.
Change-Id: Iae64b1676152fe4ea069e2021b75ad76465cf368
Reviewed-on: https://cl.tvl.fyi/c/depot/+/960
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
|
|
This isn't actually used yet, but forces the protos to be included in
the build which is useful for iteration.
Change-Id: I2abcaf297f34ae741f00ad0c929b226d5603c9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/928
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
|
|
Adds dependencies on the gRPC & protobuf libraries, and implements Nix
code to generate the C++ sources from the included proto definitions.
This is theoretically supported via CMake, but practically doesn't
work and I don't care to debug why.
Doing it like this lets us instead add a CMake library target for our
proto definitions based on the sources generated by Nix.
Pros:
* no need to deal with the gRPC CMake mess
* it works!
Cons: * iteration requires nix-shell restart
Change-Id: Ie1fe9807fc96c49cb8f7161ba59d093456062b15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/927
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
|
|
Adds initial gRPC definitions for the Nix worker protocol, which is
currently defined messily across the following files:
src/libstore/worker-protocol.hh
src/libstore/remote-store.cc
src/nix-daemon/nix-daemon.cc
The protocol definition is basically a big enum with the signatures of
the calls being implicit in the various client/server implementation
functions.
The definitions in this file are slowly reversed from these implicit
signatures, and are likely to contain an error or two which will be
weeded out when this is taken into use.
Only a handful of the calls are included in this commit, it is
intended to get us up and running first.
Change-Id: Ibc9b2ab4b91a064c8935f09f7ac72bb8150fb476
Reviewed-on: https://cl.tvl.fyi/c/depot/+/926
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
|
|
These bits are no longer required with the hashmap-backed
implementation of attribute sets.
Change-Id: I8b936d8d438a00bad4ccf8e0b4dd719c559ce8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/912
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
We don't want traces compiled out since they're an actual language
feature that're used in userspace - also their absence is breaking the
tests
Change-Id: Icaefca8f52e94001785f724fdc0c10a7586b24e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/562
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: lukegbot <bot@lukegb.com>
|
|
Change-Id: I4f44e6050fc5121072f4cde7758defe2dcbd4e92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/552
Reviewed-by: tazjin <mail@tazj.in>
|
|
Change-Id: I5ab040a62e123c57fe712b252fbf84fe5a8bc026
Reviewed-on: https://cl.tvl.fyi/c/depot/+/547
Reviewed-by: tazjin <mail@tazj.in>
|
|
Also fixes the pkgconfig files to use the corresponding CMake variables.
Change-Id: I8095b8aff39ad91e592f3edc95555c9f1f1f153d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/545
Reviewed-by: tazjin <mail@tazj.in>
|
|
This ensures that we install both glog's .a and all the .so files we
generate into a single consistent output lib path (which is, err,
lib64, but whatever).
Change-Id: Ib6ac6eacf5f56e4b719cfb586db731efc122c31b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/544
Reviewed-by: tazjin <mail@tazj.in>
|
|
Change-Id: I87eb6e59782d720015d351d8829dc7b8688e01f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/543
Reviewed-by: tazjin <mail@tazj.in>
|
|
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path
format in the tests to prevent regressions.
(cherry picked from commit c65a6fa86aef7bdf51fb4fba7bd31d265619ba3f)
|
|
sequentially
This makes the paths consistent without relying on ordering.
Co-authored-by: edef <edef@edef.eu>
(cherry picked from commit 515c0a263e137a00e82f7d981284dbe54db23247)
|
|
|
|
|
|
Completes the switch from Meson to CMake for the core build system in
Nix.
Meson was added originally because someone else had already done the
work for integrating it in Nix and it was an upgrade from the previous
setup.
However over time it became clear that Meson is not quite mature
enough for projects like Nix that have occasionally peculiar
configuration constraints.
Some issues encountered with Meson (some of these are due to the Meson
setup in Nix):
* Difficulty with generating correct compile_commands.json for
external tools like clangd
* Difficulty linking to libc++ when using clang
* Ugly shell invocations for certain parts of the build system (I want
these to be gone!!!)
This CMake setup mimics the Meson configuration, but there are some
differences (some temporary):
* headers are now included separately for each library (see a previous
commit that changes includes appropriately)
* autoheaders-style configuration is currently hardcoded. Before
blindly copying this I want to evaluate how much of it actually exists
for portability concerns that I don't have (such as support for OS
X).
* Nix is built with libc++ by default.
* [libstore] SQL schema is now inlined via a generated header, not an
included string literal
Abseil is still built as part of this build, rather than an external
dependency, because it chokes on differently configured compiler
invocations.
Note that because of the move to libc++ an unwanted behaviour is
introduced: glog log messages no longer have a body. I have yet to
debug what is going on there.
|
|
|
|
Previously all includes were anchored in one global mess of header
files. This moves the includes into filesystem "namespaces" (if you
will) for each sub-package of Nix.
Note: This commit does not introduce the relevant build system changes.
|
|
This function was a custom (and inefficient in the case of
single-character delimiters) string splitter which was used all over
the codebase. Abseil provides an appropriate replacement function.
|
|
Uses the equivalent absl::StartsWith and absl::EndsWith functions
instead.
|
|
|
|
Replaces these functions with corresponding functions from Abseil,
namely absl::StripAsciiWhitespace and absl::SimpleAtoi.
In the course of doing this some minor things I encountered along the
way were also refactored.
This also changes the signatures of the various custom readFile
functions to use absl::string_view types.
|
|
|
|
See previous commit for more details on why.
|
|
It is considered bad form to use things from includes in headers, as
these directives propagate to everywhere else and can make it
confusing.
types.hh (which is includes almost literally everywhere) had some of
these directives, which this commit removes.
|
|
|
|
This project will be dropping OS X support until the core is simplified.
|
|
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.
Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure. It just cares
about the /nix/store/[hash]-[name] part. But, when the path is deleted,
we encounter a problem. Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a. This will fail,
because the path is too long. After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it. (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)
This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash. (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)
This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2). This function takes a relative path and a directory
file descriptor. We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes. This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.
Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times. As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.
I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.
Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
(cherry picked from commit c05e20daa1abb3446e378331697938b78af2b3d7)
|
|
... this fixes nixpkgs eval!
|
|
|
|
Without this alias, the garbage-collecting allocator won't be used and
allocated attribute set values won't be visible during GC.
|
|
|
|
These were things that took me a moment to realise.
|
|
Replaces the previous implementations which performed sorting with one
that instead walks through the map (which is already sorted) and
yields values from it.
This fixes a handful of language tests because the previous
implementation did not actually yield useful values on the new implementation.
|
