about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2020-08-09 r/1624 fix(tvix): Make nix value structs inherit from GCGriffin Smith1-5/+5
All of the miscellanious structs that make up a nix Value should inherit from gc, as they contain pointers to GC'ed things as members Fixes: #42 Change-Id: I057d8c9f7dafbee7de7644ff152c9cba1aa6bc03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1696 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 r/1623 refactor(tvix): Remove UDSRemoteStoreGriffin Smith2-76/+0
Now that we've fully implemented the RPC-based store client, we can get rid of the UDSRemoteStore, whose only use was connecting to the locally running nix daemon. The RemoteStore still needs to be around to connect to remote upstream nix stores over SSH. Change-Id: I0699819803cbfe966b9a46786f2c927d8e4bf1a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1693 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 r/1622 feat(tvix): Implement all remaining RPC callsGriffin Smith8-49/+158
Implement all remaining RPC calls on the RpcSstore client, remove a few stub methods we had added that weren't actually present in the old RemoteStore implementation, and add one more RPC call for getBuildLog that is present in the store API, but that we hadn't added as a stub *or* to the proto. Fixes: #29 Change-Id: Id827f51a393ece4bc7bbecaf38aee9eb4b329770 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1692 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI
2020-08-09 r/1621 refactor(tvix): Factor-out proto utilitiesGriffin Smith3-136/+162
Factor out the shared utilities for interacting with protobufs/grpc from libstore to a proto.hh header in libproto. Change-Id: I1cb8d94867d5d4b63a9994be0b53f8f612eb8e3a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1691 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 r/1620 feat(tvix): Pass function name to SuccessOrThrowGriffin Smith2-18/+32
To aid in debugging RPC call failures, pass the name of the function being called to the SuccessOrThrow util funcion in the RpcStore Change-Id: I523dacfab896b85a3dbe6050c07ee6bd5906fa44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1690 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 r/1619 chore(3p/nix): apply google-readability-castingKane York40-125/+161
Command run: jq <compile_commands.json -r 'map(.file)|.[]' | grep -v '/generated/' | parallel clang-tidy -p compile_commands.json -checks=-*,google-readability-casting --fix Manual fixes applied in src/nix-env/nix-env.cc, src/libstore/store-api.cc Change-Id: I406b4be9368c557ca59329bf6f7002704e955f8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/1557 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2020-08-08 r/1618 fix(tvix): Wrap remaining RPCs in HandleExceptionsGriffin Smith1-21/+29
Wrap the BuildPaths and AddTextToStore RPC handlers in HandleExceptions. These were missed in the original pass due to a merge. Change-Id: Ie5be45e6098fba7a2b6b1c1be81578cb742c2880 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1689 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 r/1617 feat(web/panettone): implement shorthand issue URLsedef2-6/+16
Fix #32 Change-Id: I6ccec959201673850b4b56a44734a2874aad5856 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1648 Reviewed-by: glittershark <grfn@gws.fyi> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-08 r/1616 chore(3p/nix): don't optional-wrap EvalState::file_access_trace_fn furtheredef2-4/+3
std::function has a natural null we can't eliminate anyway, so this was effectively std::optional<std::optional<non_nullable_function>>. Change-Id: If99f139146021edb25d133dad7f0f6e125ef53df Reviewed-on: https://cl.tvl.fyi/c/depot/+/1688 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-08 r/1615 feat(tvix): Implement AddToStoreNarGriffin Smith4-33/+129
Implement both the client and server sides of AddToStoreNar, using a templated generalization of the sources and sinks we were using for AddToStore on both ends. Change-Id: I73d0ed34118c711b125851dff99a7518ced4af35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1686 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 r/1614 docs(sparse): Add RFC design for depot worktree toolKane York1-0/+37
This is meant to be a high-velocity alternative to writing an entire FUSE filesystem to facilitate smaller checkouts of the depot. The script mostly works today, except that remote dependency detection is not set up. Change-Id: Idf9048df4f7ee39eaaedcdafcd9621f3d7479947 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1644 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Reviewed-by: tazjin <mail@tazj.in>
2020-08-06 r/1613 fix(3p/nix): Use SkipEmpty in all calls to absl::StrSplitKane York28-55/+80
The behavior to return a list containing a single empty string when provided an empty string is a behavior that absl inherited from legacy code. However, the behavior expected by legacy code in Nix is the behavior provided by the SkipEmpty option. Switch all calls to use SkipEmpty, except for the call already using SkipWhitespace. See also commit 26a59482d2427f640893517f1b24dd650a5bd5da, with the partly-prophetic message: "there may be other places we need to fix this as well." Change-Id: I6e94856a12cfb1b7e4a3b4e221769ed446648861 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1687 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 r/1612 refactor(tvix): Use absl::btree_map for BasicDerivation's envVincent Ambo2-3/+4
Change-Id: I111a9a268debea322f23fdced3bed9ff3e8ed3b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1685 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1611 refactor(tvix): Use absl::btree_map for DerivationInputsVincent Ambo2-4/+2
Change-Id: If160ab1b09161969d9080d5d0d6f82e44a953e3c Reviewed-on: https://cl.tvl.fyi/c/depot/+/1684 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1610 refactor(tvix): Use absl::btree_map for DerivationOutputsVincent Ambo4-7/+22
This container implementation is much faster than std::map. We have stuck to an ordered container because it's unclear whether the accesses of this field (of which there are *many*) are actually ordering dependent. Also includes an Arbitrary implementation for absl::btree_map (for any K, V that are also Arbitrary). Change-Id: I04f58ca0ce32b9ae1759313b01508b0e44bae793 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1683 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1609 chore(tvix): Disable sandboxing by default until its time comesVincent Ambo1-15/+8
Change-Id: Ifcc776841924abcb43fb115f3de569fe500756c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1682 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1608 feat(tazjin/frog): Install sourcetrailVincent Ambo1-0/+1
Change-Id: I4858a1e374162ea1cf35aa906e0d723fe8c2c392 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1681 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-06 r/1607 docs: Add the RULEGriffin Smith1-0/+1
Change-Id: I90eddabb7e5e917524b300d5dfe86ddab4271432 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1660 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-06 r/1606 fix(whitby): Increase nrBuildUsers to 128Vincent Ambo1-0/+1
Change-Id: I3a444e163745d17d10f923c0be7565840937c53a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1662 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 r/1605 fix(whitby): I'm a trusted user, owoVincent Ambo1-0/+1
Change-Id: I2666b3cf8bdefcb5d4caeddf191dc65f6a8cb05f Reviewed-on: https://cl.tvl.fyi/c/depot/+/1661 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 r/1604 chore(3p): Bump nixpkgs channelsVincent Ambo10-21/+91
Bumps both nixos-unstable and nixos-20.03 to today's versions, as per status.nixos.org Contains minor fixes to things that broke because of the update: * tazjin/frog: hardware.u2f is a deprecated setting * glittershark/system: modSha256 in Go modules is now vendorSha256 * glittershark/owothia: removed version constraint on relude Change-Id: Ib3e9612b1b06ed547b90e4f8b0ffe5ed7fe0a5c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1642 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1603 feat(3p/nix): add --trace-file-access to nix-instantiateKane York4-0/+49
This builds on edef's work with depot-scan by adding a dedicated flag to the command. We piggyback on upstream's restricted-mode implementation, the checkSourcePath function. Change-Id: I52bb613549f40dbca1e8caa036635910c1a3d6d0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1654 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1602 chore(whitby): add edefedef1-0/+6
Change-Id: I7265259bc87594bd481c7bd455187c09b1effd1c Reviewed-on: https://cl.tvl.fyi/c/depot/+/1650 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-05 r/1601 fix(tools/depot-build): handle multiple levels of subdirectoriesedef1-1/+1
Change-Id: I99511c17d635b0222e1c933432007e6f8ea8cf85 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1655 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-05 r/1600 feat(3p/nix): Implement FindRoots, CollectGarbageGriffin Smith5-3/+117
Implement the RPC client and server handlers for the FindRoots and CollectGarbage RPC calls Change-Id: Ifa5d582c6a33bd1e7661ac2fc860505ef404dad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1656 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-05 r/1599 fix(3p/nix): fix usage error of absl::Base64UnescapeKane York1-3/+4
The source and destination strings cannot be the same string - absl will write to the destination in a streaming manner, causing the source to become invalid. Change-Id: I3578cf1f8789a51d85e0950f7987c398f0a00953 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1659 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1598 fix(3p/nix/store): fix race condition in downloaderKane York1-1/+2
In certain circumstances, the decompression thread could race ahead of the downloader thread and process the same chunk twice. Clear the data buffer while the lock is held to prevent this kind of incident. Change-Id: I19a84a0c5768d1228c6c18a7664a7b8893ef96de Reviewed-on: https://cl.tvl.fyi/c/depot/+/1658 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1597 refactor(tvix): Use SANDBOX_SHELL from environment in CMakeVincent Ambo2-3/+4
This makes the configuration work out-of-the-box using CMake in a Nix shell, rather than having to pass the additional variable to CMake on the command line. Change-Id: I04d3cc4f2e5ecf47bf2ee459d5e48588b84ae4dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/1643 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1596 feat(3p/nix): add tests for BinaryCacheStoreKane York2-0/+106
These tests are in preparation for factoring the Store away from libutil's Callback to absl::StatusOr. They use the newly added MockBinaryCacheStore. Updates: #25 Change-Id: I30c207589ec38254806ebc9a983f35668e353ae9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1595 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1595 feat(3p/nix): add MockBinaryCacheStoreKane York3-1/+170
This adds an implementation of BinaryCacheStore to be used by tests exercising both the logic inherent to BinaryCacheStore and more general Store tests. A new library target, nixstoremock, is created to indicate that this file is intended only for use in tests and not in user-facing code. Change-Id: Ib68f777238843a4f3a2303db8a69735fbc22d161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1645 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1594 feat(3p/nix): Implement AddToStore rpc clientGriffin Smith4-2/+69
Implement the AddToStore RPC client method, which uses an AddToStorePathWriterSink (the dual of the AddToStorePathReaderSource on the server side) to hook into the dumpPath machinery (which we should refactor not to use sinks or sources, but not yet) and write dumped paths as binary data over gRPC. With this commit and sandboxing disabled, the following derivation builds cleanly: derivation { name = "test"; builder = ./build.sh; system = "x86_64-linux"; } where build.sh has chmod +x and contains: #!/bin/sh echo 1 > $out Change-Id: I94cab86f0825a3a9993262a9807130645c13bf44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1638 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI
2020-08-05 r/1593 fix(3p/nix): fix clang-tidy for int-string appendsKane York1-3/+6
Change-Id: I276de7a5fd1c705c87d35dd616e5980c747190aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/1597 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1592 fix(edef/keys): add readTree headeredef1-0/+2
Change-Id: I5e1e0f58377201b591730a49f44678a33243b1d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1653 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1591 chore(edef/users/depot-scan): move to a subdiredef3-3/+1
Change-Id: If202b466aa0874048e89e0dc270ff14883510131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1652 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1590 fix(users/edef/depot-scan): properly trace readFile/readDiredef3-2/+19
Change-Id: I493e7d20504d34f83a53db50b69e86668874756e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1651 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1589 fix(users/edef/depot-scan): properly depend on pl scriptKane York2-2/+2
Change-Id: I83c75ab75dae9c904e127ad6160c7641d4523e40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1647 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1588 feat(edef): add SSH CAedef1-0/+5
Change-Id: If6839735d2730fa49363fa46e6fa059dd19bfaf7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1649 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1587 feat(edef/depot-scan): initedef2-0/+23
A small script that parses `nix-instantiate -vv` output to track the files read while instantiating a depot build. Change-Id: I5acf31d55f39c1d1acf9cdead03d33e2c8abab6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1646 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-04 r/1586 feat(nix/readTree): allow arguments to depend on tree locationedef1-3/+3
This accepts a function for args, and passes it an attrset containing the locatedAt parameter. Change-Id: I3c0f8ca00605e02a787fda88f32b06f5ef9998f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1639 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1585 chore(nix/fetchGoModule): don't move downloaded fileedef1-3/+1
unzip(1) doesn't care about the extension, unlike the unpackFile hook. Change-Id: Ia9d0e17c4b343ee5d32350329eb6267b9dc35605 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1641 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1584 chore(nix/fetchGoModule): remove overrideAttrs hackedef1-5/+3
This was based off fetchzip, but unlike fetchzip we don't need to support arbitrary archive formats, so we don't actually need the unpack hooks. Change-Id: I5655c031f420422f13e72bb79f4b4b95f6f1204e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1640 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1583 feat(tvix): Implement AddIndirectRoot, SyncWithGCGriffin Smith2-2/+32
Implement RPC handlers and client methods for the AddWithRoot and SyncWithGC calls. With this and sandboxing disabled, the following derivation is now building successfully: derivation { name = "test"; builder = "/bin/sh"; args = [ "-c" "echo 1 > $out" ]; system = "x86_64-linux"; } Co-authored-by: Vincent Ambo <mail@tazj.in> Change-Id: I245dca0dfca882e2eae5d93364287a8b79fead71 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1637 Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI
2020-08-04 r/1582 chore(tvix): Make not implemented errors nix::UnsupportedGriffin Smith1-22/+24
Without this, these errors print as "uncaught exception of type std::string", but with this they print using nix's built-in error printer, which will be nice to have as we bang out the remaining store API methods. Co-authored-by: Vincent Ambo <mail@tazj.in> Change-Id: I9bd4f77f65b27c8f97b727b98ef62e2061999e71 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1636 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-04 r/1581 fix(tvix): Partially revert "eliminate exposed global variable"Griffin Smith3-21/+39
This reverts part of commit 976a36c2e482f416acd79a624e6d96cce2564b5b, which was causing scanForReferences to hang indefinitely. I'm not sure what the original intent of the commit was since there's not really much of a commit message on it, but with this I am able to fully realise derivations. Change-Id: I620e892e2fe2244f066512286b086b3dd8f8f6c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1635 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-04 r/1580 chore: simplify buildGo expressionsedef13-15/+15
This removes redundant `map (p: p.gopkg)` expressions. Change-Id: If25d603c27d63ae011f396caf5d26cdfdb8c28d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1625 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-04 r/1579 feat(nix/buildGo): use gopkg attribute of depsedef2-3/+3
This removes the need for `map (x: x.gopkg)` in dependency lists. Change-Id: Ia19e51f14110bf22f5cdbb64fa7ce182e1b34241 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1624 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1578 feat(nix/buildGo): expose gopkg attribute on buildGo.packageedef1-7/+13
This makes the derivations more uniform, since both third-party packages generated by buildGo.external and native buildGo.package expose their libraries as gopkg attributes now. Change-Id: I547f9860082f36e3300139bf67613eb4fc600d24 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1623 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1577 fix(3p/gopkgs/cloud.google.com/go): add missing `depot` argedef1-1/+1
Change-Id: If08a43e18a41abd3e2ae7c08ceb6f48844d20df1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1626 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1576 fix(3p/nix): Don't assert on argc count for build-remoteGriffin Smith1-4/+0
This executable doesn't appear to ever actually look at argv, so throwing an error if argc is wrong seems kinda silly. Change-Id: Iabc9dad785d4c99be9e8fd4ecd954d38b876cecd Reviewed-on: https://cl.tvl.fyi/c/depot/+/1633 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1575 fix(3p/nix): Properly configure SANDBOX_SHELLGriffin Smith4-2/+23
point the SANDBOX_SHELL macro at the actual path to busybox on the build machine, or allow it to be configured at build-time with a cmake option. Change-Id: I044a1315ba9baa3bc9ceddf29f36d14f9f9ccd96 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1632 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
generated by cgit-pink 1.4.1 (git 2.44.0) at 2024-05-29T15·35+0000