about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2020-08-08 r/1616 chore(3p/nix): don't optional-wrap EvalState::file_access_trace_fn furtheredef2-4/+3
std::function has a natural null we can't eliminate anyway, so this was effectively std::optional<std::optional<non_nullable_function>>. Change-Id: If99f139146021edb25d133dad7f0f6e125ef53df Reviewed-on: https://cl.tvl.fyi/c/depot/+/1688 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-08 r/1615 feat(tvix): Implement AddToStoreNarGriffin Smith4-33/+129
Implement both the client and server sides of AddToStoreNar, using a templated generalization of the sources and sinks we were using for AddToStore on both ends. Change-Id: I73d0ed34118c711b125851dff99a7518ced4af35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1686 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 r/1614 docs(sparse): Add RFC design for depot worktree toolKane York1-0/+37
This is meant to be a high-velocity alternative to writing an entire FUSE filesystem to facilitate smaller checkouts of the depot. The script mostly works today, except that remote dependency detection is not set up. Change-Id: Idf9048df4f7ee39eaaedcdafcd9621f3d7479947 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1644 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Reviewed-by: tazjin <mail@tazj.in>
2020-08-06 r/1613 fix(3p/nix): Use SkipEmpty in all calls to absl::StrSplitKane York28-55/+80
The behavior to return a list containing a single empty string when provided an empty string is a behavior that absl inherited from legacy code. However, the behavior expected by legacy code in Nix is the behavior provided by the SkipEmpty option. Switch all calls to use SkipEmpty, except for the call already using SkipWhitespace. See also commit 26a59482d2427f640893517f1b24dd650a5bd5da, with the partly-prophetic message: "there may be other places we need to fix this as well." Change-Id: I6e94856a12cfb1b7e4a3b4e221769ed446648861 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1687 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 r/1612 refactor(tvix): Use absl::btree_map for BasicDerivation's envVincent Ambo2-3/+4
Change-Id: I111a9a268debea322f23fdced3bed9ff3e8ed3b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1685 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1611 refactor(tvix): Use absl::btree_map for DerivationInputsVincent Ambo2-4/+2
Change-Id: If160ab1b09161969d9080d5d0d6f82e44a953e3c Reviewed-on: https://cl.tvl.fyi/c/depot/+/1684 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1610 refactor(tvix): Use absl::btree_map for DerivationOutputsVincent Ambo4-7/+22
This container implementation is much faster than std::map. We have stuck to an ordered container because it's unclear whether the accesses of this field (of which there are *many*) are actually ordering dependent. Also includes an Arbitrary implementation for absl::btree_map (for any K, V that are also Arbitrary). Change-Id: I04f58ca0ce32b9ae1759313b01508b0e44bae793 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1683 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1609 chore(tvix): Disable sandboxing by default until its time comesVincent Ambo1-15/+8
Change-Id: Ifcc776841924abcb43fb115f3de569fe500756c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1682 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-06 r/1608 feat(tazjin/frog): Install sourcetrailVincent Ambo1-0/+1
Change-Id: I4858a1e374162ea1cf35aa906e0d723fe8c2c392 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1681 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-06 r/1607 docs: Add the RULEGriffin Smith1-0/+1
Change-Id: I90eddabb7e5e917524b300d5dfe86ddab4271432 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1660 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-06 r/1606 fix(whitby): Increase nrBuildUsers to 128Vincent Ambo1-0/+1
Change-Id: I3a444e163745d17d10f923c0be7565840937c53a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1662 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 r/1605 fix(whitby): I'm a trusted user, owoVincent Ambo1-0/+1
Change-Id: I2666b3cf8bdefcb5d4caeddf191dc65f6a8cb05f Reviewed-on: https://cl.tvl.fyi/c/depot/+/1661 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 r/1604 chore(3p): Bump nixpkgs channelsVincent Ambo10-21/+91
Bumps both nixos-unstable and nixos-20.03 to today's versions, as per status.nixos.org Contains minor fixes to things that broke because of the update: * tazjin/frog: hardware.u2f is a deprecated setting * glittershark/system: modSha256 in Go modules is now vendorSha256 * glittershark/owothia: removed version constraint on relude Change-Id: Ib3e9612b1b06ed547b90e4f8b0ffe5ed7fe0a5c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1642 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1603 feat(3p/nix): add --trace-file-access to nix-instantiateKane York4-0/+49
This builds on edef's work with depot-scan by adding a dedicated flag to the command. We piggyback on upstream's restricted-mode implementation, the checkSourcePath function. Change-Id: I52bb613549f40dbca1e8caa036635910c1a3d6d0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1654 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1602 chore(whitby): add edefedef1-0/+6
Change-Id: I7265259bc87594bd481c7bd455187c09b1effd1c Reviewed-on: https://cl.tvl.fyi/c/depot/+/1650 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-05 r/1601 fix(tools/depot-build): handle multiple levels of subdirectoriesedef1-1/+1
Change-Id: I99511c17d635b0222e1c933432007e6f8ea8cf85 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1655 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-05 r/1600 feat(3p/nix): Implement FindRoots, CollectGarbageGriffin Smith5-3/+117
Implement the RPC client and server handlers for the FindRoots and CollectGarbage RPC calls Change-Id: Ifa5d582c6a33bd1e7661ac2fc860505ef404dad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1656 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-05 r/1599 fix(3p/nix): fix usage error of absl::Base64UnescapeKane York1-3/+4
The source and destination strings cannot be the same string - absl will write to the destination in a streaming manner, causing the source to become invalid. Change-Id: I3578cf1f8789a51d85e0950f7987c398f0a00953 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1659 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1598 fix(3p/nix/store): fix race condition in downloaderKane York1-1/+2
In certain circumstances, the decompression thread could race ahead of the downloader thread and process the same chunk twice. Clear the data buffer while the lock is held to prevent this kind of incident. Change-Id: I19a84a0c5768d1228c6c18a7664a7b8893ef96de Reviewed-on: https://cl.tvl.fyi/c/depot/+/1658 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1597 refactor(tvix): Use SANDBOX_SHELL from environment in CMakeVincent Ambo2-3/+4
This makes the configuration work out-of-the-box using CMake in a Nix shell, rather than having to pass the additional variable to CMake on the command line. Change-Id: I04d3cc4f2e5ecf47bf2ee459d5e48588b84ae4dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/1643 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1596 feat(3p/nix): add tests for BinaryCacheStoreKane York2-0/+106
These tests are in preparation for factoring the Store away from libutil's Callback to absl::StatusOr. They use the newly added MockBinaryCacheStore. Updates: #25 Change-Id: I30c207589ec38254806ebc9a983f35668e353ae9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1595 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1595 feat(3p/nix): add MockBinaryCacheStoreKane York3-1/+170
This adds an implementation of BinaryCacheStore to be used by tests exercising both the logic inherent to BinaryCacheStore and more general Store tests. A new library target, nixstoremock, is created to indicate that this file is intended only for use in tests and not in user-facing code. Change-Id: Ib68f777238843a4f3a2303db8a69735fbc22d161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1645 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1594 feat(3p/nix): Implement AddToStore rpc clientGriffin Smith4-2/+69
Implement the AddToStore RPC client method, which uses an AddToStorePathWriterSink (the dual of the AddToStorePathReaderSource on the server side) to hook into the dumpPath machinery (which we should refactor not to use sinks or sources, but not yet) and write dumped paths as binary data over gRPC. With this commit and sandboxing disabled, the following derivation builds cleanly: derivation { name = "test"; builder = ./build.sh; system = "x86_64-linux"; } where build.sh has chmod +x and contains: #!/bin/sh echo 1 > $out Change-Id: I94cab86f0825a3a9993262a9807130645c13bf44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1638 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI
2020-08-05 r/1593 fix(3p/nix): fix clang-tidy for int-string appendsKane York1-3/+6
Change-Id: I276de7a5fd1c705c87d35dd616e5980c747190aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/1597 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 r/1592 fix(edef/keys): add readTree headeredef1-0/+2
Change-Id: I5e1e0f58377201b591730a49f44678a33243b1d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1653 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1591 chore(edef/users/depot-scan): move to a subdiredef3-3/+1
Change-Id: If202b466aa0874048e89e0dc270ff14883510131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1652 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1590 fix(users/edef/depot-scan): properly trace readFile/readDiredef3-2/+19
Change-Id: I493e7d20504d34f83a53db50b69e86668874756e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1651 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1589 fix(users/edef/depot-scan): properly depend on pl scriptKane York2-2/+2
Change-Id: I83c75ab75dae9c904e127ad6160c7641d4523e40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1647 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1588 feat(edef): add SSH CAedef1-0/+5
Change-Id: If6839735d2730fa49363fa46e6fa059dd19bfaf7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1649 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1587 feat(edef/depot-scan): initedef2-0/+23
A small script that parses `nix-instantiate -vv` output to track the files read while instantiating a depot build. Change-Id: I5acf31d55f39c1d1acf9cdead03d33e2c8abab6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1646 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-04 r/1586 feat(nix/readTree): allow arguments to depend on tree locationedef1-3/+3
This accepts a function for args, and passes it an attrset containing the locatedAt parameter. Change-Id: I3c0f8ca00605e02a787fda88f32b06f5ef9998f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1639 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1585 chore(nix/fetchGoModule): don't move downloaded fileedef1-3/+1
unzip(1) doesn't care about the extension, unlike the unpackFile hook. Change-Id: Ia9d0e17c4b343ee5d32350329eb6267b9dc35605 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1641 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1584 chore(nix/fetchGoModule): remove overrideAttrs hackedef1-5/+3
This was based off fetchzip, but unlike fetchzip we don't need to support arbitrary archive formats, so we don't actually need the unpack hooks. Change-Id: I5655c031f420422f13e72bb79f4b4b95f6f1204e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1640 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1583 feat(tvix): Implement AddIndirectRoot, SyncWithGCGriffin Smith2-2/+32
Implement RPC handlers and client methods for the AddWithRoot and SyncWithGC calls. With this and sandboxing disabled, the following derivation is now building successfully: derivation { name = "test"; builder = "/bin/sh"; args = [ "-c" "echo 1 > $out" ]; system = "x86_64-linux"; } Co-authored-by: Vincent Ambo <mail@tazj.in> Change-Id: I245dca0dfca882e2eae5d93364287a8b79fead71 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1637 Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI
2020-08-04 r/1582 chore(tvix): Make not implemented errors nix::UnsupportedGriffin Smith1-22/+24
Without this, these errors print as "uncaught exception of type std::string", but with this they print using nix's built-in error printer, which will be nice to have as we bang out the remaining store API methods. Co-authored-by: Vincent Ambo <mail@tazj.in> Change-Id: I9bd4f77f65b27c8f97b727b98ef62e2061999e71 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1636 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-04 r/1581 fix(tvix): Partially revert "eliminate exposed global variable"Griffin Smith3-21/+39
This reverts part of commit 976a36c2e482f416acd79a624e6d96cce2564b5b, which was causing scanForReferences to hang indefinitely. I'm not sure what the original intent of the commit was since there's not really much of a commit message on it, but with this I am able to fully realise derivations. Change-Id: I620e892e2fe2244f066512286b086b3dd8f8f6c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1635 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-04 r/1580 chore: simplify buildGo expressionsedef13-15/+15
This removes redundant `map (p: p.gopkg)` expressions. Change-Id: If25d603c27d63ae011f396caf5d26cdfdb8c28d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1625 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-04 r/1579 feat(nix/buildGo): use gopkg attribute of depsedef2-3/+3
This removes the need for `map (x: x.gopkg)` in dependency lists. Change-Id: Ia19e51f14110bf22f5cdbb64fa7ce182e1b34241 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1624 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1578 feat(nix/buildGo): expose gopkg attribute on buildGo.packageedef1-7/+13
This makes the derivations more uniform, since both third-party packages generated by buildGo.external and native buildGo.package expose their libraries as gopkg attributes now. Change-Id: I547f9860082f36e3300139bf67613eb4fc600d24 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1623 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1577 fix(3p/gopkgs/cloud.google.com/go): add missing `depot` argedef1-1/+1
Change-Id: If08a43e18a41abd3e2ae7c08ceb6f48844d20df1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1626 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1576 fix(3p/nix): Don't assert on argc count for build-remoteGriffin Smith1-4/+0
This executable doesn't appear to ever actually look at argv, so throwing an error if argc is wrong seems kinda silly. Change-Id: Iabc9dad785d4c99be9e8fd4ecd954d38b876cecd Reviewed-on: https://cl.tvl.fyi/c/depot/+/1633 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1575 fix(3p/nix): Properly configure SANDBOX_SHELLGriffin Smith4-2/+23
point the SANDBOX_SHELL macro at the actual path to busybox on the build machine, or allow it to be configured at build-time with a cmake option. Change-Id: I044a1315ba9baa3bc9ceddf29f36d14f9f9ccd96 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1632 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 r/1574 fix(3p/nix): pass SkipEmpty to StrSplit("", ...)Griffin Smith1-4/+8
When tokenizeString was changed to absl::StrSplit, there was a behavior change because tokenizeString on an empty string returned an empty vector - which the derivation builder (and likely a bunch of other stuff) was depending on. The canonical way of fixing this is by passing absl::SkipEmpty() to the function - there may be other places we need to fix this as well. This commit also includes some opportunistic absl::StrFormats and StrCats, because I was here anyway, but those have no semantic difference. Change-Id: Ibf9bb602284f793fa55728481f63b838fb7a41db Reviewed-on: https://cl.tvl.fyi/c/depot/+/1631 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2020-08-04 r/1573 feat(3p/nix/daemon): catch-all explicit Error-Status conversionKane York2-233/+413
We wrap every server-side proto handler with a macro that catches exceptions and turns them into proper grpc error codes. For the time being, most exceptions map to INTERNAL, the existing mapping. Change-Id: Id6ed6a279b198ad185d32562f39000ccc15eadbf Reviewed-on: https://cl.tvl.fyi/c/depot/+/1599 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 r/1572 fix(3p/nix): Initialise glog when starting nix-daemonVincent Ambo1-0/+3
Change-Id: Ic9b5bd2f66fb54675498d8328edecfd31a03997a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1630 Reviewed-by: glittershark <grfn@gws.fyi> Tested-by: BuildkiteCI
2020-08-03 r/1571 fix(3p/nix): Fix parseDrvPathWithOutputsGriffin Smith2-3/+33
At some point the behavior of this function got changed as part of our cleanup - this fixes it to behave the way the rest of the codebase expects (and how it is documented in the header) and covers it with a few tests. Change-Id: Id4c91232968e73489cd866fb4a2a84bcf20d875e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1629 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 r/1570 chore(3p/nix/daemon): replace all assertStorePath with a macroKane York1-14/+20
Two exceptions: IsValidPath needs to return success for invalid paths, and QueryAllValidPaths shouldn't need to check the paths it gets from itself. Change-Id: I4d9d4125d34e8de42f30070aec607f8a902eded7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1598 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 r/1569 refactor(3p/nix): Build nix-daemon as a separate binaryVincent Ambo4-34/+46
Skips over all the monobinary stuff and moves to a separate binary for nix-daemon. This also replaces the flag parsing logic with absl::flags. This causes a behaviour change for --help, which no longer tries to display a man page but instead shows the actual command-line help. Note: This binary no longer links to the Boehm GC. Change-Id: Ib852e994b82f2d56e91262878c10650e656427a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1622 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 r/1568 chore(3p/nix): Move legacy Nix daemon implementationVincent Ambo1-0/+6
The filename this is squatting is too good. Change-Id: I694132b94d24786df84b6b5a698679bdce818b2e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1621 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 r/1567 feat(3p/nix): Implement two more RPC callsGriffin Smith4-6/+76
Implement AddTextToStore and BuildPaths both on the client and the server Refs: #29 Change-Id: I45294c3e1c1a7489e42099d36425b7acc04e0427 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1560 Reviewed-by: kanepyork <rikingcoding@gmail.com> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-25T23·23+0000