about summary refs log tree commit diff
path: root/tools (follow)
AgeCommit message (Collapse)AuthorFilesLines
2021-12-18 r/3291 feat(tools/magrathea): add a path commandVincent Ambo1-4/+19
this command prints the absolute directory for a given target. it can be combined with shell aliases to add quick navigation commands. unfortunately due to the nature of computers implementing something like `mg cd` directly is not possible. Change-Id: Icc88eb97384812c620c49fe2de8fa331f4d7153b Reviewed-on: https://cl.tvl.fyi/c/depot/+/4395 Tested-by: BuildkiteCI Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-18 r/3290 feat(tools/magrathea): wire up parsed targets with commandsVincent Ambo1-24/+61
magrathea now does what it says on the tin - build and shell commands can be used with the targets specified on the command line. implementation notes: * string representation of target has been changed to look like the target spec format, this is now used in user-facing messages * errors returned by the target parser make the program exit with status 1 * normalisation could be done better (for example, maybe it makes sense to always do it) but it's good enough for now Change-Id: Ib85f389a5cec92b3c2f3b9c0b40764435bbcc68b Reviewed-on: https://cl.tvl.fyi/c/depot/+/4394 Tested-by: BuildkiteCI Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-17 r/3287 feat(tools/magrathea): add weirdly overengineered target spec parserVincent Ambo1-1/+105
i'm not sure what happened here, but it works (yes, the fancy target printing is completely unnecessary, but oh well): #;152> (parse-target "foo") #target(foo) #;153> (parse-target "//foo") #target(//foo) #;154> (parse-target "//foo/bar") #target(//foo/bar) #;155> (parse-target "//foo/bar/") #target(//foo/bar) #;156> (parse-target "//foo/bar:baz") #target(//foo/bar:baz) #;157> (parse-target "//foo/bar/:baz") #target(//foo/bar:baz) #;158> (parse-target "//foo/bar:") (error . "unexpected end of input while parsing virtual target") #;159> (parse-target "//foo//") (error . "unexpected root-anchor while parsing normal target") the most notable thing is that trailing slashes are allowed in the physical targets, since people may be autocompleting these on the shell from folder names. Change-Id: I32975ad77fe2a327130dc9574011fe92cce49f84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4393 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-17 r/3286 feat(tools/magrathea): bootstrap a tool for working with depotVincent Ambo2-0/+133
this is going to be a serious version of //tools/depot-build. right now it doesn't support parsing any target specs yet, so only shells and builds for the physical project of the current folder work. Change-Id: I4308e29da940571622ff9e539fbb8ededd27aca7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4335 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: grfn <grfn@gws.fyi>
2021-12-15 r/3260 fix(tvl.el): Replace fifth with nth 4Griffin Smith1-1/+1
as before, fifth doesn't exist on all emacsen, but nth definitely does Change-Id: Ic0e4e3790402d960d1546d37187758a4d9ca33c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4346 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-12-15 r/3252 chore(3p): bump NixOS channels to 2021-12-15sterni1-1/+1
* Switch from (unused) NixOS 21.05 to newly released NixOS 21.11 channel. * grfn/machines/yeren: Linux 5.14.x -> 5.15.x * 3p/gerrit, 3p/gerrit_plugin/oauth: update dependency bundle output hashes to what whitby comes up with now. Change-Id: Id3bfe3d07b9223f77ec628954783a6b0a48f4086 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4310 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2021-12-13 r/3214 fix(passively): Load known terms on startupVincent Ambo1-0/+1
Change-Id: Ia32f0a1c158255a098e7d4017bb585b5c4e3c657
2021-12-10 r/3206 fix(tvl.el): Fix use of label command in refsVincent Ambo1-4/+4
The l= is part of the command, not of the shape of commands, and the previous command concatenation logic was wrong because of that. Fix is done in the most obvious way: Make the l= part of the command. Change-Id: Ia3c08c3da60fe5fc38f29a2d94adcd123e4f3052
2021-12-10 r/3196 feat(tvl.el): Add autosubmit feature to magit-gerrit-rubberstampVincent Ambo1-4/+6
This makes this function a true rubberstamp again, leading to rubberstamped CLs automatically being merged after CI passes. This is similar to the initial functionality we had last year, where this directly submitted changes, but with the addition of the CI checks. Change-Id: I946b074b968eb18a64c4edb0043f7a4af28759b4
2021-12-10 r/3195 feat(tvl.el): Add shortcut for push with Autosubmit+1Vincent Ambo1-0/+11
This almost makes for a sort of fire&forget button, except we don't have a way to automatically pick reviewers yet :) Change-Id: I6f446270f8aaf0409ccb6321bdbb5c349079cd19
2021-12-09 r/3163 feat(tvl.el): Add magit-gerrit-cherry-pickGriffin Smith1-0/+20
Bound to `A g`, this behaves similarly to `magit-gerrit-checkout` - it prompts for a CL number, then cherry-picks the latest patchset of that CL number Change-Id: Ieef970b99d96170e8c960cc7687ead9022948f8b
2021-12-07 r/3152 fix(tvl.el): s/fourth/cadddrGriffin Smith1-1/+1
Apparently some emacsen don't have functions like `fourth` etc. Change-Id: I3d8b698685ce3b1757b427b32d8e27938cc26661
2021-12-06 r/3142 feat(tools/passively): Implement automatic passive learning in EmacsVincent Ambo2-0/+128
Adds all the functionality described in the README in cl/4066. This code is very closely related to //users/tazjin/russian/russian.el Change-Id: I14f1052cebfbe4886e75e8efc730eacbf8773f29
2021-12-03 r/3138 feat(tools/passively): Bootstrap new Emacs package 'passively'Vincent Ambo2-0/+79
Passively is a tool to help people learn information via Emacs, designed for language learning. As of this CL, the actual implementation still lives in //users/tazjin/russian/russian.el but I am generalising it here. Change-Id: Iac5a8cfc78415496637a7ba5ddc4c2a1aa6bee26
2021-11-23 r/3088 refactor(readTree): Move 'drvTargets' into readTreeVincent Ambo1-1/+1
This function is also generally useful for readTree consumers that have the concept of subtargets. Change-Id: Ic7fc03380dec6953fb288763a28e50ab3624d233
2021-11-20 r/3075 feat(tvl.el): Add magit-gerrit-checkout commandGriffin Smith1-1/+57
Add a new magit-gerrit-checkout command, which prompts for a CL number then fetches and checks out the latest patchset of that CL with a detached HEAD. Change-Id: I88b8209d40017479d97ed40ecbd5fd1ccd7cf650 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3880 Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI
2021-10-29 chore: Bump all Go dependenciesVincent Ambo3-36/+169
Result of 'go get -u && go mod tidy'
2021-10-29 chore: Bump nixpkgs pin to nixos-unstable 2021-10-29Vincent Ambo1-2/+2
2021-10-29 chore(docs): Bump included nix-1p versionVincent Ambo1-2/+2
... basically never updated this, oops.
2021-10-29 docs: Remove note about unsupported Google projectsVincent Ambo1-2/+0
I no longer work at Google and the repo has moved, so this is no longer relevant.
2021-10-12 r/2967 refactor(tools/rust-crates-advisory): move advisory-db to 3psterni1-7/+1
Change-Id: Iaaed35de078292c0c99a7c83de9ca5fdf27b8135 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3711 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-10-11 r/2965 chore(tools/cheddar): update dependenciessterni1-350/+187
Change-Id: I64c305c60d2187633cb84e8642e4639f1402325c Reviewed-on: https://cl.tvl.fyi/c/depot/+/3725 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-10-08 revert: "feat(storage): Add generic support for content-types"Jérôme Petazzoni1-10/+0
This reverts commit 7db252f36a68d875429a25e06d88fbfc804d84fd. Superseded by the implementation in #127.
2021-09-15 r/2868 refactor(rust-crates-advisory): check type instead of blacklistingsterni1-3/+2
`our-crates` can just check if the attributes in question are derivation (i. e. have an `outPath`) instead of blacklisting the `__readTree` attribute specifically. Change-Id: I472692e89c0e9eff551372c72a73ab765b0b6599
2021-09-10 r/2835 refactor(cheddar): Support multiple shortlinks & custom link setsVincent Ambo1-10/+42
This makes it possible for users of cheddar as a library to supply their own shortlinks. In practice it is unlikely anyone will do this, but the change also allows us to (relatively) easily add additional shortlinks to the set used by TVL. Note that Cheddar is primarily intended for use by TVL and the default rendering function interfaces have not changed, and will default to using TVL shortlinks. A new public function `format_markdown_with_shortlinks` has been added with which users can use an alternative set of shortlinks. This function should not be used in TVL depot code. Change-Id: I4ddab28cbcf45d07c51323b7b730b96e62922816 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3083 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-08-25 feat(prepare-image): Ensure /usr/bin/env is always presentVincent Ambo1-0/+12
This is required by common patterns in shell scripts. There are some caveats around this. Adding logic to filter whether coreutils is included in an image would slow down the Nix evaluation, so the link is currently created even in cases where it doesn't point to anything. Fixes #109
2021-08-24 r/2764 fix(tvl.el): Fix compilation warningsVincent Ambo1-5/+8
This fixes all compilation warnings except the one about 'tvl lacking a parent group, which we can look into later (it doesn't matter that much). Change-Id: Iaff5e7f5f251f0670afb0a47031ccf197de69818 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3408 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-17 r/2746 feat(tvl.el): Make target branch configurableGriffin Smith1-4/+9
I'm using gerrit at work now, and would like to use tvl.el to interact with it via Emacs, but we use a different default branch than "canon". This makes it configurable, and also marks it as safe so I can configure it in .dir-locals.el Change-Id: I66d4c7ce94351f2df863ec49dbc3e1d1d6d1547a Reviewed-on: https://cl.tvl.fyi/c/depot/+/3369 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-08-06 chore(build): Allow passing in a specific commit hash when buildingVincent Ambo1-2/+3
Required for builds where the full repository isn't available (e.g. from a tarball).
2021-07-28 r/2697 feat(tools/depot-nixpkgs-update): tool to update our pinsterni1-0/+44
Very simplistic tool to aid with updating our nixpkgs channel pins: This tool prints the correct two `*Hashes` sets to stdout, so you can easily delete the appropriate lines in `third_party/nixpkgs/default.nix`, run ./bin/depot-nixpkgs-update | wl-copy and paste it into your editor. Doing this fully automatically would be possible, but would either a) require changing `default.nix`, so it is regex-able more easily, b) touching something like rnix-parser which I have no motivation to at the moment or c) searching for the old hashes and replacing them with the new ones. This may be a simple and worthwhile improvement in the future. Change-Id: I4df44e3827ce9ff6e4fe2d336c08016d799e21a7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3252 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-07-15 docs: Update build badge in READMEVincent Ambo1-1/+1
Moves the build badge to point at Github Actions, instead of the old (failing) Travis build
2021-06-26 feat(storage): Store blob content-type in extended attributesJérôme Petazzoni4-3/+20
After the discussion in #116, this stores the blob content types in extended attributes when using the filesystem backend. If the underlying filesystem doesn't support extended attributes, storing blobs won't work; also, if extended attributes get removed, blobs won't be served anymore. We can relax this behavior if needed (i.e. log errors but still accept to store or serve blobs). However, since the Docker Engine (and possibly other container engines) won't accept to pull images from a registry that doesn't use correct content types for manifest files, it could be argued that it's better to give a hard fail. (Otherwise, the container engine gives cryptic error messages like "missing signature key".) I can change that behavior (and log errors but still store/serve blobs to the filesystem) if you think it's better.
2021-06-20 feat(ci): don't mount /var/cache/nixery from tmpfs into docker containerFlorian Klink2-2/+13
With https://github.com/google/nixery/pull/127, nixery will use extended attributes to store metadata (when using local storage). Right now, our integration test mounts a tmpfs to /var/cache/nixery. However, *user* xattrs aren't supported with tmpfs [1], so setting xattrs would fail. To workaround this, use a folder in the current working directory and hope it's backed by something supporting user xattrs (which is the case for GitHub Actions). [1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-05-25 r/2630 chore(nixpkgs): Bump channels to 2021-05-25Vincent Ambo1-2/+2
* users/grfn/system/home/yeren: remove obsolete awscli2 overrides * ops: make new isSystemUser || isNormalUser assertion happy * users/grfn/system/system/mugwump: make buildkite agents system users * users/tazjin/nixos/camden: set isSystemUser = true for git * users/tazjin/emacs: Remove missing & broken packages * third_party/openldap: remove, as the argon2 module is now enabled upstream * third_party/gerrit_plugins: Pinned new unstable hashes * third_party/nix, third_party/grpc: Disabled CI as these are broken * third_party/overlays/emacs: Bumped version to stay in sync with channel * third_party/buzz: Update LIBCLANG_PATH to reference libclang.lib, since libclang's default output no longer contains libclang.so * users/grfn/system/home: Install julia-stable instead of julia (which aliases to julia-lts), as the latter depends on an insecure version of libgit Change-Id: Iff33b0ecb0ef07a82d1de35e23c40d2f4bf0f8ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/3001 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi>
2021-05-24 r/2622 chore(3p/overlays): Remove LLVM11 bump from overlayVincent Ambo1-4/+0
* This was mostly for //third_party/nix and its dependencies which now have been set to use llvmPackages_11 manually. * For //users/grfn/achilles we also manually select the newer LLVM version. * //tools/cheddar doesn't seem to need llvm anymore. * //third_party/buzz also compiles with clang 7.1.0 * replace clang-tools everywhere with new attribute clang-tools_11 For the future we may want to have something similar again, but it may not be necessary to invest too much time into it: nixpkgs is set to upgrade their default llvmPackages to LLVM 11 as well at some point in the near future. Co-Authored-By: sterni <sternenseemann@systemli.org> Change-Id: Id83868dbc476a6c776b59518b856c933f30ea79d Reviewed-on: https://cl.tvl.fyi/c/depot/+/3135 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi>
2021-05-23 r/2612 fix(hash-password): Ensure OpenLDAP with correct module is usedVincent Ambo1-2/+2
This is fixed in upstream nixpkgs, but we're not yet at a commit where it's used, so it's important to use the OpenLDAP from //third_party Change-Id: I7c033cd23f45a95c4a4af864ffe561c496833a0d Reviewed-on: https://cl.tvl.fyi/c/depot/+/3143 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-05-17 r/2595 feat(tools): add rust-crates-advisoryProfpatsch4-0/+176
We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-05-12 r/2591 docs(cheddar): fix typo in build.rs commentsterni1-1/+1
Change-Id: Id323b4a187b3837d60ac6b8917f3e8571954b8e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3119 Reviewed-by: tazjin <mail@tazj.in> Tested-by: BuildkiteCI
2021-05-04 r/2565 feat(cheddar): Implement highlighting of CL and bug shortlinksVincent Ambo4-1/+51
Implements highlighting of shortlinks like cl/123, or b/123. Highlighting works by replacing the input Markdown using a simple regular expression replacement. We also considered parsing and replacing these links in the Markdown AST, but it would have been significantly more complex (due to arena allocation) for little upside and no obvious performance benefit. Change-Id: I53f03fb17491046d89d0b7f605929571c11ee9a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3082 Tested-by: BuildkiteCI Reviewed-by: eta <eta@theta.eu.org>
2021-05-03 r/2557 feat(cheddar): Add a build script that injects BAT_SYNTAXESVincent Ambo1-0/+50
The BAT_SYNTAXES environment variable is injected during Nix builds and points to the build result of //third_party/bat_syntaxes, which is included in the cheddar binary. During development this is a bit annoying: Opening a Nix shell will set the right variable, but it will also mess with the Cargo environment in ways that make it annoying to use (*and* require a pre-build of all dependencies as required by naersk). For an easier development flow, this adds a build script that automatically runs Nix, fetches the path to the syntax set and sets the environment variable. It is a no-op if the variable is already set. This makes for a smoother development experience when working on cheddar in a depot checkout. Change-Id: Ib5435518897e3443247edd9f362821c0b9ff756b Reviewed-on: https://cl.tvl.fyi/c/depot/+/3081 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2021-05-03 r/2556 refactor(cheddar): Split out a library with rendering logicVincent Ambo2-131/+148
Splits `main.rs` into `lib.rs` and `bin/cheddar.rs`, which enables reuse of cheddar's rendering logic in other Rust applications. Change-Id: Ifd1a44a8d1620c595550a0a497a25b0563e917ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/3060 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2021-04-30 feat(build): Run `go vet` as a step in the GitHub Actions workflowVincent Ambo2-1/+3
2021-04-30 refactor(build): Pin dependencies using Go modulesVincent Ambo5-164/+553
Drops the go2nix configuration in favour of pkgs.buildGoModule. Note that the go.sum file is bloated by issues with cyclic dependencies in some Google projects, but this large number of dependencies is not actually built.
2021-04-30 chore(build): Use current git commit hash as build versionVincent Ambo1-6/+4
2021-04-30 chore: Update default NixOS channel to nixos-20.09Vincent Ambo6-8/+8
2021-04-30 chore(ci): Remove unnecessary commands from new CI setupVincent Ambo1-7/+2
* remove a step that was not supposed to be committed ("Do we have Docker?") * remove setup of old temporary storage directory (now done in integration script test instead) * skip creation of out-link for initial Nixery build (to avoid cache-busting on the second build)
2021-04-29 docs: document unset GOOGLE_APPLICATION_CREDENTIALSFlorian Klink2-0/+8
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not set, a redirect to storage.googleapis.com is issued, which means the underlying bucket objects need to be publicly accessible. This wasn't really obvious until now, so further clarify it.
2021-04-29 feat(ci): add integration tests to GitHub Actions, remove .travis.yamlFlorian Klink3-78/+53
This copies the integration tests from `.travis.yaml` into a script, documents the assumptions it makes, and wires it into GitHub Actions. Contrary to the travis version, we don't use Nixery's GCS backend, as handing out access to the bucket used, especially for PRs, needs to be done carefully. Adding back GCS to the integration test can be done at a later point, either by using a mock server, or by only exposing the credentials for master builds (and have the test script decide on whether GOOGLE_APPLICATION_CREDENTIALS is set or not). The previous travis version had some complicated post-mortem log gathering - instead of doing this, we can just `docker run` nixery, but fork it into the background with the shell - causing it to still be able to log its output as it's running. An additional `--rm` is appended, so the container gets cleaned up on termination - this allows subsequent runs on non-CI infrastructure (like developer laptops), without having to manually clean up containers. Fixes #119.
2021-04-29 feat(ci): remove unneeded permissions: read-allFlorian Klink1-1/+0
We don't intend to label, authenticate or whatever with the GITHUB_TOKEN, so there's not really a reason to give any broader permissions than the defaults.
2021-04-28 feat(ci): Configure initial GitHub Actions setupVincent Ambo1-0/+29
Travis is being deprecated, and this might be the best option for now.