about summary refs log tree commit diff
path: root/ops/modules (follow)
AgeCommit message (Collapse)AuthorFilesLines
2021-12-19 r/3302 feat(whitby): Add buildkite agents to docker groupGriffin Smith1-1/+1
I'd like to be able to run extra CI steps that include running docker containers (to integration test things like webapps that connect to a database). To do this the buildkite agents themselves need permission to do docker things. Change-Id: I3c9a488708f0e12a508754ac41f04148ca7aedac Reviewed-on: https://cl.tvl.fyi/c/depot/+/4408 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 r/3266 feat(ops/modules): Provide some modules to all nixosesGriffin Smith1-0/+13
For modules that are gated behind a mkEnableOption, it's reasonable to just provide them to all Depot-built nixos systems without requiring people to explicitly import them. This defines a special module called `default-imports.nix` which imports these modules (currently just tvl-cache.nix and automatic-gc.nix, as I'm being rather conservative adding things here to avoid breaking anyone's system), then provides that module as one of the `modules` passed at the top-level nixos/eval-config invocation. Change-Id: I3be299ab10ae4c451ef11c514edb3c89318a2278 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4345 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2021-12-15 r/3261 feat(ops/modules): Add shared module for TVL cacheGriffin Smith1-0/+19
Add a shared nixos module for configuring whitby as a binary nix cache, and refactor tverskoy to use this module. This is enabled via an option to pave the way for including it as an import in all depot-generated nixos configs at some point in the future. Change-Id: I6dcc0e8eb48b1ac34457666dceebeedd5da6c526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4344 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: wpcarro <wpcarro@gmail.com> Autosubmit: grfn <grfn@gws.fyi>
2021-12-14 r/3244 docs(ops/irccat): link to credentials RFEFlorian Klink1-0/+4
https://cl.tvl.fyi/c/depot/+/4264 did move merging config with secrets into ExecStart=, which is tracked in an RFE upstream: https://github.com/systemd/systemd/issues/19604#issuecomment-989279884 We didn't link to this so far, neither in the commit message, nor in a comment. Let's add a comment, so people know when we can undo this. Change-Id: I7bed370b671093bb876592b4dccd562f1c256cd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4326 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: grfn <grfn@gws.fyi>
2021-12-10 r/3204 fix(tvl-buildkite): Use supported credential helper binary nameVincent Ambo1-1/+1
Git only allows binary names prefixed with `git-credential-` if the path to the helper is not absolute. Why? Who knows. Change-Id: I216b2a621f62a73f05e21def7ec8016b29ede892
2021-12-10 r/3203 refactor(ops): Move panettone secrets to agenixVincent Ambo1-0/+1
Relates to b/161 Change-Id: I508e5a0eacab668f4bd39a2c888d894b96bed093
2021-12-10 r/3200 refactor(tvl-buildkite): Prepare gerrit credentials helperVincent Ambo1-0/+6
Currently this functionality is provided by a shell script stored in /etc/secrets (which has the password value hardcoded). This needs to happen in a separate commit from the one that changes the pipeline to avoid breaking it (it needs to be deployed first). Change-Id: I680754c828ccefbacfcf0d5c813a4bc19493ba4c
2021-12-10 r/3199 refactor(ops): Move Nix cache secret to agenixVincent Ambo1-1/+1
... and also the public key, just to keep the distribution mechanism the same. Change-Id: Ief14daf9344c0fb99eeb5789c1ec9bfb1f12bee0
2021-12-10 r/3198 refactor(ops): Use besadii configuration from agenixVincent Ambo2-2/+2
We already checked this in, but this commit adds the configuration for making use of it. There are two copies of besadii's JSON configuration with different permissions. Note that the buildkite-graphql-token path needs to be updated in static-pipeline.yml, but this needs to happen in a separate commit after deploy because the pipeline will break otherwise. Change-Id: I6fab4bf1a2e679df7cf76521e2b53bd9dadbac62
2021-12-10 r/3194 fix(tvl-buildkite): Add more missing programs to agent pathVincent Ambo1-0/+2
... this option really is a pitfall! The list of programs is now the same as in the upstream module, plus curl and jq. Change-Id: I29edae4b2400a2724f62df9efa1dc184a8b0af5f
2021-12-10 r/3191 fix(ops/irccat): Avoid permissions issue with LoadCredentials=Vincent Ambo1-9/+7
The DynamicUser + Group configuration does not work as planned, thus the systemd LoadCredentials feature is used instead which makes the file (which itself is only readable by root) available in a memory-backed location only readable by the service. The secret is only available to `ExecStart` commands, so units using this feature can not be used with pre/post units and the like if those commands need secrets. To accommodate this, the merge of configuration files has been moved into the service launch script, which is now the ExecStart= process. For details take a look at https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LoadCredential=ID:PATH Change-Id: I693fe5677cc0d63c7aa485c2c7472457c5262166
2021-12-10 r/3190 fix(tvl-buildkite): Explicitly set runtimePackagesVincent Ambo1-1/+5
It turns out the lib.mkAfter call doesn't behave as expected - only *some* of the packages that are defaulted end up in the $PATH. I suspect this is actually something else, e.g. these packages are always added for some reason or another, and the option is completely overridden every time. Change-Id: I854c7198520d82b00e6338ed0fe653836226dc6d
2021-12-10 r/3185 fix(tvl-buildkite): Add missing runtimePackages backVincent Ambo1-1/+5
Turns out that the type of this option is not concatenative and it replaces the packages needed to run Buildkite if set. Change-Id: I9f52572bc165bccdd8c6518cfdf7b8967f7a50d0
2021-12-10 r/3184 refactor(ops): Move irccat secret into agenixVincent Ambo1-2/+12
The irccat module uses DynamicUser, so to grant permission to it a new group has been added for irccat. I have some vague memory of DynamicUser + Group not behaving as one would expect, but we'll see what happens. Change-Id: Iab9f6a3f1a53c4133b635458ce173250cc9a3fac
2021-12-10 r/3176 refactor(ops): Move buildkite-agent-token into agenixVincent Ambo1-1/+1
Relates to b/161 Change-Id: I5d3a698d437928966d8b78ce9e0ba226c1437655
2021-12-10 r/3175 refactor(ops): Move owothia secret into agenixVincent Ambo1-1/+1
Relates to b/161 Change-Id: I25445281b0dd3c3f3660f8bb0d8337506a1e427b
2021-12-10 r/3174 refactor(ops): Move clbot secret into agenixVincent Ambo1-1/+7
Relates to b/161 Change-Id: I7badf22ff93bb4e8b06e4dd4a8bf880b0bd48f09
2021-12-10 r/3172 feat(ops/modules): Add module for running gerrit-queueVincent Ambo1-0/+51
This is not yet including the secret configuration for gerrit-queue, and just expects the secret (gerrit username & password) to be available in /etc/secrets. Change-Id: Ia465ef7f3f521c70d606d7fdeba9aa83c7e1b98b
2021-12-10 r/3171 chore(tvl-buildkite): Add jq and curl to agent pathsVincent Ambo1-0/+1
This is required for a simplification of the build pipeline (following CL) and needs to be in a separate commit as it can not be done atomically (merging the other commit to deploy it would immediately break pipelines otherwise). Change-Id: I5d8ec8f3238f79b5518d799486bf98d1d9516c43
2021-12-07 r/3148 fix(ops): Correctly pass command name to besadii invocationsVincent Ambo2-7/+7
Ensure that besadii sees $0 as the correct command name, since that is the sole mechanism by which its functionality is switched around. There was a lingering commit that introduced this bug and hadn't been deployed in a couple of days. Maybe time to tighten deploy cycles soon ... Change-Id: Ie4284c0f6e5e06d71a71a3702ec7e092260e0ce5
2021-12-02 r/3134 chore(ops/modules): Configure besadii call sites to load configVincent Ambo2-3/+14
On whitby, the besadii config will live in /etc/secrets/besadii.json. This CL updates the call sites to pass this config path to besadii so that it can load Sourcegraph configuration. Change-Id: Ia139b9fa3b827e7a5f2386214390acc6fe19a75a
2021-12-02 r/3132 fix(ops/nixery): Temporarily stop serving depot packages in NixeryVincent Ambo1-1/+1
Change the Nixery configuration to use the plain nixpkgs package path instead of the depot path. AFAIK, nobody uses this to fetches depot packages at the moment - but plenty of people fetch non-depot packages. This means that Nixery is cache-busted less often (previously on every commit => every deploy). We'll figure out another way to have a depot Nixery later. Change-Id: Iba632333346181c3d2ce992fbab396ed0d9f86aa
2021-12-01 r/3131 fix(ops/www): Redirect tvl.fyi/blog -> tvl.fyiVincent Ambo1-0/+4
The blog index page is at the root and people may manually edit the URL. Change-Id: I6cdaaaee6223524a9e950584379cfac34f8be160
2021-12-01 r/3125 feat(besadii): Support invocation as different Gerrit hooksVincent Ambo1-1/+2
Removes besadii support for the previously used 'ref-updated' hook and instead introduces support for the 'change-merged' and 'patchset-created' hooks. These hooks more accurately capture the semantics of when besadii should trigger CI builds and using them will avoid problems such as skipping 'canon' builds if chains of CLs are submitted together. Change-Id: Ib90356c069780bf0c0250e56b927e46a5b31ce7f
2021-11-30 r/3118 fix(ops/www): Strip `.html` from TVL blog post URLsVincent Ambo1-0/+8
Change-Id: I4d1f9284ec004931c07c04d614b01f28eedea508
2021-11-21 r/3078 fix(ops/restic): Move whitby's backup to GleSYS object storageVincent Ambo1-12/+16
Since GCP nuked us, the backups are now moving to GleSYS' S3-compatible object storage. This refactors the restic module to support S3-compatible storage instead of GCP, and switches to the appropriate new secret paths. The secrets were placed on whitby manually and I verified that the backups work. This fixes b/157 Change-Id: I6a9d2b0581967605ce736605a3befb44cdeae7e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3883 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2021-10-01 r/2946 feat(whitby): serve static.tvl.{fyi|su} with max cache settingsVincent Ambo1-0/+42
The setup is explained in the comment, but TL;DR: Use the derivation hash of static files to create permanent URLs. Relates to b/151. Change-Id: Ib1ca3a1a00c90a47f4bf39c29a8b4bbf5b215e7d Reviewed-on: https://cl.tvl.fyi/c/depot/+/3664 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2021-10-01 r/2941 feat(whitby): Serve //corp/website on tvl.suVincent Ambo1-0/+20
Change-Id: I21e1ddf9a32568cac8ad2595869ac8670867efa9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3658 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-09-24 r/2914 chore(ops/git-serving): Remove josh state from whitby backupsVincent Ambo1-2/+0
As cschilling explained on cl/3563, there isn't actually anything in this state that we *need* to persist. We're still keeping it in a persistent directory on disk as this serves as an optimisation after restarts of josh. Change-Id: Ia88886792a5acac34508b5b8a669bd519ca033de Reviewed-on: https://cl.tvl.fyi/c/depot/+/3631 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-18 r/2890 refactor(whitby): Move restic path configuration into modulesVincent Ambo3-0/+8
This lets each service declare their backup paths together with the configuration for the service, which is a lot more sensible than what we had before. Fixes b/147 Change-Id: If76fe62639f4cc0e6fbb63a2959d584479d8f0fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/3583 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-18 r/2889 fix(ops/restic): types.string -> types.strVincent Ambo1-3/+3
I can never remember which is which. Change-Id: I69b8235862b8c5b49030a74bfca25aaa113273b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3582 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-16 r/2878 refactor(ops/restic): Move restic configuration into a new moduleVincent Ambo1-0/+57
Relates to b/147. First step towards giving depot modules the ability to declare their own backup directories by moving all restic configuration into a new module and adding a NixOS option for inclusion/exclusion paths for backups. This still keeps all backup paths within the whitby config. Change-Id: Ia96833668f1a3d02da892261153d8b02156b8ac0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3565 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2021-09-16 r/2877 feat(git-serving): Configure josh to serve the depot over HTTPVincent Ambo2-2/+66
Previously we served the dumb git HTTP protocol from code.tvl.fyi via cgit. This CL disables this feature and instead runs josh in the same location (by redirecting appropriately), but while also enabling partial cloning of all subtrees of the depot. For example, after this CL the following would result in an independent clone of //nix/readTree: git clone https://code.tvl.fyi/depot.git:/nix/readTree.git Note that there are no josh workspaces configured at all for now, these references are only for static depot subpaths. Please refer to the documentation for josh for more information on available kinds of josh filters. Josh state is kept in a systemd state directory in /var/lib/josh and backed up to Restic. Backing this up is necessary, as josh uses stateful information to do things like tracking merges and rewriting history per subtree appropriately to avoid cloned repositories ending up in peculiar states. Change-Id: I156f0298c2aa42e3bdbf5a0e86109070d640c56e Reviewed-on: https://cl.tvl.fyi/c/depot/+/3563 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2021-09-11 r/2847 fix(deploys.*): Folder for diffs is in /diff/Vincent Ambo1-1/+1
... this was missing before. Change-Id: I5b79cb78665f24fdb7cc6496e3782d3940dc77b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3527 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 r/2846 feat(sourcegraph): Upgrade 3.30.4 -> 3.31.2Vincent Ambo1-1/+1
This one seems a little more involved: https://docs.sourcegraph.com/admin/migration/3_31 I believe we skip that corruption issue in the previous CL though, by simply never deploying a version with that weird broken image. See b/144 Change-Id: I3bbf1b719d00905e08a92011ace5485467f504ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/3525 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2845 feat(sourcegraph): Upgrade 3.29.1 -> 3.30.4Vincent Ambo1-1/+1
See b/144 Change-Id: Ied9490f3ce6fb3fda8cbb9983416b02ea451fb44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3524 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2844 feat(sourcegraph): Upgrade 3.28.0 -> 3.29.1Vincent Ambo1-1/+1
See b/144 Change-Id: Ia62d4cbf581caaefa0dba455376eec60b8c817d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3523 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2843 fix(sourcegraph): Temporarily comment out our syntax highlighterVincent Ambo1-1/+2
We changed away from the default sourcegraph one because it didn't support Nix, but it seems that there's been a change in the interaction protocol. Change-Id: I3a2691df6a87672cf83b819143f25d93d9cd6d13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3531 Tested-by: BuildkiteCI Reviewed-by: eta <tvl@eta.st> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 r/2842 feat(sourcegraph): Upgrade 3.27.5 -> 3.28.0Vincent Ambo1-1/+1
See b/144 Change-Id: Ia09ad2af6043dcac6681c549103d1e6f52b4e0a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3522 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2841 feat(sourcegraph): Upgrade 3.26.0 -> 3.27.5Vincent Ambo1-1/+1
See b/144 Change-Id: I50d417c51b05bafcd3fe7e285f30079db8be499a Reviewed-on: https://cl.tvl.fyi/c/depot/+/3521 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-10 r/2837 feat(whitby): Serve static HTML dir for deploys.tvl.fyiGriffin Smith1-0/+20
Add a new domain and nginx virtual host at deploys.tvl.fyi, serving out of a static directory on whitby which is created by systemd-tmpfiles. This will be used to serve diffs rendered by nix-diff for pending deploys for whitby Since this contains stateful data, it is added to the restic backups on whitby. Refs: b/110 Change-Id: I5869d40800bbf5fb8fb39878a857f66ff5787830 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3144 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 r/2758 fix(tvl-sso): set memory limit to 512MLuke Granger-Brown1-0/+1
This is because I'm bored of CAS gradually consuming all the RAM on Whitby. Change-Id: Idcc14c19d99a6d3553739c5765be3faf2bdf9d84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3233 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 r/2751 feat(ops): Serve nixery.dev from whitbyVincent Ambo1-0/+21
Adds a new module for the nixery.dev domain and serves it from whitby. Note that the DNS records do *not* point to whitby yet, so deploying this will lead to a failed TLS provisioning unit - but this is intentional. Change-Id: I911f67a0aa24f8df3cb52d2cfc49a8b6132cf718 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3383 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-15 r/2735 fix(monorepo-gerrit): Enable adding new email addresses to accountsVincent Ambo1-0/+3
This is required when people change their email addresses (e.g. cl/3349) as nothing in Gerrit will update that information from the OAuth provider. Change-Id: I1eafdf22efd37898dcd0d06bb9a5d1471ffb5e31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3356 Tested-by: BuildkiteCI Reviewed-by: eta <eta@theta.eu.org> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-08-13 r/2726 feat(ops/www): Point images.tvl.* at NixeryVincent Ambo1-0/+22
Change-Id: I39f979c68e7b74f6da6a7da0f07aaa470886d451 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3346 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-12 r/2721 feat(ops/modules): Add module for running NixeryVincent Ambo1-0/+42
This sets up a very simple Nixery instance with some things lacking: * no support for garbage-collecting image fragments (yet) * no popularity setup The plan is to use this to get the ball rolling on a separate domain (e.g. images.tvl.fyi), iron things out and then look into flipping over nixery.dev Change-Id: Ic594809f9d487fec7a0f632d608752a3f9c61315 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3280 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-12 r/2719 fix(monorepo-gerrit): Pin JVM version used for GerritVincent Ambo1-0/+7
Change-Id: Ib22cdc415cbd5a8345b9589b2c34b3908996dd57 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3322 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-06-12 r/2651 refactor(ops): Break out prometheus-fail2ban-exporter moduleGriffin Smith1-0/+52
Break out the configuration for the prometheus fail2ban exporter, which is a simple python script that exports stats from fail2ban as a prometheus-scrapable textfile, from Mugwump into a reusable nixos module in //ops/nixos/modules. Change-Id: I5451c9c5de6c7bc4431150ae596a9c758bf1b693 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3136 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-05-25 r/2631 fix(wigglydonke.rs): Don't rebuild nginx config unnecessarilyVincent Ambo1-1/+1
This fix is essentially the same as the one in cl/1263. Change-Id: I27be280a610914fcfbb6d7fee7aebaa56b993812 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3158 Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2021-05-25 r/2630 chore(nixpkgs): Bump channels to 2021-05-25Vincent Ambo5-15/+8
* users/grfn/system/home/yeren: remove obsolete awscli2 overrides * ops: make new isSystemUser || isNormalUser assertion happy * users/grfn/system/system/mugwump: make buildkite agents system users * users/tazjin/nixos/camden: set isSystemUser = true for git * users/tazjin/emacs: Remove missing & broken packages * third_party/openldap: remove, as the argon2 module is now enabled upstream * third_party/gerrit_plugins: Pinned new unstable hashes * third_party/nix, third_party/grpc: Disabled CI as these are broken * third_party/overlays/emacs: Bumped version to stay in sync with channel * third_party/buzz: Update LIBCLANG_PATH to reference libclang.lib, since libclang's default output no longer contains libclang.so * users/grfn/system/home: Install julia-stable instead of julia (which aliases to julia-lts), as the latter depends on an insecure version of libgit Change-Id: Iff33b0ecb0ef07a82d1de35e23c40d2f4bf0f8ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/3001 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi>
generated by cgit-pink 1.4.1 (git 2.44.1) at 2024-06-21T04·07+0000