about summary refs log tree commit diff
path: root/ops/modules
AgeCommit message (Collapse)AuthorFilesLines
2022-01-07 r/3525 chore(cache.tvl.su): Raise cache priority to 50Vincent Ambo1-0/+5
The priority of binary caches is decided by the remotes in Nix (???), and by default nix-serve (which is *very* slow) has a lower priority than cache.nixos.org (which means that it will be preferred over the faster cache for paths that exist on both). To avoid this, override the hardcoded (????) priority by serving the nix-cache-info response directly from nginx instead. Change-Id: I15a2d6618386d16edaf69f1c9257a36bd72132d2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4823 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-04 r/3518 feat(ops): Add initial oauth2_proxy configurationVincent Ambo1-0/+52
The intent is to configure oauth2_proxy pointing at Keycloak to enable usage with nginx auth_request directives. I want to expose this as a function from within the module in which nginx server configuration blocks can be wrapped, but the function for that is currently a placeholder. Change-Id: I5ed7deb9bf1c62818f516e68c33e8c5b632fccfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/4767 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-01-04 r/3517 chore(ops): Remove login.tvl.fyi moduleVincent Ambo1-24/+0
It looks like we won't need this for oauth2_proxy when combined with nginx auth_request setups. Change-Id: I2294aee6226b4f64a27bf6592c2d18092d0268cc Reviewed-on: https://cl.tvl.fyi/c/depot/+/4766 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-01 r/3509 refactor(modules/smtprelay): Load credentials via agenixVincent Ambo1-7/+14
Change-Id: I56f6887e1fd35551cfc83ad08cafebb611f4a341 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4760 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: Profpatsch <mail@profpatsch.de> Autosubmit: tazjin <mail@tazj.in>
2021-12-26 r/3446 fix(auto-deploy): Add missing packages to pathGriffin Smith1-3/+5
Building nix derivations needs tar (provided by gnutar) and gzip on the PATH in order to extract .tar.gz archives. Change-Id: Ia2df7a3a770cfd342dfede58ad34e04805fbd1f8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4685 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-26 r/3414 feat(ops/auto-deploy): Support auto-deployWilliam Carroll2-0/+93
Automatically rebuild the current system's NixOS config from the latest checkout of depot. Change-Id: I23aa7af50e16e985ac34df214e0905e770316e5e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4390 Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: zseri <zseri.devel@ytrizja.de> Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: wpcarro <wpcarro@gmail.com> Tested-by: BuildkiteCI
2021-12-26 r/3411 chore: friendship ended with cas, now keycloak is our best friendVincent Ambo1-28/+0
Note that the login.tvl.fyi WWW configuration is still kind of hanging around until we've settled where Keycloak lives. Change-Id: Iaca4e394a7371cafa3716ca66ef09c4eca5b1520 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4626 Autosubmit: tazjin <mail@tazj.in> Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-12-26 r/3410 feat(monorepo-gerrit): Configure for Keycloak compatibilityVincent Ambo1-5/+6
Change-Id: Ic3fce02b071c09cf03e652510f16bafb795a5a1d Reviewed-on: https://cl.tvl.fyi/c/depot/+/4614 Autosubmit: tazjin <mail@tazj.in> Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: grfn <grfn@gws.fyi>
2021-12-25 r/3401 feat(whitby): Configure initial Keycloak setupVincent Ambo1-0/+24
Trialing this as an alternative to CAS that is a little easier to configure and can help us delegate authentication to other OIDC services. Change-Id: Iad63724d349334910af8fed0b148e4ba428f796b Reviewed-on: https://cl.tvl.fyi/c/depot/+/4608 Tested-by: BuildkiteCI Autosubmit: tazjin <mail@tazj.in> Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-12-19 r/3302 feat(whitby): Add buildkite agents to docker groupGriffin Smith1-1/+1
I'd like to be able to run extra CI steps that include running docker containers (to integration test things like webapps that connect to a database). To do this the buildkite agents themselves need permission to do docker things. Change-Id: I3c9a488708f0e12a508754ac41f04148ca7aedac Reviewed-on: https://cl.tvl.fyi/c/depot/+/4408 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 r/3266 feat(ops/modules): Provide some modules to all nixosesGriffin Smith1-0/+13
For modules that are gated behind a mkEnableOption, it's reasonable to just provide them to all Depot-built nixos systems without requiring people to explicitly import them. This defines a special module called `default-imports.nix` which imports these modules (currently just tvl-cache.nix and automatic-gc.nix, as I'm being rather conservative adding things here to avoid breaking anyone's system), then provides that module as one of the `modules` passed at the top-level nixos/eval-config invocation. Change-Id: I3be299ab10ae4c451ef11c514edb3c89318a2278 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4345 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: tazjin <mail@tazj.in>
2021-12-15 r/3261 feat(ops/modules): Add shared module for TVL cacheGriffin Smith1-0/+19
Add a shared nixos module for configuring whitby as a binary nix cache, and refactor tverskoy to use this module. This is enabled via an option to pave the way for including it as an import in all depot-generated nixos configs at some point in the future. Change-Id: I6dcc0e8eb48b1ac34457666dceebeedd5da6c526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4344 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: wpcarro <wpcarro@gmail.com> Autosubmit: grfn <grfn@gws.fyi>
2021-12-14 r/3244 docs(ops/irccat): link to credentials RFEFlorian Klink1-0/+4
https://cl.tvl.fyi/c/depot/+/4264 did move merging config with secrets into ExecStart=, which is tracked in an RFE upstream: https://github.com/systemd/systemd/issues/19604#issuecomment-989279884 We didn't link to this so far, neither in the commit message, nor in a comment. Let's add a comment, so people know when we can undo this. Change-Id: I7bed370b671093bb876592b4dccd562f1c256cd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4326 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: grfn <grfn@gws.fyi>
2021-12-10 r/3204 fix(tvl-buildkite): Use supported credential helper binary nameVincent Ambo1-1/+1
Git only allows binary names prefixed with `git-credential-` if the path to the helper is not absolute. Why? Who knows. Change-Id: I216b2a621f62a73f05e21def7ec8016b29ede892
2021-12-10 r/3203 refactor(ops): Move panettone secrets to agenixVincent Ambo1-0/+1
Relates to b/161 Change-Id: I508e5a0eacab668f4bd39a2c888d894b96bed093
2021-12-10 r/3200 refactor(tvl-buildkite): Prepare gerrit credentials helperVincent Ambo1-0/+6
Currently this functionality is provided by a shell script stored in /etc/secrets (which has the password value hardcoded). This needs to happen in a separate commit from the one that changes the pipeline to avoid breaking it (it needs to be deployed first). Change-Id: I680754c828ccefbacfcf0d5c813a4bc19493ba4c
2021-12-10 r/3199 refactor(ops): Move Nix cache secret to agenixVincent Ambo1-1/+1
... and also the public key, just to keep the distribution mechanism the same. Change-Id: Ief14daf9344c0fb99eeb5789c1ec9bfb1f12bee0
2021-12-10 r/3198 refactor(ops): Use besadii configuration from agenixVincent Ambo2-2/+2
We already checked this in, but this commit adds the configuration for making use of it. There are two copies of besadii's JSON configuration with different permissions. Note that the buildkite-graphql-token path needs to be updated in static-pipeline.yml, but this needs to happen in a separate commit after deploy because the pipeline will break otherwise. Change-Id: I6fab4bf1a2e679df7cf76521e2b53bd9dadbac62
2021-12-10 r/3194 fix(tvl-buildkite): Add more missing programs to agent pathVincent Ambo1-0/+2
... this option really is a pitfall! The list of programs is now the same as in the upstream module, plus curl and jq. Change-Id: I29edae4b2400a2724f62df9efa1dc184a8b0af5f
2021-12-10 r/3191 fix(ops/irccat): Avoid permissions issue with LoadCredentials=Vincent Ambo1-9/+7
The DynamicUser + Group configuration does not work as planned, thus the systemd LoadCredentials feature is used instead which makes the file (which itself is only readable by root) available in a memory-backed location only readable by the service. The secret is only available to `ExecStart` commands, so units using this feature can not be used with pre/post units and the like if those commands need secrets. To accommodate this, the merge of configuration files has been moved into the service launch script, which is now the ExecStart= process. For details take a look at https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LoadCredential=ID:PATH Change-Id: I693fe5677cc0d63c7aa485c2c7472457c5262166
2021-12-10 r/3190 fix(tvl-buildkite): Explicitly set runtimePackagesVincent Ambo1-1/+5
It turns out the lib.mkAfter call doesn't behave as expected - only *some* of the packages that are defaulted end up in the $PATH. I suspect this is actually something else, e.g. these packages are always added for some reason or another, and the option is completely overridden every time. Change-Id: I854c7198520d82b00e6338ed0fe653836226dc6d
2021-12-10 r/3185 fix(tvl-buildkite): Add missing runtimePackages backVincent Ambo1-1/+5
Turns out that the type of this option is not concatenative and it replaces the packages needed to run Buildkite if set. Change-Id: I9f52572bc165bccdd8c6518cfdf7b8967f7a50d0
2021-12-10 r/3184 refactor(ops): Move irccat secret into agenixVincent Ambo1-2/+12
The irccat module uses DynamicUser, so to grant permission to it a new group has been added for irccat. I have some vague memory of DynamicUser + Group not behaving as one would expect, but we'll see what happens. Change-Id: Iab9f6a3f1a53c4133b635458ce173250cc9a3fac
2021-12-10 r/3176 refactor(ops): Move buildkite-agent-token into agenixVincent Ambo1-1/+1
Relates to b/161 Change-Id: I5d3a698d437928966d8b78ce9e0ba226c1437655
2021-12-10 r/3175 refactor(ops): Move owothia secret into agenixVincent Ambo1-1/+1
Relates to b/161 Change-Id: I25445281b0dd3c3f3660f8bb0d8337506a1e427b
2021-12-10 r/3174 refactor(ops): Move clbot secret into agenixVincent Ambo1-1/+7
Relates to b/161 Change-Id: I7badf22ff93bb4e8b06e4dd4a8bf880b0bd48f09
2021-12-10 r/3172 feat(ops/modules): Add module for running gerrit-queueVincent Ambo1-0/+51
This is not yet including the secret configuration for gerrit-queue, and just expects the secret (gerrit username & password) to be available in /etc/secrets. Change-Id: Ia465ef7f3f521c70d606d7fdeba9aa83c7e1b98b
2021-12-10 r/3171 chore(tvl-buildkite): Add jq and curl to agent pathsVincent Ambo1-0/+1
This is required for a simplification of the build pipeline (following CL) and needs to be in a separate commit as it can not be done atomically (merging the other commit to deploy it would immediately break pipelines otherwise). Change-Id: I5d8ec8f3238f79b5518d799486bf98d1d9516c43
2021-12-07 r/3148 fix(ops): Correctly pass command name to besadii invocationsVincent Ambo2-7/+7
Ensure that besadii sees $0 as the correct command name, since that is the sole mechanism by which its functionality is switched around. There was a lingering commit that introduced this bug and hadn't been deployed in a couple of days. Maybe time to tighten deploy cycles soon ... Change-Id: Ie4284c0f6e5e06d71a71a3702ec7e092260e0ce5
2021-12-02 r/3134 chore(ops/modules): Configure besadii call sites to load configVincent Ambo2-3/+14
On whitby, the besadii config will live in /etc/secrets/besadii.json. This CL updates the call sites to pass this config path to besadii so that it can load Sourcegraph configuration. Change-Id: Ia139b9fa3b827e7a5f2386214390acc6fe19a75a
2021-12-02 r/3132 fix(ops/nixery): Temporarily stop serving depot packages in NixeryVincent Ambo1-1/+1
Change the Nixery configuration to use the plain nixpkgs package path instead of the depot path. AFAIK, nobody uses this to fetches depot packages at the moment - but plenty of people fetch non-depot packages. This means that Nixery is cache-busted less often (previously on every commit => every deploy). We'll figure out another way to have a depot Nixery later. Change-Id: Iba632333346181c3d2ce992fbab396ed0d9f86aa
2021-12-01 r/3131 fix(ops/www): Redirect tvl.fyi/blog -> tvl.fyiVincent Ambo1-0/+4
The blog index page is at the root and people may manually edit the URL. Change-Id: I6cdaaaee6223524a9e950584379cfac34f8be160
2021-12-01 r/3125 feat(besadii): Support invocation as different Gerrit hooksVincent Ambo1-1/+2
Removes besadii support for the previously used 'ref-updated' hook and instead introduces support for the 'change-merged' and 'patchset-created' hooks. These hooks more accurately capture the semantics of when besadii should trigger CI builds and using them will avoid problems such as skipping 'canon' builds if chains of CLs are submitted together. Change-Id: Ib90356c069780bf0c0250e56b927e46a5b31ce7f
2021-11-30 r/3118 fix(ops/www): Strip `.html` from TVL blog post URLsVincent Ambo1-0/+8
Change-Id: I4d1f9284ec004931c07c04d614b01f28eedea508
2021-11-21 r/3078 fix(ops/restic): Move whitby's backup to GleSYS object storageVincent Ambo1-12/+16
Since GCP nuked us, the backups are now moving to GleSYS' S3-compatible object storage. This refactors the restic module to support S3-compatible storage instead of GCP, and switches to the appropriate new secret paths. The secrets were placed on whitby manually and I verified that the backups work. This fixes b/157 Change-Id: I6a9d2b0581967605ce736605a3befb44cdeae7e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3883 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2021-10-01 r/2946 feat(whitby): serve static.tvl.{fyi|su} with max cache settingsVincent Ambo1-0/+42
The setup is explained in the comment, but TL;DR: Use the derivation hash of static files to create permanent URLs. Relates to b/151. Change-Id: Ib1ca3a1a00c90a47f4bf39c29a8b4bbf5b215e7d Reviewed-on: https://cl.tvl.fyi/c/depot/+/3664 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2021-10-01 r/2941 feat(whitby): Serve //corp/website on tvl.suVincent Ambo1-0/+20
Change-Id: I21e1ddf9a32568cac8ad2595869ac8670867efa9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3658 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-09-24 r/2914 chore(ops/git-serving): Remove josh state from whitby backupsVincent Ambo1-2/+0
As cschilling explained on cl/3563, there isn't actually anything in this state that we *need* to persist. We're still keeping it in a persistent directory on disk as this serves as an optimisation after restarts of josh. Change-Id: Ia88886792a5acac34508b5b8a669bd519ca033de Reviewed-on: https://cl.tvl.fyi/c/depot/+/3631 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-18 r/2890 refactor(whitby): Move restic path configuration into modulesVincent Ambo3-0/+8
This lets each service declare their backup paths together with the configuration for the service, which is a lot more sensible than what we had before. Fixes b/147 Change-Id: If76fe62639f4cc0e6fbb63a2959d584479d8f0fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/3583 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-18 r/2889 fix(ops/restic): types.string -> types.strVincent Ambo1-3/+3
I can never remember which is which. Change-Id: I69b8235862b8c5b49030a74bfca25aaa113273b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3582 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-16 r/2878 refactor(ops/restic): Move restic configuration into a new moduleVincent Ambo1-0/+57
Relates to b/147. First step towards giving depot modules the ability to declare their own backup directories by moving all restic configuration into a new module and adding a NixOS option for inclusion/exclusion paths for backups. This still keeps all backup paths within the whitby config. Change-Id: Ia96833668f1a3d02da892261153d8b02156b8ac0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3565 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2021-09-16 r/2877 feat(git-serving): Configure josh to serve the depot over HTTPVincent Ambo2-2/+66
Previously we served the dumb git HTTP protocol from code.tvl.fyi via cgit. This CL disables this feature and instead runs josh in the same location (by redirecting appropriately), but while also enabling partial cloning of all subtrees of the depot. For example, after this CL the following would result in an independent clone of //nix/readTree: git clone https://code.tvl.fyi/depot.git:/nix/readTree.git Note that there are no josh workspaces configured at all for now, these references are only for static depot subpaths. Please refer to the documentation for josh for more information on available kinds of josh filters. Josh state is kept in a systemd state directory in /var/lib/josh and backed up to Restic. Backing this up is necessary, as josh uses stateful information to do things like tracking merges and rewriting history per subtree appropriately to avoid cloned repositories ending up in peculiar states. Change-Id: I156f0298c2aa42e3bdbf5a0e86109070d640c56e Reviewed-on: https://cl.tvl.fyi/c/depot/+/3563 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2021-09-11 r/2847 fix(deploys.*): Folder for diffs is in /diff/Vincent Ambo1-1/+1
... this was missing before. Change-Id: I5b79cb78665f24fdb7cc6496e3782d3940dc77b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3527 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 r/2846 feat(sourcegraph): Upgrade 3.30.4 -> 3.31.2Vincent Ambo1-1/+1
This one seems a little more involved: https://docs.sourcegraph.com/admin/migration/3_31 I believe we skip that corruption issue in the previous CL though, by simply never deploying a version with that weird broken image. See b/144 Change-Id: I3bbf1b719d00905e08a92011ace5485467f504ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/3525 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2845 feat(sourcegraph): Upgrade 3.29.1 -> 3.30.4Vincent Ambo1-1/+1
See b/144 Change-Id: Ied9490f3ce6fb3fda8cbb9983416b02ea451fb44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3524 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2844 feat(sourcegraph): Upgrade 3.28.0 -> 3.29.1Vincent Ambo1-1/+1
See b/144 Change-Id: Ia62d4cbf581caaefa0dba455376eec60b8c817d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3523 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2843 fix(sourcegraph): Temporarily comment out our syntax highlighterVincent Ambo1-1/+2
We changed away from the default sourcegraph one because it didn't support Nix, but it seems that there's been a change in the interaction protocol. Change-Id: I3a2691df6a87672cf83b819143f25d93d9cd6d13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3531 Tested-by: BuildkiteCI Reviewed-by: eta <tvl@eta.st> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 r/2842 feat(sourcegraph): Upgrade 3.27.5 -> 3.28.0Vincent Ambo1-1/+1
See b/144 Change-Id: Ia09ad2af6043dcac6681c549103d1e6f52b4e0a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3522 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 r/2841 feat(sourcegraph): Upgrade 3.26.0 -> 3.27.5Vincent Ambo1-1/+1
See b/144 Change-Id: I50d417c51b05bafcd3fe7e285f30079db8be499a Reviewed-on: https://cl.tvl.fyi/c/depot/+/3521 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-10 r/2837 feat(whitby): Serve static HTML dir for deploys.tvl.fyiGriffin Smith1-0/+20
Add a new domain and nginx virtual host at deploys.tvl.fyi, serving out of a static directory on whitby which is created by systemd-tmpfiles. This will be used to serve diffs rendered by nix-diff for pending deploys for whitby Since this contains stateful data, it is added to the restic backups on whitby. Refs: b/110 Change-Id: I5869d40800bbf5fb8fb39878a857f66ff5787830 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3144 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-17T03·39+0000