feat(whitby): switch from nix-serve to harmonia for the cache r/8721

Harmonia is, ostensibly, faster and better and, most importantly, not a giant
pile of wonky Perl.

I've tested locally that Harmonia works with Nix 2.3 (on both ends), so I think
we should be good to go here.

We have a vendored copy of the upstream module for now. We need to fix Nix 2.3
compatibility in upstream for the module, but the service itself works fine.

Change-Id: I3897bb02b83bd466b6fe7077c05728ac49ea4406
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12517
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>

1 files changed, 14 insertions, 5 deletions

diff --git a/ops/machines/whitby/default.nix b/ops/machines/whitby/default.nix
index 3181ccde451c..06add2168160 100644
--- a/ops/machines/whitby/default.nix
+++ b/ops/machines/whitby/default.nix
@@ -14,6 +14,7 @@ in
     (mod "cheddar.nix")
     (mod "clbot.nix")
     (mod "gerrit-autosubmit.nix")
+    (mod "harmonia.nix")
     (mod "irccat.nix")
     (mod "josh.nix")
     (mod "journaldriver.nix")
@@ -229,12 +230,17 @@ in
       grafana.file = secretFile "grafana";
       irccat.file = secretFile "irccat";
       keycloak-db.file = secretFile "keycloak-db";
-      nix-cache-priv.file = secretFile "nix-cache-priv";
       owothia.file = secretFile "owothia";
       panettone.file = secretFile "panettone";
       smtprelay.file = secretFile "smtprelay";
       teleirc.file = secretFile "teleirc";
 
+      nix-cache-priv = {
+        file = secretFile "nix-cache-priv";
+        mode = "0440";
+        group = "harmonia";
+      };
+
       buildkite-agent-token = {
         file = secretFile "buildkite-agent-token";
         mode = "0440";
@@ -477,11 +483,14 @@ in
     ];
   };
 
-  services.nix-serve = {
+  # Run a Harmonia binary cache.
+  #
+  # TODO(tazjin): switch to upstream module after fix for Nix 2.3
+  services.depot.harmonia = {
     enable = true;
-    port = 6443;
-    secretKeyFile = config.age.secretsDir + "/nix-cache-priv";
-    bindAddress = "localhost";
+    signKeyPaths = [ (config.age.secretsDir + "/nix-cache-priv") ];
+    settings.bind = "127.0.0.1:6443";
+    settings.priority = 50;
   };
 
   services.fail2ban.enable = true;