feat(ops): introduce (head|tail)scale server at net.tvl.fyi r/6343

This runs a headscale server on sanduny which lets users join their
machines to the TVL tailscale network.

This would theoretically let people communicate with each other on the
internal network, but also more notably joined servers can advertise
exit node capability so that we can have our own "VPN network", for
starters with endpoints in Germany, UK and Russia (whitby, sanduny and
koptevo respectively).

This setup isn't fully stable yet, notably:

* The IP range used by tailscale is just the default one right now,
  I'm not sure if that should be changed or what.

* The system is stateful (on sanduny), but the state is not (yet)
  backed up anywhere. Use with caution.

* Machine joining is a manual process requiring SSH & root access to
  sanduny.

  The process is to log in to sanduny, then get a headscale shell with
  `sudo -u headscale bash`, and to use the `headscale` CLI within
  there to administrate access.

  I've opted to create a user account `tvl` for TVL-owned machines,
  and a personal account for myself and my machines.

Change-Id: I4f1be1fe8062a6c2e77203ff72fe8709f4e4dec8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8837
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI

3 files changed, 76 insertions, 0 deletions

diff --git a/ops/machines/sanduny/default.nix b/ops/machines/sanduny/default.nix
index 14f40d1fa09e..ba14fbd32a60 100644
--- a/ops/machines/sanduny/default.nix
+++ b/ops/machines/sanduny/default.nix
@@ -20,6 +20,7 @@ in
     (mod "journaldriver.nix")
     (mod "known-hosts.nix")
     (mod "tvl-cache.nix")
+    (mod "tvl-headscale.nix")
     (mod "tvl-users.nix")
     (mod "www/inbox.tvl.su.nix")
     (mod "www/self-redirect.nix")
@@ -71,6 +72,13 @@ in
   services.openssh.enable = true;
   services.fail2ban.enable = true;
 
+  # Run tailscale for the TVL net.tvl.fyi network.
+  # tailscale up --login-server https://net.tvl.fyi --accept-dns=false --advertise-exit-node
+  services.tailscale = {
+    enable = true;
+    useRoutingFeatures = "server"; # for exit-node usage
+  };
+
   # Automatically collect garbage from the Nix store.
   services.depot.automatic-gc = {
     enable = true;
diff --git a/ops/machines/whitby/default.nix b/ops/machines/whitby/default.nix
index f2a459047163..c7874cd2352d 100644
--- a/ops/machines/whitby/default.nix
+++ b/ops/machines/whitby/default.nix
@@ -624,6 +624,12 @@ in
     };
   };
 
+  # Join TVL Tailscale network at net.tvl.fyi
+  services.tailscale = {
+    enable = true;
+    useRoutingFeatures = "server"; # for exit-node usage
+  };
+
   # Allow Keycloak access to the LDAP module by forcing in the JVM
   # configuration
   systemd.services.keycloak.environment.PREPEND_JAVA_OPTS =
diff --git a/ops/modules/tvl-headscale.nix b/ops/modules/tvl-headscale.nix
new file mode 100644
index 000000000000..a07021c78861
--- /dev/null
+++ b/ops/modules/tvl-headscale.nix
@@ -0,0 +1,62 @@
+# Configuration for the coordination server for net.tvl.fyi, a
+# tailscale network run using headscale.
+#
+# All TVL members can join this network, which provides several exit
+# nodes through which traffic can be routed.
+#
+# The coordination server is currently run on sanduny.tvl.su. It is
+# managed manually, ping somebody with access ... for access.
+#
+# Servers should join using approximately this command:
+#   tailscale up --login-server https://net.tvl.fyi --accept-dns=false --advertise-exit-node
+#
+# Clients should join using approximately this command:
+#   tailscale up --login-server https://net.tvl.fyi --accept-dns=false
+{ config, pkgs, ... }:
+
+{
+  # TODO(tazjin): run embedded DERP server
+  services.headscale = {
+    enable = true;
+    port = 4725; # hscl
+
+    settings = {
+      server_url = "https://net.tvl.fyi";
+      dns_config.nameservers = [
+        "8.8.8.8"
+        "1.1.1.1"
+        "77.88.8.8"
+      ];
+
+      # TLS is handled by nginx
+      tls_cert_path = null;
+      tls_key_path = null;
+    };
+  };
+
+  environment.systemPackages = [ pkgs.headscale ]; # admin CLI
+
+  services.nginx.virtualHosts."net.tvl.fyi" = {
+    serverName = "net.tvl.fyi";
+    enableACME = true;
+    forceSSL = true;
+
+    # See https://github.com/juanfont/headscale/blob/v0.22.3/docs/reverse-proxy.md#nginx
+    extraConfig = ''
+      location / {
+        proxy_pass http://localhost:${toString config.services.headscale.port};
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection $connection_upgrade;
+        proxy_set_header Host $server_name;
+        proxy_redirect http:// https://;
+        proxy_buffering off;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
+        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
+      }
+    '';
+  };
+
+}