feat(sterni/nint): shebang interpreter for nix scripts r/2393

nint (short for nix interpreter) is a tiny wrapper around nix-instantiate
which allows to run nix scripts, i. e. nix expressions that conform to
a certain calling convention. A nix script runnable using nint must
conform to the following constraints:

* It must evaluate to a function which has a set pattern with an
  ellipsis as the single argument.

* It must produce a string as a return value or fail.

When invoked, a the expression receives the following arguments:

* `currentDir`: the current working directory as a nix path

* `argv`: a list of strings containing `argv` including `argv[0]`

* extra arguments which are manually specified which allows for
  passing along dependencies or libraries, for example:

    nint --arg depot '(import /depot {})' my-prog.nix [ argv[1] … ]

  would pass along depot to be used in `my-prog.nix`.

Such nix scripts are purely functional in a sense: The way inputs can be
taken is very limited and causing effects is also only possible in a
very limited sense (using builtins.fetchurl if TARBALL_TTL is 0,
adding files and directories to the nix store, realising derivations).
As an approximation, a program executed using nint can be thought of
as a function with the following signature:

  λ :: environment → working directory → argv → stdout

where environment includes:

* the time at the start of the program (`builtins.currentTime`)
* other information about the machine (`builtins.currentSystem` …)
* environment variables (`builtins.getEnv`)
* the file system (`builtins.readDir`, `builtins.readFile`, …) which
  is the biggest input impurity as it may change during evaluation

Additionally import from derivation and builtin fetchers are available
which introduce further impurities to be utilized.

Future work:

* Streaming I/O via lazy lists. This would allow usage of
  stdin and output before the program terminates. However this would
  require using libexpr directly or writing a custom nix interpreter.

  A description of how this would work can be found on the website of the
  esoteric programming language Lazy K: https://tromp.github.io/cl/lazy-k.html

* An effect system beyond stdin / stdout.

* Better error handling, support setting exit codes etc.

These features would require either using an alternative or custom
interpreter for nix (tvix or hnix) or to link against libexpr directly
to have more control over evaluation.

Change-Id: I61528516eb418740df355852f23425acc4d0656a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2745
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI

3 files changed, 244 insertions, 0 deletions

diff --git a/users/sterni/nint/README.md b/users/sterni/nint/README.md
new file mode 100644
index 000000000000..ddd8045f7378
--- /dev/null
+++ b/users/sterni/nint/README.md
@@ -0,0 +1,85 @@
+# nint — Nix INTerpreter
+
+`nint` is a shebang compatible interpreter for nix. It is currently
+implemented as a fairly trivial wrapper around `nix-instantiate --eval`.
+It allows to run nix expressions as command line tools if they conform
+to the following calling convention:
+
+* Every nix script needs to evaluate to a function which takes an
+  attribute set as its single argument. Ideally a set pattern with
+  an ellipsis should be used. By default `nint` passes the following
+  arguments:
+
+  * `currentDir`: the current working directory as a nix path
+  * `argv`: a list of arguments to the invokation including the
+    program name at `builtins.head argv`.
+  * Extra arguments can be manually passed as described below.
+
+* The return value should always be a string (throwing is also okay)
+  which is printed to stdout by `nint`.
+
+## Usage
+
+```
+nint [ --arg ARG VALUE … ] script.nix [ ARGS … ]
+```
+
+Instead of `--arg`, `--argstr` can also be used. They both work
+like the flags of the same name for `nix-instantiate` and may
+be specified any number of times as long as they are passed
+*before* the nix expression to run.
+
+Below is a shebang which also passes `depot` as an argument
+(note the usage of `env -S` to get around the shebang limitation
+to two arguments).
+
+```nix
+#!/usr/bin/env -S nint --arg depot /path/to/depot
+```
+
+## Limitations
+
+* No side effects except for writing to `stdout`.
+
+* Output is not streaming, i. e. even if the output is incrementally
+  calculated, nothing will be printed until the full output is available.
+  With plain nix strings we can't do better anyways.
+
+* Limited error handling for the script, no way to set the exit code etc.
+
+Some of these limitations may be possible to address in the future by using
+an alternative nix interpreter and a more elaborate calling convention.
+
+## Example
+
+Below is a (very simple) implementation of a `ls(1)`-like program in nix:
+
+```nix
+#!/usr/bin/env nint
+{ currentDir, argv, ... }:
+
+let
+  lib = import <nixpkgs/lib>;
+
+  dirs =
+    let
+      args = builtins.tail argv;
+    in
+      if args == []
+      then [ currentDir ]
+      else args;
+
+  makeAbsolute = p:
+    if builtins.isPath p
+    then p
+    else if builtins.match "^/.*" p != null
+    then p
+    else "${toString currentDir}/${p}";
+in
+
+  lib.concatStringsSep "\n"
+    (lib.flatten
+      (builtins.map
+        (d: (builtins.attrNames (builtins.readDir (makeAbsolute d))))
+        dirs)) + "\n"
+```
diff --git a/users/sterni/nint/default.nix b/users/sterni/nint/default.nix
new file mode 100644
index 000000000000..69ca7283a50f
--- /dev/null
+++ b/users/sterni/nint/default.nix
@@ -0,0 +1,49 @@
+{ depot, pkgs, ... }:
+
+let
+  inherit (depot.users.Profpatsch.writers)
+    rustSimpleBin
+    ;
+
+  inherit (pkgs)
+    buildRustCrate
+    ;
+
+  serde = buildRustCrate {
+    pname = "serde";
+    crateName = "serde";
+    version = "1.0.123";
+    sha256 = "05xl2s1vpf3p7fi2yc9qlzw88d5ap0z3qmhmd7axa6pp9pn1s5xc";
+    features = [ "std" ];
+  };
+
+  ryu = buildRustCrate {
+    pname = "ryu";
+    version = "1.0.5";
+    crateName = "ryu";
+    sha256 = "060y2ln1csix593ingwxr2y3wl236ls0ly1ffkv39h5im7xydhrc";
+  };
+
+  itoa = buildRustCrate {
+    pname = "itoa";
+    version = "0.4.7";
+    crateName = "itoa";
+    sha256 = "0079jlkcmcaw37wljrvb6r3dqq15nfahkqnl5npvlpdvkg31k11x";
+  };
+
+  serde_json = buildRustCrate {
+    pname = "serde_json";
+    version = "1.0.62";
+    crateName = "serde_json";
+    sha256 = "0sgc8dycigq0nxr4j613m4q733alfb2i10s6nz80lsbbqgrka21q";
+    dependencies = [ serde ryu itoa ];
+    features = [ "std" ];
+    edition = "2018";
+  };
+
+in
+
+  rustSimpleBin {
+    name = "nint";
+    dependencies = [ serde_json ];
+  } (builtins.readFile ./nint.rs)
diff --git a/users/sterni/nint/nint.rs b/users/sterni/nint/nint.rs
new file mode 100644
index 000000000000..3d430612851a
--- /dev/null
+++ b/users/sterni/nint/nint.rs
@@ -0,0 +1,110 @@
+extern crate serde_json;
+
+use serde_json::Value;
+use std::ffi::OsString;
+use std::os::unix::ffi::{OsStringExt, OsStrExt};
+use std::io::{Error, ErrorKind, Write};
+use std::process::Command;
+
+fn render_nix_string(s: &OsString) -> OsString {
+    let mut rendered = Vec::new();
+
+    rendered.extend(b"\"");
+
+    for b in s.as_os_str().as_bytes() {
+        match char::from(*b) {
+            '\"' => rendered.extend(b"\\\""),
+            '\\' => rendered.extend(b"\\\\"),
+            '$'  => rendered.extend(b"\\$"),
+            _    => rendered.push(*b),
+        }
+    }
+
+    rendered.extend(b"\"");
+
+    OsString::from_vec(rendered)
+}
+
+fn render_nix_list(arr: &[OsString]) -> OsString {
+    let mut rendered = Vec::new();
+
+    rendered.extend(b"[ ");
+
+    for el in arr {
+        rendered.extend(render_nix_string(el).as_os_str().as_bytes());
+        rendered.extend(b" ");
+    }
+
+    rendered.extend(b"]");
+
+    OsString::from_vec(rendered)
+}
+
+fn main() -> std::io::Result<()> {
+    let mut nix_args = Vec::new();
+
+    let mut args = std::env::args_os().into_iter();
+    let mut in_args = true;
+
+    let mut argv: Vec<OsString> = Vec::new();
+
+    // skip argv[0]
+    args.next();
+
+    loop {
+        let arg = match args.next() {
+            Some(a) => a,
+            None => break,
+        };
+
+        if !arg.to_str().map(|s| s.starts_with("-")).unwrap_or(false) {
+            in_args = false;
+        }
+
+        if in_args {
+            match(arg.to_str()) {
+                Some("--arg") | Some("--argstr") => {
+                    nix_args.push(arg);
+                    nix_args.push(args.next().unwrap());
+                    nix_args.push(args.next().unwrap());
+                    Ok(())
+                }
+                _ => Err(Error::new(ErrorKind::Other, "unknown argument")),
+            }?
+        } else {
+            argv.push(arg);
+        }
+    }
+
+    if argv.len() < 1 {
+        Err(Error::new(ErrorKind::Other, "missing argv"))
+    } else {
+        let cd = std::env::current_dir()?.into_os_string();
+
+        nix_args.push(OsString::from("--arg"));
+        nix_args.push(OsString::from("currentDir"));
+        nix_args.push(cd);
+
+        nix_args.push(OsString::from("--arg"));
+        nix_args.push(OsString::from("argv"));
+        nix_args.push(render_nix_list(&argv[..]));
+
+        nix_args.push(OsString::from("--eval"));
+        nix_args.push(OsString::from("--json"));
+
+        nix_args.push(argv[0].clone());
+
+        let run = Command::new("nix-instantiate")
+                          .args(nix_args)
+                          .output()?;
+
+        match serde_json::from_slice(&run.stdout[..]) {
+            Ok(Value::String(s)) => Ok(print!("{}", s)),
+            Ok(_) => Err(Error::new(ErrorKind::Other, "output must be a string")),
+            _ => {
+                std::io::stderr().write_all(&run.stderr[..]);
+                Err(Error::new(ErrorKind::Other, "internal nix error"))
+            },
+        }
+    }
+}