docs(nix/buildLisp): document recent changes to buildLisp r/2774

Doing this in a separate CL to avoid having to track the intermediate
changes no one will ever see in documentation as well which would be
unnecessary effort.

* Multi-implementation support introduced in cl/3292 and refined in
  cl/3368 in terms of the user interface.

* Implementation specific srcs and deps introduced in cl/3321

* Implementation passthru attrs and rename from .sbcl -> .repl was done
  in cl/3359

* ECL added in cl/3297, CCL in cl/3350

Change-Id: Ia13f2aea4e7e091c00991fcbfc601de364413979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3380
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>

1 files changed, 119 insertions, 10 deletions

diff --git a/nix/buildLisp/README.md b/nix/buildLisp/README.md
index 5ee560e056..76fe5ca9e4 100644
--- a/nix/buildLisp/README.md
+++ b/nix/buildLisp/README.md
@@ -6,18 +6,20 @@ This is a build system for Common Lisp, written in Nix.
 It aims to offer an alternative to ASDF for users who live in a
 Nix-based ecosystem. This offers several advantages over ASDF:
 
-* Simpler (logic-less) package definitions
+* Simpler (almost logic-less) package definitions
 * Easy linking of native dependencies (from Nix)
 * Composability with Nix tooling for other languages
 * Effective, per-system caching strategies
 * Easy overriding of dependencies and whatnot
+* Convenient support for multiple Common Lisp implementations
 * ... and more!
 
 The project is still in its early stages and some important
 restrictions should be highlighted:
 
-* Only SBCL is supported (though the plan is to add support for at
-  least ABCL and Clozure CL, and maybe make it extensible)
+* Extending `buildLisp` with support for a custom implementation
+  currently requires some knowledge of internals and may not be
+  considered stable yet.
 * Parallel compilation is not possible: Since buildLisp doesn't encode
   dependencies between components (i. e. source files) like ASDF,
   it must compile source files in sequence to avoid errors due to
@@ -36,6 +38,7 @@ restrictions should be highlighted:
   | `deps`    | `list<drv>`  | List of dependencies          | no        |
   | `native`  | `list<drv>`  | List of native dependencies   | no        |
   | `test`    | see "Tests"  | Specification for test suite  | no        |
+  | `implementation` | see "Implementations" | Common Lisp implementation to use | no |
 
   The output of invoking this is a directory containing a FASL file
   that is the concatenated result of all compiled sources.
@@ -50,6 +53,7 @@ restrictions should be highlighted:
   | `native`  | `list<drv>`  | List of native dependencies   | no        |
   | `main`    | `string`     | Entrypoint function           | no        |
   | `test`    | see "Tests"  | Specification for test suite  | no        |
+  | `implementation` | see "Implementations" | Common Lisp implementation to use | no |
 
   The `main` parameter should be the name of a function and defaults
   to `${name}:main` (i.e. the *exported* `main` function of the
@@ -68,13 +72,6 @@ restrictions should be highlighted:
   built-in library and returns a "package" that simply requires this
   library.
 
-* `buildLisp.sbclWith`: Creates an SBCL pre-loaded with various dependencies.
-
-  This function takes a single argument which is a list of Lisp
-  libraries programs or programs. It creates an SBCL that is
-  pre-loaded with all of that Lisp code and can be used as the host
-  for e.g. Sly or SLIME.
-
 ## Tests
 
 Both `buildLisp.library` and `buildLisp.program` take an optional argument
@@ -91,6 +88,113 @@ the `expression` parameter should be a Lisp expression and will be evaluated
 after loading all sources and dependencies (including library/program
 dependencies). It must return a non-`NIL` value if the test suite has passed.
 
+## Development REPLs
+
+`buildLisp` builds loadable variants of both `program` and `library` derivations
+(usually FASL files). Therefore it can provide a convenient way to obtain an
+instance of any implementation preloaded with `buildLisp`-derivations. This
+is especially useful to use as a host for Sly or SLIME.
+
+* `buildLisp.sbcl.lispWith`, `buildLisp.ccl.lispWith`, ...:
+  Creates a wrapper script preloading a Lisp implementation with various dependencies.
+
+  This function takes a single argument which is a list of Lisp
+  libraries programs or programs. The desired Lisp implementation
+  will load all given derivations and all their dependencies on
+  startup.
+
+  The shortcut `buildLisp.sbclWith` for `buildLisp.sbcl.lispWith` is also provided.
+
+* `repl` passthru attribute: `derivation.repl` is provided as a shortcut
+  for `buildLisp.${implementationName}.lispWith [ derivation ]`.
+  `derivation.ccl.repl`, `derivation.sbcl.repl` etc. work as well, of course
+  (see also "Implementations" section).
+
+## Implementations
+
+Both `buildLisp.library` and `buildLisp.program` allow specifying a different
+Common Lisp implementation than the default one (which is SBCL). When an
+implementation is passed, `buildLisp` makes sure all dependencies are built
+with that implementation as well since build artifacts from different
+implementation will be incompatible with each other.
+
+The argument taken by `implementation` is a special attribute set which
+describes how to do certain tasks for a given implementation, like building
+or loading a library. In case you want to use a custom implementation
+description, the precise structure needed is documented in `buildLisp`'s
+source code for now. `buildLisp` also exposes the following already
+working implementation sets:
+
+* `buildLisp.sbcl`: [SBCL][sbcl], our default implementation
+
+* `buildLisp.ccl`: [CCL][ccl], similar to SBCL, but with very good macOS support
+
+* `buildLisp.ecl`: [ECL][ecl] setup to produce statically linked binaries and
+  libraries. Note that its runtime library is LGPL, so [extra conditions][lgpl-static]
+  must be fulfilled when distributing binaries produced this way.
+
+* Support for ABCL is planned.
+
+For every of these “known” implementations, `buildLisp` will create a `passthru`
+attribute named like the implementation which points to a variant of the derivation
+built with said implementation. Say we have a derivation, `myDrv`, built using SBCL:
+While `myDrv` and `myDrv.sbcl` are built using SBCL, `myDrv.ecl`, `myDrv.ccl` etc.
+build the derivation and all its dependencies using ECL and CCL respectively.
+
+This is useful to test portability of your derivation, but is also used internally
+to speed up the “normalization” of the dependency graph. Thus it is important to
+make sure that your custom implementation's name doesn't clash with one of the
+“known” ones.
+
+## Handling Implementation Specifics
+
+When targeting multiple Common Lisp implementation, it is often necessary to
+handle differing interfaces for OS interaction or to make use of special
+implementation features. For this reason, `buildLisp` allows specifying
+dependencies and source files for specific implementations only. This can
+be utilized by having an attribute set in the list for the `deps` or `srcs`
+argument: `buildLisp` will pick the value of the attribute named like the
+used implementation or `default` and ignore the set completely if both
+are missing.
+
+```nix
+{ buildLisp, lispPkgs }:
+
+buildLisp.library {
+  name = "mylib";
+
+  srcs = [
+    # These are included always of course
+    ./package.lisp
+    ./portable-lib.lisp
+
+    # Choose right impl-* file
+    {
+      sbcl = ./impl-sbcl.lisp;
+      ccl = ./impl-ccl.lisp;
+      ecl = ./impl-ecl.lisp;
+    }
+
+    # We can also use this to inject extra files
+    { ecl = ./extra-ecl-optimizations.lisp; }
+  ];
+
+  deps = [
+    # Use SBCL's special bundled package, flexi-streams otherwise
+    {
+      sbcl = buildLisp.bundled "sb-rotate-byte";
+      default = lispPkgs.flexi-streams;
+    }
+  ];
+}
+```
+
+Additionally a `brokenOn` parameter is accepted which takes a list of
+implementation names on which the derivation is not expected to work.
+This only influences `meta.targets` which is read by depot's CI to
+check which variants (see "Implementations") of the derivation to
+build, so it may not be useful outside of depot.
+
 ## Example
 
 Using buildLisp could look like this:
@@ -119,3 +223,8 @@ in buildLisp.program {
     };
 }
 ```
+
+[sbcl]: http://www.sbcl.org/
+[ccl]: https://ccl.clozure.com/
+[ecl]: https://common-lisp.net/project/ecl/
+[lgpl-static]: https://www.gnu.org/licenses/gpl-faq.en.html#LGPLStaticVsDynamic