Restructuring the Nix manual

1 files changed, 399 insertions, 0 deletions

diff --git a/doc/manual/command-ref/nix-push.xml b/doc/manual/command-ref/nix-push.xml
new file mode 100644
index 000000000000..9c6cdfa2f268
--- /dev/null
+++ b/doc/manual/command-ref/nix-push.xml
@@ -0,0 +1,399 @@
+<refentry xmlns="http://docbook.org/ns/docbook"
+      xmlns:xlink="http://www.w3.org/1999/xlink"
+      xmlns:xi="http://www.w3.org/2001/XInclude"
+      version="5.0"
+      xml:id="sec-nix-push">
+
+<refmeta>
+  <refentrytitle>nix-push</refentrytitle>
+  <manvolnum>1</manvolnum>
+  <refmiscinfo class="source">Nix</refmiscinfo>
+  <refmiscinfo class="version"><xi:include href="version.txt" parse="text"/></refmiscinfo>
+</refmeta>
+
+<refnamediv>
+  <refname>nix-push</refname>
+  <refpurpose>generate a binary cache</refpurpose>
+</refnamediv>
+
+<refsynopsisdiv>
+  <cmdsynopsis>
+    <command>nix-push</command>
+    <arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg>
+    <arg><option>--bzip2</option></arg>
+    <arg><option>--none</option></arg>
+    <arg><option>--force</option></arg>
+    <arg><option>--link</option></arg>
+    <arg><option>--manifest</option></arg>
+    <arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg>
+    <arg><option>--url-prefix</option> <replaceable>url</replaceable></arg>
+    <arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
+  </cmdsynopsis>
+</refsynopsisdiv>
+
+
+<refsection><title>Description</title>
+
+<para>The command <command>nix-push</command> produces a
+<emphasis>binary cache</emphasis>, a directory containing compressed
+Nix archives (NARs) plus some metadata of the closure of the specified
+store paths.  This directory can then be made available through a web
+server to other Nix installations, allowing them to skip building from
+source and instead download binaries from the cache
+automatically.</para>
+
+<para><command>nix-push</command> performs the following actions.
+      
+<orderedlist>
+
+  <listitem><para>Each path in <replaceable>paths</replaceable> is
+  built (using <link
+  linkend='rsec-nix-store-realise'><command>nix-store
+  --realise</command></link>).</para></listitem>
+
+  <listitem><para>All paths in the closure of
+  <replaceable>paths</replaceable> are determined (using
+  <command>nix-store --query --requisites
+  --include-outputs</command>).  Note that since the
+  <option>--include-outputs</option> flag is used, if
+  <replaceable>paths</replaceable> includes a store derivation, you
+  get a combined source/binary distribution (e.g., source tarballs
+  will be included).</para></listitem>
+
+  <listitem><para>All store paths determined in the previous step are
+  packaged into a NAR (using <command>nix-store --dump</command>) and
+  compressed using <command>xz</command> or <command>bzip2</command>.
+  The resulting files have the extension <filename>.nar.xz</filename>
+  or <filename>.nar.bz2</filename>.  Also for each store path, Nix
+  generates a file with extension <filename>.narinfo</filename>
+  containing metadata such as the references, cryptographic hash and
+  size of each path.</para></listitem>
+
+  <listitem><para>Optionally, a single <emphasis>manifest</emphasis>
+  file is created that contains the same metadata as the
+  <filename>.narinfo</filename> files.  This is for compatibility with
+  Nix versions prior to 1.2 (see <command>nix-pull</command> for
+  details).</para></listitem>
+
+  <listitem><para>A file named <option>nix-cache-info</option> is
+  placed in the destination directory.  The existence of this file
+  marks the directory as a binary cache.</para></listitem>
+
+</orderedlist>
+
+</para>
+
+</refsection>
+
+
+<refsection><title>Options</title>
+
+<variablelist>
+
+  <varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term>
+
+    <listitem><para>Set the destination directory to
+    <replaceable>dir</replaceable>, which is created if it does not
+    exist.  This flag is required.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--bzip2</option></term>
+
+    <listitem><para>Compress NARs using <command>bzip2</command>
+    instead of <command>xz -9</command>.  The latter compresses about
+    30% better on typical archives, decompresses about twice as fast,
+    but compresses a lot slower and is not supported by Nix prior to
+    version 1.2.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--none</option></term>
+
+    <listitem><para>Do not compress NARs.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--force</option></term>
+
+    <listitem><para>Overwrite <filename>.narinfo</filename> files if
+    they already exist.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--link</option></term>
+
+    <listitem><para>By default, NARs are generated in the Nix store
+    and then copied to <replaceable>dest-dir</replaceable>.  If this
+    option is given, hard links are used instead.  This only works if
+    <replaceable>dest-dir</replaceable> is on the same filesystem as
+    the Nix store.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--manifest</option></term>
+
+    <listitem><para>Force the generation of a manifest suitable for
+    use by <command>nix-pull</command>.  The manifest is stored as
+    <filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term>
+
+    <listitem><para>Like <option>--manifest</option>, but store the
+    manifest in <replaceable>filename</replaceable>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term>
+
+    <listitem><para>Manifests are expected to contain the absolute
+    URLs of NARs.  For generating these URLs, the prefix
+    <replaceable>url</replaceable> is used.  It defaults to
+    <uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
+</refsection>
+
+
+<refsection><title>Examples</title>
+
+<para>To add the closure of Thunderbird to a binary cache:
+
+<screen>
+$ nix-push --dest /tmp/cache $(nix-build -A thunderbird)
+</screen>
+
+Assuming that <filename>/tmp/cache</filename> is exported by a web
+server as <uri>http://example.org/cache</uri>, you can then use this
+cache on another machine to speed up the installation of Thunderbird:
+
+<screen>
+$ nix-build -A thunderbird --option binary-caches http://example.org/cache
+</screen>
+
+Alternatively, you could add <literal>binary-caches =
+http://example.org/cache</literal> to
+<filename>nix.conf</filename>.</para>
+
+<para>To also include build-time dependencies (such as source
+tarballs):
+
+<screen>
+$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird)
+</screen>
+
+</para>
+
+<para>To generate a manifest suitable for <command>nix-pull</command>:
+
+<screen>
+$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest
+</screen>
+
+On another machine you can then do:
+
+<screen>
+$ nix-pull http://example.org/cache
+</screen>
+
+to cause the binaries to be used by subsequent Nix operations.</para>
+
+</refsection>
+
+
+<refsection><title>Binary cache format and operation</title>
+
+<para>A binary cache with URL <replaceable>url</replaceable> only
+denotes a valid binary cache if the file
+<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists.  If
+this file does not exist (or cannot be downloaded), the cache is
+ignored.  If it does exist, it must be a text file containing cache
+properties.  Here’s an example:
+
+<screen>
+StoreDir: /nix/store
+WantMassQuery: 1
+Priority: 10
+</screen>
+
+The properties that are currently supported are:
+
+<variablelist>
+  
+  <varlistentry><term><literal>StoreDir</literal></term>
+
+    <listitem><para>The path of the Nix store to which this binary
+    cache applies.  Binaries are not relocatable — a binary built for
+    <filename>/nix/store</filename> won’t generally work in
+    <filename>/home/alice/store</filename> — so to prevent binaries
+    from being used in a wrong store, a binary cache is only used if
+    its <literal>StoreDir</literal> matches the local Nix
+    configuration.  The default is
+    <filename>/nix/store</filename>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>WantMassQuery</literal></term>
+
+    <listitem><para>Query operations such as <command>nix-env
+    -qas</command> can cause thousands of cache queries, and thus
+    thousands of HTTP requests, to determine which packages are
+    available in binary form.  While these requests are small, not
+    every server may appreciate a potential onslaught of queries.  If
+    <literal>WantMassQuery</literal> is set to <literal>0</literal>
+    (default), “mass queries” such as <command>nix-env -qas</command>
+    will skip this cache.  Thus a package may appear not to have a
+    binary substitute.  However, the binary will still be used when
+    you actually install the package.  If
+    <literal>WantMassQuery</literal> is set to <literal>1</literal>,
+    mass queries will use this cache.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>Priority</literal></term>
+
+    <listitem><para>Each binary cache has a priority (defaulting to
+    50).  Binary caches are checked for binaries in order of ascending
+    priority; thus a higher number denotes a lower priority.  The
+    binary cache <uri>http://cache.nixos.org</uri> has priority
+    40.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
+</para>
+
+<para>Every time Nix needs to build some store path
+<replaceable>p</replaceable>, it will check each configured binary
+cache to see if it has a NAR file for <replaceable>p</replaceable>,
+until it finds one.  If no cache has a NAR, Nix will fall back to
+building the path from source (if applicable).  To see if a cache with
+URL <replaceable>url</replaceable> has a binary for
+<replaceable>p</replaceable>, Nix fetches
+<replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
+is the hash part of <replaceable>p</replaceable>.  Thus, if we have a
+cache <uri>http://cache.nixos.org</uri> and we want to obtain
+the store path
+<screen>
+/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
+</screen>
+then Nix will attempt to fetch
+<screen>
+http://cache.nixos.org/a8922c0h87iilxzzvwn2hmv8x210aqb9.narinfo
+</screen>
+(Commands such as <command>nix-env -qas</command> will issue an HTTP
+HEAD request, since it only needs to know if the
+<filename>.narinfo</filename> file exists.)  The
+<filename>.narinfo</filename> file is a simple text file that looks
+like this:
+
+<screen>
+StorePath: /nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
+URL: nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
+Compression: bzip2
+FileHash: sha256:0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70
+FileSize: 24473768
+NarHash: sha256:0s491y1h9hxj5ghiizlxk7ax6jwbha00zwn7lpyd5xg5bhf60vzg
+NarSize: 109521136
+References: 2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16 ...
+Deriver: 7akyyc87ka32xwmqza9dvyg5pwx3j212-glibc-2.7.drv
+</screen>
+
+The fields are as follows:
+
+<variablelist>
+  
+  <varlistentry><term><literal>StorePath</literal></term>
+
+    <listitem><para>The full store path, including the name part
+    (e.g., <literal>glibc-2.7</literal>).  It must match the
+    requested store path.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>URL</literal></term>
+
+    <listitem><para>The URL of the NAR, relative to the binary cache
+    URL.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>Compression</literal></term>
+
+    <listitem><para>The compression method; either
+    <literal>xz</literal> or
+    <literal>bzip2</literal>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>FileHash</literal></term>
+
+    <listitem><para>The SHA-256 hash of the compressed
+    NAR.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>FileSize</literal></term>
+
+    <listitem><para>The size of the compressed NAR.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>NarHash</literal></term>
+
+    <listitem><para>The SHA-256 hash of the uncompressed NAR.  This is
+    equal to the hash of the store path as returned by
+    <command>nix-store -q --hash
+    <replaceable>p</replaceable></command>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>NarSize</literal></term>
+
+    <listitem><para>The size of the uncompressed NAR.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>References</literal></term>
+
+    <listitem><para>The references of the store path, without the Nix
+    store prefix.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>Deriver</literal></term>
+
+    <listitem><para>The deriver of the store path, without the Nix
+    store prefix.  This field is optional.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>System</literal></term>
+
+    <listitem><para>The Nix platform type of this binary, if known.
+    This field is optional.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
+</para>
+
+<para>Thus, in our example, after recursively ensuring that the
+references exist (e.g.,
+<filename>/nix/store/2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16</filename>),
+Nix will fetch <screen>
+http://cache.nixos.org/nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
+</screen> and decompress and unpack it to
+<filename>/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7</filename>.</para>
+
+</refsection>
+
+
+</refentry>