diff options
| -rw-r--r-- | doc/manual/glossary.xml | 12 | ||||
| -rw-r--r-- | doc/manual/nix-push.xml | 406 |
2 files changed, 341 insertions, 77 deletions
diff --git a/doc/manual/glossary.xml b/doc/manual/glossary.xml index efbf96f0afa6..d74940c90b30 100644 --- a/doc/manual/glossary.xml +++ b/doc/manual/glossary.xml @@ -160,6 +160,18 @@ </glossentry> +<glossentry xml:id="gloss-nar"><glossterm>NAR</glossterm> + + <glossdef><para>A <emphasis>N</emphasis>ix + <emphasis>AR</emphasis>chive. This is a serialisation of a path in + the Nix store. It can contain regular files, directories and + symbolic links. NARs are generated and unpacked using + <command>nix-store --dump</command> and <command>nix-store + --restore</command>.</para></glossdef> + +</glossentry> + + </glosslist> diff --git a/doc/manual/nix-push.xml b/doc/manual/nix-push.xml index d2a7e063e583..170b6f55ab30 100644 --- a/doc/manual/nix-push.xml +++ b/doc/manual/nix-push.xml @@ -12,24 +12,19 @@ <refnamediv> <refname>nix-push</refname> - <refpurpose>push store paths onto a network cache</refpurpose> + <refpurpose>generate a binary cache</refpurpose> </refnamediv> <refsynopsisdiv> <cmdsynopsis> <command>nix-push</command> - <group choice='req'> - <arg choice='req'> - <arg choice='plain'><replaceable>archivesPutURL</replaceable></arg> - <arg choice='plain'><replaceable>archivesGetURL</replaceable></arg> - <arg choice='plain'><replaceable>manifestPutURL</replaceable></arg> - </arg> - <arg choice='req'> - <arg choice='plain'><option>--copy</option></arg> - <arg choice='plain'><replaceable>archivesDir</replaceable></arg> - <arg choice='plain'><replaceable>manifestFile</replaceable></arg> - </arg> - </group> + <arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg> + <arg><option>--bzip2</option></arg> + <arg><option>--force</option></arg> + <arg><option>--link</option></arg> + <arg><option>--manifest</option></arg> + <arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg> + <arg><option>--url-prefix</option> <replaceable>url</replaceable></arg> <arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg> </cmdsynopsis> </refsynopsisdiv> @@ -37,93 +32,350 @@ <refsection><title>Description</title> -<para>The command <command>nix-push</command> builds a set of store -paths (if necessary), and then packages and uploads all store paths in -the resulting closures to a server. A network cache thus populated -can subsequently be used to speed up software deployment on other -machines using the <command>nix-pull</command> command.</para> +<para>The command <command>nix-push</command> produces a +<emphasis>binary cache</emphasis>, a directory containing compressed +Nix archives (NARs) plus some metadata of the closure of the specified +store paths. This directory can then be made available through a web +server to other Nix installations, allowing them to skip building from +source and instead download binaries from the cache +automatically.</para> <para><command>nix-push</command> performs the following actions. <orderedlist> <listitem><para>Each path in <replaceable>paths</replaceable> is - realised (using <link - linkend='rsec-nix-store-realise'><literal>nix-store - --realise</literal></link>).</para></listitem> + built (using <link + linkend='rsec-nix-store-realise'><command>nix-store + --realise</command></link>).</para></listitem> - <listitem><para>All paths in the closure of the store expressions - stored in <replaceable>paths</replaceable> are determined (using - <literal>nix-store --query --requisites - --include-outputs</literal>). It should be noted that since the - <option>--include-outputs</option> flag is used, you get a combined - source/binary distribution.</para></listitem> + <listitem><para>All paths in the closure of + <replaceable>paths</replaceable> are determined (using + <command>nix-store --query --requisites + --include-outputs</command>). Note that since the + <option>--include-outputs</option> flag is used, if + <replaceable>paths</replaceable> includes a store derivation, you + get a combined source/binary distribution (e.g., source tarballs + will be included).</para></listitem> <listitem><para>All store paths determined in the previous step are - packaged and compressed into a <command>bzip</command>ped NAR - archive (extension <filename>.nar.bz2</filename>).</para></listitem> - - <listitem><para>A <emphasis>manifest</emphasis> is created that - contains information on the store paths, their eventual URLs in the - cache, and cryptographic hashes of the contents of the NAR - archives.</para></listitem> - - <listitem><para>Each store path is uploaded to the remote directory - specified by <replaceable>archivesPutURL</replaceable>. HTTP PUT - requests are used to do this. However, before a file - <varname>x</varname> is uploaded to - <literal><replaceable>archivesPutURL</replaceable>/</literal><varname>x</varname>, - <command>nix-push</command> first determines whether this upload is - unnecessary by issuing a HTTP HEAD request on - <literal><replaceable>archivesGetURL</replaceable>/</literal><varname>x</varname>. - This allows a cache to be shared between many partially overlapping - <command>nix-push</command> invocations. (We use two URLs because - the upload URL typically refers to a CGI script, while the download - URL just refers to a file system directory on the - server.)</para></listitem> - - <listitem><para>The manifest is uploaded using an HTTP PUT request - to <replaceable>manifestPutURL</replaceable>. The corresponding - URL to download the manifest can then be used by - <command>nix-pull</command>.</para></listitem> - + packaged into a NAR (using <command>nix-store --dump</command>) and + compressed using <command>xz</command> or <command>bzip2</command>. + The resulting files have the extension <filename>.nar.xz</filename> + or <filename>.nar.bz2</filename>. Also for each store path, Nix + generates a file with extension <filename>.narinfo</filename> + containing metadata such as the references, cryptographic hash and + size of each path.</para></listitem> + + <listitem><para>Optionally, a single <emphasis>manifest</emphasis> + file is created that contains the same metadata as the + <filename>.narinfo</filename> files. This is for compatibility with + Nix versions prior to 1.2 (see <command>nix-pull</command> for + details).</para></listitem> + + <listitem><para>A file named <option>nix-cache-info</option> is + placed in the destination directory. The existence of this file + marks the directory as a binary cache.</para></listitem> + </orderedlist> </para> -<!-- -<para>TODO: <option>- -copy</option></para> ---> - +</refsection> + + +<refsection><title>Options</title> + +<variablelist> + + <varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term> + + <listitem><para>Set the destination directory to + <replaceable>dir</replaceable>, which is created if it does not + exist. This flag is required.</para></listitem> + + </varlistentry> + + <varlistentry><term><option>--bzip2</option></term> + + <listitem><para>Compress NARs using <command>bzip2</command> + instead of <command>xz -9</command>. The latter compresses about + 30% better on typical archives, decompresses about twice as fast, + but compresses a lot slower and is not supported by Nix prior to + version 1.2.</para></listitem> + + </varlistentry> + + <varlistentry><term><option>--force</option></term> + + <listitem><para>Overwrite <filename>.narinfo</filename> files if + they already exist.</para></listitem> + + </varlistentry> + + <varlistentry><term><option>--link</option></term> + + <listitem><para>By default, NARs are generated in the Nix store + and then copied to <replaceable>dest-dir</replaceable>. If this + option is given, hard links are used instead. This only works if + <replaceable>dest-dir</replaceable> is on the same filesystem as + the Nix store.</para></listitem> + + </varlistentry> + + <varlistentry><term><option>--manifest</option></term> + + <listitem><para>Force the generation of a manifest suitable for + use by <command>nix-pull</command>. The manifest is stored as + <filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem> + + </varlistentry> + + <varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term> + + <listitem><para>Like <option>--manifest</option>, but store the + manifest in <replaceable>filename</replaceable>.</para></listitem> + + </varlistentry> + + <varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term> + + <listitem><para>Manifests are expected to contain the absolute + URLs of NARs. For generating these URLs, the prefix + <replaceable>url</replaceable> is used. It defaults to + <uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem> + + </varlistentry> + +</variablelist> + </refsection> <refsection><title>Examples</title> -<para>To upload files there typically is some CGI script on the server -side. This script should be be protected with a password. The -following example uploads the store paths resulting from building the -Nix expressions in <filename>foo.nix</filename>, passing appropriate -authentication information: - +<para>To add the closure of Thunderbird to a binary cache: + <screen> -$ nix-push \ - http://foo@bar:server.domain/cgi-bin/upload.pl/cache \ - http://server.domain/cache \ - http://foo@bar:server.domain/cgi-bin/upload.pl/MANIFEST \ - $(nix-instantiate foo.nix)</screen> +$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) +</screen> -This will push both sources and binaries (and any build-time -dependencies used in the build, such as compilers).</para> +Assuming that <filename>/tmp/cache</filename> is exported by a web +server as <uri>http://example.org/cache</uri>, you can then use this +cache on another machine to speed up the installation of Thunderbird: -<para>If we just want to push binaries, not sources and build-time -dependencies, we can do: - <screen> -$ nix-push <replaceable>urls</replaceable> $(nix-store -r $(nix-instantiate foo.nix))</screen> - +$ nix-build -A thunderbird --option binary-caches http://example.org/cache +</screen> + +Alternatively, you could add <literal>binary-caches = +http://example.org/cache</literal> to +<filename>nix.conf</filename>.</para> + +<para>To also include build-time dependencies (such as source +tarballs): + +<screen> +$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird) +</screen> + </para> +<para>To generate a manifest suitable for <command>nix-pull</command>: + +<screen> +$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest +</screen> + +On another machine you can then do: + +<screen> +$ nix-pull http://example.org/cache +</screen> + +to cause the binaries to be used by subsequent Nix operations.</para> + </refsection> - + + +<refsection><title>Binary cache format and operation</title> + +<para>A binary cache with URL <replaceable>url</replaceable> only +denotes a valid binary cache if the file +<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists. If +this file does not exist (or cannot be downloaded), the cache is +ignored. If it does exist, it must be a text file containing cache +properties. Here’s an example: + +<screen> +StoreDir: /nix/store +WantMassQuery: 1 +</screen> + +The properties that are currently supported are: + +<variablelist> + + <varlistentry><term><literal>StoreDir</literal></term> + + <listitem><para>The path of the Nix store to which this binary + cache applies. Binaries are not relocatable — a binary built for + <filename>/nix/store</filename> won’t generally work in + <filename>/home/alice/store</filename> — so to prevent binaries + from being used in a wrong store, a binary cache is only used if + its <literal>StoreDir</literal> matches the local Nix + configuration. The default is + <filename>/nix/store</filename>.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>WantMassQuery</literal></term> + + <listitem><para>Query operations such as <command>nix-env + -qas</command> can cause thousands of cache queries, and thus + thousands of HTTP requests, to determine which packages are + available in binary form. While these requests are small, not + every server may appreciate a potential onslaught of queries. If + <literal>WantMassQuery</literal> is set to <literal>0</literal> + (default), “mass queries” such as <command>nix-env -qas</command> + will skip this cache. Thus a package may appear not to have a + binary substitute. However, the binary will still be used when + you actually install the package. If + <literal>WantMassQuery</literal> is set to <literal>1</literal>, + mass queries will use this cache.</para></listitem> + + </varlistentry> + +</variablelist> + +</para> + +<para>Every time Nix needs to build some store path +<replaceable>p</replaceable>, it will check each configured binary +cache to see if it has a NAR file for <replaceable>p</replaceable>, +until it finds one. If no cache has a NAR, Nix will fall back to +building the path from source (if applicable). To see if a cache with +URL <replaceable>url</replaceable> has a binary for +<replaceable>p</replaceable>, Nix fetches +<replaceable>url/h</replaceable>, where <replaceable>h</replaceable> +is the hash part of <replaceable>p</replaceable>. Thus, if we have a +cache <uri>http://nixos.org/binary-cache</uri> and we want to obtain +the store path +<screen> +/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 +</screen> +then Nix will attempt to fetch +<screen> +http://nixos.org/binary-cache/cj7a81wsm1ijwwpkks3725661h3263p5.narinfo +</screen> +(Commands such as <command>nix-env -qas</command> will issue an HTTP +HEAD request, since it only needs to know if the +<filename>.narinfo</filename> file exists.) The +<filename>.narinfo</filename> file is a simple text file that looks +like this: + +<screen> +StorePath: /nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 +URL: nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2 +Compression: bzip2 +FileHash: sha256:0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21 +FileSize: 10119014 +NarHash: sha256:120mhshkcgvn48xvxz0fjbd47k615v0viv7jy4gy7dwhmqapgd9d +NarSize: 33702192 +References: cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14 +Deriver: xs32zzf2d58f6l5iz5fgwxrds2q5pnvn-glibc-2.13.drv +System: x86_64-linux +</screen> + +The fields are as follows: + +<variablelist> + + <varlistentry><term><literal>StorePath</literal></term> + + <listitem><para>The full store path, including the name part + (e.g., <literal>glibc-2.13</literal>). It must match the + requested store path.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>URL</literal></term> + + <listitem><para>The URL of the NAR, relative to the binary cache + URL.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>Compression</literal></term> + + <listitem><para>The compression method; either + <literal>xz</literal> or + <literal>bzip2</literal>.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>FileHash</literal></term> + + <listitem><para>The SHA-256 hash of the compressed + NAR.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>FileSize</literal></term> + + <listitem><para>The size of the compressed NAR.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>NarHash</literal></term> + + <listitem><para>The SHA-256 hash of the uncompressed NAR. This is + equal to the hash of the store path as returned by + <command>nix-store -q --hash + <replaceable>p</replaceable></command>.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>NarSize</literal></term> + + <listitem><para>The size of the uncompressed NAR.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>References</literal></term> + + <listitem><para>The references of the store path, without the Nix + store prefix.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>Deriver</literal></term> + + <listitem><para>The deriver of the store path, without the Nix + store prefix. This field is optional.</para></listitem> + + </varlistentry> + + <varlistentry><term><literal>System</literal></term> + + <listitem><para>The Nix platform type of this binary, if known. + This field is optional.</para></listitem> + + </varlistentry> + +</variablelist> + +</para> + +<para>Thus, in our example, after recursively ensuring that the +references exist (e.g., +<filename>/nix/store/jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14</filename>), +Nix will fetch <screen> +http://nixos.org/binary-cache/nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2 +</screen> and decompress and unpack it to +<filename>/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13</filename>.</para> + +</refsection> + + </refentry> |