about summary refs log tree commit diff
path: root/doc/manual/nix-push.xml
blob: e789bbf7d352cc9ea92468de84e64e92907a1022 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
<refentry xmlns="http://docbook.org/ns/docbook"
          xmlns:xlink="http://www.w3.org/1999/xlink"
          xmlns:xi="http://www.w3.org/2001/XInclude"
          xml:id="sec-nix-push">

<refmeta>
  <refentrytitle>nix-push</refentrytitle>
  <manvolnum>1</manvolnum>
  <refmiscinfo class="source">Nix</refmiscinfo>
  <refmiscinfo class="version"><xi:include href="version.txt" parse="text"/></refmiscinfo>
</refmeta>

<refnamediv>
  <refname>nix-push</refname>
  <refpurpose>generate a binary cache</refpurpose>
</refnamediv>

<refsynopsisdiv>
  <cmdsynopsis>
    <command>nix-push</command>
    <arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg>
    <arg><option>--bzip2</option></arg>
    <arg><option>--none</option></arg>
    <arg><option>--force</option></arg>
    <arg><option>--link</option></arg>
    <arg><option>--manifest</option></arg>
    <arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg>
    <arg><option>--url-prefix</option> <replaceable>url</replaceable></arg>
    <arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
  </cmdsynopsis>
</refsynopsisdiv>


<refsection><title>Description</title>

<para>The command <command>nix-push</command> produces a
<emphasis>binary cache</emphasis>, a directory containing compressed
Nix archives (NARs) plus some metadata of the closure of the specified
store paths.  This directory can then be made available through a web
server to other Nix installations, allowing them to skip building from
source and instead download binaries from the cache
automatically.</para>

<para><command>nix-push</command> performs the following actions.
      
<orderedlist>

  <listitem><para>Each path in <replaceable>paths</replaceable> is
  built (using <link
  linkend='rsec-nix-store-realise'><command>nix-store
  --realise</command></link>).</para></listitem>

  <listitem><para>All paths in the closure of
  <replaceable>paths</replaceable> are determined (using
  <command>nix-store --query --requisites
  --include-outputs</command>).  Note that since the
  <option>--include-outputs</option> flag is used, if
  <replaceable>paths</replaceable> includes a store derivation, you
  get a combined source/binary distribution (e.g., source tarballs
  will be included).</para></listitem>

  <listitem><para>All store paths determined in the previous step are
  packaged into a NAR (using <command>nix-store --dump</command>) and
  compressed using <command>xz</command> or <command>bzip2</command>.
  The resulting files have the extension <filename>.nar.xz</filename>
  or <filename>.nar.bz2</filename>.  Also for each store path, Nix
  generates a file with extension <filename>.narinfo</filename>
  containing metadata such as the references, cryptographic hash and
  size of each path.</para></listitem>

  <listitem><para>Optionally, a single <emphasis>manifest</emphasis>
  file is created that contains the same metadata as the
  <filename>.narinfo</filename> files.  This is for compatibility with
  Nix versions prior to 1.2 (see <command>nix-pull</command> for
  details).</para></listitem>

  <listitem><para>A file named <option>nix-cache-info</option> is
  placed in the destination directory.  The existence of this file
  marks the directory as a binary cache.</para></listitem>

</orderedlist>

</para>

</refsection>


<refsection><title>Options</title>

<variablelist>

  <varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term>

    <listitem><para>Set the destination directory to
    <replaceable>dir</replaceable>, which is created if it does not
    exist.  This flag is required.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--bzip2</option></term>

    <listitem><para>Compress NARs using <command>bzip2</command>
    instead of <command>xz -9</command>.  The latter compresses about
    30% better on typical archives, decompresses about twice as fast,
    but compresses a lot slower and is not supported by Nix prior to
    version 1.2.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--none</option></term>

    <listitem><para>Do not compress NARs.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--force</option></term>

    <listitem><para>Overwrite <filename>.narinfo</filename> files if
    they already exist.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--link</option></term>

    <listitem><para>By default, NARs are generated in the Nix store
    and then copied to <replaceable>dest-dir</replaceable>.  If this
    option is given, hard links are used instead.  This only works if
    <replaceable>dest-dir</replaceable> is on the same filesystem as
    the Nix store.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--manifest</option></term>

    <listitem><para>Force the generation of a manifest suitable for
    use by <command>nix-pull</command>.  The manifest is stored as
    <filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term>

    <listitem><para>Like <option>--manifest</option>, but store the
    manifest in <replaceable>filename</replaceable>.</para></listitem>

  </varlistentry>

  <varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term>

    <listitem><para>Manifests are expected to contain the absolute
    URLs of NARs.  For generating these URLs, the prefix
    <replaceable>url</replaceable> is used.  It defaults to
    <uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem>

  </varlistentry>

</variablelist>

</refsection>


<refsection><title>Examples</title>

<para>To add the closure of Thunderbird to a binary cache:

<screen>
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird)
</screen>

Assuming that <filename>/tmp/cache</filename> is exported by a web
server as <uri>http://example.org/cache</uri>, you can then use this
cache on another machine to speed up the installation of Thunderbird:

<screen>
$ nix-build -A thunderbird --option binary-caches http://example.org/cache
</screen>

Alternatively, you could add <literal>binary-caches =
http://example.org/cache</literal> to
<filename>nix.conf</filename>.</para>

<para>To also include build-time dependencies (such as source
tarballs):

<screen>
$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird)
</screen>

</para>

<para>To generate a manifest suitable for <command>nix-pull</command>:

<screen>
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest
</screen>

On another machine you can then do:

<screen>
$ nix-pull http://example.org/cache
</screen>

to cause the binaries to be used by subsequent Nix operations.</para>

</refsection>


<refsection><title>Binary cache format and operation</title>

<para>A binary cache with URL <replaceable>url</replaceable> only
denotes a valid binary cache if the file
<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists.  If
this file does not exist (or cannot be downloaded), the cache is
ignored.  If it does exist, it must be a text file containing cache
properties.  Here’s an example:

<screen>
StoreDir: /nix/store
WantMassQuery: 1
Priority: 10
</screen>

The properties that are currently supported are:

<variablelist>
  
  <varlistentry><term><literal>StoreDir</literal></term>

    <listitem><para>The path of the Nix store to which this binary
    cache applies.  Binaries are not relocatable — a binary built for
    <filename>/nix/store</filename> won’t generally work in
    <filename>/home/alice/store</filename> — so to prevent binaries
    from being used in a wrong store, a binary cache is only used if
    its <literal>StoreDir</literal> matches the local Nix
    configuration.  The default is
    <filename>/nix/store</filename>.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>WantMassQuery</literal></term>

    <listitem><para>Query operations such as <command>nix-env
    -qas</command> can cause thousands of cache queries, and thus
    thousands of HTTP requests, to determine which packages are
    available in binary form.  While these requests are small, not
    every server may appreciate a potential onslaught of queries.  If
    <literal>WantMassQuery</literal> is set to <literal>0</literal>
    (default), “mass queries” such as <command>nix-env -qas</command>
    will skip this cache.  Thus a package may appear not to have a
    binary substitute.  However, the binary will still be used when
    you actually install the package.  If
    <literal>WantMassQuery</literal> is set to <literal>1</literal>,
    mass queries will use this cache.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>Priority</literal></term>

    <listitem><para>Each binary cache has a priority (defaulting to
    50).  Binary caches are checked for binaries in order of ascending
    priority; thus a higher number denotes a lower priority.  The
    binary cache <uri>http://cache.nixos.org</uri> has priority
    40.</para></listitem>

  </varlistentry>

</variablelist>

</para>

<para>Every time Nix needs to build some store path
<replaceable>p</replaceable>, it will check each configured binary
cache to see if it has a NAR file for <replaceable>p</replaceable>,
until it finds one.  If no cache has a NAR, Nix will fall back to
building the path from source (if applicable).  To see if a cache with
URL <replaceable>url</replaceable> has a binary for
<replaceable>p</replaceable>, Nix fetches
<replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
is the hash part of <replaceable>p</replaceable>.  Thus, if we have a
cache <uri>http://cache.nixos.org</uri> and we want to obtain
the store path
<screen>
/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
</screen>
then Nix will attempt to fetch
<screen>
http://cache.nixos.org/a8922c0h87iilxzzvwn2hmv8x210aqb9.narinfo
</screen>
(Commands such as <command>nix-env -qas</command> will issue an HTTP
HEAD request, since it only needs to know if the
<filename>.narinfo</filename> file exists.)  The
<filename>.narinfo</filename> file is a simple text file that looks
like this:

<screen>
StorePath: /nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
URL: nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
Compression: bzip2
FileHash: sha256:0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70
FileSize: 24473768
NarHash: sha256:0s491y1h9hxj5ghiizlxk7ax6jwbha00zwn7lpyd5xg5bhf60vzg
NarSize: 109521136
References: 2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16 ...
Deriver: 7akyyc87ka32xwmqza9dvyg5pwx3j212-glibc-2.7.drv
</screen>

The fields are as follows:

<variablelist>
  
  <varlistentry><term><literal>StorePath</literal></term>

    <listitem><para>The full store path, including the name part
    (e.g., <literal>glibc-2.7</literal>).  It must match the
    requested store path.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>URL</literal></term>

    <listitem><para>The URL of the NAR, relative to the binary cache
    URL.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>Compression</literal></term>

    <listitem><para>The compression method; either
    <literal>xz</literal> or
    <literal>bzip2</literal>.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>FileHash</literal></term>

    <listitem><para>The SHA-256 hash of the compressed
    NAR.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>FileSize</literal></term>

    <listitem><para>The size of the compressed NAR.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>NarHash</literal></term>

    <listitem><para>The SHA-256 hash of the uncompressed NAR.  This is
    equal to the hash of the store path as returned by
    <command>nix-store -q --hash
    <replaceable>p</replaceable></command>.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>NarSize</literal></term>

    <listitem><para>The size of the uncompressed NAR.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>References</literal></term>

    <listitem><para>The references of the store path, without the Nix
    store prefix.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>Deriver</literal></term>

    <listitem><para>The deriver of the store path, without the Nix
    store prefix.  This field is optional.</para></listitem>

  </varlistentry>

  <varlistentry><term><literal>System</literal></term>

    <listitem><para>The Nix platform type of this binary, if known.
    This field is optional.</para></listitem>

  </varlistentry>

</variablelist>

</para>

<para>Thus, in our example, after recursively ensuring that the
references exist (e.g.,
<filename>/nix/store/2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16</filename>),
Nix will fetch <screen>
http://cache.nixos.org/nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
</screen> and decompress and unpack it to
<filename>/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7</filename>.</para>

</refsection>


</refentry>