* Release notes in Docbook; ASCII release notes (i.e., the `NEWS'

  file) is now generated from that using `w3m' and some XSL hackery.

5 files changed, 581 insertions, 5 deletions

diff --git a/doc/manual/Makefile.am b/doc/manual/Makefile.am
index 81367fd141..2cc3766120 100644
--- a/doc/manual/Makefile.am
+++ b/doc/manual/Makefile.am
@@ -37,7 +37,24 @@ manual.html: $(MANUAL_SRCS) manual.is-valid images
 	$(XSLTPROC) --nonet --xinclude --output manual.html \
 	  $(docbookxsl)/html/docbook.xsl manual.xml
 
-all-local: manual.html
+
+NEWS_OPTS = \
+ --stringparam generate.toc "article nop" \
+ --stringparam section.autolabel.max.depth 0 \
+ --stringparam header.rule 0
+
+NEWS.html: release-notes.xml
+	$(XSLTPROC) --nonet --xinclude --output $@ $(NEWS_OPTS) \
+	  $(docbookxsl)/html/docbook.xsl release-notes.xml
+
+NEWS.txt: release-notes.xml
+	$(XSLTPROC) --nonet --xinclude quote-literals.xsl release-notes.xml | \
+	  $(XSLTPROC) --nonet --output $@.tmp.html $(NEWS_OPTS) \
+	  $(docbookxsl)/html/docbook.xsl -
+	w3m -dump $@.tmp.html > $@
+
+
+all-local: manual.html NEWS.html NEWS.txt
 
 install-data-local: manual.html
 	$(INSTALL) -d $(DESTDIR)$(datadir)/nix/manual
diff --git a/doc/manual/introduction.xml b/doc/manual/introduction.xml
index 9f94b2d539..fb6e79fcb5 100644
--- a/doc/manual/introduction.xml
+++ b/doc/manual/introduction.xml
@@ -130,8 +130,8 @@ collection.  It also discusses some advanced topics, such as setting
 up a Nix-based build farm, and doing service deployment using
 Nix.</para>
 
-<note><para>Some background information on Nix can be found in three
-papers.  The ICSE 2004 paper <ulink
+<note><para>Some background information on Nix can be found in a
+number of papers.  The ICSE 2004 paper <ulink
 url='http://www.cs.uu.nl/~eelco/pubs/immdsd-icse2004-final.pdf'><citetitle>Imposing
 a Memory Management Discipline on Software
 Deployment</citetitle></ulink> discusses the hashing mechanism used to
@@ -144,7 +144,10 @@ Deployment</citetitle></ulink> gives a more general discussion of Nix
 from a system-administration perspective.  The CBSE 2005 paper <ulink
 url='http://www.cs.uu.nl/~eelco/pubs/eupfcdm-cbse2005-final.pdf'><citetitle>Efficient
 Upgrading in a Purely Functional Component Deployment Model
-</citetitle></ulink> is about transparent patch deployment in
-Nix.</para></note>
+</citetitle></ulink> is about transparent patch deployment in Nix.
+Finally, the SCM-12 paper <ulink
+url='http://www.cs.uu.nl/~eelco/pubs/servicecm-scm12-final.pdf'>
+Service Configuration Management</ulink> shows how services (e.g., web
+servers) can be deployed and managed through Nix.</para></note>
 
 </chapter>
diff --git a/doc/manual/quote-literals.xsl b/doc/manual/quote-literals.xsl
new file mode 100644
index 0000000000..acc7c8a0ba
--- /dev/null
+++ b/doc/manual/quote-literals.xsl
@@ -0,0 +1,46 @@
+<?xml version="1.0"?>
+
+<xsl:stylesheet
+  version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:str="http://exslt.org/strings"
+  extension-element-prefixes="str">
+
+  <xsl:output method="xml"/>
+  
+  <xsl:template match="function|command|literal|varname|filename|option|quote">`<xsl:apply-templates/>'</xsl:template>
+  
+  <xsl:template match="token"><xsl:text>    </xsl:text><xsl:apply-templates /><xsl:text>
+</xsl:text></xsl:template>
+
+  <xsl:template match="screen|programlisting">
+    <screen><xsl:apply-templates select="str:split(., '&#xA;')" /></screen>
+  </xsl:template>
+
+<!--  
+  <xsl:template match="section/title">
+    <title><xsl:text>FOO&#xA;BAR</xsl:text><xsl:value-of select="." /></title>
+  </xsl:template>
+-->
+ 
+  <xsl:template match="section[following::section]">
+    <section>
+      <xsl:apply-templates />
+      <screen><xsl:text>
+      </xsl:text></screen>        
+    </section>
+  </xsl:template>
+
+  <xsl:template match="*">
+    <xsl:element name="{name(.)}" namespace="{namespace-uri(.)}">
+      <xsl:copy-of select="namespace::*" />
+      <xsl:for-each select="@*">
+	<xsl:attribute name="{name(.)}" namespace="{namespace-uri(.)}">
+	  <xsl:value-of select="."/>
+	</xsl:attribute>
+      </xsl:for-each>
+      <xsl:apply-templates/>
+    </xsl:element>
+  </xsl:template>
+
+</xsl:stylesheet>
diff --git a/doc/manual/release-notes.xml b/doc/manual/release-notes.xml
new file mode 100644
index 0000000000..669fcc26f7
--- /dev/null
+++ b/doc/manual/release-notes.xml
@@ -0,0 +1,505 @@
+<?xml version="1.0"?>
+<!DOCTYPE book
+    PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN"
+    "http://www.docbook.org/xml/4.3/docbook-xml-4.3.zip"
+[
+]>
+
+<article><title>Nix Release Notes</title>
+
+
+<section><title>Release 0.9</title>
+
+<para>NOTE: this version of Nix uses Berkeley DB 4.3 instead of 4.2.
+The database is upgraded automatically, but you should be careful not
+to use old versions of Nix that still use Berkeley DB 4.2.  In
+particular, if you use a Nix installed through Nix, you should run
+
+<screen>
+$ nix-store --clear-substitutes</screen>
+
+first.</para>
+
+
+<itemizedlist>
+
+  <listitem><para>Unpacking of patch sequences is much faster now
+  since we no longer do redundant unpacking and repacking of
+  intermediate paths.</para></listitem>
+
+  <listitem><para>Nix now uses Berkeley DB 4.3.</para></listitem>
+
+  <listitem><para>The <function>derivation</function> primitive is
+  lazier.  Attributes of dependent derivations can mutually refer to
+  each other (as long as there are no data dependencies on the
+  <varname>outPath</varname> and <varname>drvPath</varname> attributes
+  computed by <function>derivation</function>).</para>
+
+  <para>For example, the expression <literal>derivation
+  attrs</literal> now evaluates to (essentially)
+
+  <programlisting>
+attrs // {
+  type = "derivation";
+  outPath = derivation! attrs;
+  drvPath = derivation! attrs;
+}</programlisting>
+
+  where <function>derivation!</function> is a primop that does the
+  actual derivation instantiation (i.e., it does what
+  <function>derivation</function> used to do).  The advantage is that
+  it allows commands such as <command>nix-env -qa</command> and
+  <command>nix-env -i</command> to be much faster since they no longer
+  need to instantiate all derivations, just the
+  <varname>name</varname> attribute.</para>
+
+  <para>Also, it allows derivations to cyclically reference each
+  other, for example,
+
+  <programlisting>
+webServer = derivation {
+  ...
+  hostName = "svn.cs.uu.nl";
+  services = [svnService];
+};
+&#x20;
+svnService = derivation {
+  ...
+  hostName = webServer.hostName;
+};</programlisting>
+
+  Previously, this would yield a black hole (infinite recursion).</para>
+  
+  </listitem>
+
+  <listitem><para><command>nix-build</command> now defaults to using
+  <filename>./default.nix</filename> if no Nix expression is
+  specified.</para></listitem>
+
+  <listitem><para><command>nix-instantiate</command>, when applied to
+  a Nix expression that evaluates to a function, will call the
+  function automatically if all its arguments have
+  defaults.</para></listitem>
+
+  <listitem><para>Nix now uses libtool to build dynamic libraries.
+  This reduces the size of executables.</para></listitem>
+
+  <listitem><para>A new list concatenation operator
+  <literal>++</literal>.  For example, <literal>[1 2 3] ++ [4 5
+  6]</literal> evaluates to <literal>[1 2 3 4 5
+  6]</literal>.</para></listitem>
+
+  <listitem><para>Some currently undocumented primops to support
+  low-level build management using Nix (i.e., using Nix as a Make
+  replacement).  See the commit messages for <literal>r3578</literal>
+  and <literal>r3580</literal>.</para></listitem>
+
+  <listitem><para>Various bug fixes and performance
+  improvements.</para></listitem>
+
+</itemizedlist>
+
+</section>
+
+
+
+<section><title>Release 0.8 (April 11, 2005)</title>
+
+<para>NOTE: the hashing scheme in Nix 0.8 changed (as detailed below).
+As a result, <command>nix-pull</command> manifests and channels built
+for Nix 0.7 and below will now work anymore.  However, the Nix
+expression language has not changed, so you can still build from
+source.  Also, existing user environments continue to work.  Nix 0.8
+will automatically upgrade the database schema of previous
+installations when it is first run.</para>
+
+<para>If you get the error message
+
+<screen>
+you have an old-style manifest `/nix/var/nix/manifests/[...]'; please
+delete it</screen>
+
+you should delete previously downloaded manifests:
+
+<screen>
+$ rm /nix/var/nix/manifests/*</screen>
+
+If <command>nix-channel</command> gives the error message
+
+<screen>
+manifest `http://catamaran.labs.cs.uu.nl/dist/nix/channels/[channel]/MANIFEST'
+is too old (i.e., for Nix &lt;= 0.7)</screen>
+
+then you should unsubscribe from the offending channel
+(<command>nix-channel --remove
+<replaceable>URL</replaceable></command>; leave out
+<literal>/MANIFEST</literal>), and subscribe to the same URL, with
+<literal>channels</literal> replaced by <literal>channels-v3</literal>
+(e.g.,
+http://catamaran.labs.cs.uu.nl/dist/nix/channels-v3/nixpkgs-unstable).</para>
+
+<para>Nix 0.8 has the following improvements:
+
+<itemizedlist>
+
+  <listitem><para>The cryptographic hashes used in store paths are now
+  160 bits long, but encoded in base-32 so that they are still only 32
+  characters long (e.g.,
+  /nix/store/csw87wag8bqlqk7ipllbwypb14xainap-atk-1.9.0).  (This is
+  actually a 160 bit truncation of a SHA-256 hash.)</para></listitem>
+
+  <listitem><para>Big cleanups and simplifications of the basic store
+  semantics.  The notion of "closure store expressions" is gone (and
+  so is the notion of "successors"); the file system references of a
+  store path are now just stored in the database.</para>
+
+  <para>For instance, given any store path, you can query its closure:
+
+  <screen>
+$ nix-store -qR $(which firefox)
+... lots of paths ...</screen>
+
+  Also, Nix now remembers for each store path the derivation that
+  built it (the "deriver"):
+
+  <screen>
+$ nix-store -qR $(which firefox)
+/nix/store/4b0jx7vq80l9aqcnkszxhymsf1ffa5jd-firefox-1.0.1.drv</screen>
+
+  So to see the build-time dependencies, you can do
+
+  <screen>
+$ nix-store -qR $(nix-store -qd $(which firefox))</screen>
+
+  or, in a nicer format:
+
+  <screen>
+$ nix-store -q --tree $(nix-store -qd $(which firefox))</screen>
+
+  </para>
+
+  <para>File system references are also stored in reverse.  For
+  instance, you can query all paths that directly or indirectly use a
+  certain Glibc:
+
+  <screen>
+$ nix-store -q --referers-closure \
+    /nix/store/8lz9yc6zgmc0vlqmn2ipcpkjlmbi51vv-glibc-2.3.4</screen>
+
+  </para>
+  
+  </listitem>
+
+  <listitem><para>The concept of fixed-output derivations has been
+  formalised.  Previously, functions such as
+  <function>fetchurl</function> in Nixpkgs used a hack (namely,
+  explicitly specifying a store path hash) to prevent changes to, say,
+  the URL of the file from propagating upwards through the dependency
+  graph, causing rebuilds of everything.  This can now be done cleanly
+  by specifying the <varname>outputHash</varname> and
+  <varname>outputHashAlgo</varname> attributes.  Nix itself checks
+  that the content of the output has the specified hash.  (This is
+  important for maintaining certain invariants necessary for future
+  work on secure shared stores.)</para></listitem>
+
+  <listitem><para>One-click installation :-) It is now possible to
+  install any top-level component in Nixpkgs directly, through the web
+  - see, e.g., http://catamaran.labs.cs.uu.nl/dist/nixpkgs-0.8/.  All
+  you have to do is associate
+  <filename>/nix/bin/nix-install-package</filename> with the MIME type
+  <literal>application/nix-package</literal> (or the extension
+  <filename>.nixpkg</filename>), and clicking on a package link will
+  cause it to be installed, with all appropriate dependencies.  If you
+  just want to install some specific application, this is easier than
+  subscribing to a channel.</para></listitem>
+
+  <listitem><para><command>nix-store -r
+  <replaceable>PATHS</replaceable></command> now builds all the
+  derivations PATHS in parallel.  Previously it did them sequentially
+  (though exploiting possible parallelism between subderivations).
+  This is nice for build farms.</para></listitem>
+
+  <listitem><para><command>nix-channel</command> has new operations
+  <option>--list</option> and
+  <option>--remove</option>.</para></listitem>
+
+  <listitem><para>New ways of installing components into user
+  environments:
+
+  <itemizedlist>
+
+    <listitem><para>Copy from another user environment:
+
+    <screen>
+$ nix-env -i --from-profile .../other-profile firefox</screen>
+
+    </para></listitem>
+
+    <listitem><para>Install a store derivation directly (bypassing the
+    Nix expression language entirely):
+
+    <screen>
+$ nix-env -i /nix/store/z58v41v21xd3...-aterm-2.3.1.drv</screen>
+
+    (This is used to implement <command>nix-install-package</command>,
+    which is therefore immune to evolution in the Nix expression
+    language.)</para></listitem>
+
+    <listitem><para>Install an already built store path directly:
+
+    <screen>
+$ nix-env -i /nix/store/hsyj5pbn0d9i...-aterm-2.3.1</screen>
+
+    </para></listitem>
+
+    <listitem><para>Install the result of a Nix expression specified
+    as a command-line argument:
+
+    <screen>
+$ nix-env -f .../i686-linux.nix -i -E 'x: x.firefoxWrapper'</screen>
+
+    The difference with the normal installation mode is that
+    <option>-E</option> does not use the <varname>name</varname>
+    attributes of derivations.  Therefore, this can be used to
+    disambiguate multiple derivations with the same
+    name.</para></listitem>
+
+  </itemizedlist></para></listitem>
+
+  <listitem><para>A hash of the contents of a store path is now stored
+  in the database after a succesful build.  This allows you to check
+  whether store paths have been tampered with: <command>nix-store
+  --verify --check-contents</command>.</para></listitem>
+
+  <listitem>
+
+    <para>Implemented a concurrent garbage collector.  It is now
+    always safe to run the garbage collector, even if other Nix
+    operations are happening simultaneously.</para>
+
+    <para>However, there can still be GC races if you use
+    <command>nix-instantiate</command> and <command>nix-store
+    --realise</command> directly to build things.  To prevent races,
+    use the <option>--add-root</option> flag of those commands.</para>
+
+  </listitem>
+
+  <listitem><para>The garbage collector now finally deletes paths in
+  the right order (i.e., topologically sorted under the
+  <quote>references</quote> relation), thus making it safe to
+  interrupt the collector without risking a store that violates the
+  closure invariant.</para></listitem>
+
+  <listitem><para>Likewise, the substitute mechanism now downloads
+  files in the right order, thus preserving the closure invariant at
+  all times.</para></listitem>
+
+  <listitem><para>The result of <command>nix-build</command> is now
+  registered as a root of the garbage collector.  If the
+  <filename>./result</filename> link is deleted, the GC root
+  disappears automatically.</para></listitem>
+
+  <listitem>
+
+    <para>The behaviour of the garbage collector can be changed
+    globally by setting options in
+    <filename>/nix/etc/nix/nix.conf</filename>.
+
+    <itemizedlist>
+
+      <listitem><para><literal>gc-keep-derivations</literal> specifies
+      whether deriver links should be followed when searching for live
+      paths.</para></listitem>
+
+      <listitem><para><literal>gc-keep-outputs</literal> specifies
+      whether outputs of derivations should be followed when searching
+      for live paths.</para></listitem>
+
+      <listitem><para><literal>env-keep-derivations</literal>
+      specifies whether user environments should store the paths of
+      derivations when they are added (thus keeping the derivations
+      alive).</para></listitem>
+
+    </itemizedlist>
+
+  </para></listitem>
+
+  <listitem><para>New <command>nix-env</command> query flags
+  <option>--drv-path</option> and
+  <option>--out-path</option>.</para></listitem>
+
+  <listitem><para><command>fetchurl</command> allows SHA-1 and SHA-256
+  in addition to MD5.  Just specify the attribute
+  <varname>sha1</varname> or <varname>sha256</varname> instead of
+  <varname>md5</varname>.</para></listitem>
+
+  <listitem><para>Manual updates.</para></listitem>
+
+</itemizedlist>
+
+</para>
+
+</section>
+
+
+
+<section><title>Release 0.7 (January 12, 2005)</title>
+
+<itemizedlist>
+
+  <listitem><para>Binary patching.  When upgrading components using
+  pre-built binaries (through nix-pull / nix-channel), Nix can
+  automatically download and apply binary patches to already installed
+  components instead of full downloads.  Patching is "smart": if there
+  is a *sequence* of patches to an installed component, Nix will use
+  it.  Patches are currently generated automatically between Nixpkgs
+  (pre-)releases.</para></listitem>
+
+  <listitem><para>Simplifications to the substitute
+  mechanism.</para></listitem>
+
+  <listitem><para>Nix-pull now stores downloaded manifests in
+  /nix/var/nix/manifests.</para></listitem>
+
+  <listitem><para>Metadata on files in the Nix store is canonicalised
+  after builds: the last-modified timestamp is set to 0 (00:00:00
+  1/1/1970), the mode is set to 0444 or 0555 (readable and possibly
+  executable by all; setuid/setgid bits are dropped), and the group is
+  set to the default.  This ensures that the result of a build and an
+  installation through a substitute is the same; and that timestamp
+  dependencies are revealed.</para></listitem>
+
+</itemizedlist>
+
+</section>
+
+
+
+<section><title>Release 0.6 (November 14, 2004)</title>
+
+<itemizedlist>
+
+  <listitem>
+    <para>Rewrite of the normalisation engine.
+
+    <itemizedlist>
+
+      <listitem><para>Multiple builds can now be performed in parallel
+      (option <option>-j</option>).</para></listitem>
+
+      <listitem><para>Distributed builds.  Nix can now call a shell
+      script to forward builds to Nix installations on remote
+      machines, which may or may not be of the same platform
+      type.</para></listitem>
+
+      <listitem><para>Option <option>--fallback</option> allows
+      recovery from broken substitutes.</para></listitem>
+
+      <listitem><para>Option <option>--keep-going</option> causes
+      building of other (unaffected) derivations to continue if one
+      failed.</para></listitem>
+
+    </itemizedlist>
+
+    </para>
+
+  </listitem>
+
+  <listitem><para>Improvements to the garbage collector (i.e., it
+  should actually work now).</para></listitem>
+
+  <listitem><para>Setuid Nix installations allow a Nix store to be
+  shared among multiple users.</para></listitem>
+
+  <listitem><para>Substitute registration is much faster
+  now.</para></listitem>
+
+  <listitem><para>A utility <command>nix-build</command> to build a
+  Nix expression and create a symlink to the result int the current
+  directory; useful for testing Nix derivations.</para></listitem>
+
+  <listitem><para>Manual updates.</para></listitem>
+
+  <listitem>
+
+    <para><command>nix-env</command> changes:
+
+    <itemizedlist>
+
+      <listitem><para>Derivations for other platforms are filtered out
+      (which can be overriden using
+      <option>--system-filter</option>).</para></listitem>
+
+      <listitem><para><option>--install</option> by default now
+      uninstall previous derivations with the same
+      name.</para></listitem>
+
+      <listitem><para><option>--upgrade</option> allows upgrading to a
+      specific version.</para></listitem>
+
+      <listitem><para>New operation
+      <option>--delete-generations</option> to remove profile
+      generations (necessary for effective garbage
+      collection).</para></listitem>
+
+      <listitem><para>Nicer output (sorted,
+      columnised).</para></listitem>
+
+    </itemizedlist>
+    
+    </para>
+
+  </listitem>
+
+  <listitem><para>More sensible verbosity levels all around (builder
+  output is now shown always, unless <option>-Q</option> is
+  given).</para></listitem>
+
+  <listitem>
+
+    <para>Nix expression language changes:
+
+    <itemizedlist>
+      
+      <listitem><para>New language construct: <literal>with
+      <replaceable>E1</replaceable>;
+      <replaceable>E2</replaceable></literal> brings all attributes
+      defined in the attribute set <replaceable>E1</replaceable> in
+      scope in <replaceable>E2</replaceable>.</para></listitem>
+
+      <listitem><para>Added a <function>map</function>
+      function.</para></listitem>
+
+      <listitem><para>Various new operators (e.g., string
+      concatenation).</para></listitem>
+
+    </itemizedlist>
+
+    </para>
+
+  </listitem>
+
+  <listitem><para>Expression evaluation is much
+  faster.</para></listitem>
+
+  <listitem><para>An Emacs mode for editing Nix expressions (with
+  syntax highlighting and indentation) has been
+  added.</para></listitem>
+
+  <listitem><para>Many bug fixes.</para></listitem>
+
+</itemizedlist>
+
+</section>
+
+
+
+<section><title>Release 0.5 and earlier</title>
+
+<para>Please refer to the Subversion commit log messages.</para>
+
+</section>
+
+
+
+</article>
diff --git a/doc/manual/writing-nix-expressions.xml b/doc/manual/writing-nix-expressions.xml
index d9c78dbecb..222092f638 100644
--- a/doc/manual/writing-nix-expressions.xml
+++ b/doc/manual/writing-nix-expressions.xml
@@ -1047,6 +1047,11 @@ weakest binding).</para>
         <replaceable>id</replaceable>.</entry>
       </row>
       <row>
+        <entry><replaceable>e1</replaceable> ++ <replaceable>e2</replaceable></entry>
+        <entry>right</entry>
+        <entry>List concatenation.</entry>
+      </row>
+      <row>
         <entry><replaceable>e1</replaceable> + <replaceable>e2</replaceable></entry>
         <entry>left</entry>
         <entry>String or path concatenation.</entry>