about summary refs log tree commit diff
diff options
context:
space:
mode:
Diffstat
-rw-r--r--tvix/docs/src/SUMMARY.md3
-rw-r--r--tvix/docs/src/nix-daemon/changelog.md2
-rw-r--r--tvix/docs/src/nix-daemon/handshake.md32
-rw-r--r--tvix/docs/src/nix-daemon/logging.md42
-rw-r--r--tvix/docs/src/nix-daemon/operations.md176
-rw-r--r--tvix/docs/src/nix-daemon/serialization.md202
6 files changed, 303 insertions, 154 deletions
diff --git a/tvix/docs/src/SUMMARY.md b/tvix/docs/src/SUMMARY.md
index a912832eb3b7..d64aa4b4b1b2 100644
--- a/tvix/docs/src/SUMMARY.md
+++ b/tvix/docs/src/SUMMARY.md
@@ -39,7 +39,8 @@
 - [Nix language version history](./lang-version.md)
 - [Value Pointer Equality](./value-pointer-equality.md)
 - [Daemon Protocol](./nix-daemon/index.md)
-  - [Changelog](./nix-daemon/changelog.md)
+  - [Handshake](./nix-daemon/handshake.md)
   - [Logging](./nix-daemon/logging.md)
   - [Operations](./nix-daemon/operations.md)
   - [Serialization](./nix-daemon/serialization.md)
+  - [Changelog](./nix-daemon/changelog.md)
diff --git a/tvix/docs/src/nix-daemon/changelog.md b/tvix/docs/src/nix-daemon/changelog.md
index bc99dc6af00d..41c168374c50 100644
--- a/tvix/docs/src/nix-daemon/changelog.md
+++ b/tvix/docs/src/nix-daemon/changelog.md
@@ -34,7 +34,7 @@ So this means that any version of Nix 2.x can't talk to Nix 0.x.
 | --------------- | -- | -------------- | -------- | ----------- | ----- |
 | *Quit           | 0  | [a711689368][a711689368] || 0.11 | Became dead code in [7951c3c54][7951c3c54] (Nix 0.11) and removed in [d3c61d83b][d3c61d83b] (Nix 1.8) |
 | IsValidPath     | 1  | [a711689368][a711689368] || 0.11 ||
-| HasSubstitutes  | 3  | [0565b5f2b3][0565b5f2b3] || 0.11 ||
+| HasSubstitutes  | 3  | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [09a6321aeb][09a6321aeb]<br>Nix 1.2 Protocol 1.12 |
 | QueryPathHash   | 4  | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [e0204f8d46][e0204f8d46]<br>Nix 2.0 Protocol 1.16 |
 | QueryReferences | 5  | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [e0204f8d46][e0204f8d46]<br>Nix 2.0 Protocol 1.16 |
 | QueryReferrers  | 6  | [0565b5f2b3][0565b5f2b3] || 0.11 ||
diff --git a/tvix/docs/src/nix-daemon/handshake.md b/tvix/docs/src/nix-daemon/handshake.md
new file mode 100644
index 000000000000..0a436372b3ff
--- /dev/null
+++ b/tvix/docs/src/nix-daemon/handshake.md
@@ -0,0 +1,32 @@
+
+
+## client -> server
+- 0x6e697863 :: [Int](#int) (hardcoded, 'nixc' in ASCII)
+
+## server -> client
+- 0x6478696f :: [Int](#int) (hardcoded, 'dxio' in ASCII)
+- protocolVersion :: [Int](#int)
+
+## client -> server
+- clientVersion :: [Int](#int)
+
+### If clientVersion is 1.14 or later
+- sendCpu :: [Bool](#bool) (hardcoded to false in client)
+#### If sendCpu is true
+- cpuAffinity :: [Int](#int) (obsolete and ignored)
+
+### If clientVersion is 1.11 or later
+- reserveSpace :: [Bool](#bool) (obsolete, ignored and set to false)
+
+
+## server -> client
+
+### If clientVersion is 1.33 or later
+- nixVersion :: String
+
+### If clientVersion is 1.35 or later
+- trusted :: OptTrusted
+
+## server -> client
+- send logs
+- [operation](./operations.md) :: Int
\ No newline at end of file
diff --git a/tvix/docs/src/nix-daemon/logging.md b/tvix/docs/src/nix-daemon/logging.md
index 70fe0882fa6d..c2828b13c21a 100644
--- a/tvix/docs/src/nix-daemon/logging.md
+++ b/tvix/docs/src/nix-daemon/logging.md
@@ -13,20 +13,21 @@ While not in this state between request reading and response sending all
 messages and activities are buffered until next time the logger can send data.
 
 The logging messages supported are:
-- `STDERR_LAST`
-- `STDERR_ERROR`
-- `STDERR_NEXT`
-- `STDERR_READ`
-- `STDERR_WRITE`
-- `STDERR_START_ACTIVITY`
-- `STDERR_STOP_ACTIVITY`
-- `STDERR_RESULT`
+- [`STDERR_LAST`](#stderr_last)
+- [`STDERR_ERROR`](#stderr_error)
+- [`STDERR_NEXT`](#stderr_next)
+- [`STDERR_READ`](#stderr_read)
+- [`STDERR_WRITE`](#stderr_write)
+- [`STDERR_START_ACTIVITY`](#stderr_start_activity)
+- [`STDERR_STOP_ACTIVITY`](#stderr_stop_activity)
+- [`STDERR_RESULT`](#stderr_result)
 
 
 ### `STDERR_LAST`
 Marks the end of the logs, normal processing can resume.
 
-- 0x616c7473 :: [UInt64][se-UInt64] (hardcoded)
+- 0x616c7473 :: [UInt64][se-UInt64] (hardcoded, 'alts' in ASCII)
+
 
 ### `STDERR_ERROR`
 This also marks the end of this log "session" and so it
@@ -34,20 +35,20 @@ has the same effect as `STDERR_LAST`.
 On the client the error is thrown as an exception and no response is read.
 
 #### If protocol version is 1.26 or newer
-- 0x63787470 :: [UInt64][se-UInt64] (hardcoded)
+- 0x63787470 :: [UInt64][se-UInt64] (hardcoded, 'cxtp' in ASCII)
 - error :: [Error][se-Error]
 
 #### If protocol version is older than 1.26
-- 0x63787470 :: [UInt64][se-UInt64] (hardcoded)
-- msg :: [String][se-String]
+- 0x63787470 :: [UInt64][se-UInt64] (hardcoded, 'cxtp' in ASCII)
+- msg :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 - exitStatus :: [Int][se-Int]
 
 
 ### `STDERR_NEXT`
 Normal string log message.
 
-- 0x6f6c6d67 :: [UInt64][se-UInt64] (hardcoded)
-- msg :: [String][se-String]
+- 0x6f6c6d67 :: [UInt64][se-UInt64] (hardcoded, 'olmg' in ASCII)
+- msg :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 
 
 ### `STDERR_READ`
@@ -56,15 +57,17 @@ It works by sending a desired buffer length and then on the receiver stream it
 reads bytes buffer of that length. If it receives 0 bytes it sees this as an
 unexpected EOF.
 
-- 0x64617461 :: [UInt64][se-UInt64] (hardcoded)
+- 0x64617461 :: [UInt64][se-UInt64] (hardcoded, 'data' in ASCII)
 - desiredLen :: [Size][se-Size]
 
+
 ### `STDERR_WRITE`
 Writer interface used by ExportPath. Simply writes a buffer.
 
 - 0x64617416 :: [UInt64][se-UInt64] (hardcoded)
 - buffer :: [Bytes][se-Bytes]
 
+
 ### `STDERR_START_ACTIVITY`
 Begins an activity. In other tracing frameworks this would be called a span.
 
@@ -73,11 +76,11 @@ versions of the protocol instead of sending an `STDERR_START_ACTIVITY`
 the level is checked against enabled logging level and the text field is
 sent as a simple log message with `STDERR_NEXT`.
 
-- 0x53545254 :: [UInt64][se-UInt64] (hardcoded)
+- 0x53545254 :: [UInt64][se-UInt64] (hardcoded, 'STRT' in ASCII)
 - act :: [UInt64][se-UInt64]
 - level :: [Verbosity][se-Verbosity]
 - type :: [ActivityType][se-ActivityType]
-- text :: [String][se-String]
+- text :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 - fields :: [List][se-List] of [Field][se-Field]
 - parent :: [UInt64][se-UInt64]
 
@@ -92,7 +95,7 @@ Just sends `ActivityId`.
 Implemented in protocol 1.20. When backwards compatible with older versions of
 the protocol and this message would have been sent it is instead ignored.
 
-- 0x53544f50 :: [UInt64][se-UInt64] (hardcoded)
+- 0x53544f50 :: [UInt64][se-UInt64] (hardcoded, 'STOP' in ASCII)
 
 
 ### `STDERR_RESULT`
@@ -101,14 +104,13 @@ Sends results for a given activity.
 Implemented in protocol 1.20. When backwards compatible with older versions of
 the protocol and this message would have been sent it is instead ignored.
 
-- 0x52534c54 :: [UInt64][se-UInt64] (hardcoded)
+- 0x52534c54 :: [UInt64][se-UInt64] (hardcoded, 'RSLT' in ASCII)
 - act :: [UInt64][se-UInt64]
 - type :: [ResultType][se-ResultType]
 - fields :: [List][se-List] of [Field][se-Field]
 
 
 
-
 [se-UInt64]: ./serialization.md#uint64
 [se-Int]: ./serialization.md#int
 [se-Size]: ./serialization.md#size
diff --git a/tvix/docs/src/nix-daemon/operations.md b/tvix/docs/src/nix-daemon/operations.md
index 0683ab070973..80708c9104b5 100644
--- a/tvix/docs/src/nix-daemon/operations.md
+++ b/tvix/docs/src/nix-daemon/operations.md
@@ -14,11 +14,9 @@
 | [FindRoots](#findroots)                                     | 14 |
 | [SetOptions](#setoptions)                                   | 19 |
 | [CollectGarbage](#collectgarbage)                           | 20 |
-| [QuerySubstitutablePathInfo](#querysubstitutablepathinfo)   | 21 |
 | [QueryAllValidPaths](#queryallvalidpaths)                   | 23 |
 | [QueryPathInfo](#querypathinfo)                             | 26 |
 | [QueryPathFromHashPart](#querypathfromhashpart)             | 29 |
-| [QuerySubstitutablePathInfos](#querysubstitutablepathinfos) | 30 |
 | [QueryValidPaths](#queryvalidpaths)                         | 31 |
 | [QuerySubstitutablePaths](#querysubstitutablepaths)         | 32 |
 | [QueryValidDerivers](#queryvalidderivers)                   | 33 |
@@ -40,17 +38,19 @@
 
 ## Obsolete operations
 
-| Operation                                                 | Id |
-| --------------------------------------------------------- | -- |
-| [QueryPathHash](#querypathhash)                           | 4  |
-| [QueryReferences](#queryreferences)                       | 5  |
-| [AddTextToStore](#addtexttostore)                         | 8  |
-| [SyncWithGC](#syncwithgc)                                 | 13 |
-| [ExportPath](#exportpath)                                 | 16 |
-| [QueryDeriver](#queryderiver)                             | 18 |
-| [QueryDerivationOutputs](#queryderivationoutputs)         | 22 |
-| [ImportPaths](#importpaths)                               | 27 |
-| [QueryDerivationOutputNames](#queryderivationoutputnames) | 28 |
+| Operation                                                   | Id |
+| ----------------------------------------------------------- | -- |
+| [QueryPathHash](#querypathhash)                             | 4  |
+| [QueryReferences](#queryreferences)                         | 5  |
+| [AddTextToStore](#addtexttostore)                           | 8  |
+| [SyncWithGC](#syncwithgc)                                   | 13 |
+| [ExportPath](#exportpath)                                   | 16 |
+| [QueryDeriver](#queryderiver)                               | 18 |
+| [QuerySubstitutablePathInfo](#querysubstitutablepathinfo)   | 21 |
+| [QueryDerivationOutputs](#queryderivationoutputs)           | 22 |
+| [ImportPaths](#importpaths)                                 | 27 |
+| [QueryDerivationOutputNames](#queryderivationoutputnames)   | 28 |
+| [QuerySubstitutablePathInfos](#querysubstitutablepathinfos) | 30 |
 
 
 ## Removed operations
@@ -81,6 +81,7 @@ As the name says checks that a store path is valid i.e. in the store.
 
 This is a pretty core operation used everywhere.
 
+
 ### Inputs
 path :: [StorePath][se-StorePath]
 
@@ -92,6 +93,9 @@ isValid :: [Bool][se-Bool]
 
 **Id:** 3<br>
 **Introduced:** Nix 0.11<br>
+**Obsolete** Protocol 1.12, Nix 1.2<br>
+
+Replaced by QuerySubstitutablePaths.
 
 Checks if we can substitute the input path from a substituter. Uses
 QuerySubstitutablePaths under the hood :/
@@ -115,7 +119,7 @@ Retrieves the base16 NAR hash of a given store path.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-hash :: [String][se-String] (base16-encoded NAR hash without algorithm prefix)
+hash :: [NARHash][se-NARHash]
 
 
 ## QueryReferences
@@ -130,7 +134,7 @@ Retrieves the references of a given path
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-references :: [List][se-List] of [StorePath][se-StorePath]
+references :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryReferrers
@@ -144,7 +148,7 @@ Retrieves the referrers of a given path.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-referrers :: [List][se-List] of [StorePath][se-StorePath]
+referrers :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## AddToStore
@@ -156,16 +160,16 @@ Add a new path to the store.
 
 ### Before protocol version 1.25
 #### Inputs
-- baseName :: [String][se-String]
+- baseName :: [StorePathName][se-StorePathName]
 - fixed :: [Bool64][se-Bool64]
 - recursive :: [FileIngestionMethod][se-FileIngestionMethod]
-- hashAlgo :: [String][se-String]
+- hashAlgo :: [HashAlgorithm][se-HashAlgorithm]
 - NAR dump
 
 If fixed is `true`, hashAlgo is forced to `sha256` and recursive is forced to
-`Recursive`.
+`NixArchive`.
 
-Only `Flat` and `Recursive` values are supported for the recursive input
+Only `Flat` and `NixArchive` values are supported for the recursive input
 parameter.
 
 #### Outputs
@@ -173,9 +177,9 @@ path :: [StorePath][se-StorePath]
 
 ### Protocol version 1.25 or newer
 #### Inputs
-- name :: [String][se-String]
+- name :: [StorePathName][se-StorePathName]
 - camStr :: [ContentAddressMethodWithAlgo][se-ContentAddressMethodWithAlgo]
-- refs :: [List][se-List] of [StorePath][se-StorePath]
+- refs :: [Set][se-Set] of [StorePath][se-StorePath]
 - repairBool :: [Bool64][se-Bool64]
 - [Framed][se-Framed] NAR dump
 
@@ -197,9 +201,9 @@ to [AddToStore](#addtostore). And so this corresponds to calling
 wrapped as a NAR.
 
 ### Inputs
-- suffix :: [String][se-String]
+- suffix :: [StorePathName][se-StorePathName]
 - text :: [Bytes][se-Bytes]
-- refs :: [List][se-List] of [StorePath][se-StorePath]
+- refs :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### Outpus
 path :: [StorePath][se-StorePath]
@@ -213,15 +217,15 @@ path :: [StorePath][se-StorePath]
 Build (or substitute) a list of derivations.
 
 ### Inputs
-paths :: [List][se-List] of [DerivedPath][se-DerivedPath]
+paths :: [Set][se-Set] of [DerivedPath][se-DerivedPath]
 
 #### Protocol 1.15 or newer
-mode :: [BuildMode][se-BuildMode]
+mode :: [BuildMode][se-BuildMode] (defaults to Normal)
 
 Check that connection is trusted before allowing Repair mode.
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## EnsurePath
@@ -235,7 +239,7 @@ Checks if a path is valid. Note: it may be made valid by running a substitute.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## AddTempRoot
@@ -253,7 +257,7 @@ deleting store paths that the client is actively doing something with.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## AddIndirectRoot
@@ -267,10 +271,10 @@ created by [AddPermRoot](#addpermroot).
 Only ever sent on the local unix socket nix daemon.
 
 ### Inputs
-path :: [String][se-String]
+path :: [Path][se-Path]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## SyncWithGC
@@ -308,7 +312,7 @@ thing.
 Find the GC roots.
 
 ### Outputs
-roots :: [Map][se-Map] of [String][se-String] to [StorePath][se-StorePath]
+roots :: [Map][se-Map] of [Path][se-Path] to [StorePath][se-StorePath]
 
 The key is the link pointing to the given store path.
 
@@ -333,7 +337,8 @@ Export a store path in the binary format nix-store --import expects. See impleme
 - sign :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 
 ### Outputs
-Uses `STDERR_WRITE` to send dump in export format
+Uses [`STDERR_WRITE`](./logging.md#stderr_write) to send dump in
+[export format][se-ExportFormat]
 
 After dump it outputs.
 
@@ -373,18 +378,18 @@ Only ever used right after the handshake.
 
 ### Inputs
 
-- keepFailed :: [Int][se-Int]
-- keepGoing :: [Int][se-Int]
-- tryFallback :: [Int][se-Int]
+- keepFailed :: [Bool][se-Bool]
+- keepGoing :: [Bool][se-Bool]
+- tryFallback :: [Bool][se-Bool]
 - verbosity :: [Verbosity][se-Verbosity]
-- maxbuildJobs :: [Int][se-Int]
-- maxSilentTime :: [Int][se-Int]
+- maxBuildJobs :: [Int][se-Int]
+- maxSilentTime :: [Time][se-Time]
 - useBuildHook :: [Bool][se-Bool] (ignored and hardcoded to true in client)
 - verboseBuild :: [Verbosity][se-Verbosity]
 - logType :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 - printBuildTrace :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 - buildCores :: [Int][se-Int]
-- useSubstitutes :: [Int][se-Int]
+- useSubstitutes :: [Bool][se-Bool]
 
 ### Protocol 1.12 or newer
 otherSettings :: [Map][se-Map] of [String][se-String] to [String][se-String]
@@ -399,7 +404,7 @@ Find the GC roots.
 
 ### Inputs
 - action :: [GCAction][se-GCAction]
-- pathsToDelete :: [List][se-List] of [StorePath][se-StorePath]
+- pathsToDelete :: [Set][se-Set] of [StorePath][se-StorePath]
 - ignoreLiveness :: [Bool64][se-Bool64]
 - maxFreed :: [UInt64][se-UInt64]
 - removed :: [Int][se-Int] (ignored and hardcoded to 0 in client)
@@ -407,18 +412,19 @@ Find the GC roots.
 - removed :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 
 ### Outputs
-- pathsDeleted :: [List][se-List] of [String][se-String]
+- pathsDeleted :: [Set][se-Set] of [Path][se-Path]
 - bytesFreed :: [UInt64][se-UInt64]
-- 0 :: [UInt64][se-UInt64] (hardcoded)
+- 0 :: [UInt64][se-UInt64] (hardcoded, obsolete and ignored by client)
 
-Depending on the action pathsDeleted is, the GC roots, or the paths that would
-be or have been deleted.
+Depending on the value of the action input the value of output pathsDeleted
+is either, the GC roots, or the paths that would be or have been deleted.
 
 
 ## QuerySubstitutablePathInfo
 
 **Id:** 21<br>
 **Introduced:** Protocol 1.02, Nix 0.12<br>
+**Obsolete:** Protocol 1.12, Nix 1.2<br>
 
 Retrieves the various substitutable paths infos for a given path.
 
@@ -429,10 +435,7 @@ path :: [StorePath][se-StorePath]
 found :: [Bool][se-Bool]
 
 #### If found is true
-- deriver :: [OptStorePath][se-OptStorePath]
-- references :: [List][se-List] of [StorePath][se-StorePath]
-- downloadSize :: [UInt64][se-UInt64]
-- narSize :: [UInt64][se-UInt64]
+- info :: [SubstitutablePathInfo][se-SubstitutablePathInfo]
 
 
 ## QueryDerivationOutputs
@@ -447,7 +450,7 @@ Retrieves all the outputs paths of a given derivation.
 path :: [StorePath][se-StorePath] (must point to a derivation)
 
 ### Outputs
-derivationOutputs :: [List][se-List] of [StorePath][se-StorePath]
+derivationOutputs :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryAllValidPaths
@@ -458,7 +461,7 @@ derivationOutputs :: [List][se-List] of [StorePath][se-StorePath]
 Retrieves all the valid paths contained in the store.
 
 ### Outputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryFailedPaths (removed)
@@ -498,7 +501,7 @@ success :: [Bool64][se-Bool64]
 pathInfo :: [UnkeyedValidPathInfo][se-UnkeyedValidPathInfo]
 
 #### If protocol version is older than 1.17
-If info not found return error with `STDERR_ERROR`
+If info not found return error with [`STDERR_ERROR`](./logging.md#stderr_error)
 
 pathInfo :: [UnkeyedValidPathInfo][se-UnkeyedValidPathInfo]
 
@@ -516,7 +519,7 @@ before the metadata about the store path and so you would typically have
 to store the NAR in memory or temporarily on disk before processing it.
 
 ### Inputs
-List of NAR dumps coming from the ExportPaths operations.
+[List of NAR dumps][se-ImportPaths] coming from one or more ExportPath operations.
 
 ### Outputs
 importedPaths :: [List][se-List] of [StorePath][se-StorePath]
@@ -534,7 +537,7 @@ Retrieves the name of the outputs of a given derivation. EG. out, dev, etc.
 path :: [StorePath][se-StorePath] (must be a derivation path)
 
 ### Outputs
-names :: [List][se-List] of [String][se-String]
+names :: [Set][se-Set] of [OutputName][se-OutputName]
 
 
 ## QueryPathFromHashPart
@@ -542,11 +545,10 @@ names :: [List][se-List] of [String][se-String]
 **Id:** 29<br>
 **Introduced:** Protocol 1.11, Nix 1.1<br>
 
-Retrieves a store path from a base16 (input) hash. Returns "" if no path was
-found.
+Retrieves a store path from a nixbase32 (input) hash.
 
 ### Inputs
-hashPart :: [String][se-String]  (must be a base-16 hash)
+hashPart :: [StorePathHash][se-StorePathHash]
 
 ### Outputs
 path :: [OptStorePath][se-OptStorePath]
@@ -556,18 +558,22 @@ path :: [OptStorePath][se-OptStorePath]
 
 **Id:** 30<br>
 **Introduced:** Protocol 1.12*, Nix 1.2<br>
+**Obsolete:** Protocol 1.19*, Nix 2.0<br>
 
 Retrieves the various substitutable paths infos for set of store paths.
 
+Only ever used in the fallback for QueryMissing which means that if protocol is 1.19 or later
+it is never sent and is therefore obsolete after that.
+
 ### Inputs
 #### If protocol version is 1.22 or newer
 paths :: [Map][se-Map] of [StorePath][se-StorePath] to [OptContentAddress][se-OptContentAddress] 
 
 #### If protocol version older than 1.22
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### Outputs
-infos :: [List][se-List] of [SubstitutablePathInfo][se-SubstitutablePathInfo]
+infos :: [Map][se-Map] of [StorePath][se-StorePath] to [SubstitutablePathInfo][se-SubstitutablePathInfo]
 
 
 ## QueryValidPaths
@@ -578,13 +584,13 @@ infos :: [List][se-List] of [SubstitutablePathInfo][se-SubstitutablePathInfo]
 Takes a list of store paths and returns a new list only containing the valid store paths
 
 ## Inputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### If protocol version is 1.27 or newer
 substitute :: [Bool][se-Bool] (defaults to false if not sent)
 
 ## Outputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QuerySubstitutablePaths
@@ -599,10 +605,10 @@ In versions of the protocol prior to 1.12 [HasSubstitutes](#hassubstitutes)
 is used to implement the functionality that this operation provides.
 
 ### Inputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### Outputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryValidDerivers
@@ -616,7 +622,7 @@ Retrieves the derivers of a given path.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-derivers :: [List][se-List] of [StorePath][se-StorePath]
+derivers :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## OptimiseStore
@@ -627,7 +633,7 @@ derivers :: [List][se-List] of [StorePath][se-StorePath]
 Optimise store by hardlinking files with the same content.
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## VerifyStore
@@ -677,14 +683,14 @@ buildResult :: [BuildResult][se-BuildResult]
 **Id:** 37<br>
 **Introduced:** Protocol 1.16, Nix 2.0<br>
 
-Add the signatures associated to a given path.
+Add the signatures associated to a given path. Used by `nix store copy-sigs` and `nix store sign`.
 
 ### Inputs
 - path :: [StorePath][se-StorePath]
-- signatures :: [List][se-List] of [String][se-String]
+- signatures :: [Set][se-Set] of [Signature][se-Signature]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## NarFromPath
@@ -717,12 +723,12 @@ Dumps a path as a NAR
 ### Inputs
 - path :: [StorePath][se-StorePath]
 - deriver :: [OptStorePath][se-OptStorePath]
-- narHash :: [String][se-String] SHA256 NAR hash base 16
-- references :: [List][se-List] of [StorePath][se-StorePath]
+- narHash :: [NARHash][se-NARHash]
+- references :: [Set][se-Set] of [StorePath][se-StorePath]
 - registrationTime :: [Time][se-Time]
 - narSize :: [UInt64][se-UInt64]
 - ultimate :: [Bool64][se-Bool64]
-- signatures :: [List][se-List] of [String][se-String]
+- signatures :: [Set][se-Set] of [Signature][se-Signature]
 - ca :: [OptContentAddress][se-OptContentAddress]
 - repair :: [Bool64][se-Bool64]
 - dontCheckSigs :: [Bool64][se-Bool64]
@@ -731,7 +737,7 @@ Dumps a path as a NAR
 [Framed][se-Framed] NAR dump
 
 #### If protocol version is between 1.21 and 1.23
-NAR dump sent using `STDERR_READ`
+NAR dump sent using [`STDERR_READ`](./logging.md#stderr_read)
 
 #### If protocol version is older than 1.21
 NAR dump sent raw on stream
@@ -746,9 +752,9 @@ NAR dump sent raw on stream
 targets :: [List][se-List] of [DerivedPath][se-DerivedPath]
 
 ### Outputs
-- willBuild :: [List][se-List] of [StorePath][se-StorePath]
-- willSubstitute :: [List][se-List] of [StorePath][se-StorePath]
-- unknown :: [List][se-List] of [StorePath][se-StorePath]
+- willBuild :: [Set][se-Set] of [StorePath][se-StorePath]
+- willSubstitute :: [Set][se-Set] of [StorePath][se-StorePath]
+- unknown :: [Set][se-Set] of [StorePath][se-StorePath]
 - downloadSize :: [UInt64][se-UInt64]
 - narSize :: [UInt64][se-UInt64]
 
@@ -764,7 +770,7 @@ Retrieves an associative map outputName -> storePath for a given derivation.
 path :: [StorePath][se-StorePath]  (must be a derivation path)
 
 ### Outputs
-outputs :: [Map][se-Map] of [String][se-String] to [OptStorePath][se-OptStorePath]
+outputs :: [Map][se-Map] of [OutputName][se-OutputName] to [OptStorePath][se-OptStorePath]
 
 
 ## RegisterDrvOutput
@@ -795,10 +801,10 @@ outputId :: [DrvOutput][se-DrvOutput]
 
 ### Outputs
 #### If protocol is 1.31 or newer
-realisations :: [List][se-List] of [Realisation][se-Realisation]
+realisations :: [Set][se-Set] of [Realisation][se-Realisation]
 
 #### If protocol is older than 1.31
-outPaths :: [List][se-List] of [BaseStorePath][se-BaseStorePath]
+outPaths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## AddMultipleToStore
@@ -816,7 +822,7 @@ for each small NAR was costly.
 ### Inputs
 - repair :: [Bool64][se-Bool64]
 - dontCheckSigs :: [Bool64][se-Bool64]
-- [Framed][se-Framed] stream of add multiple NAR dump
+- [Framed][se-Framed] stream of [add multiple NAR dump][se-AddMultipleToStore]
 
 
 ## AddBuildLog
@@ -827,11 +833,11 @@ for each small NAR was costly.
 Attach some build logs to a given build.
 
 ### Inputs
-- path :: [String][se-String] (might be [BaseStorePath][se-BaseStorePath])
+- path :: [BaseStorePath][se-BaseStorePath]
 - [Framed][se-Framed] stream of log lines
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## BuildPathsWithResults
@@ -856,10 +862,10 @@ results :: [List][se-List] of [KeyedBuildResult][se-KeyedBuildResult]
 
 ### Inputs
 - storePath :: [StorePath][se-StorePath]
-- gcRoot :: [String][se-String]
+- gcRoot :: [Path][se-Path]
 
 ### Outputs
-gcRoot :: [String][se-String]
+gcRoot :: [Path][se-Path]
 
 
 
@@ -884,6 +890,7 @@ gcRoot :: [String][se-String]
 [se-DrvOutput]: ./serialization.md#drvoutput
 [se-Realisation]: ./serialization.md#realisation
 [se-List]: ./serialization.md#list-of-x
+[se-Set]: ./serialization.md#set-of-x
 [se-Map]: ./serialization.md#map-of-x-to-y
 [se-SubstitutablePathInfo]: ./serialization.md#substitutablepathinfo
 [se-ValidPathInfo]: ./serialization.md#validpathinfo
@@ -891,4 +898,7 @@ gcRoot :: [String][se-String]
 [se-BuildResult]: ./serialization.md#buildmode
 [se-KeyedBuildResult]: ./serialization.md#keyedbuildresult
 [se-BasicDerivation]: ./serialization.md#basicderivation
-[se-Framed]: ./serialization.md#framed
\ No newline at end of file
+[se-Framed]: ./serialization.md#framed
+[se-AddMultipleToStore]: ./serialization.md#addmultipletostore-format
+[se-ExportFormat]: ./serialization.md#export-path-format
+[se-ImportPaths]: ./serialization.md#import-paths-format
\ No newline at end of file
diff --git a/tvix/docs/src/nix-daemon/serialization.md b/tvix/docs/src/nix-daemon/serialization.md
index a2694a4dea30..1042c956ba71 100644
--- a/tvix/docs/src/nix-daemon/serialization.md
+++ b/tvix/docs/src/nix-daemon/serialization.md
@@ -6,7 +6,7 @@ Little endian byte order
 
 - len :: [UInt64](#uint64)
 - len bytes of content
-- padding with zeros to ensure 64 bit alignment of content with padding
+- padding with zeros to ensure 64 bit alignment of content + padding
 
 
 ## Int serializers
@@ -16,6 +16,9 @@ Little endian byte order
 
 ### Int64
 [UInt64](#uint64) cast to C `int64_t` with upper bounds checking.
+This means that negative numbers can be written but not read.
+Since this is only used for cpuSystem and cpuUser it is fine that
+negative numbers aren't supported.
 
 ### UInt8
 [UInt64](#uint64) cast to C `uint8_t` with upper bounds checking.
@@ -25,7 +28,8 @@ Little endian byte order
 
 ### Time
 [UInt64](#uint64) cast to C `time_t` with upper bounds checking.
-s
+This means that negative numbers can be written but not read.
+
 ### Bool
 Sent as an [Int](#int) where 0 is false and everything else is true.
 
@@ -35,10 +39,10 @@ Sent as an [UInt64](#uint64) where 0 is false and everything else is true.
 ### FileIngestionMethod
 An [UInt8](#uint8) enum with the following possible values:
 
-| Name      | Int |
-| --------- | --- |
-| Flat      |  0  |
-| Recursive |  1  |
+| Name       | Int |
+| ---------- | --- |
+| Flat       |  0  |
+| NixArchive |  1  |
 
 ### BuildMode
 An [Int](#int) enum with the following possible values:
@@ -137,6 +141,15 @@ An [Int](#int) enum with the following possible values:
 | Int    |  0  |
 | String |  1  |
 
+### OptTrusted
+An [UInt8](#uint8) optional enum with the following possible values:
+
+| Name             | Int |
+| ---------------- | --- |
+| None             |  0  |
+| Some(Trusted)    |  1  |
+| Some(NotTrusted) |  2  |
+
 
 ## Bytes serializers
 
@@ -144,51 +157,107 @@ An [Int](#int) enum with the following possible values:
 Simply a [Bytes](#bytes) that has some UTF-8 string like semantics sometimes.
 
 ### StorePath
-String representation of a full store path.
+[String](#string) representation of a full store path including the store directory.
 
 ### BaseStorePath
-String representation of the basename of a store path. That is the store path
+[String](#string) representation of the basename of a store path. That is the store path
 without the /nix/store prefix.
 
+### StorePathName
+[String](#string) representation of the name part of a base store path. This is the part
+of the store path after the nixbase32 hash and '-'
+
+It must have the following format:
+- Deny ".", "..", or those strings followed by '-'
+- Otherwise check that each character is 0-9, a-z, A-Z or one of +-._?=
+
+### StorePathHash
+[String](#string) representation of the hash part of a base store path. This is the part
+of the store path at the beginning and before the '-' and is in nixbase32 format.
+
+
+### OutputName
+[String](#string) representation of the name of a derivation output.
+This is usually combined with the name in the derivation to form the store path name for the
+store path with this output.
+
+Since output name is usually combined to form a store path name its format must follow the
+same rules as [StorePathName](#storepathname):
+- Deny ".", "..", or those strings followed by '-'
+- Otherwise check that each character is 0-9, a-z, A-Z or one of +-._?=
+
+
 ### OptStorePath
 Optional store path.
 
 If no store path this is serialized as the empty string otherwise it is the same as
 [StorePath](#storepath).
 
+### Path
+[String](#string) representation of an absolute path.
+
+### NARHash
+[String](#string) base16-encoded NAR SHA256 hash without algorithm prefix.
+
+### Signature
+[String](#string) with a signature for the given store path or realisation. This should be
+in the format `name`:`base 64 encoded signature` but this is not enforced in the protocol.
+
+### HashAlgorithm
+[String](#string) with one of the following values:
+- md5
+- sha1
+- sha256
+- sha512
+
+### HashDigest
+[String](#string) with a hash digest in any encoding
+
+### OptHashDigest
+Optional version of [HashDigest](#hashdigest) where empty string means
+no value.
+
+
 ### ContentAddressMethodWithAlgo
-One of the following strings:
-- text:`hash algorithm`
-- fixed:r:`hash algorithm`
-- fixed:`hash algorithm`
+[String](#string) with one of the following formats:
+- text:[HashAlgorithm](#hashalgorithm)
+- fixed:r:[HashAlgorithm](#hashalgorithm)
+- fixed:[HashAlgorithm](#hashalgorithm)
 
-### DerivedPath
-#### If protocol is 1.30 or newer
-        return DerivedPath::parseLegacy(store, s);
-#### If protocol is older than 1.30
-        return parsePathWithOutputs(store, s).toDerivedPath();
+### OptContentAddressMethodWithAlgo
+Optional version of [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo)
+where empty string means no value.
 
 ### ContentAddress
-String with the format:
-- [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo):`hash`
+[String](#string) with the format:
+- [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo):[HashDigest](#hashdigest)
 
 ### OptContentAddress
 Optional version of [ContentAddress](#contentaddress) where empty string means
 no content address.
 
+### DerivedPath
+#### If protocol is 1.30 or newer
+output-names = [OutputName](#outputname), { "," [OutputName](#outputname) }<br>
+output-spec = "*" | output-names<br>
+derived-path = [StorePath](#storepath), [ "!", output-spec ]<br>
+
+#### If protocol is older than 1.30
+[StorePath](#storepath), [ "!", [OutputName](#outputname), { "," [OutputName](#outputname) } ]
+
 ### DrvOutput
-String with format:
-- `hash with any prefix`!`output name`
+[String](#string) with format:
+- `hash with any prefix` "!" [OutputName](#outputname)
 
 ### Realisation
-A JSON object sent as a string.
+A JSON object sent as a [String](#string).
 
 The JSON object has the following keys:
-| Key                   | Value                   |
-| --------------------- | ----------------------- |
-| id                    | [DrvOutput](#drvoutput) |
-| outPath               | [StorePath](#storepath) |
-| signatures            | Array of String         |
+| Key                   | Value                            |
+| --------------------- | -------------------------------- |
+| id                    | [DrvOutput](#drvoutput)          |
+| outPath               | [StorePath](#storepath)          |
+| signatures            | Array of [Signature](#signature) |
 | dependentRealisations | Object where key is [DrvOutput](#drvoutput) and value is [StorePath](#storepath) |
 
 
@@ -200,20 +269,22 @@ A list is encoded as a [Size](#size) length n followed by n encodings of x
 ### Map of x to y
 A map is encoded as a [Size](#size) length n followed by n encodings of pairs of x and y
 
+### Set of x
+A set is encoded as a [Size](#size) length n followed by n encodings of x
 
 ### BuildResult
 - status :: [BuildStatus](#buildstatus)
 - errorMsg :: [String](#string)
 
 #### Protocol 1.29 or newer
-- timesBuilt :: [Int](#int)
-- isNonDeterministic :: [Bool64](#bool64)
-- startTime :: [Time](#time)
-- stopTime :: [Time](#time)
+- timesBuilt :: [Int](#int) (defaults to 0)
+- isNonDeterministic :: [Bool64](#bool64) (defaults to false)
+- startTime :: [Time](#time) (defaults to 0)
+- stopTime :: [Time](#time) (defaults to 0)
 
 #### Protocol 1.37 or newer
-- cpuUser :: [OptMicroseconds](#optmicroseconds)
-- cpuSystem :: [OptMicroseconds](#optmicroseconds)
+- cpuUser :: [OptMicroseconds](#optmicroseconds) (defaults to none)
+- cpuSystem :: [OptMicroseconds](#optmicroseconds) (defaults to none)
 
 #### Protocol 1.28 or newer
 builtOutputs ::  [Map](#map-of-x-to-y) of [DrvOutput](#drvoutput) to [Realisation](#realisations)
@@ -232,23 +303,22 @@ Optional microseconds.
 
 
 ### SubstitutablePathInfo
-- storePath :: [StorePath](#storepath)
 - deriver :: [OptStorePath](#optstorepath)
-- references :: [List](#list-of-x) of [StorePath](#storepath)
+- references :: [Set][#set-of-x] of [StorePath](#storepath)
 - downloadSize :: [UInt64](#uint64)
 - narSize :: [UInt64](#uint64)
 
 
 ### UnkeyedValidPathInfo
 - deriver :: [OptStorePath](#optstorepath)
-- narHash :: [String](#string) SHA256 NAR hash base16 encoded
-- references :: [List](#list-of-x) of [StorePath](#storepath)
+- narHash :: [NARHash](#narhash)
+- references :: [Set](#set-of-x) of [StorePath](#storepath)
 - registrationTime :: [Time](#time)
 - narSize :: [UInt64](#uint64)
 
 #### If protocol version is 1.16 or above
-- ultimate :: [Bool64](#bool64)
-- signatures :: [List](#list-of-x) of [String](#string)
+- ultimate :: [Bool64](#bool64) (defaults to false)
+- signatures :: [Set](#set-of-x) of [Signature](#signature)
 - ca :: [OptContentAddress](#optcontentaddress)
 
 
@@ -257,13 +327,13 @@ Optional microseconds.
 - info :: [UnkeyedValidPathInfo](#unkeyedvalidpathinfo)
 
 ### DerivationOutput
-- path :: [String](#string)
-- hashAlgo :: [String](#string)
-- hash :: [String](#string)
+- path :: [OptStorePath](#optstorepath)
+- hashAlgo :: [OptContentAddressMethodWithAlgo](#optcontentaddressmethodwithalgo)
+- hash :: [OptHashDigest](#opthashdigest)
 
 ### BasicDerivation
-- outputs :: [Map](#map-of-x-to-y) of [String](#string) to [DerivationOutput](#derivationoutput)
-- inputSrcs :: [List](#list-of-x) of [StorePath](#storepath)
+- outputs :: [Map](#map-of-x-to-y) of [OutputName](#outputname) to [DerivationOutput](#derivationoutput)
+- inputSrcs :: [Set](#set-of-x) of [StorePath](#storepath)
 - platform :: [String](#string)
 - builder :: [String](#string)
 - args :: [List](#list-of-x) of [String](#string)
@@ -271,13 +341,13 @@ Optional microseconds.
 
 ### TraceLine
 - havePos :: [Size](#size) (hardcoded to 0)
-- hint :: [String](#string)
+- hint :: [String](#string) (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 
 ### Error
 - type :: [String](#string) (hardcoded to `Error`)
 - level :: [Verbosity](#verbosity)
 - name :: [String](#string) (removed and hardcoded to `Error`)
-- msg :: [String](#string)
+- msg :: [String](#string) (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 - havePos :: [Size](#size) (hardcoded to 0)
 - traces :: [List](#list-of-x) of [TraceLine](#traceline)
 
@@ -297,9 +367,43 @@ At protocol 1.23 [AddToStoreNar](./operations.md#addtostorenar) introduced a
 framed streaming for sending the NAR dump and later versions of the protocol
 also used this framing for other operations.
 
-At its core the framed streaming is just a series of [Bytes](#bytes) of
-varying length and terminated by an empty [Bytes](#bytes).
+At its core the framed streaming is just a series of [UInt64](#uint64) `size`
+followed by `size` bytes. The stream is terminated when `size` is zero.
+
+Unlike [Bytes](#bytes), frames are *NOT* padded.
 
 This method of sending data has the advantage of not having to parse the data
 to find where it ends. Older versions of the protocol would potentially parse
-the NAR twice.
\ No newline at end of file
+the NAR twice.
+
+
+## AddMultipleToStore format
+
+Paths must be topologically sorted.
+
+- expected :: [UInt64](#uint64)
+
+### Repeated expected times
+- info :: [ValidPathInfo](#validpathinfo)
+- NAR dump
+
+
+## Export path format
+- NAR dump
+- 0x4558494es :: [Int](#int) (hardcoded, 'EXIN' in ASCII)
+- path :: [StorePath](#storepath)
+- references :: [Set](#set-of-x) of [StorePath](#storepath)
+- deriver :: [OptStorePath](#optstorepath)
+- hasSignature :: [Int](#int) (hardcoded to 0 in newer versions)
+
+#### If hasSignature is 1
+- signature :: [String](#string) (ignored)
+
+
+## Import paths format
+
+- hasNext :: [UInt64](#uint64)
+
+### While hasNext is 1
+- [Export path format](#export-path-format)
+- hasNext :: [UInt64](#uint64)
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T11ยท55+0000