diff options
Diffstat (limited to 'tvix/docs/src/nix-daemon')
| -rw-r--r-- | tvix/docs/src/nix-daemon/changelog.md | 2 | ||||
| -rw-r--r-- | tvix/docs/src/nix-daemon/handshake.md | 32 | ||||
| -rw-r--r-- | tvix/docs/src/nix-daemon/logging.md | 42 | ||||
| -rw-r--r-- | tvix/docs/src/nix-daemon/operations.md | 176 | ||||
| -rw-r--r-- | tvix/docs/src/nix-daemon/serialization.md | 202 |
5 files changed, 301 insertions, 153 deletions
diff --git a/tvix/docs/src/nix-daemon/changelog.md b/tvix/docs/src/nix-daemon/changelog.md index bc99dc6af00d..41c168374c50 100644 --- a/tvix/docs/src/nix-daemon/changelog.md +++ b/tvix/docs/src/nix-daemon/changelog.md @@ -34,7 +34,7 @@ So this means that any version of Nix 2.x can't talk to Nix 0.x. | --------------- | -- | -------------- | -------- | ----------- | ----- | | *Quit | 0 | [a711689368][a711689368] || 0.11 | Became dead code in [7951c3c54][7951c3c54] (Nix 0.11) and removed in [d3c61d83b][d3c61d83b] (Nix 1.8) | | IsValidPath | 1 | [a711689368][a711689368] || 0.11 || -| HasSubstitutes | 3 | [0565b5f2b3][0565b5f2b3] || 0.11 || +| HasSubstitutes | 3 | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [09a6321aeb][09a6321aeb]<br>Nix 1.2 Protocol 1.12 | | QueryPathHash | 4 | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [e0204f8d46][e0204f8d46]<br>Nix 2.0 Protocol 1.16 | | QueryReferences | 5 | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [e0204f8d46][e0204f8d46]<br>Nix 2.0 Protocol 1.16 | | QueryReferrers | 6 | [0565b5f2b3][0565b5f2b3] || 0.11 || diff --git a/tvix/docs/src/nix-daemon/handshake.md b/tvix/docs/src/nix-daemon/handshake.md new file mode 100644 index 000000000000..0a436372b3ff --- /dev/null +++ b/tvix/docs/src/nix-daemon/handshake.md @@ -0,0 +1,32 @@ + + +## client -> server +- 0x6e697863 :: [Int](#int) (hardcoded, 'nixc' in ASCII) + +## server -> client +- 0x6478696f :: [Int](#int) (hardcoded, 'dxio' in ASCII) +- protocolVersion :: [Int](#int) + +## client -> server +- clientVersion :: [Int](#int) + +### If clientVersion is 1.14 or later +- sendCpu :: [Bool](#bool) (hardcoded to false in client) +#### If sendCpu is true +- cpuAffinity :: [Int](#int) (obsolete and ignored) + +### If clientVersion is 1.11 or later +- reserveSpace :: [Bool](#bool) (obsolete, ignored and set to false) + + +## server -> client + +### If clientVersion is 1.33 or later +- nixVersion :: String + +### If clientVersion is 1.35 or later +- trusted :: OptTrusted + +## server -> client +- send logs +- [operation](./operations.md) :: Int \ No newline at end of file diff --git a/tvix/docs/src/nix-daemon/logging.md b/tvix/docs/src/nix-daemon/logging.md index 70fe0882fa6d..c2828b13c21a 100644 --- a/tvix/docs/src/nix-daemon/logging.md +++ b/tvix/docs/src/nix-daemon/logging.md @@ -13,20 +13,21 @@ While not in this state between request reading and response sending all messages and activities are buffered until next time the logger can send data. The logging messages supported are: -- `STDERR_LAST` -- `STDERR_ERROR` -- `STDERR_NEXT` -- `STDERR_READ` -- `STDERR_WRITE` -- `STDERR_START_ACTIVITY` -- `STDERR_STOP_ACTIVITY` -- `STDERR_RESULT` +- [`STDERR_LAST`](#stderr_last) +- [`STDERR_ERROR`](#stderr_error) +- [`STDERR_NEXT`](#stderr_next) +- [`STDERR_READ`](#stderr_read) +- [`STDERR_WRITE`](#stderr_write) +- [`STDERR_START_ACTIVITY`](#stderr_start_activity) +- [`STDERR_STOP_ACTIVITY`](#stderr_stop_activity) +- [`STDERR_RESULT`](#stderr_result) ### `STDERR_LAST` Marks the end of the logs, normal processing can resume. -- 0x616c7473 :: [UInt64][se-UInt64] (hardcoded) +- 0x616c7473 :: [UInt64][se-UInt64] (hardcoded, 'alts' in ASCII) + ### `STDERR_ERROR` This also marks the end of this log "session" and so it @@ -34,20 +35,20 @@ has the same effect as `STDERR_LAST`. On the client the error is thrown as an exception and no response is read. #### If protocol version is 1.26 or newer -- 0x63787470 :: [UInt64][se-UInt64] (hardcoded) +- 0x63787470 :: [UInt64][se-UInt64] (hardcoded, 'cxtp' in ASCII) - error :: [Error][se-Error] #### If protocol version is older than 1.26 -- 0x63787470 :: [UInt64][se-UInt64] (hardcoded) -- msg :: [String][se-String] +- 0x63787470 :: [UInt64][se-UInt64] (hardcoded, 'cxtp' in ASCII) +- msg :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD) - exitStatus :: [Int][se-Int] ### `STDERR_NEXT` Normal string log message. -- 0x6f6c6d67 :: [UInt64][se-UInt64] (hardcoded) -- msg :: [String][se-String] +- 0x6f6c6d67 :: [UInt64][se-UInt64] (hardcoded, 'olmg' in ASCII) +- msg :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD) ### `STDERR_READ` @@ -56,15 +57,17 @@ It works by sending a desired buffer length and then on the receiver stream it reads bytes buffer of that length. If it receives 0 bytes it sees this as an unexpected EOF. -- 0x64617461 :: [UInt64][se-UInt64] (hardcoded) +- 0x64617461 :: [UInt64][se-UInt64] (hardcoded, 'data' in ASCII) - desiredLen :: [Size][se-Size] + ### `STDERR_WRITE` Writer interface used by ExportPath. Simply writes a buffer. - 0x64617416 :: [UInt64][se-UInt64] (hardcoded) - buffer :: [Bytes][se-Bytes] + ### `STDERR_START_ACTIVITY` Begins an activity. In other tracing frameworks this would be called a span. @@ -73,11 +76,11 @@ versions of the protocol instead of sending an `STDERR_START_ACTIVITY` the level is checked against enabled logging level and the text field is sent as a simple log message with `STDERR_NEXT`. -- 0x53545254 :: [UInt64][se-UInt64] (hardcoded) +- 0x53545254 :: [UInt64][se-UInt64] (hardcoded, 'STRT' in ASCII) - act :: [UInt64][se-UInt64] - level :: [Verbosity][se-Verbosity] - type :: [ActivityType][se-ActivityType] -- text :: [String][se-String] +- text :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD) - fields :: [List][se-List] of [Field][se-Field] - parent :: [UInt64][se-UInt64] @@ -92,7 +95,7 @@ Just sends `ActivityId`. Implemented in protocol 1.20. When backwards compatible with older versions of the protocol and this message would have been sent it is instead ignored. -- 0x53544f50 :: [UInt64][se-UInt64] (hardcoded) +- 0x53544f50 :: [UInt64][se-UInt64] (hardcoded, 'STOP' in ASCII) ### `STDERR_RESULT` @@ -101,14 +104,13 @@ Sends results for a given activity. Implemented in protocol 1.20. When backwards compatible with older versions of the protocol and this message would have been sent it is instead ignored. -- 0x52534c54 :: [UInt64][se-UInt64] (hardcoded) +- 0x52534c54 :: [UInt64][se-UInt64] (hardcoded, 'RSLT' in ASCII) - act :: [UInt64][se-UInt64] - type :: [ResultType][se-ResultType] - fields :: [List][se-List] of [Field][se-Field] - [se-UInt64]: ./serialization.md#uint64 [se-Int]: ./serialization.md#int [se-Size]: ./serialization.md#size diff --git a/tvix/docs/src/nix-daemon/operations.md b/tvix/docs/src/nix-daemon/operations.md index 0683ab070973..80708c9104b5 100644 --- a/tvix/docs/src/nix-daemon/operations.md +++ b/tvix/docs/src/nix-daemon/operations.md @@ -14,11 +14,9 @@ | [FindRoots](#findroots) | 14 | | [SetOptions](#setoptions) | 19 | | [CollectGarbage](#collectgarbage) | 20 | -| [QuerySubstitutablePathInfo](#querysubstitutablepathinfo) | 21 | | [QueryAllValidPaths](#queryallvalidpaths) | 23 | | [QueryPathInfo](#querypathinfo) | 26 | | [QueryPathFromHashPart](#querypathfromhashpart) | 29 | -| [QuerySubstitutablePathInfos](#querysubstitutablepathinfos) | 30 | | [QueryValidPaths](#queryvalidpaths) | 31 | | [QuerySubstitutablePaths](#querysubstitutablepaths) | 32 | | [QueryValidDerivers](#queryvalidderivers) | 33 | @@ -40,17 +38,19 @@ ## Obsolete operations -| Operation | Id | -| --------------------------------------------------------- | -- | -| [QueryPathHash](#querypathhash) | 4 | -| [QueryReferences](#queryreferences) | 5 | -| [AddTextToStore](#addtexttostore) | 8 | -| [SyncWithGC](#syncwithgc) | 13 | -| [ExportPath](#exportpath) | 16 | -| [QueryDeriver](#queryderiver) | 18 | -| [QueryDerivationOutputs](#queryderivationoutputs) | 22 | -| [ImportPaths](#importpaths) | 27 | -| [QueryDerivationOutputNames](#queryderivationoutputnames) | 28 | +| Operation | Id | +| ----------------------------------------------------------- | -- | +| [QueryPathHash](#querypathhash) | 4 | +| [QueryReferences](#queryreferences) | 5 | +| [AddTextToStore](#addtexttostore) | 8 | +| [SyncWithGC](#syncwithgc) | 13 | +| [ExportPath](#exportpath) | 16 | +| [QueryDeriver](#queryderiver) | 18 | +| [QuerySubstitutablePathInfo](#querysubstitutablepathinfo) | 21 | +| [QueryDerivationOutputs](#queryderivationoutputs) | 22 | +| [ImportPaths](#importpaths) | 27 | +| [QueryDerivationOutputNames](#queryderivationoutputnames) | 28 | +| [QuerySubstitutablePathInfos](#querysubstitutablepathinfos) | 30 | ## Removed operations @@ -81,6 +81,7 @@ As the name says checks that a store path is valid i.e. in the store. This is a pretty core operation used everywhere. + ### Inputs path :: [StorePath][se-StorePath] @@ -92,6 +93,9 @@ isValid :: [Bool][se-Bool] **Id:** 3<br> **Introduced:** Nix 0.11<br> +**Obsolete** Protocol 1.12, Nix 1.2<br> + +Replaced by QuerySubstitutablePaths. Checks if we can substitute the input path from a substituter. Uses QuerySubstitutablePaths under the hood :/ @@ -115,7 +119,7 @@ Retrieves the base16 NAR hash of a given store path. path :: [StorePath][se-StorePath] ### Outputs -hash :: [String][se-String] (base16-encoded NAR hash without algorithm prefix) +hash :: [NARHash][se-NARHash] ## QueryReferences @@ -130,7 +134,7 @@ Retrieves the references of a given path path :: [StorePath][se-StorePath] ### Outputs -references :: [List][se-List] of [StorePath][se-StorePath] +references :: [Set][se-Set] of [StorePath][se-StorePath] ## QueryReferrers @@ -144,7 +148,7 @@ Retrieves the referrers of a given path. path :: [StorePath][se-StorePath] ### Outputs -referrers :: [List][se-List] of [StorePath][se-StorePath] +referrers :: [Set][se-Set] of [StorePath][se-StorePath] ## AddToStore @@ -156,16 +160,16 @@ Add a new path to the store. ### Before protocol version 1.25 #### Inputs -- baseName :: [String][se-String] +- baseName :: [StorePathName][se-StorePathName] - fixed :: [Bool64][se-Bool64] - recursive :: [FileIngestionMethod][se-FileIngestionMethod] -- hashAlgo :: [String][se-String] +- hashAlgo :: [HashAlgorithm][se-HashAlgorithm] - NAR dump If fixed is `true`, hashAlgo is forced to `sha256` and recursive is forced to -`Recursive`. +`NixArchive`. -Only `Flat` and `Recursive` values are supported for the recursive input +Only `Flat` and `NixArchive` values are supported for the recursive input parameter. #### Outputs @@ -173,9 +177,9 @@ path :: [StorePath][se-StorePath] ### Protocol version 1.25 or newer #### Inputs -- name :: [String][se-String] +- name :: [StorePathName][se-StorePathName] - camStr :: [ContentAddressMethodWithAlgo][se-ContentAddressMethodWithAlgo] -- refs :: [List][se-List] of [StorePath][se-StorePath] +- refs :: [Set][se-Set] of [StorePath][se-StorePath] - repairBool :: [Bool64][se-Bool64] - [Framed][se-Framed] NAR dump @@ -197,9 +201,9 @@ to [AddToStore](#addtostore). And so this corresponds to calling wrapped as a NAR. ### Inputs -- suffix :: [String][se-String] +- suffix :: [StorePathName][se-StorePathName] - text :: [Bytes][se-Bytes] -- refs :: [List][se-List] of [StorePath][se-StorePath] +- refs :: [Set][se-Set] of [StorePath][se-StorePath] ### Outpus path :: [StorePath][se-StorePath] @@ -213,15 +217,15 @@ path :: [StorePath][se-StorePath] Build (or substitute) a list of derivations. ### Inputs -paths :: [List][se-List] of [DerivedPath][se-DerivedPath] +paths :: [Set][se-Set] of [DerivedPath][se-DerivedPath] #### Protocol 1.15 or newer -mode :: [BuildMode][se-BuildMode] +mode :: [BuildMode][se-BuildMode] (defaults to Normal) Check that connection is trusted before allowing Repair mode. ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## EnsurePath @@ -235,7 +239,7 @@ Checks if a path is valid. Note: it may be made valid by running a substitute. path :: [StorePath][se-StorePath] ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## AddTempRoot @@ -253,7 +257,7 @@ deleting store paths that the client is actively doing something with. path :: [StorePath][se-StorePath] ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## AddIndirectRoot @@ -267,10 +271,10 @@ created by [AddPermRoot](#addpermroot). Only ever sent on the local unix socket nix daemon. ### Inputs -path :: [String][se-String] +path :: [Path][se-Path] ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## SyncWithGC @@ -308,7 +312,7 @@ thing. Find the GC roots. ### Outputs -roots :: [Map][se-Map] of [String][se-String] to [StorePath][se-StorePath] +roots :: [Map][se-Map] of [Path][se-Path] to [StorePath][se-StorePath] The key is the link pointing to the given store path. @@ -333,7 +337,8 @@ Export a store path in the binary format nix-store --import expects. See impleme - sign :: [Int][se-Int] (ignored and hardcoded to 0 in client) ### Outputs -Uses `STDERR_WRITE` to send dump in export format +Uses [`STDERR_WRITE`](./logging.md#stderr_write) to send dump in +[export format][se-ExportFormat] After dump it outputs. @@ -373,18 +378,18 @@ Only ever used right after the handshake. ### Inputs -- keepFailed :: [Int][se-Int] -- keepGoing :: [Int][se-Int] -- tryFallback :: [Int][se-Int] +- keepFailed :: [Bool][se-Bool] +- keepGoing :: [Bool][se-Bool] +- tryFallback :: [Bool][se-Bool] - verbosity :: [Verbosity][se-Verbosity] -- maxbuildJobs :: [Int][se-Int] -- maxSilentTime :: [Int][se-Int] +- maxBuildJobs :: [Int][se-Int] +- maxSilentTime :: [Time][se-Time] - useBuildHook :: [Bool][se-Bool] (ignored and hardcoded to true in client) - verboseBuild :: [Verbosity][se-Verbosity] - logType :: [Int][se-Int] (ignored and hardcoded to 0 in client) - printBuildTrace :: [Int][se-Int] (ignored and hardcoded to 0 in client) - buildCores :: [Int][se-Int] -- useSubstitutes :: [Int][se-Int] +- useSubstitutes :: [Bool][se-Bool] ### Protocol 1.12 or newer otherSettings :: [Map][se-Map] of [String][se-String] to [String][se-String] @@ -399,7 +404,7 @@ Find the GC roots. ### Inputs - action :: [GCAction][se-GCAction] -- pathsToDelete :: [List][se-List] of [StorePath][se-StorePath] +- pathsToDelete :: [Set][se-Set] of [StorePath][se-StorePath] - ignoreLiveness :: [Bool64][se-Bool64] - maxFreed :: [UInt64][se-UInt64] - removed :: [Int][se-Int] (ignored and hardcoded to 0 in client) @@ -407,18 +412,19 @@ Find the GC roots. - removed :: [Int][se-Int] (ignored and hardcoded to 0 in client) ### Outputs -- pathsDeleted :: [List][se-List] of [String][se-String] +- pathsDeleted :: [Set][se-Set] of [Path][se-Path] - bytesFreed :: [UInt64][se-UInt64] -- 0 :: [UInt64][se-UInt64] (hardcoded) +- 0 :: [UInt64][se-UInt64] (hardcoded, obsolete and ignored by client) -Depending on the action pathsDeleted is, the GC roots, or the paths that would -be or have been deleted. +Depending on the value of the action input the value of output pathsDeleted +is either, the GC roots, or the paths that would be or have been deleted. ## QuerySubstitutablePathInfo **Id:** 21<br> **Introduced:** Protocol 1.02, Nix 0.12<br> +**Obsolete:** Protocol 1.12, Nix 1.2<br> Retrieves the various substitutable paths infos for a given path. @@ -429,10 +435,7 @@ path :: [StorePath][se-StorePath] found :: [Bool][se-Bool] #### If found is true -- deriver :: [OptStorePath][se-OptStorePath] -- references :: [List][se-List] of [StorePath][se-StorePath] -- downloadSize :: [UInt64][se-UInt64] -- narSize :: [UInt64][se-UInt64] +- info :: [SubstitutablePathInfo][se-SubstitutablePathInfo] ## QueryDerivationOutputs @@ -447,7 +450,7 @@ Retrieves all the outputs paths of a given derivation. path :: [StorePath][se-StorePath] (must point to a derivation) ### Outputs -derivationOutputs :: [List][se-List] of [StorePath][se-StorePath] +derivationOutputs :: [Set][se-Set] of [StorePath][se-StorePath] ## QueryAllValidPaths @@ -458,7 +461,7 @@ derivationOutputs :: [List][se-List] of [StorePath][se-StorePath] Retrieves all the valid paths contained in the store. ### Outputs -paths :: [List][se-List] of [StorePath][se-StorePath] +paths :: [Set][se-Set] of [StorePath][se-StorePath] ## QueryFailedPaths (removed) @@ -498,7 +501,7 @@ success :: [Bool64][se-Bool64] pathInfo :: [UnkeyedValidPathInfo][se-UnkeyedValidPathInfo] #### If protocol version is older than 1.17 -If info not found return error with `STDERR_ERROR` +If info not found return error with [`STDERR_ERROR`](./logging.md#stderr_error) pathInfo :: [UnkeyedValidPathInfo][se-UnkeyedValidPathInfo] @@ -516,7 +519,7 @@ before the metadata about the store path and so you would typically have to store the NAR in memory or temporarily on disk before processing it. ### Inputs -List of NAR dumps coming from the ExportPaths operations. +[List of NAR dumps][se-ImportPaths] coming from one or more ExportPath operations. ### Outputs importedPaths :: [List][se-List] of [StorePath][se-StorePath] @@ -534,7 +537,7 @@ Retrieves the name of the outputs of a given derivation. EG. out, dev, etc. path :: [StorePath][se-StorePath] (must be a derivation path) ### Outputs -names :: [List][se-List] of [String][se-String] +names :: [Set][se-Set] of [OutputName][se-OutputName] ## QueryPathFromHashPart @@ -542,11 +545,10 @@ names :: [List][se-List] of [String][se-String] **Id:** 29<br> **Introduced:** Protocol 1.11, Nix 1.1<br> -Retrieves a store path from a base16 (input) hash. Returns "" if no path was -found. +Retrieves a store path from a nixbase32 (input) hash. ### Inputs -hashPart :: [String][se-String] (must be a base-16 hash) +hashPart :: [StorePathHash][se-StorePathHash] ### Outputs path :: [OptStorePath][se-OptStorePath] @@ -556,18 +558,22 @@ path :: [OptStorePath][se-OptStorePath] **Id:** 30<br> **Introduced:** Protocol 1.12*, Nix 1.2<br> +**Obsolete:** Protocol 1.19*, Nix 2.0<br> Retrieves the various substitutable paths infos for set of store paths. +Only ever used in the fallback for QueryMissing which means that if protocol is 1.19 or later +it is never sent and is therefore obsolete after that. + ### Inputs #### If protocol version is 1.22 or newer paths :: [Map][se-Map] of [StorePath][se-StorePath] to [OptContentAddress][se-OptContentAddress] #### If protocol version older than 1.22 -paths :: [List][se-List] of [StorePath][se-StorePath] +paths :: [Set][se-Set] of [StorePath][se-StorePath] ### Outputs -infos :: [List][se-List] of [SubstitutablePathInfo][se-SubstitutablePathInfo] +infos :: [Map][se-Map] of [StorePath][se-StorePath] to [SubstitutablePathInfo][se-SubstitutablePathInfo] ## QueryValidPaths @@ -578,13 +584,13 @@ infos :: [List][se-List] of [SubstitutablePathInfo][se-SubstitutablePathInfo] Takes a list of store paths and returns a new list only containing the valid store paths ## Inputs -paths :: [List][se-List] of [StorePath][se-StorePath] +paths :: [Set][se-Set] of [StorePath][se-StorePath] ### If protocol version is 1.27 or newer substitute :: [Bool][se-Bool] (defaults to false if not sent) ## Outputs -paths :: [List][se-List] of [StorePath][se-StorePath] +paths :: [Set][se-Set] of [StorePath][se-StorePath] ## QuerySubstitutablePaths @@ -599,10 +605,10 @@ In versions of the protocol prior to 1.12 [HasSubstitutes](#hassubstitutes) is used to implement the functionality that this operation provides. ### Inputs -paths :: [List][se-List] of [StorePath][se-StorePath] +paths :: [Set][se-Set] of [StorePath][se-StorePath] ### Outputs -paths :: [List][se-List] of [StorePath][se-StorePath] +paths :: [Set][se-Set] of [StorePath][se-StorePath] ## QueryValidDerivers @@ -616,7 +622,7 @@ Retrieves the derivers of a given path. path :: [StorePath][se-StorePath] ### Outputs -derivers :: [List][se-List] of [StorePath][se-StorePath] +derivers :: [Set][se-Set] of [StorePath][se-StorePath] ## OptimiseStore @@ -627,7 +633,7 @@ derivers :: [List][se-List] of [StorePath][se-StorePath] Optimise store by hardlinking files with the same content. ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## VerifyStore @@ -677,14 +683,14 @@ buildResult :: [BuildResult][se-BuildResult] **Id:** 37<br> **Introduced:** Protocol 1.16, Nix 2.0<br> -Add the signatures associated to a given path. +Add the signatures associated to a given path. Used by `nix store copy-sigs` and `nix store sign`. ### Inputs - path :: [StorePath][se-StorePath] -- signatures :: [List][se-List] of [String][se-String] +- signatures :: [Set][se-Set] of [Signature][se-Signature] ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## NarFromPath @@ -717,12 +723,12 @@ Dumps a path as a NAR ### Inputs - path :: [StorePath][se-StorePath] - deriver :: [OptStorePath][se-OptStorePath] -- narHash :: [String][se-String] SHA256 NAR hash base 16 -- references :: [List][se-List] of [StorePath][se-StorePath] +- narHash :: [NARHash][se-NARHash] +- references :: [Set][se-Set] of [StorePath][se-StorePath] - registrationTime :: [Time][se-Time] - narSize :: [UInt64][se-UInt64] - ultimate :: [Bool64][se-Bool64] -- signatures :: [List][se-List] of [String][se-String] +- signatures :: [Set][se-Set] of [Signature][se-Signature] - ca :: [OptContentAddress][se-OptContentAddress] - repair :: [Bool64][se-Bool64] - dontCheckSigs :: [Bool64][se-Bool64] @@ -731,7 +737,7 @@ Dumps a path as a NAR [Framed][se-Framed] NAR dump #### If protocol version is between 1.21 and 1.23 -NAR dump sent using `STDERR_READ` +NAR dump sent using [`STDERR_READ`](./logging.md#stderr_read) #### If protocol version is older than 1.21 NAR dump sent raw on stream @@ -746,9 +752,9 @@ NAR dump sent raw on stream targets :: [List][se-List] of [DerivedPath][se-DerivedPath] ### Outputs -- willBuild :: [List][se-List] of [StorePath][se-StorePath] -- willSubstitute :: [List][se-List] of [StorePath][se-StorePath] -- unknown :: [List][se-List] of [StorePath][se-StorePath] +- willBuild :: [Set][se-Set] of [StorePath][se-StorePath] +- willSubstitute :: [Set][se-Set] of [StorePath][se-StorePath] +- unknown :: [Set][se-Set] of [StorePath][se-StorePath] - downloadSize :: [UInt64][se-UInt64] - narSize :: [UInt64][se-UInt64] @@ -764,7 +770,7 @@ Retrieves an associative map outputName -> storePath for a given derivation. path :: [StorePath][se-StorePath] (must be a derivation path) ### Outputs -outputs :: [Map][se-Map] of [String][se-String] to [OptStorePath][se-OptStorePath] +outputs :: [Map][se-Map] of [OutputName][se-OutputName] to [OptStorePath][se-OptStorePath] ## RegisterDrvOutput @@ -795,10 +801,10 @@ outputId :: [DrvOutput][se-DrvOutput] ### Outputs #### If protocol is 1.31 or newer -realisations :: [List][se-List] of [Realisation][se-Realisation] +realisations :: [Set][se-Set] of [Realisation][se-Realisation] #### If protocol is older than 1.31 -outPaths :: [List][se-List] of [BaseStorePath][se-BaseStorePath] +outPaths :: [Set][se-Set] of [StorePath][se-StorePath] ## AddMultipleToStore @@ -816,7 +822,7 @@ for each small NAR was costly. ### Inputs - repair :: [Bool64][se-Bool64] - dontCheckSigs :: [Bool64][se-Bool64] -- [Framed][se-Framed] stream of add multiple NAR dump +- [Framed][se-Framed] stream of [add multiple NAR dump][se-AddMultipleToStore] ## AddBuildLog @@ -827,11 +833,11 @@ for each small NAR was costly. Attach some build logs to a given build. ### Inputs -- path :: [String][se-String] (might be [BaseStorePath][se-BaseStorePath]) +- path :: [BaseStorePath][se-BaseStorePath] - [Framed][se-Framed] stream of log lines ### Outputs -1 :: [Int][se-Int] (hardcoded) +1 :: [Int][se-Int] (hardcoded and ignored by client) ## BuildPathsWithResults @@ -856,10 +862,10 @@ results :: [List][se-List] of [KeyedBuildResult][se-KeyedBuildResult] ### Inputs - storePath :: [StorePath][se-StorePath] -- gcRoot :: [String][se-String] +- gcRoot :: [Path][se-Path] ### Outputs -gcRoot :: [String][se-String] +gcRoot :: [Path][se-Path] @@ -884,6 +890,7 @@ gcRoot :: [String][se-String] [se-DrvOutput]: ./serialization.md#drvoutput [se-Realisation]: ./serialization.md#realisation [se-List]: ./serialization.md#list-of-x +[se-Set]: ./serialization.md#set-of-x [se-Map]: ./serialization.md#map-of-x-to-y [se-SubstitutablePathInfo]: ./serialization.md#substitutablepathinfo [se-ValidPathInfo]: ./serialization.md#validpathinfo @@ -891,4 +898,7 @@ gcRoot :: [String][se-String] [se-BuildResult]: ./serialization.md#buildmode [se-KeyedBuildResult]: ./serialization.md#keyedbuildresult [se-BasicDerivation]: ./serialization.md#basicderivation -[se-Framed]: ./serialization.md#framed \ No newline at end of file +[se-Framed]: ./serialization.md#framed +[se-AddMultipleToStore]: ./serialization.md#addmultipletostore-format +[se-ExportFormat]: ./serialization.md#export-path-format +[se-ImportPaths]: ./serialization.md#import-paths-format \ No newline at end of file diff --git a/tvix/docs/src/nix-daemon/serialization.md b/tvix/docs/src/nix-daemon/serialization.md index a2694a4dea30..1042c956ba71 100644 --- a/tvix/docs/src/nix-daemon/serialization.md +++ b/tvix/docs/src/nix-daemon/serialization.md @@ -6,7 +6,7 @@ Little endian byte order - len :: [UInt64](#uint64) - len bytes of content -- padding with zeros to ensure 64 bit alignment of content with padding +- padding with zeros to ensure 64 bit alignment of content + padding ## Int serializers @@ -16,6 +16,9 @@ Little endian byte order ### Int64 [UInt64](#uint64) cast to C `int64_t` with upper bounds checking. +This means that negative numbers can be written but not read. +Since this is only used for cpuSystem and cpuUser it is fine that +negative numbers aren't supported. ### UInt8 [UInt64](#uint64) cast to C `uint8_t` with upper bounds checking. @@ -25,7 +28,8 @@ Little endian byte order ### Time [UInt64](#uint64) cast to C `time_t` with upper bounds checking. -s +This means that negative numbers can be written but not read. + ### Bool Sent as an [Int](#int) where 0 is false and everything else is true. @@ -35,10 +39,10 @@ Sent as an [UInt64](#uint64) where 0 is false and everything else is true. ### FileIngestionMethod An [UInt8](#uint8) enum with the following possible values: -| Name | Int | -| --------- | --- | -| Flat | 0 | -| Recursive | 1 | +| Name | Int | +| ---------- | --- | +| Flat | 0 | +| NixArchive | 1 | ### BuildMode An [Int](#int) enum with the following possible values: @@ -137,6 +141,15 @@ An [Int](#int) enum with the following possible values: | Int | 0 | | String | 1 | +### OptTrusted +An [UInt8](#uint8) optional enum with the following possible values: + +| Name | Int | +| ---------------- | --- | +| None | 0 | +| Some(Trusted) | 1 | +| Some(NotTrusted) | 2 | + ## Bytes serializers @@ -144,51 +157,107 @@ An [Int](#int) enum with the following possible values: Simply a [Bytes](#bytes) that has some UTF-8 string like semantics sometimes. ### StorePath -String representation of a full store path. +[String](#string) representation of a full store path including the store directory. ### BaseStorePath -String representation of the basename of a store path. That is the store path +[String](#string) representation of the basename of a store path. That is the store path without the /nix/store prefix. +### StorePathName +[String](#string) representation of the name part of a base store path. This is the part +of the store path after the nixbase32 hash and '-' + +It must have the following format: +- Deny ".", "..", or those strings followed by '-' +- Otherwise check that each character is 0-9, a-z, A-Z or one of +-._?= + +### StorePathHash +[String](#string) representation of the hash part of a base store path. This is the part +of the store path at the beginning and before the '-' and is in nixbase32 format. + + +### OutputName +[String](#string) representation of the name of a derivation output. +This is usually combined with the name in the derivation to form the store path name for the +store path with this output. + +Since output name is usually combined to form a store path name its format must follow the +same rules as [StorePathName](#storepathname): +- Deny ".", "..", or those strings followed by '-' +- Otherwise check that each character is 0-9, a-z, A-Z or one of +-._?= + + ### OptStorePath Optional store path. If no store path this is serialized as the empty string otherwise it is the same as [StorePath](#storepath). +### Path +[String](#string) representation of an absolute path. + +### NARHash +[String](#string) base16-encoded NAR SHA256 hash without algorithm prefix. + +### Signature +[String](#string) with a signature for the given store path or realisation. This should be +in the format `name`:`base 64 encoded signature` but this is not enforced in the protocol. + +### HashAlgorithm +[String](#string) with one of the following values: +- md5 +- sha1 +- sha256 +- sha512 + +### HashDigest +[String](#string) with a hash digest in any encoding + +### OptHashDigest +Optional version of [HashDigest](#hashdigest) where empty string means +no value. + + ### ContentAddressMethodWithAlgo -One of the following strings: -- text:`hash algorithm` -- fixed:r:`hash algorithm` -- fixed:`hash algorithm` +[String](#string) with one of the following formats: +- text:[HashAlgorithm](#hashalgorithm) +- fixed:r:[HashAlgorithm](#hashalgorithm) +- fixed:[HashAlgorithm](#hashalgorithm) -### DerivedPath -#### If protocol is 1.30 or newer - return DerivedPath::parseLegacy(store, s); -#### If protocol is older than 1.30 - return parsePathWithOutputs(store, s).toDerivedPath(); +### OptContentAddressMethodWithAlgo +Optional version of [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo) +where empty string means no value. ### ContentAddress -String with the format: -- [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo):`hash` +[String](#string) with the format: +- [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo):[HashDigest](#hashdigest) ### OptContentAddress Optional version of [ContentAddress](#contentaddress) where empty string means no content address. +### DerivedPath +#### If protocol is 1.30 or newer +output-names = [OutputName](#outputname), { "," [OutputName](#outputname) }<br> +output-spec = "*" | output-names<br> +derived-path = [StorePath](#storepath), [ "!", output-spec ]<br> + +#### If protocol is older than 1.30 +[StorePath](#storepath), [ "!", [OutputName](#outputname), { "," [OutputName](#outputname) } ] + ### DrvOutput -String with format: -- `hash with any prefix`!`output name` +[String](#string) with format: +- `hash with any prefix` "!" [OutputName](#outputname) ### Realisation -A JSON object sent as a string. +A JSON object sent as a [String](#string). The JSON object has the following keys: -| Key | Value | -| --------------------- | ----------------------- | -| id | [DrvOutput](#drvoutput) | -| outPath | [StorePath](#storepath) | -| signatures | Array of String | +| Key | Value | +| --------------------- | -------------------------------- | +| id | [DrvOutput](#drvoutput) | +| outPath | [StorePath](#storepath) | +| signatures | Array of [Signature](#signature) | | dependentRealisations | Object where key is [DrvOutput](#drvoutput) and value is [StorePath](#storepath) | @@ -200,20 +269,22 @@ A list is encoded as a [Size](#size) length n followed by n encodings of x ### Map of x to y A map is encoded as a [Size](#size) length n followed by n encodings of pairs of x and y +### Set of x +A set is encoded as a [Size](#size) length n followed by n encodings of x ### BuildResult - status :: [BuildStatus](#buildstatus) - errorMsg :: [String](#string) #### Protocol 1.29 or newer -- timesBuilt :: [Int](#int) -- isNonDeterministic :: [Bool64](#bool64) -- startTime :: [Time](#time) -- stopTime :: [Time](#time) +- timesBuilt :: [Int](#int) (defaults to 0) +- isNonDeterministic :: [Bool64](#bool64) (defaults to false) +- startTime :: [Time](#time) (defaults to 0) +- stopTime :: [Time](#time) (defaults to 0) #### Protocol 1.37 or newer -- cpuUser :: [OptMicroseconds](#optmicroseconds) -- cpuSystem :: [OptMicroseconds](#optmicroseconds) +- cpuUser :: [OptMicroseconds](#optmicroseconds) (defaults to none) +- cpuSystem :: [OptMicroseconds](#optmicroseconds) (defaults to none) #### Protocol 1.28 or newer builtOutputs :: [Map](#map-of-x-to-y) of [DrvOutput](#drvoutput) to [Realisation](#realisations) @@ -232,23 +303,22 @@ Optional microseconds. ### SubstitutablePathInfo -- storePath :: [StorePath](#storepath) - deriver :: [OptStorePath](#optstorepath) -- references :: [List](#list-of-x) of [StorePath](#storepath) +- references :: [Set][#set-of-x] of [StorePath](#storepath) - downloadSize :: [UInt64](#uint64) - narSize :: [UInt64](#uint64) ### UnkeyedValidPathInfo - deriver :: [OptStorePath](#optstorepath) -- narHash :: [String](#string) SHA256 NAR hash base16 encoded -- references :: [List](#list-of-x) of [StorePath](#storepath) +- narHash :: [NARHash](#narhash) +- references :: [Set](#set-of-x) of [StorePath](#storepath) - registrationTime :: [Time](#time) - narSize :: [UInt64](#uint64) #### If protocol version is 1.16 or above -- ultimate :: [Bool64](#bool64) -- signatures :: [List](#list-of-x) of [String](#string) +- ultimate :: [Bool64](#bool64) (defaults to false) +- signatures :: [Set](#set-of-x) of [Signature](#signature) - ca :: [OptContentAddress](#optcontentaddress) @@ -257,13 +327,13 @@ Optional microseconds. - info :: [UnkeyedValidPathInfo](#unkeyedvalidpathinfo) ### DerivationOutput -- path :: [String](#string) -- hashAlgo :: [String](#string) -- hash :: [String](#string) +- path :: [OptStorePath](#optstorepath) +- hashAlgo :: [OptContentAddressMethodWithAlgo](#optcontentaddressmethodwithalgo) +- hash :: [OptHashDigest](#opthashdigest) ### BasicDerivation -- outputs :: [Map](#map-of-x-to-y) of [String](#string) to [DerivationOutput](#derivationoutput) -- inputSrcs :: [List](#list-of-x) of [StorePath](#storepath) +- outputs :: [Map](#map-of-x-to-y) of [OutputName](#outputname) to [DerivationOutput](#derivationoutput) +- inputSrcs :: [Set](#set-of-x) of [StorePath](#storepath) - platform :: [String](#string) - builder :: [String](#string) - args :: [List](#list-of-x) of [String](#string) @@ -271,13 +341,13 @@ Optional microseconds. ### TraceLine - havePos :: [Size](#size) (hardcoded to 0) -- hint :: [String](#string) +- hint :: [String](#string) (If logger is JSON, invalid UTF-8 is replaced with U+FFFD) ### Error - type :: [String](#string) (hardcoded to `Error`) - level :: [Verbosity](#verbosity) - name :: [String](#string) (removed and hardcoded to `Error`) -- msg :: [String](#string) +- msg :: [String](#string) (If logger is JSON, invalid UTF-8 is replaced with U+FFFD) - havePos :: [Size](#size) (hardcoded to 0) - traces :: [List](#list-of-x) of [TraceLine](#traceline) @@ -297,9 +367,43 @@ At protocol 1.23 [AddToStoreNar](./operations.md#addtostorenar) introduced a framed streaming for sending the NAR dump and later versions of the protocol also used this framing for other operations. -At its core the framed streaming is just a series of [Bytes](#bytes) of -varying length and terminated by an empty [Bytes](#bytes). +At its core the framed streaming is just a series of [UInt64](#uint64) `size` +followed by `size` bytes. The stream is terminated when `size` is zero. + +Unlike [Bytes](#bytes), frames are *NOT* padded. This method of sending data has the advantage of not having to parse the data to find where it ends. Older versions of the protocol would potentially parse -the NAR twice. \ No newline at end of file +the NAR twice. + + +## AddMultipleToStore format + +Paths must be topologically sorted. + +- expected :: [UInt64](#uint64) + +### Repeated expected times +- info :: [ValidPathInfo](#validpathinfo) +- NAR dump + + +## Export path format +- NAR dump +- 0x4558494es :: [Int](#int) (hardcoded, 'EXIN' in ASCII) +- path :: [StorePath](#storepath) +- references :: [Set](#set-of-x) of [StorePath](#storepath) +- deriver :: [OptStorePath](#optstorepath) +- hasSignature :: [Int](#int) (hardcoded to 0 in newer versions) + +#### If hasSignature is 1 +- signature :: [String](#string) (ignored) + + +## Import paths format + +- hasNext :: [UInt64](#uint64) + +### While hasNext is 1 +- [Export path format](#export-path-format) +- hasNext :: [UInt64](#uint64) |