From c3572048d5c5fa4fc89425c17eeb4116d6584c65 Mon Sep 17 00:00:00 2001 From: Florian Klink Date: Wed, 5 Jun 2024 10:15:04 +0200 Subject: docs(tvix/docs/TODO): expand Store Composition config section Write down some of the thoughts after brainstorming with yuka. Change-Id: I01c94474dc643b8c4993db80e50d3ec65f5c17f4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11749 Autosubmit: flokli Tested-by: BuildkiteCI Reviewed-by: yuka --- tvix/docs/src/TODO.md | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) (limited to 'tvix') diff --git a/tvix/docs/src/TODO.md b/tvix/docs/src/TODO.md index 0f1bcee27b..c686f8ab64 100644 --- a/tvix/docs/src/TODO.md +++ b/tvix/docs/src/TODO.md @@ -107,9 +107,33 @@ logs etc, but this is something requiring a lot of designing. ### Store composition - Combinators: list-by-priority, first-come-first-serve, cache - - How do describe hierarchies. URL format too one-dimensional, but we might get - quite far with a similar "substituters" concept that Nix uses, to construct - the composed stores. + - Store composition hierarchies (@yuka). + - URL format too one-dimensional. + - We want to have nice and simple user-facing substituter config, including + sensible default wrappers for caching, retries, fallbacks, as well as + granular control for power-users. + - Current design idea: + - Have a concept similar to rclone config (map with store aliases as + keys, allowing to refer to stores by their alias from other parts of + the config). + It allows both referring to by name, as well as ad-hoc definition: + https://rclone.org/docs/#syntax-of-remote-paths + - Each store needs to be aware of its "instance name", so it can be + included in logs, metrics, … + - Have a "instantiation function" traversing such a config data structure, + creating store instances and plugging them together, ultimately returning + a dyn …Service interface. + - No reconfiguration/reconcilation for now + - Making URLs the primary data format would get ugly quite easily (hello + multiple layers of escaping!), so best to convert the existing URL + syntax to our new config format on the fly and then use one codepath + to instantiate/assemble. Similarly, something like the "user-facing + substituter config" mentioned above could aalso be converted to such a + config format under the hood. + - Maybe add a ?cache=$other_url parameter support to the URL syntax, to + easily wrap a store with a caching frontend, using $other_url as the + "near" store URL. + ### Store Config There's already serde for some store options (bigtable uses `serde_qs`). We might also have common options global over all backends, like chunking -- cgit 1.4.1