1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
|
---
author:
- Florian Klink
date: 2023-09-09
title: "tvix-store: A content-addressed file system and sync protocol"
theme: moon
revealOptions:
transition: 'fade'
---
## tvix-store
### A content-addressed file system and sync protocol
2023-09-13
Florian Klink / flokli
---
## Whoami
- <!-- .element: class="fragment" -->
flokli
- <!-- .element: class="fragment" -->
Nix/NixOS contributor
- maintain systemd, nss and more low-level stuff there
- <!-- .element: class="fragment" -->
Freelance Nix/DevOps consultant
Note: more Kubernetes/DevOps exposure with work
---
## What is tvix-store?
- <!-- .element: class="fragment" -->
A new implementation of a content-addressed "storage system"
- <!-- .element: class="fragment" -->
part of the Tvix Project, a (WIP) reimplementation of Nix and auxillary components in Rust
- <!-- .element: class="fragment" -->
Storage model: think about git trees and its Merkle DAG…
- <!-- .element: class="fragment" -->
… but with nicer wire format (`.proto`) and hash function (blake3)
---
## Storage model
- <!-- .element: class="fragment" -->
Once you know the root: everything else is content-addressed
- <!-- .element: class="fragment" -->
No timestamps, no uid/gid, no xattrs, only one way to represent the same tree
- <!-- .element: class="fragment" -->
Automatic dedup of identical subtrees in different file system trees
- <!-- .element: class="fragment" -->
Automatic dedup of identical blobs (and you can do more chunking underneath too)
---
## Storage model (cont.)
- <!-- .element: class="fragment" -->
Granular seekable access into blobs
- <!-- .element: class="fragment" -->
verified streaming thanks to BLAKE3 and Bao, faulty data is detected early on
- <!-- .element: class="fragment" -->
Everything below can be retrieved from anyone without having to trust (P2P substitution, CDNs, …)
---
## Usecases
- <!-- .element: class="fragment" -->
File system tree delivery
- <!-- .element: class="fragment" -->
Container image delivery
- <!-- .element: class="fragment" -->
Backing store for VCS
- <!-- .element: class="fragment" -->
Granular access into large datasets
---
## Status
- <!-- .element: class="fragment" -->
In-memory backend, a local K/V backend (Sled)
- <!-- .element: class="fragment" -->
FUSE filesystem
- <!-- .element: class="fragment" -->
A gRPC API to transfer things, bindings for golang and rust
- <!-- .element: class="fragment" -->
some object storage backends in development (GCS, NATS)
- <!-- .element: class="fragment" -->
FUTUREWORK: more storage backends / store composition / in-kernel module?
Notes: of course you can use your own network protocol too, like HTTP CAS or iroh....plug different stores together to represent caches, blobfs
---
## Contributing
- <!-- .element: class="fragment" -->
Join the IRC channel (`#tvl` on `hackint`), bridged to Matrix and XMPP
- <!-- .element: class="fragment" -->
Check our issue tracker
- <!-- .element: class="fragment" -->
Try to use it and tell us how you broke it!
Note: if this sounds useful to you, reach out!
---
# Thanks!
<style>
.container{
display: flex;
}
.col{
flex: 1;
}
</style>
<div class="container">
<div class="col">
Florian Klink / <a href="https://flokli.de">flokli.de</a><br />
<img src="qrcode-flokli.svg" />
</div>
<div class="col">
Tvix / <a href="https://tvix.dev">tvix.dev</a><br />
<img src="qrcode-tvix.svg" />
</div>
</div>
---
## Structure
[tvix-store graph](tvix-store-graph-blob-directory.svg)
|
