tvix-store

A content-addressed file system and sync protocol

2023-09-13

Florian Klink / flokli

---

Whoami

• flokli
• Nix/NixOS contributor
  • maintain systemd, nss and more low-level stuff there
• Freelance Nix/DevOps consultant

Note: more Kubernetes/DevOps exposure with work

---

What is tvix-store?

• A new implementation of a content-addressed "storage system"
  • part of the Tvix Project, a (WIP) reimplementation of Nix and auxillary components in Rust
  • Storage model: think about git trees and its Merkle DAG…
  • … but with nicer wire format (.proto) and hash function (blake3)

---

Storage model

• Once you know the root: everything else is content-addressed
  • No timestamps, no uid/gid, no xattrs, only one way to represent the same tree
• Automatic dedup of identical subtrees in different file system trees
• Automatic dedup of identical blobs (and you can do more chunking underneath too)

---

Storage model (cont.)

• Granular seekable access into blobs
• verified streaming thanks to BLAKE3 and Bao, faulty data is detected early on
• Everything below can be retrieved from anyone without having to trust (P2P substitution, CDNs, …)

---

Usecases

• File system tree delivery
• Container image delivery
• Backing store for VCS
• Granular access into large datasets

---

Status

• In-memory backend, a local K/V backend (Sled)
• FUSE filesystem
• A gRPC API to transfer things, bindings for golang and rust
• some object storage backends in development (GCS, NATS)
• FUTUREWORK: more storage backends / store composition / in-kernel module?

Notes: of course you can use your own network protocol too, like HTTP CAS or iroh....plug different stores together to represent caches, blobfs

---

Contributing

• Join the IRC channel (#tvl on hackint), bridged to Matrix and XMPP
• Check our issue tracker
• Try to use it and tell us how you broke it!

Note: if this sounds useful to you, reach out!

---

Thanks!

<style> .container{ display: flex; } .col{ flex: 1; } </style>

---

Structure

tvix-store graph