From 85421b7f973e64ce9ec6f8d675854b0970967794 Mon Sep 17 00:00:00 2001 From: Connor Brewster Date: Mon, 22 Jan 2024 20:37:22 -0600 Subject: feat(tvix/store/nar-info): Support async blob upload Right now all blob uploads are performed synchronously, this means if a NAR contains many small files, the import time is dominated by round trip time to the blob service. For small files, we can buffer them in memory and upload them asynchronously to the blob service. Before returning we make sure to join all the uploads to make sure they complete successfully before responding OK. This reduces time to import a bash-interactive closure on my machine from 1m19s to 7s. Change-Id: Ica3695c159e6c8ad8769281ac20d037e3143e856 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10679 Tested-by: BuildkiteCI Reviewed-by: flokli Autosubmit: Connor Brewster --- tvix/nar-bridge/go.mod | 1 + 1 file changed, 1 insertion(+) (limited to 'tvix/nar-bridge/go.mod') diff --git a/tvix/nar-bridge/go.mod b/tvix/nar-bridge/go.mod index 175f7edcfde9..deb6943e23df 100644 --- a/tvix/nar-bridge/go.mod +++ b/tvix/nar-bridge/go.mod @@ -18,6 +18,7 @@ require ( go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go.opentelemetry.io/otel/sdk v1.22.0 go.opentelemetry.io/otel/sdk/metric v1.22.0 + golang.org/x/sync v0.4.0 google.golang.org/grpc v1.60.1 google.golang.org/protobuf v1.32.0 lukechampine.com/blake3 v1.2.1 -- cgit 1.4.1