depot - monorepo for the virus lounge

Age	Commit message (Collapse)	Author	Files	Lines
2023-11-12	r/7003 feat(users/flokli/archeology): add AWS config to shell	Florian Klink	1	-1/+14
	This allows using awscli inside a shell. Clickhouse AWS SSO integration still seems broken unfortunately, even with https://github.com/ClickHouse/ClickHouse/pull/54347 included in our bump - it seems it's coming up with another token file path than the AWS SDK: > SSOCredentialsProvider: Unable to open token file on path: /home/flokli/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json This is the sha1sum of the sso_start_url, not the sha1sum of the session-name (nixos / f2f059b8b7298f1ad52636d67cef8b719aa83bf5). Change-Id: Ia1bdec03c4f269a7415c42c90c1f4fd3d928f770 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10012 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-11	r/6994 fix(users/flokli/archaeology): don't use file but column compression	Florian Klink	1	-2/+5
	Clickhouse also has column compression, configurable with the output_format_parquet_compression_method setting. It defaults to lz4, and the previous setting got a a zstd-compressed parquet file with lz4 data. Set output_format_parquet_compression_method to zstd instead, and sort by timestamp before assembling the parquet file. The existing files were updated to the same format with the following query: ``` SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd' ``` Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-11	r/6992 fix(users/flokli/archeology): make clickhouse use ambient AWS creds	Florian Klink	1	-1/+22
	Rather than picking up from clickhouse-specific config files, this gets it to pick up from the ambient environment, which is closer to (but not the same as) the AWS default credentials chain. Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11	r/6991 feat(users/flokli/archeology): show clickhouse-local progress	Florian Klink	1	-1/+2
	This behaviour might change (or not), see https://github.com/ClickHouse/ ClickHouse/pull/42003, but as of now, a `--progress` will provide some progress. Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-11	r/6990 feat(users/flokli/archeology): add shell	Florian Klink	1	-0/+5
	Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11	r/6989 feat(users/flokli/archeology): init parse-bucket-logs	Florian Klink	4	-0/+55
	Change-Id: I096b6fed8c73ddd5a417f5183cc113356ffd98c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9983 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>