about summary refs log tree commit diff
path: root/users/flokli/nixos
AgeCommit message (Collapse)AuthorFilesLines
2024-11-24 r/8962 feat(users/flokli/nixos-tvix-cache): re-enable http2Florian Klink1-20/+5
With nar-bridge supporting zstd content-encoding, we don't need the nginx zstd module and can re-enable http2. We also need to propagate the Accept-Encoding sent by the client to nar-bridge, so it actually knows it can send zstd. This reduces the time measured in the microbenchmark from ~13s to this: ``` hyperfine 'rm -rf /tmp/cache; nix copy --from https://nixos.tvix.store/ --to "file:///tmp/cache?compression=none" /nix/store/jlkypcf54nrh4n6r0l62ryx93z752hb2-firefox-132.0' Benchmark 1: rm -rf /tmp/cache; nix copy --from https://nixos.tvix.store/ --to "file:///tmp/cache?compression=none" /nix/store/jlkypcf54nrh4n6r0l62ryx93z752hb2-firefox-132.0 Time (mean ± σ): 4.880 s ± 0.207 s [User: 4.661 s, System: 2.377 s] Range (min … max): 4.700 s … 5.274 s 10 runs ``` Change-Id: Id092307423636163ae95ef87ec8fa558b83ce0bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12835 Reviewed-by: Jörg Thalheim <joerg@thalheim.io> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-11-23 r/8958 refactor(users/flokli/nixos-tvix-cache): absorb otlpcollector into alloyFlorian Klink1-49/+33
We don't need a separate instance of opentelemetry-collector, alloy can also do this job for us. Change-Id: I1b671ba57d70b080f7db112e1afcfe2e0cbdd13e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12829 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com> Tested-by: BuildkiteCI
2024-11-23 r/8957 fix(users/flokli/nixos-tvix-cache): bump max_traces_per_userFlorian Klink1-0/+2
These are quite bursty, and I've seen messages about getting rate limited. Change-Id: I73058140957cb5718971fa432c003c2d1b0305e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12828 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-11-23 r/8956 feat(users/flokli/nixos/nixos-tvix-cache): also collect system metricszimbatm1-0/+26
Use grafana-alloy to collect system metrics. Change-Id: I592e64ca722701d4f12e69a531a434b54954955a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12827 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-11-23 r/8950 feat(users/flokli/nixos/nixos-tvix-cache): collect metricsFlorian Klink1-57/+61
This enables routing of metrics to an instance of VictoriaMetrics, and configures opentelemetry-collector to route metrics there. Change-Id: If765191a4cc70ddcaad821d45132b96a10a12148 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12812 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
2024-11-23 r/8949 feat(users/flokli/nixos/nixos-tvix-cache): initFlorian Klink8-0/+555
This is a fetch-through mirror of cache.nixos.org, hosted by NumTide. The current machine is a SX65 Hetzner dedicated server with 4x22TB SATA disks, and 2x1TB NVMe disks. The goals of this machine: - Exercise tvix-store and nar-bridge code - Collect usage metrics (see https://nixos.tvix.store/grafana) - Identify bottlenecks - Replace cache.nixos.org? Be however aware that there's zero availability guarantees. Since Tvix doesn't support garbage collection yet, we either will delete data or order a bigger box. Change-Id: Id24baa18cae1629a06caaa059c0c75d4a01659d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12811 Tested-by: BuildkiteCI Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com> Reviewed-by: flokli <flokli@flokli.de>
2024-11-22 r/8946 fix(users/flokli/nixos): add source_upFlorian Klink1-0/+2
This got lost somehow, but is necessary to keep `mg` in `$PATH`. Change-Id: I2100d68225284bfe825bcc5ab01628891ebd09a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12810 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 r/8603 feat(users/flokli/nixos): add awscli to shellFlorian Klink1-0/+1
Change-Id: I665ba215de12fad58b91604700c09a87444ac3ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/12381 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de>
2024-04-06 r/7858 chore(3p/sources): Bump channels & overlayssterni1-2/+1
- agenix has not been updated (https://github.com/ryantm/agenix/pull/241). - Re-enable now fixed dependency of flokli/archeology-ec2. Change-Id: I4e0399e5b5dbaf5e504076e029013f165dd4d191 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11363 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-03 r/7842 chore(users/flokli/nixos): restore archeology-ec2 buildFlorian Klink1-1/+2
Temporarily remove parquet-tools, so this builds again. Can be re-added once https://github.com/NixOS/nixpkgs/pull/301032 landed upstream. Change-Id: Ie74f014eb8158d5f529a5f1c55788a4edc5c805d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11347 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-01-30 r/7456 chore(users/flokli/nixos): drop archeology nixos configFlorian Klink4-64/+0
The machine at Hetzner is gone, only the one in EC2 remains. Change-Id: Ia7266d56ef1174267b95086c51e6d80015c2f905 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10711 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 r/7012 feat(users/flokli/archeology): install parquet-toolsedef1-0/+1
Change-Id: I64cd83fbce920eeabace5b49ef623c033d98a8be Reviewed-on: https://cl.tvl.fyi/c/depot/+/10000 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-14 r/7011 feat(users/flokli/archeology): install DuckDBedef1-0/+1
Change-Id: I76bc20711c7e59d184659db134ba224cfcd7f6cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9999 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-13 r/7009 feat(users/flokli/archeology): turn on task_delayacctedef1-0/+6
More ClickHouse perf stats ^_^ Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-12 r/6996 feat(users/flokli/nixos/archeology-ec2): automate bucket log parsingFlorian Klink2-0/+80
This adds a `parse-bucket-logs.{service,timer}`, running once every night at 3AM UTC, figuring out the last time it was run and parsing bucket logs for all previous days. It invokes the `archeology-parse-bucket-logs` script to produce a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for that day (inside a temporary directory), then on success uploads the produced parquet file to `s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`. Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-11 r/6993 feat(users/flokli/nixos/archeology-ec2): add parse-bucket-logsFlorian Klink1-0/+4
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`. It can be invoked like this: ``` archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd ```` … and will produce a zstd-compressed Parquet file for (roughly) that time range. As the EC2 instance credentials don't give access to the logs bucket (yet), other AWS credentials need to be provided. This can be accomplished by using "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from "Option 2: Manually add a profile to your AWS credentials file (Short- term credentials)" in AWS IAM Identity Center. Processing logs for a one-hour range takes a minute or two, the resulting zstd-compressed Parquet file is around 40-80M in size. Processing logs for a whole day takes some 25mins, due to the sheer amount of data (12 GB of raw log data, distributed among 450k individual files, 20Mio log lines), but at least clickhouse isn't able to parse the resulting parquet file back in: > Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached For future automation tasks, it's probably better to run this once an hour, and further join the data later on. Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-30 r/6914 feat(users/flokli/nixos): add direnv supportFlorian Klink2-6/+7
Expose `deps` separately, add a direnv with PATH_add for it to bring tooling into $PATH. Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6913 feat(users/flokli/nixos): use lazydeps for shellFlorian Klink1-2/+7
Avoid having to re-enter the shell whenever the config is changed. Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6912 feat(users/flokli/archeology): add awscli, htop, kitty terminfoFlorian Klink1-1/+8
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6911 feat(users/flokli): use nix-copy-closure instead of nix copyFlorian Klink1-1/+1
nix copy seems to stall on the EC2 box for unknown reasons. Change-Id: I30639a52758814968d3b54d716522fb88db80cfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6910 feat(users/flokli): expose readTree targets to CIFlorian Klink1-1/+1
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6909 refactor(users/flokli): move common stuff to `archeology` profileFlorian Klink3-29/+24
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6907 feat(users/flokli): add archeology-ec2Florian Klink4-3/+70
This add the EC2 box config to the repo. Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-29 r/6905 feat(users/flokli/nixos): add deploy scriptFlorian Klink1-1/+16
This adds a deploy-archeology script. I tried getting morph to work first, but passing it a depot.ops.nixos.nixosFor seems to be very hard - the NixOS module system doesn't like the arguments it's called with. Replace morph with a 3 line bash script, which assumes your ssh_config contains config for an `archeology` host. Change-Id: I2bf694c60ded39c201efbbb899f3b5512aa4d0f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9835 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-29 r/6904 refactor(users/flokli): drop some optional args and complexityFlorian Klink3-9/+7
Change-Id: Ifdcac829ede4ec469a7ce1b608e78bae11f2766b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9834 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-29 r/6903 feat(users/flokli/nixos): init archeologyFlorian Klink5-0/+88
Change-Id: Ic31cb8030179ff37b1cc3d3d9241e2582cfe3e5e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9833 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2025-01-06T08·29+0000