about summary refs log tree commit diff
path: root/users/flokli/nixos
AgeCommit message (Collapse)AuthorFilesLines
2023-11-13 r/7009 feat(users/flokli/archeology): turn on task_delayacctedef1-0/+6
More ClickHouse perf stats ^_^ Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-12 r/6996 feat(users/flokli/nixos/archeology-ec2): automate bucket log parsingFlorian Klink2-0/+80
This adds a `parse-bucket-logs.{service,timer}`, running once every night at 3AM UTC, figuring out the last time it was run and parsing bucket logs for all previous days. It invokes the `archeology-parse-bucket-logs` script to produce a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for that day (inside a temporary directory), then on success uploads the produced parquet file to `s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`. Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-11 r/6993 feat(users/flokli/nixos/archeology-ec2): add parse-bucket-logsFlorian Klink1-0/+4
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`. It can be invoked like this: ``` archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd ```` … and will produce a zstd-compressed Parquet file for (roughly) that time range. As the EC2 instance credentials don't give access to the logs bucket (yet), other AWS credentials need to be provided. This can be accomplished by using "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from "Option 2: Manually add a profile to your AWS credentials file (Short- term credentials)" in AWS IAM Identity Center. Processing logs for a one-hour range takes a minute or two, the resulting zstd-compressed Parquet file is around 40-80M in size. Processing logs for a whole day takes some 25mins, due to the sheer amount of data (12 GB of raw log data, distributed among 450k individual files, 20Mio log lines), but at least clickhouse isn't able to parse the resulting parquet file back in: > Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached For future automation tasks, it's probably better to run this once an hour, and further join the data later on. Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-30 r/6914 feat(users/flokli/nixos): add direnv supportFlorian Klink2-6/+7
Expose `deps` separately, add a direnv with PATH_add for it to bring tooling into $PATH. Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6913 feat(users/flokli/nixos): use lazydeps for shellFlorian Klink1-2/+7
Avoid having to re-enter the shell whenever the config is changed. Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6912 feat(users/flokli/archeology): add awscli, htop, kitty terminfoFlorian Klink1-1/+8
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6911 feat(users/flokli): use nix-copy-closure instead of nix copyFlorian Klink1-1/+1
nix copy seems to stall on the EC2 box for unknown reasons. Change-Id: I30639a52758814968d3b54d716522fb88db80cfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6910 feat(users/flokli): expose readTree targets to CIFlorian Klink1-1/+1
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6909 refactor(users/flokli): move common stuff to `archeology` profileFlorian Klink3-29/+24
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6907 feat(users/flokli): add archeology-ec2Florian Klink4-3/+70
This add the EC2 box config to the repo. Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-29 r/6905 feat(users/flokli/nixos): add deploy scriptFlorian Klink1-1/+16
This adds a deploy-archeology script. I tried getting morph to work first, but passing it a depot.ops.nixos.nixosFor seems to be very hard - the NixOS module system doesn't like the arguments it's called with. Replace morph with a 3 line bash script, which assumes your ssh_config contains config for an `archeology` host. Change-Id: I2bf694c60ded39c201efbbb899f3b5512aa4d0f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9835 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-29 r/6904 refactor(users/flokli): drop some optional args and complexityFlorian Klink3-9/+7
Change-Id: Ifdcac829ede4ec469a7ce1b608e78bae11f2766b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9834 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-29 r/6903 feat(users/flokli/nixos): init archeologyFlorian Klink5-0/+88
Change-Id: Ic31cb8030179ff37b1cc3d3d9241e2582cfe3e5e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9833 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-01T14·14+0000