about summary refs log tree commit diff
path: root/users
AgeCommit message (Collapse)AuthorFilesLines
2023-12-02 r/7105 feat(tazjin/aoc2023): day 1 solutionVincent Ambo1-0/+64
String splits all the way! Change-Id: I7e7f64fb082ee2e04d895f3b7e52210610d06d39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10184 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2023-12-01 r/7104 feat(tazjin/aoc2023): day 1 solutionVincent Ambo1-0/+52
The second task was very annoying because you had to guess the actual rules (overlapping words), as they're not explained correctly in the task. My solution hardcodes those cases. Change-Id: Idf24579a78a1b8ede368504d3ff0c58c9978f069 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10183 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-01 r/7103 feat(tazjin/emacs): experimentally install orderless & corfuVincent Ambo1-0/+4
Change-Id: Iabf2ad7562e5a9008cbf7aa701716b204e7239f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10182 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-12-01 r/7102 feat(tazjin/nixos): annual attempt to use FirefoxVincent Ambo1-0/+1
Lets see if it's good now. Chrom(e|ium) (and by extension (pun intended) derivatives) are getting rid of ad blockers, so need to do *something*. Change-Id: Id7c69d38ddb0ad929aed9e6fbae4d9919ed0731f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10177 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-01 r/7101 chore(tazjin/emacs): remove company-modeVincent Ambo5-20/+3
I have a suspicion that some strange behaviour I occasionally get is related to company mode. Change-Id: I26f25c31967ae092d15248a806acdf4f28cb4c10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10176 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-12-01 r/7100 refactor(tazjin/emacs): remove telephone-lineVincent Ambo4-31/+5
I don't really like this package, it's kinda wonky, and now that part of my mode line logic is in the tab-bar, it's no longer needed. Change-Id: I4791a75e5ce2f0c49ef0d239cadf6a4f81c73636 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10171 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-12-01 r/7099 fix(grfn/xanthous): Fix buildAspen Smith3-6/+12
Bunch of miscellaneous stuff due to breakages in dependencies Change-Id: I807cfa875148e7e5b691f2be0b58dc0a08f3c3ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/9003 Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2023-12-01 r/7098 chore(grfn/wigglydonke.rs): Add cf 2fa verify tokenAspen Smith1-0/+1
I lost a yubikey :( this is the way to recover my cloudflare account Change-Id: Iaffb567ec4a072baa93e78073b47e99a1dc2bb36 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10181 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2023-11-30 r/7097 chore(sterni/machines): move minecraft from edwin to ingeborgsterni3-1/+1
Change-Id: I6917a9633c998148d6e5d23b17d949ee007898e5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10180 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-30 r/7096 feat(sterni/machines): prevent deploy if hostname doesn't matchsterni1-1/+5
Change-Id: I5a850e0d98069483e89d90022b624feba60ceebc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10179 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-30 r/7095 feat(sterni/ingeborg): make secrets availablesterni5-22/+30
Change-Id: I2a39a6e5125e95c4ea10836694b594d068ceda73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10178 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-30 r/7094 feat(sterni/ingeborg): set up monitoring via netdatasterni4-0/+156
Main objective was to get SMART/md monitoring working, alerts go (via some awful glue code) to #sterni.lv on hackint. Bot nick should also be registered in the future. Change-Id: Ia73c5a64ee9f6df62f5fbe21fc1606477e3d6e73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10174 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-30 r/7093 chore(sterni/edwin): disable creative minecraft serversterni1-1/+1
We currently don't need it actively and it is easy to re-enable if needed. Due to spawn chunks simulation it is not really idling either. Change-Id: I2e4e5ff2271fd61ee1affec27a614244d4a87fcf Reviewed-on: https://cl.tvl.fyi/c/depot/+/10173 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-30 r/7092 fix(sterni/mod/minecraft-fabric): correctly set service enable optsterni1-1/+1
This actually allows disabling a service for the first time, since the configuration structure for the service is created irrespective of the enable option. Change-Id: I08e5e67565d6fe210fb0f65600b8750433ce4712 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10175 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-30 r/7091 fix(sterni/edwin): launch mirror jobs after network-online.targetsterni1-1/+1
This will hopefully prevent failures on system deploys where DNS is briefly not available, so git(1) fails to resolve github.com. Thanks flokli for the tip. Change-Id: I6096e9f3655cbe28ca2a71142de22337814e0be1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10172 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-30 r/7089 chore(grfn/system): Death to kolideAspen Smith4-53/+0
Change-Id: Ie0433ad182ff2292968572d45822a4137c3066eb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10168 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi>
2023-11-28 r/7079 refactor(tazjin/emacs): move global-mode-line info into tab-barVincent Ambo3-39/+7
This removes the wonky hack with detecting the bottom right window, and gives me saner, unified handling of this display in the tab bar (of which there is only one!) Change-Id: Id21c6b2472d0c89fc4d000a10a9e90d2ddba86b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10165 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-26 r/7072 feat(tazjin/emacs): i3-style jumping back&forth for tab-bar-modeVincent Ambo1-1/+15
Change-Id: I19cab91646241edc38a58bf5546c9dd8161abeb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10130 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-26 r/7070 feat(sterni/ingeborg): enable btrfs auto scrubsterni3-26/+3
While we are at it, rename disk-checkup.nix to btrfs-auto-scrub.nix and move it into //ops/modules. I originally wanted to have additionally disk health related services in that module, but the btrfs scrub functionality is nicely self-contained and reusable, so I think it makes sense to have this in a more central location. Change-Id: Iabdd62838eef009540ca71abafd921afda2a9b47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10128 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-25 r/7069 feat(sterni/machines/ingeborg): boot-strapsterni3-0/+153
Network configuration and initrd setup is basically the same as with edwin, but we are using md for Software RAID this time as well as LVM over two partitions with LUKS: - sda2 <-- RAID1 --> sdb2 (boot-raid) └ boot partition, ext4 (encrypted-container-raid) - sda3 <-- RAID1 --> sdb3 └ LUKS container └ Volume Group vgmain ├ Logical Volume vgmain/swap │ └ swap └ Logical Volume vgmain/root └ btrfs So we no longer rely on btrfs raid1 due to question marks over its reliability (I personally did not have any problems though). This also means that we have less LUKS containers we need to unlock when booting (kind of neglible improvement). The biggest improvement is that we have redundancy for the swap, so a disk failure shouldn't cause memory corruption/loss. Change-Id: I14f065b659857415917d9a60a7ec019e687f8d1c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10127 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-25 r/7068 chore(sterni/edwin): allow 10G of logs (there's enough space)sterni1-1/+1
Change-Id: I576107047d8213e718e720d3a7a1dcc1dff39122 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10126 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-25 r/7067 refactor(sterni/edwin): move generic settings into modulesterni2-64/+81
These settings would also be applied to a machine that'd replace edwin, so it's useful to have them outside edwin's default.nix. Change-Id: I4e8f464118a103645e53909a87c6ee4446022fa3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10125 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-11-25 r/7066 feat(users/flokli/archivist): initFlorian Klink2-0/+29
This introduces a new structure, as activities are moved to a separate AWS Account for reasons (tm). Change-Id: Ic310eca2dc0d4ee81bae8944238b26910067336a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10124 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-25 r/7062 chore(tazjin/emacs): remove all workspace related functionalityVincent Ambo2-191/+1
This removes everything related to workspaces, as well as the current RandR configuration, creating a (cleaner) slate for switching to tabs. This was supposed to be committed earlier, but got broken by clbot (see b/333). Change-Id: I2d110bca0d6629a505699210f0aba12882f83d48 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10115 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-25 r/7061 feat(tazjin/emacs): initial tab-bar-mode configurationVincent Ambo2-3/+23
Sets up tab-bar-mode for EXWM, and adds simple initial configuration and shortcuts to make it behave sanely. I had to assign tab switching shortcuts myself, as the EXWM input key stuff doesn't work correctly with the built-in way of setting these shortcuts in tab-bar-mode. Change-Id: Icd96f03ae7bd2bd5f7c2d59f9aca7a1ca2e7f788 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10116 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-25 r/7060 feat(tazjin/emacs): dynamically assign EXWM workspaces to monitorsVincent Ambo1-2/+37
Adds an `exwm-assign-workspaces` function that automatically creates and assigns workspaces to each currently connected monitor. The first workspace (index 0) is always on the primary monitor. This function should be idempotent and can be called at any point to synchronise X outputs and what EXWM is displaying on them. This works because tabs are disconnected from workspaces completely, so I don't have to care about what's going on on other workspaces anymore. Still missing: * functions to connect/disconnect outputs * switching to other outputs from within emacs commands (i.e. without the mouse) Change-Id: I7c24aa1b45218fe35de6939e799852b5d11d1272 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10119 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-25 r/7058 feat(tazjin/emacs): add a monitor focus switch shortcutVincent Ambo1-0/+14
This one is a bit stupid because I couldn't figure out a way to determine the active workspace. It's definitely possible (either through some XCB calls, or through state management in screen change hooks), but for now this is fine. Change-Id: I5e4c531b248caa0021664bad9dc196bef60cfbac Reviewed-on: https://cl.tvl.fyi/c/depot/+/10122 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-25 r/7057 fix(tazjin/emacs): pick new primary monitor when disabling currentVincent Ambo1-3/+15
By default, this will be the (only) remaining monitor. In N>2 situations, ask the user. Change-Id: Id68fcf60d56d0414d1072b8ffeef72c608678f52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10121 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-25 r/7056 feat(tazjin/emacs): add interactive commands for configuring screensVincent Ambo1-0/+52
Uses a bunch of weird xrandr invocations and completing reads to configure screens the way I want. Note that this has a known bug where disconnecting a primary screen will *not* make one of the remaining screens primary. Change-Id: Ide5322df446685cc4740d4ddd7b6ca8682375050 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10120 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-22 r/7044 fix(3p/exwm): fix EXWM build with depot sourcesVincent Ambo1-1/+1
Something changed in the machinery and broke the overriding. I didn't notice this, as a I was temporarily using an unpatched EXWM. Change-Id: I1a4e8ea63bd116d86a430e680c2b631474e9a0fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/10047 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-19 r/7036 feat(users/flokli/keyboard): align 3rd layer with thinkpadFlorian Klink1-1/+2
Make these keys behave a bit more like the Fn+F* keys on the thinkpad keyboards. Seems there's currently no trivial way to get mic mute, keyboard mute and wifi toggle sent out, but considering the thinkpad usb keyboard is able to, this should be possible somehow here too - but not today, left for a followup. Change-Id: I529a958c78116dd9f7250c938e2e7989b296d6c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10076 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 r/7013 fix(users/flokli/archeology/parse_bucket_logs): fix regex and skipFlorian Klink1-1/+2
It seems the regex is not perfect, it choked on a single log line: ``` Nov 13 03:10:19 archeology-ec2 59nkrwmih3ywaxrgxqj79pn395fs6m17-parse-bucket-logs-continuously[11105]: Code: 117. DB::Exception: Line "d57bd890fbd1ae16625bdb8168064125e013198099b7e1b3c24878a4d03c3ab8 nix-cache [12/Nov/2023:09:13:02 +0000] xxx.xx.xxx.xxx - VB7SJVZ108DSSN67 REST.POST.OBJECT index.html "POST /index.html HTTP/1.1" 405 MethodNotAllowed 348 - 4 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 0bFdGKbi0n9JHXU1a2hijcJwmYdc6lG2xgbdozc3wS6mlUkBE7ssrQCHIDdOLebo78o2cGbhivY= - ECDHE-RSA-AES128-GCM-SHA256 - nix-cache.s3.amazonaws.com TLSv1.2 - -" doesn't match the regexp.: (in file/uri log/2023-11-12-10-19-50-80805A702ECF65EB): (at row 5) ``` This was due to the user-agent field. The regex is now fixed. The request itself is fun (someone trying to POST an index.html to the bucket), and we should probably filter this on the Fastly side already, not via IAM, In any case, there's no point failing to parse if a single line doesn't match the regex - we can just skip them. For the sake of completeness, logs for that day have been reprocessed and reuploaded. Change-Id: Id98a7167a381cda06d150ad5118ee9e70ead277e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10034 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 r/7012 feat(users/flokli/archeology): install parquet-toolsedef1-0/+1
Change-Id: I64cd83fbce920eeabace5b49ef623c033d98a8be Reviewed-on: https://cl.tvl.fyi/c/depot/+/10000 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-14 r/7011 feat(users/flokli/archeology): install DuckDBedef1-0/+1
Change-Id: I76bc20711c7e59d184659db134ba224cfcd7f6cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9999 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-13 r/7010 feat(users/grfn/web): Update work sectionAspen Smith1-4/+8
I no longer work at ReadySet. Change-Id: Idc19e2d68846551b6cd94f84594712692ebe35a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9976 Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: grfn <grfn@gws.fyi>
2023-11-13 r/7009 feat(users/flokli/archeology): turn on task_delayacctedef1-0/+6
More ClickHouse perf stats ^_^ Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-13 r/7008 feat(users/flokli/keyboard): add missing KC_PSCRFlorian Klink1-1/+1
Change-Id: I005defb868151ecec95e710523db3d23c859e489 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10021 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-13 r/7007 feat(users/flokli/keyboard): initFlorian Klink3-0/+116
This packages up my keyboard firmware used for the Keychron K6 Pro. We add a custom keymap to the `keyboards/keychron/k6_pro/ansi/rgb/ keymaps` directory, a copy from the `default` one (with a modified `keymap.c`), and then build that as a makefile target. `via` is *disabled*, as their keybindings take priority over keymap.c. Luckily, only `qmk` seems to be sufficient to build it. A simple `:flash` target/script is provided as well, it relies on some udev rules set in the global system (`hardware.keyboard.qmk.enable = true`). Change-Id: I9f7a7a992e13516c32033127f94e37aec62d6b67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10020 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-12 r/7003 feat(users/flokli/archeology): add AWS config to shellFlorian Klink1-1/+14
This allows using awscli inside a shell. Clickhouse AWS SSO integration still seems broken unfortunately, even with https://github.com/ClickHouse/ClickHouse/pull/54347 included in our bump - it seems it's coming up with another token file path than the AWS SDK: > SSOCredentialsProvider: Unable to open token file on path: /home/flokli/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json This is the sha1sum of the sso_start_url, not the sha1sum of the session-name (nixos / f2f059b8b7298f1ad52636d67cef8b719aa83bf5). Change-Id: Ia1bdec03c4f269a7415c42c90c1f4fd3d928f770 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10012 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-12 r/7000 chore(tazjin/tgsa): bump cargo dependenciesVincent Ambo1-226/+294
Fixes: - RUSTSEC-2023-0044 - RUSTSEC-2023-0018 Change-Id: Ifc1acce5696f9ec584ac7790d3a99f8ad7d28707 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10014 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-12 r/6999 chore(tazjin/yddns): bump cargo dependenciesVincent Ambo1-360/+312
Fixes RUSTSEC-2023-0053. Change-Id: I6b9fc31dad405b7f9fc21c27fc7beee3687a4572 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10013 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-12 r/6997 feat(tazjin/blog): import blog post on emacs buffer switching thingVincent Ambo2-0/+24
This was previously only in my Telegram channel, but it might as well be on the blog itself. Change-Id: I301ebeaa4dd1875f3858cee5259a5c689b950790 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10009 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-12 r/6996 feat(users/flokli/nixos/archeology-ec2): automate bucket log parsingFlorian Klink2-0/+80
This adds a `parse-bucket-logs.{service,timer}`, running once every night at 3AM UTC, figuring out the last time it was run and parsing bucket logs for all previous days. It invokes the `archeology-parse-bucket-logs` script to produce a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for that day (inside a temporary directory), then on success uploads the produced parquet file to `s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`. Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-11 r/6994 fix(users/flokli/archaeology): don't use file but column compressionFlorian Klink1-2/+5
Clickhouse also has column compression, configurable with the output_format_parquet_compression_method setting. It defaults to lz4, and the previous setting got a a zstd-compressed parquet file with lz4 data. Set output_format_parquet_compression_method to zstd instead, and sort by timestamp before assembling the parquet file. The existing files were updated to the same format with the following query: ``` SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd' ``` Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-11 r/6993 feat(users/flokli/nixos/archeology-ec2): add parse-bucket-logsFlorian Klink1-0/+4
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`. It can be invoked like this: ``` archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd ```` … and will produce a zstd-compressed Parquet file for (roughly) that time range. As the EC2 instance credentials don't give access to the logs bucket (yet), other AWS credentials need to be provided. This can be accomplished by using "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from "Option 2: Manually add a profile to your AWS credentials file (Short- term credentials)" in AWS IAM Identity Center. Processing logs for a one-hour range takes a minute or two, the resulting zstd-compressed Parquet file is around 40-80M in size. Processing logs for a whole day takes some 25mins, due to the sheer amount of data (12 GB of raw log data, distributed among 450k individual files, 20Mio log lines), but at least clickhouse isn't able to parse the resulting parquet file back in: > Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached For future automation tasks, it's probably better to run this once an hour, and further join the data later on. Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 r/6992 fix(users/flokli/archeology): make clickhouse use ambient AWS credsFlorian Klink1-1/+22
Rather than picking up from clickhouse-specific config files, this gets it to pick up from the ambient environment, which is closer to (but not the same as) the AWS default credentials chain. Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 r/6991 feat(users/flokli/archeology): show clickhouse-local progressFlorian Klink1-1/+2
This behaviour might change (or not), see https://github.com/ClickHouse/ ClickHouse/pull/42003, but as of now, a `--progress` will provide some progress. Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-11 r/6990 feat(users/flokli/archeology): add shellFlorian Klink1-0/+5
Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 r/6989 feat(users/flokli/archeology): init parse-bucket-logsFlorian Klink4-0/+55
Change-Id: I096b6fed8c73ddd5a417f5183cc113356ffd98c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9983 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-06 r/6969 feat(users/grfn/web): Purge cloudflare cache on deployAspen Smith3-9/+39
Change-Id: Iebaff500bc7bbb92ff4344f185a3b446f9ce9eb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9975 Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-18T12·18+0000