about summary refs log tree commit diff
path: root/users/flokli
AgeCommit message (Collapse)AuthorFilesLines
2024-04-06 r/7858 chore(3p/sources): Bump channels & overlayssterni1-2/+1
- agenix has not been updated (https://github.com/ryantm/agenix/pull/241). - Re-enable now fixed dependency of flokli/archeology-ec2. Change-Id: I4e0399e5b5dbaf5e504076e029013f165dd4d191 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11363 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-03 r/7842 chore(users/flokli/nixos): restore archeology-ec2 buildFlorian Klink1-1/+2
Temporarily remove parquet-tools, so this builds again. Can be re-added once https://github.com/NixOS/nixpkgs/pull/301032 landed upstream. Change-Id: Ie74f014eb8158d5f529a5f1c55788a4edc5c805d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11347 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-03-19 r/7739 feat(users/flokli/ipu6-softisp): expose qcam to $PATHFlorian Klink1-0/+7
This allows testing the camera connection without pipewire in between: ``` LIBCAMERA_LOG_LEVELS=*:DEBUG qcam Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway. [0:26:31.326112005] [51496] INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/nix/store/src/ipa' to the IPA search path [0:26:31.326354338] [51496] DEBUG IPAModule ipa_module.cpp:334 ipa_ipu3.so: IPA module /nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_ipu3.so is signed [0:26:31.326417399] [51496] DEBUG IPAManager ipa_manager.cpp:245 Loaded IPA module '/nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_ipu3.so' [0:26:31.326484670] [51496] DEBUG IPAModule ipa_module.cpp:334 ipa_soft_simple.so: IPA module /nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_soft_simple.so is signed [0:26:31.326518633] [51496] DEBUG IPAManager ipa_manager.cpp:245 Loaded IPA module '/nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_soft_simple.so' [0:26:31.326559430] [51496] INFO Camera camera_manager.cpp:284 libcamera v0.2.0 [0:26:31.326728389] [51505] DEBUG Camera camera_manager.cpp:69 Starting camera manager [0:26:31.345213320] [51505] DEBUG DeviceEnumerator device_enumerator.cpp:230 New media device "intel-ipu6" created from /dev/media1 [0:26:31.345249850] [51505] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media1 due to 36 missing dependencies [0:26:31.350335759] [51505] DEBUG DeviceEnumerator device_enumerator_udev.cpp:322 All dependencies for media device /dev/media1 found [0:26:31.350365700] [51505] DEBUG DeviceEnumerator device_enumerator.cpp:258 Added device /dev/media1: intel-ipu6 [0:26:31.350738881] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'SimplePipelineHandler' [0:26:31.350765332] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'PipelineHandlerIPU3' [0:26:31.350779457] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'PipelineHandlerUVC' ``` Change-Id: I30b4ede89f90e2455bf1313b3c7c4d638089e5db Reviewed-on: https://cl.tvl.fyi/c/depot/+/11180 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-19 r/7738 chore(users/flokli/ipu6-softisp): refresh libcamera patchesFlorian Klink33-3652/+3858
Refresh them with the patches from https://patchwork.libcamera.org/cover/19663/. This is still based off v0.2.0. Change-Id: I875fd64e3bb71a95c92af1108a23d27c0f3494e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11179 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de>
2024-03-19 r/7737 feat(users/flokli/ipu6-softisp): test pipewire & wireplumberFlorian Klink1-0/+7
This ensures we also test building pipewire. Change-Id: I7527ca2e4259f26fecd453f5da5d91bbd3862bb3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11178 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-18 r/7734 chore(3p/sources): bump channels & overlays (2024-03-17)Vincent Ambo2-1037/+1974
In hope that iwlwifi works again on this commit, and I don't actually have to debug it. Includes following changes: * users/aspen: home-manager is shuffling around pinentry options again * users/flokli: rebase ipu6-softisp patches to Linux 6.8 make cl/11097 a separate patch * ops/modules: remove unused (and now broken) v4l2loopback module Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: I763f1f075778f2ed8db7803f87248c9dabde4213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11174 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-03-07 r/7654 feat(users/flokli/ipu6-softisp): provide example closureFlorian Klink1-0/+50
This introduces a NixOS configuration using config.nix as a NixOS module, and also checks the firmware is present in the location it's expected to. Change-Id: I3ec4333d73fe1b28e10c589c1cf351c59372bb99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11099 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-07 r/7653 fix(users/flokli/ipu6-softisp): skip subdirs from readTreeFlorian Klink2-0/+0
There's no readTree targets in there. Change-Id: I267e3c363ccf1084adb40738361ea1ce702e2a8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11098 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-03-07 r/7652 chore(users/flokli/ipu6-softisp): align with linux-firmwareFlorian Klink2-6/+4
The firmware files ended up in linux-firmware, and https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=dbc294d3e49aa63a61bb4026b5784bae5b75aa2b moved them to a ipu/ subpath, so update our patches to find them there, and stop using the pkgs.ipu6-camera-bins package. https://github.com/NixOS/nixpkgs/pull/290216 bumped our linux-firmware image past this, so this should work for people tracking nixos-unstable. Change-Id: Ic3eb9fb364c93ee0d10951451a59f7e98888b5d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11097 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 r/7618 feat(users/flokli/kb/dilemma): fix grave keyFlorian Klink1-1/+1
Change-Id: I5037c95d1a69cc55e766466821a9406695641c0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11036 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-26 r/7613 feat(users/flokli/kb/dilemma): fix clipboardFlorian Klink1-4/+4
It seems linux, or my desktop environment don't support the clipboard-related keycodes. Replace them with Ctrl-C and Ctrl-V, which is good enough. Change-Id: Ibac27b6e022157341ba6eb9f8f6ff62f9e97137b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11034 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-26 r/7606 feat(users/flokli/kb/dilemma): add RALTFlorian Klink1-2/+2
äöüß. Change-Id: I2e1df9dc504f892e9176cabf4a1479cd25a9c59f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11030 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-02-24 r/7601 chore: update links to tvixboltFlorian Klink1-1/+1
Use bolt.tvix.dev everywhere. Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-13 r/7505 chore(users/flokli/ipu6-softisp): simplify expressionFlorian Klink1-23/+4
nixos-unstable got the bump to libcamera 0.2, no need to also do this here. We only need to use the patched libcamera for pipewire and wireplumber running on the system. Change-Id: I4ba5d30f68a5ded56cba07a5ad4aaa84d2711b8f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10825 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-13 r/7504 chore(users/flokli/ipu6-softisp): inline kernel patchesFlorian Klink2-4/+17144
"media: ov2740: Fix hts value" got backported into linux-stable: ``` commit c5883a8a3676b94fe7cefde97c8eec50bb879eb2 Author: Hans de Goede <hdegoede@redhat.com> Date: Mon Dec 4 13:39:42 2023 +0100 media: ov2740: Fix hts value [ Upstream commit 3735228bbe3511f844e03dfcc4003fadb59dde23 ] HTS must be more then width, so the 1080 value clearly is wrong, this is then corrected with some weird math dividing clocks in to_pixels_per_line() which results in the hts getting multiplied by 2, resulting in 2160. Instead just directly set hts to the correct value of 2160 and drop to_pixels_per_line(). Signed-off-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl> Signed-off-by: Sasha Levin <sashal@kernel.org> ``` So drop that patch from the list of kernel patches to apply (and vendor in the patches). Change-Id: I314ccb524d156c0f445ed923ce0600f27b8cc699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10813 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de>
2024-02-10 r/7493 fix(users/flokli/kb/dillemma): update commentsFlorian Klink1-10/+10
Some of the comments didn't yet account for this layout now being colemak, and the function keys swapped to match stock Miryoku more closely. Change-Id: I38ae92153f80855ac21cc62cd603b7020b5be4ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/10787 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de>
2024-01-30 r/7456 chore(users/flokli/nixos): drop archeology nixos configFlorian Klink4-64/+0
The machine at Hetzner is gone, only the one in EC2 remains. Change-Id: Ia7266d56ef1174267b95086c51e6d80015c2f905 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10711 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de>
2024-01-30 r/7454 feat(users/flokli/ipu6-softisp): initFlorian Klink27-0/+5847
This code adds support for the ipu6 webcams via libcamera, based on the work in https://copr.fedorainfracloud.org/coprs/jwrdegoede/ipu6-softisp/. It's supposed to be included in your NixOS configuration imports. Change-Id: Ifb71999ad61161fa23506b97cb449f73fb1270e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10709 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de>
2024-01-25 r/7450 feat(users/flokli/kb/dilemma): switch to qmk upstreamFlorian Klink1-4/+4
Looks like the develop branch at least has everything that's needed. Change-Id: Ic25d571f4e6fe2a45f8f11f917622fe50a39ce07 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10690 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-23 r/7440 feat(users/flokli/kb/dilemma): enable tapsFlorian Klink2-0/+26
Change-Id: I08e4718879da3e832f38265b5741f3e0bfc586cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10682 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-22 r/7436 feat(users/flokli/kb/dilemma): enable kinetic mouse modeFlorian Klink1-0/+1
Change-Id: I959edf0996ecf0993131b554b2732cd3a265d4d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10680 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-20 r/7428 feat(users/flokli/kb/dilemma): configure miryokuFlorian Klink1-70/+81
The original layout is pretty far from Miryoku. It's not using Colemak DH, but QWERTY, and the rest also looks quite different. Align with the official Miryoku reference, from https://github.com/manna-harbour/miryoku/tree/master/docs/reference. Notable differences: - Button layer left out (normally activated by holding a bottom row pinkie key) - Media layer is missing external power and bluetooth-related keys (understandably). Change-Id: Icd8b2222c9a68650d9d4ee81aa96df0f4bc3bc99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10673 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-20 r/7427 feat(users/flokli/kb/dilemma): disable VIAFlorian Klink3-25/+0
Change-Id: I922e79233b4c9f7ee68352d155be830f4ca66644 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10671 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-20 r/7426 feat(users/flokli/kb/dilemma): init custom layoutFlorian Klink4-3/+242
So far, this is still a copy from the "via" target, but makes customization possible. Change-Id: I9e23808b72b874901a5f8c074763346181b05c1f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10670 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-19 r/7424 feat(users/flokli/keyboards): add dilemma 3x5_3Florian Klink1-0/+39
This builds the stock "via" layout, which, in addition to via support, contains some other usefulness from miryoku. Change-Id: I967a9bfcc87e108d4c0a2cb37b512d0b39912f7a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10666 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-19 r/7423 chore(users/flokli): move keyboard to keyboards/k6_proFlorian Klink3-0/+0
Change-Id: I8039053ee1fb7ed4beb564e20f98adcfc9e79fc6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10665 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-29 r/7284 feat(users/flokli/keyboard): enable kinetic modeFlorian Klink1-1/+1
Change-Id: Ieae46ff2fa4617c113852763980a5c55d3fb0b22 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10459 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-25 r/7066 feat(users/flokli/archivist): initFlorian Klink2-0/+29
This introduces a new structure, as activities are moved to a separate AWS Account for reasons (tm). Change-Id: Ic310eca2dc0d4ee81bae8944238b26910067336a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10124 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-19 r/7036 feat(users/flokli/keyboard): align 3rd layer with thinkpadFlorian Klink1-1/+2
Make these keys behave a bit more like the Fn+F* keys on the thinkpad keyboards. Seems there's currently no trivial way to get mic mute, keyboard mute and wifi toggle sent out, but considering the thinkpad usb keyboard is able to, this should be possible somehow here too - but not today, left for a followup. Change-Id: I529a958c78116dd9f7250c938e2e7989b296d6c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10076 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 r/7013 fix(users/flokli/archeology/parse_bucket_logs): fix regex and skipFlorian Klink1-1/+2
It seems the regex is not perfect, it choked on a single log line: ``` Nov 13 03:10:19 archeology-ec2 59nkrwmih3ywaxrgxqj79pn395fs6m17-parse-bucket-logs-continuously[11105]: Code: 117. DB::Exception: Line "d57bd890fbd1ae16625bdb8168064125e013198099b7e1b3c24878a4d03c3ab8 nix-cache [12/Nov/2023:09:13:02 +0000] xxx.xx.xxx.xxx - VB7SJVZ108DSSN67 REST.POST.OBJECT index.html "POST /index.html HTTP/1.1" 405 MethodNotAllowed 348 - 4 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 0bFdGKbi0n9JHXU1a2hijcJwmYdc6lG2xgbdozc3wS6mlUkBE7ssrQCHIDdOLebo78o2cGbhivY= - ECDHE-RSA-AES128-GCM-SHA256 - nix-cache.s3.amazonaws.com TLSv1.2 - -" doesn't match the regexp.: (in file/uri log/2023-11-12-10-19-50-80805A702ECF65EB): (at row 5) ``` This was due to the user-agent field. The regex is now fixed. The request itself is fun (someone trying to POST an index.html to the bucket), and we should probably filter this on the Fastly side already, not via IAM, In any case, there's no point failing to parse if a single line doesn't match the regex - we can just skip them. For the sake of completeness, logs for that day have been reprocessed and reuploaded. Change-Id: Id98a7167a381cda06d150ad5118ee9e70ead277e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10034 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 r/7012 feat(users/flokli/archeology): install parquet-toolsedef1-0/+1
Change-Id: I64cd83fbce920eeabace5b49ef623c033d98a8be Reviewed-on: https://cl.tvl.fyi/c/depot/+/10000 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-14 r/7011 feat(users/flokli/archeology): install DuckDBedef1-0/+1
Change-Id: I76bc20711c7e59d184659db134ba224cfcd7f6cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9999 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-13 r/7009 feat(users/flokli/archeology): turn on task_delayacctedef1-0/+6
More ClickHouse perf stats ^_^ Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-13 r/7008 feat(users/flokli/keyboard): add missing KC_PSCRFlorian Klink1-1/+1
Change-Id: I005defb868151ecec95e710523db3d23c859e489 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10021 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-13 r/7007 feat(users/flokli/keyboard): initFlorian Klink3-0/+116
This packages up my keyboard firmware used for the Keychron K6 Pro. We add a custom keymap to the `keyboards/keychron/k6_pro/ansi/rgb/ keymaps` directory, a copy from the `default` one (with a modified `keymap.c`), and then build that as a makefile target. `via` is *disabled*, as their keybindings take priority over keymap.c. Luckily, only `qmk` seems to be sufficient to build it. A simple `:flash` target/script is provided as well, it relies on some udev rules set in the global system (`hardware.keyboard.qmk.enable = true`). Change-Id: I9f7a7a992e13516c32033127f94e37aec62d6b67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10020 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-12 r/7003 feat(users/flokli/archeology): add AWS config to shellFlorian Klink1-1/+14
This allows using awscli inside a shell. Clickhouse AWS SSO integration still seems broken unfortunately, even with https://github.com/ClickHouse/ClickHouse/pull/54347 included in our bump - it seems it's coming up with another token file path than the AWS SDK: > SSOCredentialsProvider: Unable to open token file on path: /home/flokli/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json This is the sha1sum of the sso_start_url, not the sha1sum of the session-name (nixos / f2f059b8b7298f1ad52636d67cef8b719aa83bf5). Change-Id: Ia1bdec03c4f269a7415c42c90c1f4fd3d928f770 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10012 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-12 r/6996 feat(users/flokli/nixos/archeology-ec2): automate bucket log parsingFlorian Klink2-0/+80
This adds a `parse-bucket-logs.{service,timer}`, running once every night at 3AM UTC, figuring out the last time it was run and parsing bucket logs for all previous days. It invokes the `archeology-parse-bucket-logs` script to produce a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for that day (inside a temporary directory), then on success uploads the produced parquet file to `s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`. Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-11 r/6994 fix(users/flokli/archaeology): don't use file but column compressionFlorian Klink1-2/+5
Clickhouse also has column compression, configurable with the output_format_parquet_compression_method setting. It defaults to lz4, and the previous setting got a a zstd-compressed parquet file with lz4 data. Set output_format_parquet_compression_method to zstd instead, and sort by timestamp before assembling the parquet file. The existing files were updated to the same format with the following query: ``` SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd' ``` Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-11 r/6993 feat(users/flokli/nixos/archeology-ec2): add parse-bucket-logsFlorian Klink1-0/+4
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`. It can be invoked like this: ``` archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd ```` … and will produce a zstd-compressed Parquet file for (roughly) that time range. As the EC2 instance credentials don't give access to the logs bucket (yet), other AWS credentials need to be provided. This can be accomplished by using "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from "Option 2: Manually add a profile to your AWS credentials file (Short- term credentials)" in AWS IAM Identity Center. Processing logs for a one-hour range takes a minute or two, the resulting zstd-compressed Parquet file is around 40-80M in size. Processing logs for a whole day takes some 25mins, due to the sheer amount of data (12 GB of raw log data, distributed among 450k individual files, 20Mio log lines), but at least clickhouse isn't able to parse the resulting parquet file back in: > Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached For future automation tasks, it's probably better to run this once an hour, and further join the data later on. Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 r/6992 fix(users/flokli/archeology): make clickhouse use ambient AWS credsFlorian Klink1-1/+22
Rather than picking up from clickhouse-specific config files, this gets it to pick up from the ambient environment, which is closer to (but not the same as) the AWS default credentials chain. Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 r/6991 feat(users/flokli/archeology): show clickhouse-local progressFlorian Klink1-1/+2
This behaviour might change (or not), see https://github.com/ClickHouse/ ClickHouse/pull/42003, but as of now, a `--progress` will provide some progress. Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-11 r/6990 feat(users/flokli/archeology): add shellFlorian Klink1-0/+5
Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 r/6989 feat(users/flokli/archeology): init parse-bucket-logsFlorian Klink4-0/+55
Change-Id: I096b6fed8c73ddd5a417f5183cc113356ffd98c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9983 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-30 r/6914 feat(users/flokli/nixos): add direnv supportFlorian Klink2-6/+7
Expose `deps` separately, add a direnv with PATH_add for it to bring tooling into $PATH. Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6913 feat(users/flokli/nixos): use lazydeps for shellFlorian Klink1-2/+7
Avoid having to re-enter the shell whenever the config is changed. Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6912 feat(users/flokli/archeology): add awscli, htop, kitty terminfoFlorian Klink1-1/+8
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6911 feat(users/flokli): use nix-copy-closure instead of nix copyFlorian Klink1-1/+1
nix copy seems to stall on the EC2 box for unknown reasons. Change-Id: I30639a52758814968d3b54d716522fb88db80cfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6910 feat(users/flokli): expose readTree targets to CIFlorian Klink1-1/+1
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6909 refactor(users/flokli): move common stuff to `archeology` profileFlorian Klink3-29/+24
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6907 feat(users/flokli): add archeology-ec2Florian Klink4-3/+70
This add the EC2 box config to the repo. Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-01T08·58+0000