| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
It prevent some shortcuts from being used the "natural way", like
only being able to type Ctrl-Alt-T with the left hand.
Luckily, RALT isn't needed that frequently, so we can expose it right
next to GACS (on the G and M keys).
Change-Id: Iccba3b1f6a5e2b01195f87471fd9972967b4e175
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12572
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I54fb10f587f5cfadd2b047e7596020094177959d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12571
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I665ba215de12fad58b91604700c09a87444ac3ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12381
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
|
|
* Treewide: re-run depotfmt
* //third_party/nixpkgs:html5validator: build with Python 3.11,
dependency openstackdocstheme doesn't support 3.12
* //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap
module API change: https://github.com/NixOS/nixpkgs/pull/318599
* //tvix/*-go: regenerate protobuf files
* //third_party/nixpkgs:treefmt: Remove patch for merged pull request
* //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches
Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
* agenix has not been updated (https://github.com/ryantm/agenix/pull/241).
* wasm-bindgen bumped to 0.2.92 in Rust WASM projects
* 3p/lisp: port lispPackages from stable channel
The Lisp package set we are using (`pkgs.lispPackages`) is the "old
old" package set, whereas we were supposed to have been using
`pkgs.lispPackages_new` (which is the "old new" package set).
Either way we missed that train, and now there's a "new new" package
set, but with a twist: Lisp packages in nixpkgs are now tied to
their compilers, so the most generic way to access them seems to be
from `pkgs.sbclPackages`.
Switching to the packages from the "new new" package set doesn't
work: Lots of stuff stops building if we just switch the sources
over, and not everything is trivially fixable.
For now we stay on the lispPackages from the stable channel. We need
to look into the migration later.
Or rewrite panettone.
* tvix: update generated protobuf files
* 3p/nixpkgs: pick trunk from stable channel; newer versions try to
read files and do network I/O during build, but don't print enough
details in error messages to figure out why.
* 3p/overlays: remove tdlib override (nixpkgs is currently new enough)
* 3p/overlays: override telega.el sources while updates are lagging in
nixpkgs
* users/flokli/ipu6-softisp: update firmware paths, which NixOS now
stores zstd-compressed.
Change-Id: I5a7a6c8b5d0688461bca92b9e6d654356d3a1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I9709aa2b663df5d3fd95d48e463e9b087e4d4ce1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11629
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Ia0b4784384b0f1dda0f849f14e5619274dec1dd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
`qmk flash` disappeared from the `qmk` command itself.
See `qmk_cli/script_qmk.py`:
```
This program can be run from anywhere, with or without a qmk_firmware
repository. If a qmk_firmware repository can be located we will use that
to augment our available subcommands.
```
By pointing QMK_HOME to our firmware sources, we can make it re-appear.
Change-Id: I5348fff546e864773063fb1f712bf2abd198972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11627
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
|
|
- agenix has not been updated
(https://github.com/ryantm/agenix/pull/241).
- Re-enable now fixed dependency of flokli/archeology-ec2.
Change-Id: I4e0399e5b5dbaf5e504076e029013f165dd4d191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11363
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Temporarily remove parquet-tools, so this builds again.
Can be re-added once https://github.com/NixOS/nixpkgs/pull/301032 landed
upstream.
Change-Id: Ie74f014eb8158d5f529a5f1c55788a4edc5c805d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11347
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
This allows testing the camera connection without pipewire in between:
```
LIBCAMERA_LOG_LEVELS=*:DEBUG qcam
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
[0:26:31.326112005] [51496] INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/nix/store/src/ipa' to the IPA search path
[0:26:31.326354338] [51496] DEBUG IPAModule ipa_module.cpp:334 ipa_ipu3.so: IPA module /nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_ipu3.so is signed
[0:26:31.326417399] [51496] DEBUG IPAManager ipa_manager.cpp:245 Loaded IPA module '/nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_ipu3.so'
[0:26:31.326484670] [51496] DEBUG IPAModule ipa_module.cpp:334 ipa_soft_simple.so: IPA module /nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_soft_simple.so is signed
[0:26:31.326518633] [51496] DEBUG IPAManager ipa_manager.cpp:245 Loaded IPA module '/nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_soft_simple.so'
[0:26:31.326559430] [51496] INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[0:26:31.326728389] [51505] DEBUG Camera camera_manager.cpp:69 Starting camera manager
[0:26:31.345213320] [51505] DEBUG DeviceEnumerator device_enumerator.cpp:230 New media device "intel-ipu6" created from /dev/media1
[0:26:31.345249850] [51505] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media1 due to 36 missing dependencies
[0:26:31.350335759] [51505] DEBUG DeviceEnumerator device_enumerator_udev.cpp:322 All dependencies for media device /dev/media1 found
[0:26:31.350365700] [51505] DEBUG DeviceEnumerator device_enumerator.cpp:258 Added device /dev/media1: intel-ipu6
[0:26:31.350738881] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'SimplePipelineHandler'
[0:26:31.350765332] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'PipelineHandlerIPU3'
[0:26:31.350779457] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'PipelineHandlerUVC'
```
Change-Id: I30b4ede89f90e2455bf1313b3c7c4d638089e5db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11180
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Refresh them with the patches from
https://patchwork.libcamera.org/cover/19663/.
This is still based off v0.2.0.
Change-Id: I875fd64e3bb71a95c92af1108a23d27c0f3494e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11179
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
|
|
This ensures we also test building pipewire.
Change-Id: I7527ca2e4259f26fecd453f5da5d91bbd3862bb3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11178
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
In hope that iwlwifi works again on this commit, and I don't actually
have to debug it.
Includes following changes:
* users/aspen: home-manager is shuffling around pinentry options again
* users/flokli: rebase ipu6-softisp patches to Linux 6.8
make cl/11097 a separate patch
* ops/modules: remove unused (and now broken) v4l2loopback module
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I763f1f075778f2ed8db7803f87248c9dabde4213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11174
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This introduces a NixOS configuration using config.nix as a NixOS
module, and also checks the firmware is present in the location it's
expected to.
Change-Id: I3ec4333d73fe1b28e10c589c1cf351c59372bb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11099
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
There's no readTree targets in there.
Change-Id: I267e3c363ccf1084adb40738361ea1ce702e2a8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11098
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
The firmware files ended up in linux-firmware, and
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=dbc294d3e49aa63a61bb4026b5784bae5b75aa2b
moved them to a ipu/ subpath, so update our patches to find them there,
and stop using the pkgs.ipu6-camera-bins package.
https://github.com/NixOS/nixpkgs/pull/290216 bumped our linux-firmware
image past this, so this should work for people tracking nixos-unstable.
Change-Id: Ic3eb9fb364c93ee0d10951451a59f7e98888b5d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11097
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I5037c95d1a69cc55e766466821a9406695641c0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11036
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
It seems linux, or my desktop environment don't support the
clipboard-related keycodes.
Replace them with Ctrl-C and Ctrl-V, which is good enough.
Change-Id: Ibac27b6e022157341ba6eb9f8f6ff62f9e97137b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11034
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
äöüß.
Change-Id: I2e1df9dc504f892e9176cabf4a1479cd25a9c59f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11030
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Use bolt.tvix.dev everywhere.
Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
nixos-unstable got the bump to libcamera 0.2, no need to also do this
here.
We only need to use the patched libcamera for pipewire and wireplumber
running on the system.
Change-Id: I4ba5d30f68a5ded56cba07a5ad4aaa84d2711b8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10825
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
"media: ov2740: Fix hts value" got backported into linux-stable:
```
commit c5883a8a3676b94fe7cefde97c8eec50bb879eb2
Author: Hans de Goede <hdegoede@redhat.com>
Date: Mon Dec 4 13:39:42 2023 +0100
media: ov2740: Fix hts value
[ Upstream commit 3735228bbe3511f844e03dfcc4003fadb59dde23 ]
HTS must be more then width, so the 1080 value clearly is wrong,
this is then corrected with some weird math dividing clocks in
to_pixels_per_line() which results in the hts getting multiplied by 2,
resulting in 2160.
Instead just directly set hts to the correct value of 2160 and
drop to_pixels_per_line().
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Signed-off-by: Sasha Levin <sashal@kernel.org>
```
So drop that patch from the list of kernel patches to apply (and vendor
in the patches).
Change-Id: I314ccb524d156c0f445ed923ce0600f27b8cc699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10813
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Some of the comments didn't yet account for this layout now being
colemak, and the function keys swapped to match stock Miryoku more
closely.
Change-Id: I38ae92153f80855ac21cc62cd603b7020b5be4ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10787
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
|
|
The machine at Hetzner is gone, only the one in EC2 remains.
Change-Id: Ia7266d56ef1174267b95086c51e6d80015c2f905
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10711
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
|
|
This code adds support for the ipu6 webcams via libcamera, based on the work in
https://copr.fedorainfracloud.org/coprs/jwrdegoede/ipu6-softisp/.
It's supposed to be included in your NixOS configuration imports.
Change-Id: Ifb71999ad61161fa23506b97cb449f73fb1270e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10709
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
|
|
Looks like the develop branch at least has everything that's needed.
Change-Id: Ic25d571f4e6fe2a45f8f11f917622fe50a39ce07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10690
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I08e4718879da3e832f38265b5741f3e0bfc586cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10682
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I959edf0996ecf0993131b554b2732cd3a265d4d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10680
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
The original layout is pretty far from Miryoku. It's not using Colemak
DH, but QWERTY, and the rest also looks quite different.
Align with the official Miryoku reference, from
https://github.com/manna-harbour/miryoku/tree/master/docs/reference.
Notable differences:
- Button layer left out (normally activated by holding a bottom row
pinkie key)
- Media layer is missing external power and bluetooth-related keys
(understandably).
Change-Id: Icd8b2222c9a68650d9d4ee81aa96df0f4bc3bc99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10673
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I922e79233b4c9f7ee68352d155be830f4ca66644
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10671
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
So far, this is still a copy from the "via" target, but makes
customization possible.
Change-Id: I9e23808b72b874901a5f8c074763346181b05c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10670
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This builds the stock "via" layout, which, in addition to via support,
contains some other usefulness from miryoku.
Change-Id: I967a9bfcc87e108d4c0a2cb37b512d0b39912f7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10666
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I8039053ee1fb7ed4beb564e20f98adcfc9e79fc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10665
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ieae46ff2fa4617c113852763980a5c55d3fb0b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10459
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
This introduces a new structure, as activities are moved to a separate
AWS Account for reasons (tm).
Change-Id: Ic310eca2dc0d4ee81bae8944238b26910067336a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10124
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Make these keys behave a bit more like the Fn+F* keys on the thinkpad
keyboards.
Seems there's currently no trivial way to get mic mute, keyboard mute
and wifi toggle sent out, but considering the thinkpad usb keyboard is
able to, this should be possible somehow here too - but not today, left
for a followup.
Change-Id: I529a958c78116dd9f7250c938e2e7989b296d6c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10076
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
It seems the regex is not perfect, it choked on a single log line:
```
Nov 13 03:10:19 archeology-ec2 59nkrwmih3ywaxrgxqj79pn395fs6m17-parse-bucket-logs-continuously[11105]: Code: 117. DB::Exception: Line "d57bd890fbd1ae16625bdb8168064125e013198099b7e1b3c24878a4d03c3ab8 nix-cache [12/Nov/2023:09:13:02 +0000] xxx.xx.xxx.xxx - VB7SJVZ108DSSN67 REST.POST.OBJECT index.html "POST /index.html HTTP/1.1" 405 MethodNotAllowed 348 - 4 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 0bFdGKbi0n9JHXU1a2hijcJwmYdc6lG2xgbdozc3wS6mlUkBE7ssrQCHIDdOLebo78o2cGbhivY= - ECDHE-RSA-AES128-GCM-SHA256 - nix-cache.s3.amazonaws.com TLSv1.2 - -" doesn't match the regexp.: (in file/uri log/2023-11-12-10-19-50-80805A702ECF65EB): (at row 5)
```
This was due to the user-agent field. The regex is now fixed.
The request itself is fun (someone trying to POST an index.html to the
bucket), and we should probably filter this on the Fastly side already,
not via IAM,
In any case, there's no point failing to parse if a single line doesn't
match the regex - we can just skip them.
For the sake of completeness, logs for that day have been reprocessed
and reuploaded.
Change-Id: Id98a7167a381cda06d150ad5118ee9e70ead277e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10034
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I64cd83fbce920eeabace5b49ef623c033d98a8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10000
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I76bc20711c7e59d184659db134ba224cfcd7f6cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9999
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
More ClickHouse perf stats ^_^
Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I005defb868151ecec95e710523db3d23c859e489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10021
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This packages up my keyboard firmware used for the Keychron K6 Pro.
We add a custom keymap to the `keyboards/keychron/k6_pro/ansi/rgb/
keymaps` directory, a copy from the `default` one (with a modified
`keymap.c`), and then build that as a makefile target.
`via` is *disabled*, as their keybindings take priority over keymap.c.
Luckily, only `qmk` seems to be sufficient to build it.
A simple `:flash` target/script is provided as well, it relies on some
udev rules set in the global system
(`hardware.keyboard.qmk.enable = true`).
Change-Id: I9f7a7a992e13516c32033127f94e37aec62d6b67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10020
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This allows using awscli inside a shell.
Clickhouse AWS SSO integration still seems broken unfortunately, even
with https://github.com/ClickHouse/ClickHouse/pull/54347 included in
our bump - it seems it's coming up with another token file path than the
AWS SDK:
> SSOCredentialsProvider: Unable to open token file on path: /home/flokli/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json
This is the sha1sum of the sso_start_url, not the sha1sum of the
session-name (nixos / f2f059b8b7298f1ad52636d67cef8b719aa83bf5).
Change-Id: Ia1bdec03c4f269a7415c42c90c1f4fd3d928f770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10012
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
This adds a `parse-bucket-logs.{service,timer}`, running once every
night at 3AM UTC, figuring out the last time it was run and parsing
bucket logs for all previous days.
It invokes the `archeology-parse-bucket-logs` script to produce
a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for
that day (inside a temporary directory), then on success uploads the
produced parquet file to
`s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`.
Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
Clickhouse also has column compression, configurable with the
output_format_parquet_compression_method setting.
It defaults to lz4, and the previous setting got a a zstd-compressed
parquet file with lz4 data.
Set output_format_parquet_compression_method to zstd instead, and sort
by timestamp before assembling the parquet file.
The existing files were updated to the same format with the following query:
```
SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd'
```
Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`.
It can be invoked like this:
```
archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd
````
… and will produce a zstd-compressed Parquet file for (roughly) that
time range.
As the EC2 instance credentials don't give access to the logs bucket
(yet), other AWS credentials need to be provided.
This can be accomplished by using "AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from
"Option 2: Manually add a profile to your AWS credentials file (Short-
term credentials)" in AWS IAM Identity Center.
Processing logs for a one-hour range takes a minute or two, the
resulting zstd-compressed Parquet file is around 40-80M in size.
Processing logs for a whole day takes some 25mins, due to the sheer
amount of data (12 GB of raw log data, distributed among 450k individual
files, 20Mio log lines), but at least clickhouse isn't able to parse the
resulting parquet file back in:
> Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached
For future automation tasks, it's probably better to run this once an
hour, and further join the data later on.
Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Rather than picking up from clickhouse-specific config files, this gets
it to pick up from the ambient environment, which is closer to (but not
the same as) the AWS default credentials chain.
Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This behaviour might change (or not), see https://github.com/ClickHouse/
ClickHouse/pull/42003, but as of now, a `--progress` will provide some
progress.
Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
