Age | Commit message (Collapse) | Author | Files | Lines |
|
This adds a `parse-bucket-logs.{service,timer}`, running once every
night at 3AM UTC, figuring out the last time it was run and parsing
bucket logs for all previous days.
It invokes the `archeology-parse-bucket-logs` script to produce
a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for
that day (inside a temporary directory), then on success uploads the
produced parquet file to
`s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`.
Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
Clickhouse also has column compression, configurable with the
output_format_parquet_compression_method setting.
It defaults to lz4, and the previous setting got a a zstd-compressed
parquet file with lz4 data.
Set output_format_parquet_compression_method to zstd instead, and sort
by timestamp before assembling the parquet file.
The existing files were updated to the same format with the following query:
```
SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd'
```
Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`.
It can be invoked like this:
```
archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd
````
… and will produce a zstd-compressed Parquet file for (roughly) that
time range.
As the EC2 instance credentials don't give access to the logs bucket
(yet), other AWS credentials need to be provided.
This can be accomplished by using "AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from
"Option 2: Manually add a profile to your AWS credentials file (Short-
term credentials)" in AWS IAM Identity Center.
Processing logs for a one-hour range takes a minute or two, the
resulting zstd-compressed Parquet file is around 40-80M in size.
Processing logs for a whole day takes some 25mins, due to the sheer
amount of data (12 GB of raw log data, distributed among 450k individual
files, 20Mio log lines), but at least clickhouse isn't able to parse the
resulting parquet file back in:
> Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached
For future automation tasks, it's probably better to run this once an
hour, and further join the data later on.
Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Rather than picking up from clickhouse-specific config files, this gets
it to pick up from the ambient environment, which is closer to (but not
the same as) the AWS default credentials chain.
Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This behaviour might change (or not), see https://github.com/ClickHouse/
ClickHouse/pull/42003, but as of now, a `--progress` will provide some
progress.
Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I096b6fed8c73ddd5a417f5183cc113356ffd98c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9983
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: Iebaff500bc7bbb92ff4344f185a3b446f9ce9eb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9975
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
|
|
Change-Id: I67906dc2933fd46f4babe1542a498da131788f21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9974
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
|
|
Change-Id: Ibcf21617a1cee66c157644712918bcd9bb390228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9827
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
|
|
Change-Id: Icdee0caa3a053958750d4b81853dbc3eee5b193a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9973
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
|
|
Change-Id: Id0e8883c1b621afb99edde3d256cccd057e89097
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9972
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
|
|
First pass at an xdotool-based command to edit the current text input in
emacs
Change-Id: I1e04612478292fe83083d197d481e034a9fce97f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9971
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
|
|
Change-Id: Iaf223e238713a48f8c50b0c58a8800a82ba24eed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9913
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
|
|
Change-Id: I762f1cb72b9a87850bb2afb9a129a13428fe22e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9914
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I436aeba6da6a0d5096452529d7dd0be11a466e8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9905
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
Change-Id: Ia6db1213bb62cfba475ee9d1654763d56c98fc7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9897
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: I87e4b709906333ff9316792effb1082973dc918c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9896
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|
|
This can apparently work around some of the CPU throttling bugs on
~modern~ computers.
Change-Id: I807ece85d3eba53857a1cb1e73a33f7924538e96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9895
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Expose `deps` separately, add a direnv with PATH_add for it to bring
tooling into $PATH.
Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Avoid having to re-enter the shell whenever the config is changed.
Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
nix copy seems to stall on the EC2 box for unknown reasons.
Change-Id: I30639a52758814968d3b54d716522fb88db80cfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
This add the EC2 box config to the repo.
Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This adds a deploy-archeology script.
I tried getting morph to work first, but passing it a
depot.ops.nixos.nixosFor seems to be very hard - the NixOS module system
doesn't like the arguments it's called with.
Replace morph with a 3 line bash script, which assumes your ssh_config
contains config for an `archeology` host.
Change-Id: I2bf694c60ded39c201efbbb899f3b5512aa4d0f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9835
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
Change-Id: Ifdcac829ede4ec469a7ce1b608e78bae11f2766b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9834
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
Change-Id: Ic31cb8030179ff37b1cc3d3d9241e2582cfe3e5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9833
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Ic05378013aa327616a822336881d9a76c8a22d58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9829
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
And of course I managed to move the cache creation into the handlers,
instead of doing it before starting the webserver.
And now I managed to create a hopeless mess of callbacks, but oh well.
Change-Id: I73c3aeced71923c7372496286a279e326b20c388
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9813
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
I’ve been wanting to experiment with this stuff for a while,
abstracting away a handler type.
The existentials for parser and body took a bit of mucking about, but
in the end hiding the variable behind a `Body` constructor did the
trick.
Now every handler has its own cache, which means we can start caching
arbitrary results.
Change-Id: If57230c47f97ef4c548683f2c2f27660817a31f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9812
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Now this is getting cool. After 5 minutes we will ask the backend
again (which takes like 3 seconds), but then we compare the old cached
result with the new result and only send it back to the client iff it
changed.
So the client will still have to wait for the roundtrip time, but
doesn’t have to pay for the content. Plus, it gets some info that
upstream hasn’t been updated.
Change-Id: I6dba40321949da5da6a16b2e799d939573c77ba7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9811
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
This is a dumb experiment to see how hard it is to respect cache
headers; turns out, medium hard but doable.
Sets the correct expiry time according to the cache, plus respects
`If-Modified-Since` which is a tiny bit harder.
Change-Id: I9e6166af0fa254df2beb0f3919187b91a407487b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9810
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Change-Id: Ia272460d098d2b25d3890853a3cd8e29ffb31545
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9809
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Okay, so I guess you also have to seq the cache and everything in
between the IORef and the data.
Change-Id: I4c79c99afbd09e83e9d7a01d58b31b36862e4d11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9807
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Otherwise the table might potentially hold onto data from the website
request, it’s hard to say.
Change-Id: I786478bd1ce2d9775b3d0b57565d79666ef8a96f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9806
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Change-Id: I51aeacbd553e56a69f9c9347296e758418325b20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9805
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I3e8b7ed9993268fab49050fb6894e3cc21e4a318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9804
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Back at my bullshit.
Mostly copied the setup from whatcd-resolver.
Change-Id: I9edd4387ee73c18816b1692d5338735536cce70f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9803
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
|
|
This adds the materials for the lightning talk held at All Systems Go
Conference 2023, Berlin.
Talk lives at https://media.ccc.de/v/all-systems-go-2023-245-tvix-store
Change-Id: I114b1aec9f1953c148dd29ca88888c16b9fc655d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This adds the materials for the talk held at NixCon 2023, Darmstadt.
It seems it's not really possible to export a PDF out of this,
at least I couldn't get reveal-md to do it properly.
The video recordings live on
https://media.ccc.de/v/nixcon-2023-35254-tvix and
https://www.youtube.com/watch?v=j67prAPYScY .
Change-Id: Id27dc8c8637ffd455981141383c54d8d1484679e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9750
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We can cross-reference all of these to schema.org, it should work for
most of the fields.
Change-Id: I38d8dbc7e964764886ddd156c4148bcf3ee376f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9744
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Ideally we can figure out how to search for single songs by grepping
through musicbrainz. For this we kinda need the jsonld results, so
this is a first step which visualizes the structure and makes it
easy-ish to lazily traverse it.
Change-Id: Ieca21674dee8e8c2dacbab4f2f15ccbe067da647
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9743
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
We want to use this quite generic parser type for other things as
well.
Change-Id: I890b43c58e479bdf2d179a724280ef1d8748fafa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9742
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Change-Id: I8abcb479b0f5c0bd6ed1abc3c9618c2362ff835a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9740
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Change-Id: I5094b64f202eeedb57510a25850bba2edd9ec36f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9725
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I9eeaffad53aba278cd9082c5b4c3e7b1c5f6cc22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9585
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: tazjin <tazjin@tvl.su>
|
|
nixpkgs changed something in how it deals with configuration of the
package set itself when that is externally instantiated (like in
depot)
It seems like we can work around this mostly by just ... deleting some
code, as all instances of this were for allowing unfree code, which
we've already set on the top-level anyways.
* //users/sterni: fix nixpkgs config assertion to point at
pkgs.config
* //users/wpcarro: disable locate service, which is broken in nixpkgs
Change-Id: Iacf6f1c8fd5b5289e7265e155d74f8269a858ceb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9541
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I3cfb425e4dac0a467e3917df996e9800a3ebe875
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9576
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
|