diff options
author | William Carroll <wpcarro@gmail.com> | 2022-10-26T16·52-0700 |
---|---|---|
committer | wpcarro <wpcarro@gmail.com> | 2022-10-26T22·29+0000 |
commit | ea49a948d173bb1fc88d3b1cf13bd56541600944 (patch) | |
tree | 7e3585662c278e136550bf27524285e58bb51608 /users/wpcarro/website/blog | |
parent | d588f1313687be7162132973b2ef473a200a1b9b (diff) |
feat(wpcarro/blog): NixOS disk clean-up (note to self) r/5205
Another note to Future Me Change-Id: Icf2edbbc6118b2b689ff403c38e91a69089a27a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7099 Reviewed-by: wpcarro <wpcarro@gmail.com> Tested-by: BuildkiteCI
Diffstat (limited to 'users/wpcarro/website/blog')
-rw-r--r-- | users/wpcarro/website/blog/posts.nix | 7 | ||||
-rw-r--r-- | users/wpcarro/website/blog/posts/nixos-disk-full-note.md | 113 |
2 files changed, 120 insertions, 0 deletions
diff --git a/users/wpcarro/website/blog/posts.nix b/users/wpcarro/website/blog/posts.nix index ba0c6abb62f1..27a184a22ca7 100644 --- a/users/wpcarro/website/blog/posts.nix +++ b/users/wpcarro/website/blog/posts.nix @@ -71,4 +71,11 @@ content = ./posts/git-filter-repo-note.md; draft = false; } + { + key = "nixos-disk-full-note"; + title = "disk full (note to self)"; + date = 1666801882; + content = ./posts/nixos-disk-full-note.md; + draft = false; + } ] diff --git a/users/wpcarro/website/blog/posts/nixos-disk-full-note.md b/users/wpcarro/website/blog/posts/nixos-disk-full-note.md new file mode 100644 index 000000000000..4bbd3f58e2fb --- /dev/null +++ b/users/wpcarro/website/blog/posts/nixos-disk-full-note.md @@ -0,0 +1,113 @@ +## Background + +Every now and then NixOS hosts runs out of disk space. This happened to my IRC +server recently... + +> No problem. Let's free-up some space with Nix's garbage-collection: +> - me + +```shell +λ nix-collect-garbage -d # failed due lack of disk space +``` + +Ironically Nix needs to do an SQLite transaction before deleting stuff and +SQLite can't do that if there's no space. This is especially funny because the +SQLite is probably a `DELETE`. + +## Solution + +First let's verify that our disk is indeed at capacity: + +```shell +λ df -h +Filesystem Size Used Avail Use% Mounted on +devtmpfs 399M 0 399M 0% /dev +tmpfs 3.9G 0 3.9G 0% /dev/shm +tmpfs 2.0G 3.7M 2.0G 1% /run +tmpfs 3.9G 408K 3.9G 1% /run/wrappers +/dev/disk/by-label/nixos 9.9G 9.9G 0G 100% / +tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup +tmpfs 797M 0 797M 0% /run/user/0 +``` + +Looks like `/dev/disk/by-label/nixos` is at `100%`. Now let's find some easy +targets to free-up space so that we can run `nix-collect-garbage -d`... + +```shell +λ du -hs /* 2>/dev/null +8.0K /bin +12M /boot +0 /dev +200K /etc +68K /home +16K /lost+found +9.0G /nix +0 /proc +1.2M /root +2.9M /run +4.0K /srv +0 /sys +44K /tmp +12K /usr +1.2G /var +``` + +Okay: `/var` looks like an easy candidate. Let's recurse into that directory: + +```shell +λ du -hs /var/* +40K /var/cache +12K /var/db +4.0K /var/empty +4.0K /var/google-users.d +211M /var/lib +0 /var/lock +918M /var/log +0 /var/run +4.0K /var/spool +44K /var/tmp +λ du -hs /var/log/* # /var/log looks promising +60M /var/log/btmp +82M /var/log/btmp.1 +776M /var/log/journal # ah-ha! journald. Let's clean-up some logs +8.0K /var/log/lastlog +1.1M /var/log/nginx +4.0K /var/log/private +12K /var/log/wtmp +``` + +To retain at most 1w's worth of logs: + +```shell +λ journalctl --vacuum-time=1w +``` + +...or if you'd prefer to retain only 100M's worth of logs: + +```shell +λ journalctl --vacuum-size=100M +``` + +Now Nix should be able to garbage-collect! + +```shell +λ nix-collect-garbage -d +``` + +And lastly verify that it WAI'd: + +``` +λ df -h +Filesystem Size Used Avail Use% Mounted on +devtmpfs 399M 0 399M 0% /dev +tmpfs 3.9G 0 3.9G 0% /dev/shm +tmpfs 2.0G 3.7M 2.0G 1% /run +tmpfs 3.9G 408K 3.9G 1% /run/wrappers +/dev/disk/by-label/nixos 9.9G 5.1G 4.3G 55% / +tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup +tmpfs 797M 0 797M 0% /run/user/0 +``` + +## Closing Thoughts + +Why doesn't Nix just reserve enough space to be able to GC itself? Not sure... |