diff options
| -rw-r--r-- | users/wpcarro/website/blog/posts.nix | 7 | ||||
| -rw-r--r-- | users/wpcarro/website/blog/posts/nixos-disk-full-note.md | 113 |
2 files changed, 120 insertions, 0 deletions
diff --git a/users/wpcarro/website/blog/posts.nix b/users/wpcarro/website/blog/posts.nix index ba0c6abb62f1..27a184a22ca7 100644 --- a/users/wpcarro/website/blog/posts.nix +++ b/users/wpcarro/website/blog/posts.nix @@ -71,4 +71,11 @@ content = ./posts/git-filter-repo-note.md; draft = false; } + { + key = "nixos-disk-full-note"; + title = "disk full (note to self)"; + date = 1666801882; + content = ./posts/nixos-disk-full-note.md; + draft = false; + } ] diff --git a/users/wpcarro/website/blog/posts/nixos-disk-full-note.md b/users/wpcarro/website/blog/posts/nixos-disk-full-note.md new file mode 100644 index 000000000000..4bbd3f58e2fb --- /dev/null +++ b/users/wpcarro/website/blog/posts/nixos-disk-full-note.md @@ -0,0 +1,113 @@ +## Background + +Every now and then NixOS hosts runs out of disk space. This happened to my IRC +server recently... + +> No problem. Let's free-up some space with Nix's garbage-collection: +> - me + +```shell +λ nix-collect-garbage -d # failed due lack of disk space +``` + +Ironically Nix needs to do an SQLite transaction before deleting stuff and +SQLite can't do that if there's no space. This is especially funny because the +SQLite is probably a `DELETE`. + +## Solution + +First let's verify that our disk is indeed at capacity: + +```shell +λ df -h +Filesystem Size Used Avail Use% Mounted on +devtmpfs 399M 0 399M 0% /dev +tmpfs 3.9G 0 3.9G 0% /dev/shm +tmpfs 2.0G 3.7M 2.0G 1% /run +tmpfs 3.9G 408K 3.9G 1% /run/wrappers +/dev/disk/by-label/nixos 9.9G 9.9G 0G 100% / +tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup +tmpfs 797M 0 797M 0% /run/user/0 +``` + +Looks like `/dev/disk/by-label/nixos` is at `100%`. Now let's find some easy +targets to free-up space so that we can run `nix-collect-garbage -d`... + +```shell +λ du -hs /* 2>/dev/null +8.0K /bin +12M /boot +0 /dev +200K /etc +68K /home +16K /lost+found +9.0G /nix +0 /proc +1.2M /root +2.9M /run +4.0K /srv +0 /sys +44K /tmp +12K /usr +1.2G /var +``` + +Okay: `/var` looks like an easy candidate. Let's recurse into that directory: + +```shell +λ du -hs /var/* +40K /var/cache +12K /var/db +4.0K /var/empty +4.0K /var/google-users.d +211M /var/lib +0 /var/lock +918M /var/log +0 /var/run +4.0K /var/spool +44K /var/tmp +λ du -hs /var/log/* # /var/log looks promising +60M /var/log/btmp +82M /var/log/btmp.1 +776M /var/log/journal # ah-ha! journald. Let's clean-up some logs +8.0K /var/log/lastlog +1.1M /var/log/nginx +4.0K /var/log/private +12K /var/log/wtmp +``` + +To retain at most 1w's worth of logs: + +```shell +λ journalctl --vacuum-time=1w +``` + +...or if you'd prefer to retain only 100M's worth of logs: + +```shell +λ journalctl --vacuum-size=100M +``` + +Now Nix should be able to garbage-collect! + +```shell +λ nix-collect-garbage -d +``` + +And lastly verify that it WAI'd: + +``` +λ df -h +Filesystem Size Used Avail Use% Mounted on +devtmpfs 399M 0 399M 0% /dev +tmpfs 3.9G 0 3.9G 0% /dev/shm +tmpfs 2.0G 3.7M 2.0G 1% /run +tmpfs 3.9G 408K 3.9G 1% /run/wrappers +/dev/disk/by-label/nixos 9.9G 5.1G 4.3G 55% / +tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup +tmpfs 797M 0 797M 0% /run/user/0 +``` + +## Closing Thoughts + +Why doesn't Nix just reserve enough space to be able to GC itself? Not sure... |