前言

最近博客莫名其妙 502 gateway | Nginx,不知道什么情况,于是乎上后台查看,发现是因为40G的磁盘被狠狠地塞满,很好奇为什么会酱紫,于是开始排查。

正文

莫名其妙的发现博客开始很卡,然后重启了一次服务器之后,502 gateway | Nginx了,遂开始排查。

先排查halo本身是什么问题systemctl status halo

halo.service - Halo Service
   Loaded: loaded (/etc/systemd/system/halo.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-10-09 21:00:56 CST; 825ms ago
     Docs: https://docs.halo.run
  Process: 21680 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=1/FAILURE)
 Main PID: 21682 (java)
   CGroup: /system.slice/halo.service
           └─21682 /usr/bin/java -server -Xms256m -Xmx256m -jar /home/halo/app/halo.jar

Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez systemd[1]: Started Halo Service.
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez systemd[1]: Starting Halo Service...
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez java[21682]: OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez java[21682]: 21682
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez java[21682]: Try using the -Djava.io.tmpdir= option to select an alternate temp location.

发现其中有一条Insufficient space for shared memory file,于是df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   39G     0 100% /
devtmpfs        456M     0  456M   0% /dev
tmpfs           466M     0  466M   0% /dev/shm
tmpfs           466M   42M  425M   9% /run
tmpfs           466M     0  466M   0% /sys/fs/cgroup
tmpfs            94M     0   94M   0% /run/user/0

发现怎么莫名其妙的40G被塞满了,检查了一下halo的文件大小,du -sh /home/halo/*,发现才370M,为什么370M能塞满40G的盘!

370M    /home/halo/.halo/

奇了个怪,为什么370M能塞满40G的盘du -sh /* 2>/dev/null,看看是什么塞满了:

0       /bin
140M    /boot
0       /dev
37M     /etc
447M    /home
0       /lib
0       /lib64
16K     /lost+found
4.0K    /media
40K     /mnt
4.0K    /opt
8.0K    /patch
0       /proc
580K    /root
44M     /run
0       /sbin
4.0K    /srv
0       /sys
72K     /tmp
2.4G    /usr
722M    /var
35G     /www

这个/www怎么这么大,仔细看看du -sh /www/*

816K    /www/backup
33G     /www/server
1.1G    /www/swap
216M    /www/wwwlogs
28K     /www/wwwroot

这个/www/server怎么这么大,仔细看看du -sh /www/server/*

4.0K    /www/server/lib.pl
496M    /www/server/nginx
33G     /www/server/panel
4.0K    /www/server/php
8.0K    /www/server/stop

这个/www/server/panel怎么这么大, 仔细看看du -sh /www/server/panel/*

14M     /www/server/panel/BTPanel
4.0K    /www/server/panel/BT-Panel
4.0K    /www/server/panel/BT-Task
3.2M    /www/server/panel/class
52K     /www/server/panel/config
536K    /www/server/panel/data
4.0K    /www/server/panel/default.pl
12K     /www/server/panel/init.sh
48K     /www/server/panel/install
28K     /www/server/panel/license.txt
33G     /www/server/panel/logs
84K     /www/server/panel/plugin
44K     /www/server/panel/__pycache__
270M    /www/server/panel/pyenv
4.0K    /www/server/panel/requirements.txt
184K    /www/server/panel/rewrite
4.0K    /www/server/panel/runconfig.py
4.0K    /www/server/panel/runserver.py
72K     /www/server/panel/script
12K     /www/server/panel/ssl
24K     /www/server/panel/task.py
4.0K    /www/server/panel/tmp
28K     /www/server/panel/tools.py
168K    /www/server/panel/vhost

这个/www/server/panel/logs怎么这么大,仔细看看du -sh /www/server/panel/logs/*

0       /www/server/panel/logs/access.log
0       /www/server/panel/logs/certbot.log
33G     /www/server/panel/logs/error.log
4.0K    /www/server/panel/logs/panel.pid
1.9M    /www/server/panel/logs/request
33M     /www/server/panel/logs/task.log
4.0K    /www/server/panel/logs/task.pid

这个/www/server/panel/logs/error.log怎么这么大,仔细看看tail -n 100 /www/server/panel/logs/error.log

发现都是一些"Too many open files" 错误,感觉不是很重要,全部删掉!

> /www/server/panel/logs/error.log

再检查df -h

/dev/vda1        40G  5.8G   32G  16% /
devtmpfs        859M     0  859M   0% /dev
tmpfs           869M     0  869M   0% /dev/shm
tmpfs           869M   33M  836M   4% /run
tmpfs           869M     0  869M   0% /sys/fs/cgroup
/dev/vda2        20G  398M   19G   3% /mnt/newdisk
tmpfs           174M     0  174M   0% /run/user/0

舒服!/dev/vda2是中间多加的20G,但是好像挂载方面没有处理好,就先不管了~!

总结

如果出问题了,就先去看看,内存够不够!直接df -h

参考

[1] 自己
[2] ChatGPT

Q.E.D.


立志做一个有趣的碳水化合物