前言
最近博客莫名其妙 502 gateway | Nginx,不知道什么情况,于是乎上后台查看,发现是因为40G的磁盘被狠狠地塞满,很好奇为什么会酱紫,于是开始排查。
正文
莫名其妙的发现博客开始很卡,然后重启了一次服务器之后,502 gateway | Nginx了,遂开始排查。
先排查halo本身是什么问题systemctl status halo
:
halo.service - Halo Service
Loaded: loaded (/etc/systemd/system/halo.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2024-10-09 21:00:56 CST; 825ms ago
Docs: https://docs.halo.run
Process: 21680 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=1/FAILURE)
Main PID: 21682 (java)
CGroup: /system.slice/halo.service
└─21682 /usr/bin/java -server -Xms256m -Xmx256m -jar /home/halo/app/halo.jar
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez systemd[1]: Started Halo Service.
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez systemd[1]: Starting Halo Service...
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez java[21682]: OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez java[21682]: 21682
Oct 09 21:00:56 izt4nb5f64bwzkgfhzkm2ez java[21682]: Try using the -Djava.io.tmpdir= option to select an alternate temp location.
发现其中有一条Insufficient space for shared memory file
,于是df -h
:
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 39G 0 100% /
devtmpfs 456M 0 456M 0% /dev
tmpfs 466M 0 466M 0% /dev/shm
tmpfs 466M 42M 425M 9% /run
tmpfs 466M 0 466M 0% /sys/fs/cgroup
tmpfs 94M 0 94M 0% /run/user/0
发现怎么莫名其妙的40G被塞满了,检查了一下halo的文件大小,du -sh /home/halo/*
,发现才370M,为什么370M能塞满40G的盘!
370M /home/halo/.halo/
奇了个怪,为什么370M能塞满40G的盘du -sh /* 2>/dev/null
,看看是什么塞满了:
0 /bin
140M /boot
0 /dev
37M /etc
447M /home
0 /lib
0 /lib64
16K /lost+found
4.0K /media
40K /mnt
4.0K /opt
8.0K /patch
0 /proc
580K /root
44M /run
0 /sbin
4.0K /srv
0 /sys
72K /tmp
2.4G /usr
722M /var
35G /www
这个/www
怎么这么大,仔细看看du -sh /www/*
:
816K /www/backup
33G /www/server
1.1G /www/swap
216M /www/wwwlogs
28K /www/wwwroot
这个/www/server
怎么这么大,仔细看看du -sh /www/server/*
:
4.0K /www/server/lib.pl
496M /www/server/nginx
33G /www/server/panel
4.0K /www/server/php
8.0K /www/server/stop
这个/www/server/panel
怎么这么大, 仔细看看du -sh /www/server/panel/*
:
14M /www/server/panel/BTPanel
4.0K /www/server/panel/BT-Panel
4.0K /www/server/panel/BT-Task
3.2M /www/server/panel/class
52K /www/server/panel/config
536K /www/server/panel/data
4.0K /www/server/panel/default.pl
12K /www/server/panel/init.sh
48K /www/server/panel/install
28K /www/server/panel/license.txt
33G /www/server/panel/logs
84K /www/server/panel/plugin
44K /www/server/panel/__pycache__
270M /www/server/panel/pyenv
4.0K /www/server/panel/requirements.txt
184K /www/server/panel/rewrite
4.0K /www/server/panel/runconfig.py
4.0K /www/server/panel/runserver.py
72K /www/server/panel/script
12K /www/server/panel/ssl
24K /www/server/panel/task.py
4.0K /www/server/panel/tmp
28K /www/server/panel/tools.py
168K /www/server/panel/vhost
这个/www/server/panel/logs
怎么这么大,仔细看看du -sh /www/server/panel/logs/*
:
0 /www/server/panel/logs/access.log
0 /www/server/panel/logs/certbot.log
33G /www/server/panel/logs/error.log
4.0K /www/server/panel/logs/panel.pid
1.9M /www/server/panel/logs/request
33M /www/server/panel/logs/task.log
4.0K /www/server/panel/logs/task.pid
这个/www/server/panel/logs/error.log
怎么这么大,仔细看看tail -n 100 /www/server/panel/logs/error.log
:
发现都是一些"Too many open files" 错误,感觉不是很重要,全部删掉!
> /www/server/panel/logs/error.log
再检查df -h
:
/dev/vda1 40G 5.8G 32G 16% /
devtmpfs 859M 0 859M 0% /dev
tmpfs 869M 0 869M 0% /dev/shm
tmpfs 869M 33M 836M 4% /run
tmpfs 869M 0 869M 0% /sys/fs/cgroup
/dev/vda2 20G 398M 19G 3% /mnt/newdisk
tmpfs 174M 0 174M 0% /run/user/0
舒服!/dev/vda2
是中间多加的20G,但是好像挂载方面没有处理好,就先不管了~!
总结
如果出问题了,就先去看看,内存够不够!直接df -h
。
参考
[1] 自己
[2] ChatGPT
Q.E.D.