$ free -h
total used free shared buff/cache available
Mem: 3.7G 1.4G 208M 448M 2.1G 1.6G
Swap: 5.9G 1.4G 4.4G
du -t 100M /var
......
3539156 /var/log/journal/963c2c41b08343f7b063dddac6b2e486
3539164 /var/log/journal
3693500 /var/log
6652860 /var
journalctl
查看占用多少空间journalctl --disk-usage
这些日志文件记录了很长时间以来的systemd情况
Archived and active journals take up 3.3G on disk.
wget http://7xir15.com1.z0.glb.clouddn.com/hcache
chmod +x hcache
mv hcache /usr/local/bin/
$ hcache -top 10
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-00000000003a4e8c-000570e896680ef6.journal | 92274688 | 22528 | 14304 | 063.494 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system.journal | 33554432 | 8192 | 8192 | 100.000 |
| /usr/sbin/mysqld | 25451672 | 6214 | 4223 | 067.959 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-00000000001940fb-0005686620d70b71.journal | 125829120 | 30720 | 4112 | 013.385 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-0000000000362fbe-0005701ac8be4787.journal | 92274688 | 22528 | 4111 | 018.248 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-000000000034cfbe-00056fd57730cfc5.journal | 92274688 | 22528 | 4111 | 018.248 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-00000000002c2918-00056dc7738c98f3.journal | 109051904 | 26624 | 4111 | 015.441 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-0000000000337279-00056f9069ba25de.journal | 92274688 | 22528 | 4111 | 018.248 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-0000000000019c5c-00056095dcb1966a.journal | 75497472 | 18432 | 4111 | 022.304 |
| /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-0000000000378f6c-0005705ec5504902.journal | 92274688 | 22528 | 4109 | 018.240 |
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
我看了下,这里面从8.1M~128M日志大小不等..一共用了3.4G. 刚好是内存的2.1G + swap的1.4G
hcache -pid 2903 -bname
我通过以上方式查看了日志,以及实时观察一段时间日志之后发现以下两个错误
supervisord
启动失败7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz systemd[1]: supervisord.service holdoff time over, scheduling restart.
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz systemd[1]: Starting Supervisor daemon...
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz supervisord[3908]: Error: could not find config file /etc/supervisord/supervisord.conf
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz supervisord[3908]: For help, use /usr/bin/supervisord -h
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz systemd[1]: supervisord.service: control process exited, code=exited status=2
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz systemd[1]: Failed to start Supervisor daemon.
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz systemd[1]: Unit supervisord.service entered failed state.
7月 18 17:59:28 izm5e5n9efbv0mqjokcoesz systemd[1]: supervisord.service failed.
他总是提示找不到 /etc/supervisord/supervisord.conf 这个配置,但是我明明是 supervisord -c /etc/supervisor/supervisord.conf 指定配置文件启动的啊,
root 4699 0.0 0.0 112676 1000 pts/8 S+ 18:18 0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn supervis
root 19766 0.0 0.4 223428 16348 ? Ss 6月28 6:43 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
上图中看到,supervisord确实是启动起来了,而且配置文件用的是 /etc/supervisor/supervisord.conf
这时候问了公司运维小哥哥..他说我自己又用 systemctl 在启动 supervisord,然后 systemctl 找的默认配置是 /etc/supervisord/supervisord.conf,所以才会一直报错. 我一想 那应该是我安装 supervisord 的时候自启动了吧..所以才会一直启动
果然是在一直启动失败...
$ systemctl status supervisord
● supervisord.service - Supervisor daemon
Loaded: loaded (/usr/lib/systemd/system/supervisord.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since 三 2018-07-18 18:14:21 CST; 14s ago
Process: 4569 ExecStart=/usr/bin/supervisord -c /etc/supervisord/supervisord.conf (code=exited, status=2)
7月 18 18:14:21 izm5e5n9efbv0mqjokcoesz systemd[1]: supervisord.service: control process exited, code=exited status=2
7月 18 18:14:21 izm5e5n9efbv0mqjokcoesz systemd[1]: Failed to start Supervisor daemon.
7月 18 18:14:21 izm5e5n9efbv0mqjokcoesz systemd[1]: Unit supervisord.service entered failed state.
7月 18 18:14:21 izm5e5n9efbv0mqjokcoesz systemd[1]: supervisord.service failed.
systemctl stop supervisord
systemctl
管理服务7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz systemd[1]: Started Session 126297 of user root.
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz CROND[3915]: (root) CMD (/usr/lib64/sa/sa1 1 1)
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz systemd[1]: Starting Session 126297 of user root.
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz systemd[1]: Started Session 126299 of user root.
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz systemd[1]: Starting Session 126299 of user root.
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz systemd[1]: Started Session 126298 of user root.
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz systemd[1]: Starting Session 126298 of user root.
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz CROND[3917]: (root) CMD (/usr/bin/php /mnt/luoyang/lequ-back/artisan lq:publish_story_list >> /dev/null 2>&1)
7月 18 18:00:02 izm5e5n9efbv0mqjokcoesz CROND[3919]: (root) CMD (/usr/bin/php /mnt/luoyang/lequ-back/artisan lq:robot_star_for_song_comment >> /dev/null 2>&1)
......
*/1 * * * * /usr/bin/php /mnt/luoyang/lequ-back/artisan lq:publish_story_list >> /dev/null 2>&1
*/1 * * * * /usr/bin/php /mnt/luoyang/lequ-back/artisan lq:robot_star_for_song_comment >> /dev/null 2>&1
因为 /dev/null 2>&1 会完整输出信息 所以这个日志是正常的
journald
日志过大的问题,清理journald
日志为什么日志那么大,我们都知道了..怎么优化.让日志不再那么大
如果使用 –vacuum-size 选项,则可硬性指定日志的总体体积,意味着其会不断删除旧有记录直到所占容量符合要
journalctl --vacuum-size=100M
# 保留一年之后的日志
journalctl --vacuum-time=1years
# 保留2天之后的日志
journalctl --vacuum-time=2d
# 这个基本是清除所有日志了
sudo journalctl --vacuum-time=1seconds
Deleted archived journal /var/log/journal/963c2c41b08343f7b063dddac6b2e486/user-1001@9d4943ca800341b9ae4fefe796f6b506-00000000003a07e8-000570dab51e6f5f.journal (8.0M).
Deleted archived journal /var/log/journal/963c2c41b08343f7b063dddac6b2e486/system@786cc8e2be604dc3816ebd7d224b2411-00000000003a4e8c-000570e896680ef6.journal (88.0M).
/etc/systemd/journald.conf
# 这个配置可以保持你的日志在600M
SystemMaxUse=600M