日常检查服务的时候，从portainer那里进去看容器日志的时候，发现右上角出现红色的感叹号：unable to retrieve container logs。
因为之前没出现过这样的问题，所以就先上服务器上用命令docker logs -f containerid看日志，发现日志也是动不了，还是停留在某个时间的日志记录上。
想了一下不应该是服务的日志打印出问题，先照着google搜索了一遍，发现都没有跟我的问题相匹配的。因为日志有时能收集显示，有些日志不可以，应该是跟docker设置的日志引擎有问题。
本来是想整一套efk的，但是感觉现在日志量还不够大，所以并没有修改docker的日志引擎，还是默认的journald
[root@ad-official xiaoxiao]# docker info|grep logging warning: you're not using the default sep profile logging driver: journald
journald的官方文档上有这么一个说明：
man journald.conf ... ratelimitinterval=, ratelimitburst= configures the rate limiting that is applied to all messages generated on the system. if, in the time interval defined by ratelimitinterval=, more messages than specified in ratelimitburst= are logged by a service, all further messages within the interval are dropped until the interval is over. a message about the number of dropped messages is generated. this rate limiting is applied per-service, so that two services which log do not interfere with each other's limits. defaults to 1000 messages in 30s. the time specification for ratelimitinterval= may be specified in the following units: "s", "min", "h", "ms", "us". to turn off any kind of rate limiting, set either value to 0. ...
这里写了默认30秒内只能接收1000条日志，看到这里就能明白了，因为前阵子刚在docker发布了一个单日日志文件大小差不多达到3g的服务，导致到了其他服务的日志也受到了影响，大量的日志被journald丢弃，所以我们修改一下配置就没有问题了。
打开/etc/systemd/journald.conf文件，将ratelimitburst从默认的1000修改成5000，根据自己目前的日志输出量进行调整：
[root@ad-official log]# cat /etc/systemd/journald.conf # this file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the gnu lesser general public license as published by # the free software foundation; either version 2.1 of the license, or # (at your option) any later version. # # entries in this file show the pile time defaults. # you can change settings by editing this file. # defaults can be restored by simply deleting this file. # # see journald.conf(5) for details. [journal] #storage=auto press=yes #seal=yes #splitmode=uid #syncintervalsec=5m #ratelimitinterval=30s ratelimitburst=5000 #systemmaxuse= #systemkeepfree= #systemmaxfilesize= #runtimemaxuse= #runtimekeepfree= #runtimemaxfilesize= #maxretentionsec= #maxfilesec=1month forwardtosyslog=no #forwardtokmsg=no #forwardtoconsole=no forwardtowall=no #ttypath=/dev/console #maxlevelstore=debug #maxlevelsyslog=debug #maxlevelkmsg=notice #maxlevelconsole=info #maxlevelwall=emerg #linemax=48k
顺便将forwardtosyslog和forwardtowall设置成no，因为默认是yes，会导致我们清理了journal的日志文件，而syslog中的没有清除掉，慢慢的就会将磁盘占满。
然后重启一下journald就可以恢复正常使用啦：systemctl restart systemd-journald.service
