Truncate and rotate docker container logs

Problem

Each time I would run docker-compose logs -f it would take a long time to get to the bottom of the log file.

Solution

Edit the daemon.json file and configure the max-size and max-file parameters.

root@whiscardz:~# vim /etc/docker/daemon.json
{
  "log-driver": "json-file",  
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

Run the following to reload the updated daemon.json file:

$ systemctl daemon-reload

$ systemctl restart docker

Then recreate the docker containers:

root@whiscardz:/opt/docker-elk# docker-compose up -d
Recreating elasticsearch ... done
Recreating logstash      ... done
Recreating kibana        ... done

References

Docker container log rotation

Configure the default logging driver

Options

Json Options

How to Clear Log Files of A Docker Container

logstash java.lang.OutOfMemoryError: Java heap space

Problem

ELK visualizations were not loading. Looked at the logs and saw the following error:

logstash java.lang.OutOfMemoryError: Java heap space

Solution

Added the JAVA OPTS setting to the docker-compose.yml file:

    environment:
      - "LS_JAVA_OPTS=-Xms5120m -Xmx5120m"
      - "LS_HEAP_SIZE=5g"

Optionally stop the ELK containers and then run docker-compose up -d so that the containers are recreated with the new parameters:

root@whiscardz:/opt/docker-elk# docker-compose stop
Stopping logstash      ... done
Stopping kibana        ... done
Stopping elasticsearch ... done
root@whiscardz:/opt/docker-elk# docker-compose up -d
Recreating elasticsearch ... done
Recreating logstash      ... done
Recreating kibana        ... done
root@whiscardz:/opt/docker-elk#

How to check if heap was updated:

Exec into the logstash container and then run either of the following commands:

root@whiscardz:/opt/docker-elk# docker exec -it mylogstashcontainerid /bin/bash
bash-4.2$ ps -ef | grep logstash
logstash       1       0 18 09:41 ?        00:58:45 /bin/java -Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ -Xms5120m -Xmx5120m

OR

root@whiscardz:/opt/docker-elk# docker exec -it mylogstashcontainerid /bin/bash
bash-4.2$ jps -lvm
1 org.logstash.Logstash -Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ -Xms5120m -Xmx5120m
196 jdk.jcmd/sun.tools.jps.Jps -lvm -Dapplication.home=/usr/lib/jvm/java-11-openjdk-11.0.7.10-4.el7_8.x86_64 -Xms8m -Djdk.module.main=jdk.jcmd
bash-4.2$

References

How to change the JVM parameter for a Docker container

Setting JVM options for an Elasticsearch service run in a Docker container

Manually set the heap size

10 Important JVM Options for Production JAVA Application System

Getting the parameters of a running JVM

Java.lang.OutOfMemoryError: Java heap space on Elasticsearch in AWS