Docker container with a heavy service dies after restart, how to find a reason?

Posted on by

Dockerfile has a configured CMD command for start (without ENTRYPOINT):

ARG wildfly_version

FROM jboss/wildfly:$wildfly_version

...

CMD ["/bin/bash", "/opt/jboss/init.sh"]

init.sh script starts WildFly, checks if tmp folder exists, if so deploys lots of applications, then removes tmp folder and runs command to make container runnable:

/opt/jboss/wildfly/bin/standalone.sh 
    -c standalone-full.xml 
    -b 0.0.0.0 & sleep 30 && 

# this folder does not exist, when container is restarted
[ -d "/opt/jboss/tmp" ] && /opt/jboss/deploy.sh && 

# remove /tmp to also mark a deployment has been finished successfully
rm -rf /opt/jboss/tmp && 

# keep the container running in detached mode, run in the foreground:
tail -f /dev/null

A workaround with a tmp folder, allows to avoid step-by-step deploying of previously deployed packages after a container restart.

After the first start, container works stable, and shows the following MEM and CPU usage:

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
8f6269f5b650        bm_wf               9.53%               5.345GiB / 15.52GiB   34.44%              11.5MB / 6.34MB     1.11GB / 1.51GB     1249

But after a container restart, I can see, how memory is increased from about 100Mb till 4-5Gb and then dies. Via docker logs command, I can see only Wildfly logs, and each restart try shows only messages how applications are deployed. Each time different messages. No system errors, only info from Wildfly. Based on it I made a conclusion, that that a reason of the issue is not with Wilfly itself.

docker events shows:

020-09-08T11:41:27.243803353+02:00 network connect 7467b7c9524a0f0abfee9d2aaddb65c39fde9faddaa205e9662d9bff533dacda (container=8f6269f5b650f3667bbacad4baaa11942e87bb97d8c85117ddefa8b0d1b1c17f, name=bridge, type=bridge)
2020-09-08T11:41:27.327371724+02:00 network connect 4484c2b8bfcc37e1a278aae3fae86a32597eb99f9ef1f29f85568ac1a3ab1df6 (container=8f6269f5b650f3667bbacad4baaa11942e87bb97d8c85117ddefa8b0d1b1c17f, name=bm_network, type=bridge)
2020-09-08T11:41:27.836003355+02:00 container start 8f6269f5b650f3667bbacad4baaa11942e87bb97d8c85117ddefa8b0d1b1c17f (image=bm_wf:10.1.0.Final, name=bm_wf, org.label-schema.build-date=20181205, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS)
2020-09-08T11:41:58.064390117+02:00 container die 8f6269f5b650f3667bbacad4baaa11942e87bb97d8c85117ddefa8b0d1b1c17f (exitCode=1, image=bm_wf:10.1.0.Final, name=bm_wf, org.label-schema.build-date=20181205, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS)
2020-09-08T11:41:58.622899034+02:00 network disconnect 7467b7c9524a0f0abfee9d2aaddb65c39fde9faddaa205e9662d9bff533dacda (container=8f6269f5b650f3667bbacad4baaa11942e87bb97d8c85117ddefa8b0d1b1c17f, name=bridge, type=bridge)
2020-09-08T11:41:58.622970586+02:00 network disconnect 4484c2b8bfcc37e1a278aae3fae86a32597eb99f9ef1f29f85568ac1a3ab1df6 (container=8f6269f5b650f3667bbacad4baaa11942e87bb97d8c85117ddefa8b0d1b1c17f, name=bm_network, type=bridge)

What else can I investigate to be able to restart a container and use it?

Source: Docker Questions