ELK consumes too much resources on the docker version

[silverdaz]

....crashes after a while.

[silverdaz]

Here is a possible explanation of the issue, but Dmytro pointed out in the page that:

Elasticsearch alone needs at least 2GB of RAM to run.

With Docker for Mac, the amount of RAM dedicated to Docker can be set using the UI: see How to increase docker-machine memory Mac (Stack Overflow).```

That's not a good sign...

[silverdaz]

Link from Stefan: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode

[blankdots]

A brief look into this, as I managed to run it by using the link above for docker-cli-run (discovered that by looking into docker logs for elastic).
The key differences are the amount of memory and the OS (using RHEL 7), the amount of memory showed as limit is 7.5GB, and also there seems to be a spike at first, as elk goes up and after a minute or so it is down to ~11%:

An idea is that there are some prerequisites as pointed here: https://elk-docker.readthedocs.io/#prerequisites As elastic alone requires 2GB RAM - and it is safe to think that 2 elastic instances = 4GB minimum

Also tried running a vanilla docker-elk: https://github.com/deviantony/docker-elk and revealed a similar pattern there is a spike (200-300+%) after that it drops down (to 10-20%) - as illustrated by gif:

Increasing memory should help (MacOS needs to be confirmed), and @dtitov pointed https://docs.travis-ci.com/user/reference/overview/ in order to try how this on travis.

There is a https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html guide, not sure it is the scope of this issue.

[blankdots]

Follow up: by looking over the logs of docker-compose up noticed that elasticsearch failed to start with message:

ega-elasticsearch-swe1 | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ega-elasticsearch-fin1 | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

This confirms #252 (comment)
Thus on linux setting: sudo sysctl -w vm.max_map_count=262144 should do the trick (even on travis, adding it in the before_install), and on macOS same solution as pointed in #252 (comment)

Should we to look more into this?

[dtitov]

Just fix it, if you know how to 😺

[silverdaz]

Solved by PR#254