I began writing this guide early in 2017 to develop an ElasticSearch, Logstash, and Kibana (ELK) stack in the cloud on the cheap (less than $15 per month). The idea was to use my Google Cloud free trial to create as small of instances as I could to run the stack to analyze a few days worth of Nginx logs. I ended up creating three f1-micro instances to run the processes. The only issue I ran into happened during installation of the node modules needed for XPath for the Kibana instance. The installation would run out of memory and required that I scale up the instance to one with at least 3GB of RAM during installation and scale down once it was complete.

Before I could finish this experiment, life/work got in the way and my free trial ended. Instead of completely scrapping this project and letting all of the knowledge go to waste, I decided to publish it, unparsed and unedited, in the hopes that it will help others. Thus, I would advice treating this post like a Gist and for informational purposes only.

In the field of software development, some projects fall behind and are inevitably abandoned. Instead of dwelling on what could have been, grasp the knowledge acquired and apply it to future endeavors.

ELK Stack on the Cheap

Add the Elastic PPA to each Ubuntu instance:

wget -qO - | sudo apt-key add -
echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Install and Configure Elasticsearch

sudo apt install elasticsearch openjdk-8-jre

sudo vim /etc/elasticsearch/elasticsearch.yml
sudo vim /etc/elasticsearch/jvm.options
# -Xms128m -Xmx128m
sudo ./usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
sudo systemctl start elasticsearch.service
sudo systemctl status elasticsearch.service
curl localhost:9200 # TODO: Need to authenticate with x-pack

Encrypt cluster traffic

sudo ./usr/share/elasticsearch/bin/x-pack/syskeygen
# /etc/elasticsearch/x-pack/system_key
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/x-pack/system_key

Encrypt Kibana traffic

sudo /usr/share/elasticsearch/bin/x-pack/certgen
# /etc/elasticsearch/x-pack/
sudo cp ca/ca.* /etc/elasticsearch/x-pack/

Install and Configure Kibana

sudo apt install openjdk-8-jre kibana logstash
sudo vim /etc/kibana/kibana.yml
# Edit,, elasticsearch.url
# elasticsearch.ssl.cert: /etc/kibana/ca.crt
# elasticsearch.ssl.key: /etc/kibana/ca.key

sudo vim /etc/default/kibana


sudo ./usr/share/kibana/bin/kibana-plugin install x-pack
# Memory needed to complete: 1.38g
sudo systemctl start kibana.service
sudo systemctl status kibana.service
journalctl # Make sure kibana is able to connect to elasticsearch. Constant restarting.



sudo vim /etc/kibana/kibana.yml
sudo systemctl restart kibana.service


# queue.page_capacity, pipeline.batch.size, pipeline.batch.delay
sudo vim /etc/logstash/jvm.options
# -Xms64m
# -Xmx128m

SSL Certificates

# subjectAltName = IP: logstash_ip
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt

Copy /etc/pki/tls/certs/logstash-forwarder.crt to filebeat server.