ELK Stack on the Cheap

Preface

I began writing this guide early in 2017 to develop an ElasticSearch, Logstash, and Kibana (ELK) stack in the cloud on the cheap (less than $15 per month). The idea was to use my Google Cloud free trial to create as small of instances as I could to run the stack to analyze a few days worth of Nginx logs. I ended up creating three f1-micro instances to run the processes. The only issue I ran into happened during installation of the node modules needed for XPath for the Kibana instance. The installation would run out of memory and required that I scale up the instance to one with at least 3GB of RAM during installation and scale down once it was complete.

Before I could finish this experiment, life/work got in the way and my free trial ended. Instead of completely scrapping this project and letting all of the knowledge go to waste, I decided to publish it, unparsed and unedited, in the hopes that it will help others. Thus, I would advice treating this post like a Gist and for informational purposes only.

In the field of software development, some projects fall behind and are inevitably abandoned. Instead of dwelling on what could have been, grasp the knowledge acquired and apply it to future endeavors.

ELK Stack on the Cheap

Add the Elastic PPA to each Ubuntu instance:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Install and Configure Elasticsearch

sudo apt install elasticsearch openjdk-8-jre

sudo vim /etc/elasticsearch/elasticsearch.yml
# cluster.name, node.name, network.host
sudo vim /etc/elasticsearch/jvm.options
# -Xms128m -Xmx128m
sudo ./usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
sudo systemctl start elasticsearch.service
sudo systemctl status elasticsearch.service
curl localhost:9200 # TODO: Need to authenticate with x-pack

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html

Encrypt cluster traffic

sudo ./usr/share/elasticsearch/bin/x-pack/syskeygen
# /etc/elasticsearch/x-pack/system_key
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/x-pack/system_key

Encrypt Kibana traffic

sudo /usr/share/elasticsearch/bin/x-pack/certgen
# /etc/elasticsearch/x-pack/certificate-bundle.zip
sudo cp ca/ca.* /etc/elasticsearch/x-pack/

https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html

Install and Configure Kibana

sudo apt install openjdk-8-jre kibana logstash
sudo vim /etc/kibana/kibana.yml
# Edit server.host, server.name, elasticsearch.url
# elasticsearch.ssl.cert: /etc/kibana/ca.crt
# elasticsearch.ssl.key: /etc/kibana/ca.key

https://www.elastic.co/guide/en/kibana/4.0/_setting_kibana_server_properties.html

sudo vim /etc/default/kibana

X-Pack

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

sudo ./usr/share/kibana/bin/kibana-plugin install x-pack
# Memory needed to complete: 1.38g
sudo systemctl start kibana.service
sudo systemctl status kibana.service
journalctl # Make sure kibana is able to connect to elasticsearch. Constant restarting.

Logon

elastic/changeme
kibana/changeme
https://www.elastic.co/guide/en/x-pack/current/kibana.html

sudo vim /etc/kibana/kibana.yml
sudo systemctl restart kibana.service

Logstash

/etc/logstash/logstash.yml
# queue.page_capacity, pipeline.batch.size, pipeline.batch.delay
sudo vim /etc/logstash/jvm.options
# -Xms64m
# -Xmx128m

SSL Certificates

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04#generate-ssl-certificates

/etc/ssl/openssl.cnf
# subjectAltName = IP: logstash_ip
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt

Copy /etc/pki/tls/certs/logstash-forwarder.crt to filebeat server.

https://www.elastic.co/guide/en/x-pack/current/logstash.html

Nginx

https://app.logz.io/#/apps?q=nginx
http://logz.io/blog/nginx-log-analysis/
https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

More

http://www.ragingcomputer.com/2014/02/securing-elasticsearch-kibana-with-nginx