Preface

I began writing this guide early in 2017 to develop an ElasticSearch, Logstash, and Kibana (ELK) stack in the cloud on the cheap (less than $15 per month). The idea was to use my Google Cloud free trial to create as small of instances as I could to run the stack to analyze a few days worth of Nginx logs. I ended up creating three f1-micro instances to run the processes. The only issue I ran into happened during installation of the node modules needed for XPath for the Kibana instance. The installation would run out of memory and required that I scale up the instance to one with at least 3GB of RAM during installation and scale down once it was complete.

Before I could finish this experiment, life/work got in the way and my free trial ended. Instead of completely scrapping this project and letting all of the knowledge go to waste, I decided to publish it, unparsed and unedited, in the hopes that it will help others. Thus, I would advice treating this post like a Gist and for informational purposes only.

In the field of software development, some projects fall behind and are inevitably abandoned. Instead of dwelling on what could have been, grasp the knowledge acquired and apply it to future endeavors.

ELK Stack on the Cheap

Add the Elastic PPA to each Ubuntu instance:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Install and Configure Elasticsearch

sudo apt install elasticsearch openjdk-8-jre

sudo vim /etc/elasticsearch/elasticsearch.yml
# cluster.name, node.name, network.host

sudo vim /etc/elasticsearch/jvm.options
# -Xms128m -Xmx128m

sudo ./usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack

sudo systemctl start elasticsearch.service
sudo systemctl status elasticsearch.service

curl localhost:9200 # TODO: Need to authenticate with x-pack

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html

Encrypt cluster traffic

sudo ./usr/share/elasticsearch/bin/x-pack/syskeygen
# /etc/elasticsearch/x-pack/system_key
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/x-pack/system_key

Encrypt Kibana traffic

sudo /usr/share/elasticsearch/bin/x-pack/certgen
# /etc/elasticsearch/x-pack/certificate-bundle.zip

sudo cp ca/ca.* /etc/elasticsearch/x-pack/

https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html

Install and Configure Kibana

sudo apt install openjdk-8-jre kibana logstash

sudo vim /etc/kibana/kibana.yml
# Edit server.host, server.name, elasticsearch.url
# elasticsearch.ssl.cert: /etc/kibana/ca.crt
# elasticsearch.ssl.key: /etc/kibana/ca.key

https://www.elastic.co/guide/en/kibana/4.0/_setting_kibana_server_properties.html

sudo vim /etc/default/kibana

X-Pack

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

sudo ./usr/share/kibana/bin/kibana-plugin install x-pack
# Memory needed to complete: 1.38g

sudo systemctl start kibana.service
sudo systemctl status kibana.service
journalctl # Make sure kibana is able to connect to elasticsearch. Constant restarting.

Logon

elastic/changeme
kibana/changeme
https://www.elastic.co/guide/en/x-pack/current/kibana.html

sudo vim /etc/kibana/kibana.yml
sudo systemctl restart kibana.service

Logstash

/etc/logstash/logstash.yml
# queue.page_capacity, pipeline.batch.size, pipeline.batch.delay

sudo vim /etc/logstash/jvm.options
# -Xms64m
# -Xmx128m

SSL Certificates

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04#generate-ssl-certificates

/etc/ssl/openssl.cnf
# subjectAltName = IP: logstash_ip

sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private

sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt

Copy /etc/pki/tls/certs/logstash-forwarder.crt to filebeat server.

https://www.elastic.co/guide/en/x-pack/current/logstash.html