ELK Stack on the Cheap
Preface
I began writing this guide early in 2017 to develop an ElasticSearch, Logstash, and Kibana (ELK) stack in the cloud on the cheap (less than $15 per month). The idea was to use my Google Cloud free trial to create as small of instances as I could to run the stack to analyze a few days worth of Nginx logs. I ended up creating three f1-micro instances to run the processes. The only issue I ran into happened during installation of the node modules needed for XPath for the Kibana instance. The installation would run out of memory and required that I scale up the instance to one with at least 3GB of RAM during installation and scale down once it was complete.
Before I could finish this experiment, life/work got in the way and my free trial ended. Instead of completely scrapping this project and letting all of the knowledge go to waste, I decided to publish it, unparsed and unedited, in the hopes that it will help others. Thus, I would advice treating this post like a Gist and for informational purposes only.
In the field of software development, some projects fall behind and are inevitably abandoned. Instead of dwelling on what could have been, grasp the knowledge acquired and apply it to future endeavors.
ELK Stack on the Cheap
Add the Elastic PPA to each Ubuntu instance:
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
Install and Configure Elasticsearch
sudo apt install elasticsearch openjdk-8-jre
sudo vim /etc/elasticsearch/elasticsearch.yml
# cluster.name, node.name, network.host
sudo vim /etc/elasticsearch/jvm.options
# -Xms128m -Xmx128m
sudo ./usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
sudo systemctl start elasticsearch.service
sudo systemctl status elasticsearch.service
curl localhost:9200 # TODO: Need to authenticate with x-pack
https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
Encrypt cluster traffic
sudo ./usr/share/elasticsearch/bin/x-pack/syskeygen
# /etc/elasticsearch/x-pack/system_key
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/x-pack/system_key
Encrypt Kibana traffic
sudo /usr/share/elasticsearch/bin/x-pack/certgen
# /etc/elasticsearch/x-pack/certificate-bundle.zip
sudo cp ca/ca.* /etc/elasticsearch/x-pack/
https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html
Install and Configure Kibana
sudo apt install openjdk-8-jre kibana logstash
sudo vim /etc/kibana/kibana.yml
# Edit server.host, server.name, elasticsearch.url
# elasticsearch.ssl.cert: /etc/kibana/ca.crt
# elasticsearch.ssl.key: /etc/kibana/ca.key
https://www.elastic.co/guide/en/kibana/4.0/_setting_kibana_server_properties.html
sudo vim /etc/default/kibana
X-Pack
https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
sudo ./usr/share/kibana/bin/kibana-plugin install x-pack
# Memory needed to complete: 1.38g
sudo systemctl start kibana.service
sudo systemctl status kibana.service
journalctl # Make sure kibana is able to connect to elasticsearch. Constant restarting.
Logon
elastic/changeme
kibana/changeme
https://www.elastic.co/guide/en/x-pack/current/kibana.html
sudo vim /etc/kibana/kibana.yml
sudo systemctl restart kibana.service
Logstash
/etc/logstash/logstash.yml
# queue.page_capacity, pipeline.batch.size, pipeline.batch.delay
sudo vim /etc/logstash/jvm.options
# -Xms64m
# -Xmx128m
SSL Certificates
/etc/ssl/openssl.cnf
# subjectAltName = IP: logstash_ip
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt
Copy /etc/pki/tls/certs/logstash-forwarder.crt to filebeat server.
https://www.elastic.co/guide/en/x-pack/current/logstash.html
Nginx
https://app.logz.io/#/apps?q=nginx
http://logz.io/blog/nginx-log-analysis/
https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04
More
http://www.ragingcomputer.com/2014/02/securing-elasticsearch-kibana-with-nginx