I began writing this guide early in 2017 to develop an ElasticSearch, Logstash, and Kibana (ELK) stack in the cloud on the cheap (less than $15 per month). The idea was to use my Google Cloud free trial to create as small of instances as I could to run the stack to analyze a few days worth of Nginx logs. I ended up creating three f1-micro instances to run the processes. The only issue I ran into happened during installation of the node modules needed for XPath for the Kibana instance. The installation would run out of memory and required that I scale up the instance to one with at least 3GB of RAM during installation and scale down once it was complete.
Before I could finish this experiment, life/work got in the way and my free trial ended. Instead of completely scrapping this project and letting all of the knowledge go to waste, I decided to publish it, unparsed and unedited, in the hopes that it will help others. Thus, I would advice treating this post like a Gist and for informational purposes only.
In the field of software development, some projects fall behind and are inevitably abandoned. Instead of dwelling on what could have been, grasp the knowledge acquired and apply it to future endeavors.
ELK Stack on the Cheap
Add the Elastic PPA to each Ubuntu instance:
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
Install and Configure Elasticsearch
sudo apt install elasticsearch openjdk-8-jre sudo vim /etc/elasticsearch/elasticsearch.yml # cluster.name, node.name, network.host
sudo vim /etc/elasticsearch/jvm.options # -Xms128m -Xmx128m
sudo ./usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
sudo systemctl start elasticsearch.service sudo systemctl status elasticsearch.service
curl localhost:9200 # TODO: Need to authenticate with x-pack
Encrypt cluster traffic
sudo ./usr/share/elasticsearch/bin/x-pack/syskeygen # /etc/elasticsearch/x-pack/system_key sudo chown elasticsearch:elasticsearch /etc/elasticsearch/x-pack/system_key
Encrypt Kibana traffic
sudo /usr/share/elasticsearch/bin/x-pack/certgen # /etc/elasticsearch/x-pack/certificate-bundle.zip
sudo cp ca/ca.* /etc/elasticsearch/x-pack/
Install and Configure Kibana
sudo apt install openjdk-8-jre kibana logstash
sudo vim /etc/kibana/kibana.yml # Edit server.host, server.name, elasticsearch.url # elasticsearch.ssl.cert: /etc/kibana/ca.crt # elasticsearch.ssl.key: /etc/kibana/ca.key
sudo vim /etc/default/kibana
sudo ./usr/share/kibana/bin/kibana-plugin install x-pack # Memory needed to complete: 1.38g
sudo systemctl start kibana.service sudo systemctl status kibana.service journalctl # Make sure kibana is able to connect to elasticsearch. Constant restarting.
sudo vim /etc/kibana/kibana.yml sudo systemctl restart kibana.service
/etc/logstash/logstash.yml # queue.page_capacity, pipeline.batch.size, pipeline.batch.delay
sudo vim /etc/logstash/jvm.options # -Xms64m # -Xmx128m
/etc/ssl/openssl.cnf # subjectAltName = IP: logstash_ip
sudo mkdir -p /etc/pki/tls/certs sudo mkdir /etc/pki/tls/private
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt
Copy /etc/pki/tls/certs/logstash-forwarder.crt to filebeat server.