Install Elasticsearch with Kibana and Logstash on Ubuntu

Elasticsearch is an open-source, distributed search and analytics engine that is designed to handle large amounts of data. It is built on top of the Apache Lucene search engine library and is part of the Elastic Stack (formerly known as the ELK Stack), which includes Kibana and Logstash.

It comes with several new features and improvements over the previous versions, including:

1. Frozen Indices: This feature allows users to store less frequently accessed data in a more cost-effective manner while still being able to search and analyze it.
2. Cross-Cluster Replication: This feature enables users to replicate data across multiple clusters, which is useful for disaster recovery, data migration, and scaling.
3. Improved Security: includes several improvements to its security features, including support for encrypted communication between nodes, fine-grained role-based access control, and improved auditing capabilities.
4. SQL Support: now includes support for SQL, which allows users to query and analyze data using SQL syntax.

Prerequisites

1. OpenJDK 11 must be installed
2. Nginx must be installed on your server, click here for the guide.

Setting up Elasticsearch: Installation and Configuration

Elasticsearch components are not included in the default package repositories of Ubuntu. Nevertheless, you can install them via APT by adding Elastic’s package source list. To prevent package spoofing and ensure system security, all packages are signed with the signing key. Your package manager will consider the key-authenticated packages as trustworthy. To proceed with the installation, you need to import the public GPG key and add the Elastic package source list in this step.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch |sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg

echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update your APT packages index with the new Elastic source:

apt-get update

Use the following command to install:

sudo apt install elasticsearch

Elasticsearch has been installed and is now ready for configuration. You may utilize your preferred text editor to modify the main configuration file, which is the elasticsearch.yml file. In this case, we will use nano:

nano /etc/elasticsearch/elasticsearch.yml

Locate the line in the configuration file that specifies “network.host”, remove the comment symbol, and substitute its value with “localhost” as demonstrated below:

Start the Elasticsearch service:

sudo systemctl start elasticsearch

sudo systemctl enable elasticsearch

curl -X GET "localhost:9200"

Setting up Kibana Dashboard

Kibana is an open-source data visualization and exploration tool designed to work with Elasticsearch. It provides users with a user-friendly web interface for analyzing and visualizing data stored in Elasticsearch indices. Kibana offers a wide range of features such as data filtering, aggregation, and visualization tools that enable users to interactively explore and understand their data. It is commonly used in conjunction with Elasticsearch for log analysis, business intelligence, and other analytical use cases.

sudo apt install kibana
sudo systemctl enable kibana
sudo systemctl start kibana

In order to enable external access to Kibana, which is currently set up to only listen on localhost, a reverse proxy needs to be established. For this purpose, Nginx will be used, assuming it has already been installed on the server.

The first step involves creating an administrative Kibana user using the openssl command. This account will be used to access the Kibana web interface. For example, we will name this account “virtonoadmin”. However, to ensure greater security, it is recommended to choose a non-standard username that would be difficult to guess.

Executing the following command will create the administrative Kibana user and password, and store them in the “htpasswd.users” file. Nginx will be configured to require this username and password and read this file in the next steps:

echo "virtonoadmin:openssl passwd -apr1" | sudo tee -a /etc/nginx/htpasswd.users

nano /etc/nginx/sites-available/domain

And paste the following:

server {
    listen 80;

    server_name your_domain; MAKE SURE TO REPLACE WITH YOUR DOMAIN

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

sudo ln -s /etc/nginx/sites-available/domain /etc/nginx/sites-enabled/domain
sudo nginx -t
sudo systemctl reload nginx

To access the Kibana server’s status page, navigate to http://domain/status

Setting up Logstash

Use the following command to install Logstash:

apt install logstash

Generate a configuration file named “02-beats-input.conf” in which you will configure your Filebeat input:

nano /etc/logstash/conf.d/02-beats-input.conf

Add the following lines to the conf file:

input {
  beats {
    port => 5044
  }
}

Then, generate a configuration file named “30-elasticsearch-output.conf” and enter the following:

nano /etc/logstash/conf.d/30-elasticsearch-output.conf


output {
  if [@metadata][pipeline] {
	elasticsearch {
  	hosts => ["localhost:9200"]
  	manage_template => false
  	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  	pipeline => "%{[@metadata][pipeline]}"
	}
  } else {
	elasticsearch {
  	hosts => ["localhost:9200"]
  	manage_template => false
  	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
	}
  }
}

Check The Config Validation:

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Config Validation Result: OK. Exiting Logstash

Once your configuration test is deemed successful, commence and activate Logstash to apply the configuration modifications:

sudo systemctl start logstash

sudo systemctl enable logstash