Install elasticsearch, logstash and kibana

INSTALL ELASTICSEARCH, LOGSTASH AND KIBANA
Logstash / Kibana setup has four main components:
 LOGSTASH: The server component of Logstash that processes incoming logs.
 ELASTICSEARCH: Stores all of the logs.
 KIBANA: Web interface for searching and visualizing logs, which will be proxied through Nginx.
 LOGSTASH FORWARDER: Installed on servers that will send their logs to Logstash, Logstash
Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to
communicate with Logstash.

SETTINGUP REPOS
Date:
date -s "9 AUG 2013 11:32:08"
## RHEL/CentOS 6 64-Bit ##
wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm
Preparing... ###################################### [100%]
1:epel-release ###################################### [100%]
sed-i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/epel.repo
INSTALLIGN JDK 1.8
Elasticsearch and Logstash require Java, so we will install that now. We will install Oracle Java 8 update
40 because Elasticsearch recommends at least Java 8 update 20. It should, however, work fine with
OpenJDK, if you decide to go that route. Following the steps in this section means that you accept the
Oracle Binary License Agreement for Java SE.
The correct JDK must be installedon all the nodesin your cluster.To manually deploythe JDK:
1. CHECK THE VERSION. FROM A TERMINAL WINDOW,TYPE:
java –version
2. (OPTIONAL) UNINSTALLTHE JAVA PACKAGEIF THE JDK VERSIONIS LESS THAN V1.6 UPDATE 31.
rpm -qa | grepjdk or
rpm -qa | grepjre or
rpm -qa | grepjava
yum remove {java-1.x.0-jdk-1.x.0.0-1.45.1.11.1.el6.x86_64} or
rpm -e {java-1.x.0-jdk-1.x.0.0-1.45.1.11.1.el6.x86_64
3. (OPTIONAL) VERIFYTHAT THE DEFAULT JAVAPACKAGE IS UNINSTALLED.
which java
4. DOWNLOADTHE ORACLE 64-BIT JDK JDK-6U31-LINUX-X64.BIN FROM THE ORACLE DOWNLOAD SITE.
cd /usr
sudo wget--no-cookies--no-check-certificate --header"Cookie:gpw_e24=http%3A%2F%2Fwww.oracle.com%2F;
oraclelicense=accept-securebackup-cookie"
"http://download.oracle.com/otn-pub/java/jdk/8u40-b25/jre-8u40-linux-x64.tar.gz"
tar -zxvf jre-8u40-linux-x64.tar.gz

5. CREATE SYMBOLIC LINKS (SYMLINKS) TO THE JDK.
mkdir /usr/java
ln -s/usr/jre1.8.0_40 /usr/java/default
ln -s/usr/java/default/bin/java /usr/bin/java
6. SET UP YOUR ENVIRONMENTTO DEFINE JAVA_HOME TO PUT THE JAVAVIRTUAL MACHINE AND THE JAVA
COMPILER ON YOUR path.
vim /etc/profile
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
INSTALL ELASTICSEARCH
Elasticsearch is an open source search server; it offers a real-time distributed search and analytics with Restful web
interface. Elasticsearch stores all the logs sent by the logstash server and displays the messages when the kibana4
requests for full filling user request over the web interface.
Run the followingcommandtoimportthe ElasticsearchpublicGPGkeyintorpm:
rpm --importhttp://packages.elasticsearch.org/GPG-KEY-elasticsearch
Create and edita newyumrepositoryfilefor Elasticsearch:
vim /etc/yum.repos.d/elasticsearch.repo
Addthe followingrepositoryconfiguration:
[elasticsearch-1.4]
name=Elasticsearchrepositoryfor1.4.xpackages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
Save and exit.
Install Elasticsearch1.4.4withthiscommand:
yum -y install elasticsearch-1.4.4
Elasticsearchisnowinstalled.Let'seditthe configuration:

vim /etc/elasticsearch/elasticsearch.yml
You will wantto restrictoutside accessto your Elasticsearchinstance (port9200), so outsiderscan't read your data or
shutdownyourElasticsearchclusterthroughthe HTTPAPI.Findthelinethatspecifies network.host,uncommentit,and
replace its value with "localhost"
network.host: localhost
Save and exit elasticsearch.yml.
Now start Elasticsearch:
servive elasticsearch restart
Then run the following command to start Elasticsearch automatically on boot up:
chkconfig elasticsearch on
Wait,at leasta minute toletthe Elasticsearchgetfullyrestarted,otherwisetestingwill fail.Elastisearchshouldbe now
listening on 9200 for processing HTTP request; we can use CURL to get the response.
[root@kibana ~]# curl -X GET http://localhost:9200
{
"status" : 200,
"name" : "Red Nine",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.4",
"build_hash" : "c88f77ffc81301dfa9dfd81ca2232f09588bd512",
"build_timestamp" : "2015-02-19T13:05:36Z",
"build_snapshot" : false,
"lucene_version" : "4.10.3"
},
"tagline" : "You Know, for Search"

}
INSTALL LOGSTASH
Logstash is an open source tool, used for collecting logs, parsing and storing them searching.
The Logstash package shares the same GPGKey as Elasticsearch,and we already installed that public key, so let's create
and edit a new Yum repository file for Logstash:
vim /etc/yum.repos.d/logstash.repo
Add the following repository configuration:
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
enabled=1
Save and exit.
Install Logstash 1.5 with this command:
yum -y install logstash
Logstash is installed but it is not configured yet.
CREATE SSL CERTIFICATE:
Logstash-forwarderwhichwill be installedonclient-servertoshipthe logsrequiresSSLcertificatetovalidateidentityof
logstashserver.We have a two optionsto create a SSL certificate andit dependsonlogstash-forwarderconfiguration;
if the hostname (“servers”: [ “kibana.vm:5050″ ]) is used
Before creating a certificate, make sure you have A record for logstash server; ensure that client servers are able to
resolve the hostname of thelogstashserver.If youdonothaveDNS,kindlyaddthe hostentryforlogstashserver;where
172.17.100.210 is the ip address of logstash server and itzgeek is the hostname of your logstash server.
vim /etc/hosts
172.17.100.201 kibana.vm
Let’screate a SSl certificate.

Goto OpenSSLdirectory.
cd /etc/pki/tls
Execute the followingcommandtocreate a SSL certificate,replace “red”one inwithyourreal logstashserver.
openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout private/logstash-forwarder.key -out
certs/logstash-forwarder.crt -subj /CN=kibana.vm
Thislogstash-forwarder.crtshouldbe copied toall clientserversthosewhosendlogstologstashserver.
CONFIGURE LOGSTASH:
Logstash configurationfilescanbe foundin /etc/logstash/conf.d/,justanemptyfolder.We wouldneedto create file,
logstashconfigurationfilesconsistof three sectioninput,filterandoutput;all threesectioncanbe foundeitherinsingle
file or each section will have separate files ends with .conf.
Here we will use a single file to place an input, filter and output sections.
vim /etc/logstash/conf.d/logstash_syslogs.conf
input {
lumberjack {
port => 5050
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
date {

match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { host => localhost index => "logstash-%{+YYYY.MM.dd}" }
stdout { codec => rubydebug }
}
Now start the logstash service.
servive logstash restart
chkconfig logstash on
Logstash server logs are stored in the following file, will help us to troubleshoot the issues.
touch /var/log/logstash/logstash.log
tail -f /var/log/logstash/logstash.log
SETUP LOGSTASH FORWARDER (ADD CLIENTSERVERS)
Sendlogsto yourLogstash Server.ForinstructionsoninstallingLogstashForwarderonDebian-basedLinux
distributions.
INSTALL LOGSTASH FORWARDER PACKAGE
On Client Server, create run the following command to import the Elasticsearch public GPG key into rpm:
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
Create and edit a new yum repository file for Logstash Forwarder:
vim /etc/yum.repos.d/logstash-forwarder.repo
Add the following repository configuration:
[logstash-forwarder]
name=logstash-forwarder repository

baseurl=http://packages.elasticsearch.org/logstashforwarder/centos
gpgcheck=1
enabled=1
Save and exit.
Theninstall the LogstashForwarderpackage:
yum -y install logstash-forwarder
Now copy the Logstash server's SSL certificate into the appropriate location (/etc/pki/tls/certs):
COPY SSLCERTIFICATE ANDLOGSTASH FORWARDER PACKAGE
On Logstash Server,copy the SSL certificate to ClientServer(substitutethe clientserver'sIPaddress,andyourown
login):
scp -r /etc/pki/tls/certs/logstash-forwarder.crt root@server_clinet_IP: /etc/pki/tls/certs/
After providingthe logincredentials, ensure that the certificate copy was successful. It is required for communication
between the client servers and the Logstash server.
CONFIGURE LOGSTASH FORWARDER
On Client Server, create and edit Logstash Forwarder configuration file, which is in JSON format:
vim /etc/logstash-forwarder.conf
Underthe network section,addthe followinglinesintothe file,substitutinginyourLogstashServer'sprivate IPaddress
for logstash_server_private_IP:
"servers": [ "logstash_server_private_IP:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
Under the files section (between the square brackets), add the following lines,
{
"paths": [

"/var/log/messages",
"/var/log/secure"
],
"fields": { "type": "syslog" }
}
Save and quit. This configures Logstash Forwarder to connect to your Logstash Server on port 5000 (the port that we
specifiedaninputforearlier),andusesthe SSLcertificate thatwe createdearlier.The pathssectionspecifieswhichlog
files to send (here we specify syslog and auth.log), and the type section specifies that these logs are of type "syslog*
(which is the type that our filter is looking for).
Note that this is where you wouldadd more files/types to configure LogstashForwarder to other log files to Logstash
on port 5000.
Now restart Logstash Forwarder to put our changes into place:
Logstash-forwarder is client software which ship logs to a logstash server; it should be installed on all client servers.
Logstash-forwardercan be downloadedfromofficial website oryoucan use the followingcommandtodownloaditin
terminal and install it.
service logstash-forwarder restart
NowLogstash Forwarderis sendingyoursyslog messages andsecure filestoyour LogstashServer!Repeatthissection
for all of the other servers that you wish to gather logs for.
You can look at a log file in case of any issue.
tail -f /var/log/logstash-forwarder/logstash-forwarder.err
INSTALL KIBANA
Download Kibana 4 to your home directory with the following command:
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
Extract Kibana archive with tar:
tar -zxvf kibana-*.tar.gz
Open the Kibana configuration file for editing:
vim ~/kibana-4*/config/kibana.yml

In the Kibana configuration file, find the line that specifies host, and replace the IP address ("0.0.0.0" by default) with
"localhost":
host: "localhost"
# The host to bind the server to.
host: "localhost"
# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "http://localhost:9200"
Before we can use the Kibana web interface, we have to set up a reverse proxy. Let's do that now, with Nginx.
Because we configuredKibanatolistenon localhost,we mustset up a reverse proxytoallow external accesstoit. We
will use Nginx for this purpose.
Note: If you already have an Nginx instance that you want to use, feel free to use that instead. Just make sure to
configure Kibana so it is reachable by your Nginx server (you probably want to change the host value, in
/opt/kibana/config/kibana.yml, to your Kibana server's private IP address). Also, it is recommended that you enable
SSL/TLS.
Add the EPEL repository to yum:
yum -y install epel-release
Now use yum to install Nginx and httpd-tools:
yum -y install nginx httpd-tools
Use htpasswdtocreate anadminuser,called"kibanaadmin"(youshoulduse anothername),thatcanaccessthe Kibana
web interface:
htpasswd -c /etc/nginx/htpasswd.users kibanaadmin
Entera passwordat the prompt.Rememberthislogin,asyouwill needittoaccessthe Kibanawebinterface. Nowopen
the Nginx configuration file in your favorite editor.
vim /etc/nginx/nginx.conf
Configure Nginx:
Find the default server block(starts with server {),the last configuration block inthe file,and delete it. When you are
done, the last two lines in the file should look like this:
user nginx;

worker_processes 1;
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
# Load config files from the /etc/nginx/conf.d directory
# The default server is in conf.d/default.conf
server {
listen 80;

server_name 172.17.100.201;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.users;
location / {
proxy_pass http://localhost:5601;
#proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
include /etc/nginx/conf.d/*.conf;
}
ThisconfiguresNginx todirectyourserver'sHTTPtrafficto the Kibanaapplication,whichislisteningon localhost:5601.
Also, Nginx will use the htpasswd.users file, that we created earlier, and require basic authentication.
Now start and enable Nginx to put our changes into effect:
/etc/rc.d/init.d/nginx start
chkconfig nginx on
Kibana is now accessible via your FQDN or the public IP address of your Logstash Server i.e.
http://logstash_server_public_ip/.If yougothere in a webbrowser,afterenteringthe "kibanaadmin"credentials,you
shouldsee aKibanawelcomepage whichwillaskyoutoconfigure anindex pattern.Let'sgetbacktothatlater,afterwe
install all of the other components.
Starting Kibana:
nohup sh kibana > /dev/null 2>&1 &

Juts login into the Kibana URL: http://172.17.100.201/
You will geta followingpage where youhave tomaplogstashindex touse kibana.Scroll downonTime -fieldname and
select
That’s all,youhave successfully configuredELKstackfor centralizedlogmanagement.

Indexes are stored on disk as configured in elasticsearch.yml with the configuration option path.data;
localhostonport 9200 is the defaultconnectionportforthe HTTP REST interface,the pathof the url generallydefines
an action to be taken (like searching for documents);
What exactlydo you meanwiththe algorithm?Elasticsearchisa search engine,itusesLucene to read documentsand
index their properties to enable search.
ELASTICSEARCHDATA STORAGE LOCATION

Install elasticsearch, logstash and kibana

More Related Content

What's hot (20)

Similar to Install elasticsearch, logstash and kibana (20)

More from Chanaka Lasantha (20)

Recently uploaded (20)

Install elasticsearch, logstash and kibana