Cloud Computing

5 Mins Read

Lagging Behind Because of Logs? ELK Stack to the Rescue!

One of the common mistakes done by most of the professionals is not using valuable data called ‘Logs’. Because of the quantity of logs generated, the chances of using them becomes very less. Logs are used only to debug in case of failure or issues, but it can be used for much more

For Example:

The collections and analyzing of the log becomes extremely difficult because of the diversity generated. For example we have access logs, error logs, application logs etc. which are associated with an application or a server.

In this blog, I will be demonstrating how to install and configure ELK Stack.
ELK stands for: Elasticsearch, Logstash and Kibana.

Before we begin, let’s have a quick overview of the overall architecture with their components, followed by the implementation procedure.
 

Architecture of ELK Stack:

 

blog-archi

  1. ElasticSearch:
    • It is an Indexing, Storage and Retrieval engine
    • Powerful open-source full-text search library
    • A Document is the unit of search and index
    • Fast search against large volumes
    • De-normalized document storage: Fast, direct access to the data
    • Broadly distributed and highly scalable
  2. Logstash:
    • Log input slicer and dicer and output writer
    • Centralize Data Processing of all types
    • Normalize Varying Schema
    • Extend to Custom Log Formats
  3. Kibana:
    • Data Visualizer
    • Kibana is an open source data visualization plugin for ElasticSearch
    • Smooth integration with ElasticSearch
    • Give shape to the artifacts
    • Sophisticated Analytics
    • Flexible Interface
    • Visualize Data from different sources

Working:
The ELK stack architecture is very simple and clearly specifies the flow of the process.Various logs from different locations will be pulled by the Logstash (If you install Nginx for allowing external access then the logs will go to Nginx first), it will process the logs.
Logstash is the center where all the logs are processed and differentiated. Logs are then pushed to ElasticSearch, which is a Retrieval engine, it will index all the logs as per index pattern and will store it to be further accessed by Kibana.
Kibana is a Web UI through which we will do all the activities such as visualizing and analyzing, creating index patterns, etc.

Prerequisites:

Making ELK Stack Up and Running:

Step 1: Launching EC2 Instance and all Installations

 Install Java 8

Install ElasticSearch 

 Install Logstash 

Install Kibana 

Step 2: Configurations
 Configure Logstash:

NOTE: This will make Kibana accessible to instance_ip only. If we want to allow external access, then need to use Nginx as reverse proxy.

To allow external access following are the steps to configure with Nginx

This will prompt for a password that you will need to access Kibana dashboard along with kibadmin user

This configuration will make nginx to direct the server’s HTTP traffic to kibana which is listening on localhost:5601. This will enable to access kibana dashboard with elasticsearch server’s public ip.
Restart nginx to apply changes that we made

Step 3: Access Kibana Dashboard

kibana-start

This is the dashboard that we will get.

kibana-new-log

In this way, we get the logs.But, there are many options to view logs in different formats and to filter them.

kibana-piechart

Here, you can see the logs of nginx. There are many such options on the Kibana dashboard that you can explore.

Conclusion:

Implementing ELK Stack will provide you with the following benefits:

  1. Simple and quick way to manage logs
  2. Easy analysis of logs
  3. Deep dive into logs (based on timestamp)
  4. Various types of forms to view logs (bar chart, pie diagram, etc.)

You just need to create index pattern as per your need and you are ready to go.

Feel free to ask your questions below and I will get back to you on them.

Need professional assistance or consulting services for your ELK Stack project? Kindly visit here  Please comment and share if you liked the article.

WRITTEN BY CloudThat

SHARE

Comments

    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!