Open Source TCP or Netflow Log Server Using Graylog

Open Source
TCP or
Netflow Log
Server
Using
Graylog

Agenda
How to create netflow log
server using Graylog
[alternative of nfdump/nfsen]
Create your own TCP based log
server for Mikrotik NAT devices
with zero cost and how to meet
regulatory requirements.
Analyze traffic flow and
create custom dashboard for
visualization.
Finally exporting your
netflow or tcp logs data to
grafana for further analysis

Why we
need a
log
server?
• Consolidates logs from multiple
sources into a single platform.
• Simplifies log analysis and
troubleshooting.
• Ensures compliance with industry
regulations by maintaining audit
trails.
• Enhances performance monitoring
and issue diagnosis.
• Reduces the time and effort
required to search and analyze
logs.
Open Source Solutions: Graylog and
ELK Stack, NFsen are excellent
open-source options for handling
NetFlow data, offering flexibility
and powerful visualization.
Paid Solutions: like SolarWinds
NetFlow Traffic Analyzer, PRTG
Network Monitor, and ManageEngine
NetFlow Analyzer etc provide

Sample
Network
with a
log
Server

Netflow
Sample
Configurati
on:
Mikrotik

TCP logs sample
configuration:
Mikrotik

Overview
of
Graylog
Graylog is an open-source
log management tool.
Real-time search and
analysis, alerting,
dashboards, and more.
Scalability, flexibility,
user-friendly interface,
and strong community
support

Understand
ing TCP
and
Netflow
Logs
Netflow Logs:
NetFlow is a network protocol developed
by Cisco for collecting and monitoring IP
traffic data. It helps network
administrators analyze traffic patterns,
monitor performance, and troubleshoot
issues.
• Key Components:
• NetFlow Exporter: Collects and
exports flow data.
• NetFlow Collector: Receives and
stores flow data.
• NetFlow Analyzer: Analyzes and
visualizes the data.
• NetFlow Versions:
• NetFlow v5: Widely used, standard
version.
• NetFlow v9: Template-based, more
flexible and extensible.

Understand
ing TCP
and
Netflow
Logs
• Here are the most important fields:
• Source IP Address: The IP address of the device
that sent the packets.
• Destination IP Address: The IP address of the
device that received the packets.
• Source Port: The port number on the source
device.
• Destination Port: The port number on the
destination device.
• Layer 3 Protocol Type: The type of protocol used
(e.g., TCP, UDP).
• Type of Service (ToS): Quality of service
information.
• Input Interface: The network interface that
received the packets.
• Output Interface: The network interface that
sent the packets.
• Packet Count: The number of packets in the flow.
• Byte Count: The total number of bytes in the
flow.
• Flow Start Timestamp: The time when the flow
started.
• Flow End Timestamp: The time when the flow
ended.
• Next Hop IP Address: The IP address of the next
hop router.
• Source AS (Autonomous System): The AS number of
the source device.
• Destination AS: The AS number of the destination
device.

Understand
ing TCP
and
Netflow
Logs
TCP Logs:
TCP logs capture detailed information
about TCP connections and their states.
These logs help network administrators
monitor traffic, diagnose issues, and
ensure reliable network communication.
• Common TCP Flags:
• SYN (Synchronize): Initiates a
connection.
• ACK (Acknowledgment): Acknowledges
received data.
• FIN (Finish): Indicates the sender is
finished sending data.
• RST (Reset): Abruptly terminates a
connection.
• PSH (Push): Indicates that data
should be pushed to the receiving
application.
• URG (Urgent): Indicates that urgent
data is being sent.

Understand
ing TCP
and
Netflow
Logs
1.Source IP Address: The IP address of the
device that initiated the connection.
2.Destination IP Address: The IP address
of the device that received the
connection request.
3.Source Port: The port number on the
source device.
4.Destination Port: The port number on the
destination device.
5.Timestamp: The date and time when the
log entry was recorded.
6.TCP Flags: Indicators of the state of
the TCP connection, such as SYN, ACK,
FIN, and RST.
7.Sequence Numbers: Used to ensure data is
transmitted and received in the correct
order.
8.Acknowledgment Numbers: Confirm receipt
of packets.
9.Window Size: Indicates the amount of
data that can be sent before receiving an
acknowledgment.
10.Connection State: Describes the current
state of the TCP connection (e.g.,
established, closed).

Setting
Up
Graylog
for TCP
and
Netflow
Logs
• System Requirements:
• Hardware and software
requirements for installing
Graylog.
• Installation Steps:
• Installing prerequisites
(Java, MongoDB,
Elasticsearch).
• Installing Graylog.
• Configuration:
• Basic configuration steps for
Graylog.
• Setting up inputs for TCP and
Netflow logs.
https://go2docs.graylog.org/current/downloading
_and_installing_graylog/installing_graylog.html

Collecti
ng and
Processi
ng Logs
Configuring Inputs:
Inputs in Graylog are used to receive log messages from various
sources. Configuring inputs is the first step in collecting and
analyzing log data.
1. Accessing Inputs
• Navigate to the Graylog web interface.
• Go to System > Inputs.
2. Selecting Input Type
• Click on Select input.
• Choose the type of input based on the log source
(e.g., Syslog, NetFlow UDP).
3. Configuring Input Details
• Fill in the necessary details:
• Title: Name of the input.
• Port: The port on which Graylog will
listen for incoming messages.
• Bind Address: The IP address to bind the
input to.
• Additional Parameters: Depending on the
input type, additional configurations may
be required.
4. Starting the Input
• Click Launch new input to start the input.
• Verify that the input is running and receiving
messages.

Collecting and
Processing Logs
• Example: Configuring a Syslog UDP
Input
• Title: Syslog UDP
• Port: 514
• Bind Address: 0.0.0.0 (all network
interfaces)
• Save and launch the input.

Collecti
ng and
Processi
ng Logs
Log Parsing:
Log Parsing is the process of analyzing and
extracting meaningful data from log files. It
involves converting raw log data into structured
information for easier analysis and monitoring.
1. Creating Extractors in Graylog
• Navigate to System > Inputs.
• Select the input you want to add an extractor
to.
• Click on Manage Extractors.
2. Defining an Extractor
• Choose an extractor type (e.g., GROK, JSON,
Regex).
• Define the parsing rule to extract fields from
log messages.
• Test the extractor with sample log data to
ensure accuracy.
3. Saving and Using Extractors
• Save the extractor.
• Graylog will apply the extractor to incoming
log messages on the selected input.
• Extracted fields will be available for
searching and analysis.

Collecting
and
Processing
Logs
prerouting: in:<pppoe-ram-ruhul@v> out:(unknown
0), connection-state:established,snat proto TCP
(ACK,FIN), 10.64.4.28:55842-
>23.212.164.120:443, NAT (10.64.4.28:55842-
>103.234.202.0:55842)->23.212.164.120:443
grok_pattern: NAT
(%{IP:internal_ip}:%{NUMBER:internal_port}-
>%{IP:nat_ip}:%{NUMBER:nat_port})-
>%{IP:destination_ip}:%{NUMBER:destination_port
}, len %{NUMBER:length}

Collecti
ng and
Processi
ng Logs
Log Storage and Retention:
1. Setting Up Log Storage
• Use Elasticsearch as the primary storage backend for
Graylog.
• Ensure Elasticsearch is configured for high availability
and scalability.
2. Configuring Retention Policies in Graylog
• Navigate to System > Indices.
• Define retention policies based on index sets.
• Options include:
• Time-based retention: Retain logs for a specified
period.
• Size-based retention: Retain logs until a certain
index size is reached.
• Combined retention: Use both time and size-based
retention.
3. Implementing Log Rotation
• Configure index rotation strategies:
• Time-based rotation: Rotate indices daily, weekly,
etc.
• Size-based rotation: Rotate indices when they reach
a specific size.
4. Monitoring Storage and Retention
• Use Graylog's built-in monitoring tools to track storage
usage and retention compliance.
• Set up alerts for when storage limits are approaching.

Collecting
and
Processing
Logs
Log Storage and Retention:
Example: Rotate indices daily and
retain logs for 30 days.

Analyzin
g Logs
with
Graylog
Search and Filter:
• Search: The ability to query log data to
find specific information.
• Filter: Narrowing down search results
to show only relevant log entries.
• Accessing the Search Interface,
Navigate to the Graylog web interface.
Click on Search in the top menu.
• Performing a Basic Search
• Enter keywords in the search bar.
• Use common fields like message,
source, and timestamp
• Using Time Range Filters
• Select a predefined time range (e.g.,
last 5 minutes, last 24 hours).
• Customize the time range using the
date and time picker.

Analyzing
Logs with
Graylog
• Example:
• Filter by log level: level:ERROR
• Filter by IP address: source_ip:192.168.1.1
• Search for login failures: message:login AND
message:failed
• Filter by user or IP address: user:admin

Analyzing Logs with Graylog
Creating Dashboards:
1. Accessing the Dashboard Interface
• Navigate to the Graylog web interface.
• Click on Dashboards in the top menu.
2. Creating a New Dashboard
• Click on Create new dashboard.
• Provide a name and description for the dashboard.

Analyzing
Logs with
Graylog
Alerting and Notifications:
Automated triggers for specific events or
conditions is possible and also
notification of those alert via
Email/API/Slack is support. Some features
are only supported by Graylog Enterprise
Edition.

Share
Dashboard
with others
• Add Users/Teams: Enter the usernames or teams you want to share the
dashboard with.
• Assign Roles:
• Viewer: Can view the dashboard.
• Editor: Can edit the dashboard.
• Owner: Full control over the dashboard, including sharing
settings.

Use Cases and Examples [TCP
flow Dashboard]

Use Cases and Examples
[Netflow Dashboard]

Use Cases and Examples
[Grafana Dashboard]

Best Practices and Tips
Security Considerations:
• Ensuring secure log transmission and storage.
Performance Optimization:
• Tips for optimizing Graylog performance.
Community and Support:
• Leveraging the Graylog community for support and collaboration.

Open Source TCP or Netflow Log Server Using Graylog

More Related Content

What's hot (11)

Similar to Open Source TCP or Netflow Log Server Using Graylog (20)

More from Bangladesh Network Operators Group (20)

Recently uploaded (20)

Open Source TCP or Netflow Log Server Using Graylog