OSMC 2013 | Monitoring network traffic using ntopng by Luca Deri

© 2013 - ntop.org
Monitoring Network Trafﬁc using ntopng
Luca Deri <deri@ntop.org>

© 2013 - ntop.org
Outlook
• What are the main activities of ntop.org ?
• ntop’s view on network monitoring.
• From ntop to ntopng.
• ntopng architecture and design.
• Using ntopng.
• Advanced monitoring with ntopng.
• Future roadmap items.
2

© 2013 - ntop.org
Introduction
3

© 2013 - ntop.org
About ntop.org [1/2]
• Private company devoted to development of
open source network traffic monitoring
applications.
• ntop (circa 1998) is the first app we released and
it is a web-based network monitoring application.
• Today our products range from traffic
monitoring, high-speed packet processing, deep-
packet inspection, and IDS/IPS acceleration.
4

© 2013 - ntop.org
About ntop.org [2/2]
• Our software is powering many commercial
products...
5

© 2013 - ntop.org
Problem Statement [1/2]
• In 90s standard-based (e.g. RMON) and
proprietary solutions (e.g. HP OpenView) were
dominating the market.
• The explosion of the Internet(working) reduced
the number of vendors but increased the number
of monitoring devices and protocols.
• ntop tried to fill the lack of simple traffic
monitoring application releasing the original ntop
application aimed at simplifying the analysis of
network traffic.
6

© 2013 - ntop.org
Problem Statement [2/2]
• Today the market is partially driven by hardware
manufacturers that manufacture costly, limited
(usually no analysis beyond packet header) and
often not extensible solutions.
• Consequences:
◦Monitoring evolution is capped by hardware vendors.
◦Commercial probes monitor what the vendor wants
(e.g. Cisco TelePresence) and not what the user needs.
◦Often people believe in speed=hardware, so they buy
hardware probes that are unlike to provide timely
answers to continuously evolving monitoring needs.
7

© 2013 - ntop.org
ntop Goals
• Provide better, yet price effective, traffic
monitoring solution by enabling users to have
increased traffic visibility.
• Go beyond standard metrics and increase traffic
visibility by analyzing key protocols in detail.
• Provide users comprehensive and accurate traffic
reports able to offer at a fraction of price what
many commercial products do together.
• Promote open-source software, while protecting
selected IPRs.
8

© 2013 - ntop.org
ntop’s Approach to Trafﬁc Monitoring
• Ability to capture, process and (optionally)
transmit trafﬁc at line rate, any packet size.
• Leverage on modern multi-core/NUMA
architectures in order to promote scalability.
• Use commodity hardware for producing
affordable, long-living (no vendor lock), scalable
(use new hardware by the time it is becoming
available).
• Use open-source to spread the software, and let
the community test it on unchartered places.
9

© 2012 - ntop.org
Network Health Monitoring is
Complex
• Traditional network monitoring systems can limit their
supervision activities to “home” networks.
• Mobile users, intra-VM data exchange, or cloud services
are often invisible to network monitoring systems.
• Traditional ﬁrewalls are becoming blind as:
◦ Generic (e.g. HTTP) protocols can be used to tunnel non-
hypertext services (e.g. music streaming).
◦ HTTPS is not checked so often it ﬂows unrestricted.
• Cloud services access can’t be monitored with simple
“periodic pings” as provides IPs are often unresponsive
and change according to our location.
10

© 2013 - ntop.org
Limitations of (Many) Monitoring Systems
• Visibility limited to packet header (payload
agnostic).
• Packet encapsulations (e.g. GRE, PPP, GTP) are
not always handled, so that we don’t know what
happens inside tunnels.
• Unable to monitor intra-virtual machines (VMs)
traffic (no cloud-friendly).
• Windows PCs are not first-class citizens.
• Inability to map user traffic to actions performed
or user’s interests.
11

© 2013 - ntop.org
Some History
• In 1998, the original
ntop has been created.
• It was a C-based app
embedding a web server
able to capture trafﬁc
and analyze it.
• Contrary to many tools available at that time,
ntop used a web GUI to report trafﬁc activities.
• It is available for Unix and Windows under GPL.
12

© 2013 - ntop.org
ntop Architecture
13
HTTP/HTTPS RRD
Cisco NetFlow
InMon sFlow

© 2013 - ntop.org
Why was ntop obsolete?
• Its original LAN-oriented design prevented ntop
from handling more than a few hundred Mbit.
• The GUI was an old (no fancy HTML 5)
monolithic piece written in C so changing/
extending a page required a programmer.
• ntop could not be used as web-less monitoring
engine to be integrated with other apps.
• Many components were designed in 1998, and it
was time to start over (spaghetti code).
14

© 2013 - ntop.org
ntopng Design Goals
• Clean separation between the monitoring engine
and the reporting facilities.
• Robust, crash-free engine (ntop was not really so).
• Platform scriptability for enabling extensions or
changes at runtime without restart.
• Realtime: most monitoring tools aggregate data (5
mins usually) and present it when it’s too late.
• Many new features including HTML 5-based
dynamic GUI, categorization, DPI.
15

© 2013 - ntop.org
ntopng Architecture
16
• Three different and self-contained components,
communicating with clean API calls.
Users
HTTP
Lua-based Web Reports
nDPI-based C++
Monitoring Engine
PF_RING Kernel Module
and Drivers
Internet
Trafﬁc
Lua API Calls
PF_RING C API Calls(Linux) Kernel
Data Cache

© 2013 - ntop.org
17
PF_RING (Circa 2003)
Read
Index
Write
Index
Incoming Packets
Outgoing Packets Userspace
Kernel
Socket
(ring)
Network
Adapter
mmap()
Socket
(ring)
PF_RING
Application A Application Z

© 2013 - ntop.org
Vanilla vs DNA (Direct NIC Access)
18
Device Driver
Application
DMA
PF_RING
Userland
Kernel
Circular
Buffer
NAPI
Polling
PF_RING
Polling
PF_RING DNAVanilla PF_RING
Device Driver
Application
DMA
Accelerated
Cards
Userland
Kernel
NIC
Memory
Map FPGA
Application
Polling

© 2013 - ntop.org
PF_RING DNA: Features
• DNA is a Linux kernel-bypassing technology developed
by ntop/Silicom for 1/10 Gbit network adapters.
• It can receive/transmit traffic at line rate any packet size
(14.88 Mpps, 60+4 bytes) using x86 servers.
• Feature-rich set of companion tools distributed in
source format (e.g. line-rate traffic generation and
pcap traffic replay) so that people can customize
and use existing (pcap-based) apps as low-cost
alternatives to hardware-based tools (e.g. traffic
generator).
19

© 2013 - ntop.org
ntopng Monitoring Engine
• Coded in C++ and based the concept of ﬂow (se
of packets with the same 6-tuple).
• Flows are inspected with a home-grown DPI-
library named nDPI aiming to discovering the
“real” application protocol (no ports are used).
• Information is clustered per:
 (Capture) Network Device
 Flow
 Host
 High-level Aggregations
20

© 2013 - ntop.org
Information Lifecycle
• All information (e.g. hosts and flows) is stored in
memory.
• Using command line options, users can specify
how many hosts/flows can be kept in memory.
• Idle flows and are periodically purged to free
memory.
• Hosts are serialized and stored in JSON format
in Redis for 1 hour, so that in case new traffic is
detected ntopng can restore them from cache.
21

© 2013 - ntop.org
Packet Processing Journey
1.Packet capture: PF_RING (Linux) or libpcap.
2.Packet decoding: no IP traffic is accounted.
3.IPv4/v6 Traffic only:
1.Map the packet to a 6-tuple flow and increment stats.
2.Identify source/destination hosts and increment stats.
3.Use nDPI to identify the flow application protocol
1.UDP flows are identified in no more than 2 packets.
2.TCP Flows can be identified in up to 15 packets in total,
otherwise the flow is marked as “Unknown”.
4.Move to the next packet.
22

© 2013 - ntop.org
The need for DPI in Monitoring [1/2]
• Limit traffic analysis at packet header level it no
longer enough (nor cool).
• Network administrators want to know the real
protocol without relying on the port being used.
• Selected protocols can be “precisely
dissected” (e.g. HTTP) in order to extract
information, but on the rest of the traffic it is
necessary to tell network administrators what is
the protocol flowing in their network.
23

© 2013 - ntop.org
The need for DPI in Monitoring [2/2]
• DPI (Deep Packet Inspection) is a technique for inspecting
the packet payload for the purpose of extracting metadata
(e.g. protocol).
• There are many DPI toolkits available but they are not
what we looked for as:
◦ They are proprietary (you need to sign an NDA to use them), and
costly for both purchase and maintenance.
◦ Adding a new protocol requires vendor support (i.e. it has a high
cost and might need time until the vendor supports it) = you’re
locked-in.
• On a nutshell DPI is a requirement but the market does
not offer an alternative for open-source.
24

© 2013 - ntop.org
Say hello to nDPI
• ntop has decided to develop its own GPL DPI
toolkit in order to build an open DPI layer for
ntop and third party applications.
• Supported protocols (~170) include:
◦P2P (Skype, BitTorrent)
◦Messaging (Viber,Whatsapp, MSN,The Facebook)
◦Multimedia (YouTube, Last.gm, iTunes)
◦Conferencing (Webex, CitrixOnLine)
◦Streaming (Zattoo, Icecast, Shoutcast, Netﬂix)
◦Business (VNC, RDP, Citrix, *SQL)
25

© 2013 - ntop.org
nDPI on ntopng: Sample Report
26

© 2013 - ntop.org
Is nDPI Computationally Expensive?
• nDPI requires
◦Full packet payload capture (free with PF_RING DNA).
◦Inspection of the packet payload.
• The above activities are not cheap, but they are
performed only at the beginning of the
connection: the longer is the connection the
cheaper.
• nDPI increases the application load of less than
5-10% so it is almost unnoticeable.
27

© 2013 - ntop.org
ntopng and Redis
• Redis is an open source key-value in-memory
database.
• ntop uses it to cache data such as:
◦Conﬁguration and user preferences information.
◦DNS name resolution (numeric to symbolic).
◦Volatile monitoring data (e.g. hosts JSON
representation).
• Some information is persistent (e.g. preferences)
and some is volatile: ntopng can tell redis how
long a given value must be kept in cache.
28

© 2013 - ntop.org
Welcome to the MicroCloud [1/2]
29
Cloud Node
(redis)
Cloud Node
(redis)
Cloud Node
(redis)
Cloud Get/Put
Cloud Get/Put
ntopng
nProbentopng
nProbe
nProbe
ntopng
Cloud Get/Put
Cloud Get/Put
MicroCloud
Monitoring Application
Cloud Subscribe
Cloud Getn
nnn
n
n

© 2013 - ntop.org
Welcome to the MicroCloud [2/2]
• The microcloud is a semi-persistent cache, used
to aggregate heterogeneous data onto a single
place for promoting data correlation and thus
better traffic visibility.
• ntopng is a first-class microcloud citizen, but
additional applications (e.g.WinLogon, Captive
Portals) can fill the cloud or access (read-only)
data present on it.
30

© 2013 - ntop.org
Using the MicroCloud
• Typical usage scenarios include:
◦Store the DNS cache and HTTP host, so that we can
associate the correct symbolic hostname to a flow
(often a issue in particular with CDNs).
◦Keep coherency across tunnels (e.g. GTP) so that
probes can associate traffic directions properly.
◦Store user-data (e.g. Radius IP/User association) so that
the probe, can report collectors both the username
who produced a given flow and the symbolic flow
hostnames.
◦Trigger data capture on demand (via publish/subscribe).
31

© 2013 - ntop.org
Lua-based ntopng Scriptability [1/3]
• A design principle of ntopng has been the clean
separation of the GUI from engine (in ntop it was
all mixed).
• This means that ntopng can (also) be used (via
HTTP) to feed data into third party apps such as
Nagios or OpenNMS.
• All data export from the engine happens via Lua.
• Lua methods invoke the ntopng C++ API in order
to interact with the monitoring engine.
32

© 2013 - ntop.org
• /scripts/callback/
scripts are executed
periodically to perform
speciﬁc actions.
• /scripts/lua/ scripts
are executed only by the
web GUI.
• Example:
http://ntopng:3000/lua/flow_stats.lua
33

© 2013 - ntop.org
• ntopng defines (in C++) two Lua classes:
◦interface
 Hook to objects that describe flows and hosts.
 Access to live monitoring data.
◦ntop
 General functions used to interact with ntopng configuration.
• Lua objects are usually in “read-only” mode
◦C++ sets their data, Lua reads data (e.g. host.name).
◦Some Lua methods (e.g. interface.restoreHost()) can
however modify the information stored in the engine.
34

© 2013 - ntop.org
Multithreading in ntopng
• ntopng can monitor simultaneously various
network interfaces, each using a thread.
• Packet processing per-interface is lockless.
• Lua scripts can access monitoring data while
packets are being processed: locks occur in very
few places and are not per-packet.
• Multiple Lua scripts and HTTP requests can be
served (on different threads) while packets are
being processed.
35

© 2013 - ntop.org
Using ntopng
36

© 2013 - ntop.org
Logging into ntopng
37

© 2013 - ntop.org
Dashboard
38

© 2013 - ntop.org
Available Menu Items
39

© 2013 - ntop.org
Dynamic Web Interface
40

© 2013 - ntop.org
Activity Map
• 1 second resolution host and aggregation activity
• Compressed bitmap
• Saved persistently on disk (Local Hosts only)
46
> ls -l client14.dropbox.com
4 -rw-rw-rw- 1 nobody nogroup 24 Oct 11 02:31 client14.dropbox.com

© 2013 - ntop.org
Traffic Aggregations [1/2]
• nDPI extracts specific attributes from traffic that
ntopng aggregates (if configured):
◦DNS/Whois responses
◦HTTP host names
◦Operating System (from HTTP headers)
• Aggregations can be enabled (they are off by
default) and are handled just as flows and hosts.
47

© 2013 - ntop.org
Historical Activities
• All relevant counters are saved on disk in RRD.
• Interface counters are saved with 1 second
resolution. Hosts counters every 5 minutes.
52
RRD values correlated
with top talkers
Ajax-based charts
(no RRD graphs)

© 2013 - ntop.org
Using ntopng as a Live Data Source
• ntopng is a server able to serve data to third
party applications via HTTP.
• Data is exported via JSON.
• This mechanism can be extended via Lua scripts.
53

© 2013 - ntop.org
Using ntopng with NetFlow/sFlow
• ntopng can handle ﬂows (Net/sFlow) via nProbe.
• Data Collector (ntopng)
◦ ntopng -i tcp://127.0.0.1:5556
• Probe (nProbe)
◦ nprobe --zmq "tcp://*:5556" -i eth1 -n none (probe mode)
◦ nprobe --zmq "tcp://*:5556" -i none -n none --collector-port
2055 (sFlow/NetFlow collector mode)
55

© 2013 - ntop.org
Embedding ntopng [1/2]
• Historically we have started our ﬁrst embed
attempt in 2003 with the Cyclades TS100.
• The nBox was used to analyze trafﬁc then sent to
ntop for representation.
• After 10 years we tried again with ntopng.
56

© 2013 - ntop.org
Embedding ntopng [2/2]
• The ntopng code compiles smoothly for cheap
(36 Euro) boxes such as the BeagleBone Black.
• You can now create
your personal/cheap
trafﬁc analyzer
without having to
use a PC.
• Post 1.1 release we
will optimize support
for these devices.
57

© 2013 - ntop.org
Final Remarks
• Over the past 15 years ntop created a software
framework for efficiently monitoring traffic.
• “We have a story to tell you, not just hacks”.
• Commodity hardware, with adequate software,
can now match the performance and flexibility
that markets require.With the freedom of open
source.
• Available under GNU GPLv3 from
http://www.ntop.org/.
58

OSMC 2013 | Monitoring network traffic using ntopng by Luca Deri

More Related Content

Similar to OSMC 2013 | Monitoring network traffic using ntopng by Luca Deri (20)

Recently uploaded (20)

OSMC 2013 | Monitoring network traffic using ntopng by Luca Deri