SlideShare a Scribd company logo

Zabbix LLD from a C Module by Jan-Piet Mens

NETWAYS, profile picture
NETWAYS

The document discusses extending Zabbix's functionality through loadable C modules and low-level discovery (LLD) methods. It explains how to create custom modules to support various monitoring tasks and outlines the process for implementing LLD to manage items and triggers efficiently. Additionally, it provides examples of using MQTT for real-time data transport and monitoring applications.

Software
1 of 70
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
Zabbix loadable C modules and Low-Level Discovery (LLD) Jan-Piet Mens OSMC, November 2019 @jpmens
@jpmens: consultant, part-time admin, trainer, small-scale fiddler, loves plain text, and things which work. Contributes to Ansible, dreamed up OwnTracks, and chases bugs in open source DNS servers.
This is OwnTracks
Zabbix cannot, out of the box, support every possible thing you may want to monitor; let's extend it. extensible
extending Zabbix system.run[] User Parameters External checks Loadable modules Beware fork(), invalid results: unsupported items
system.run[] # agent EnableRemoteCommands=1 LogRemoteCommands=1 76775:20190326:173402.315 Executing command '/usr/local/bin/zdisc.sh one two three' #!/bin/sh echo $# Runs on agent
user parameters #!/bin/sh echo 16 exit 0 # agent
 UserParameter=jjolie.cn[*],echo "$1 Jolie" UserParameter=jjolie.uid,/usr/local/bin/uidnumber.sh Runs on agent
external checks Runs on server # server
 ExternalScripts=/usr/lib/zabbix/externalscripts
loadable modules Written in the C language, compiled, loaded dynamically by server, proxy, and/or agent
what can modules do? Basically anything: sockets, SystemV IPC, files, databases, IoT, sensors, … Beware permissions, context, errors
custom module # server, proxy, agent LoadModulePath=/usr/lib/zabbix/agent
 LoadModule=owntracks-module.so
 LoadModule=two.so
make install Build inside / outside source tree owntracks-module.so: owntracks-module.c gcc -fPIC -shared -o owntracks-module.so owntracks-module.c install owntracks-module.so /usr/lib/zabbix/agent/
module.h --- zabbix-4.0.5/include/module.h 2019-02-25 10:36:39.000000000 +0100 +++ module.h 2019-03-26 15:03:59.000000000 +0100 @@ -20,7 +20,8 @@ #ifndef ZABBIX_MODULE_H #define ZABBIX_MODULE_H -#include "zbxtypes.h" +#include <inttypes.h> +#define zbx_uint64_t uint64_t #define ZBX_MODULE_OK 0 #define ZBX_MODULE_FAIL -1
the API int zbx_module_api_version(void) int zbx_module_init(void) ZBX_METRIC *zbx_module_item_list(void) void zbx_module_item_timeout(int timeout) ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) int zbx_module_uninit(void)
returning a value #define SET_UI64_RESULT(res,val) #define SET_DBL_RESULT(res, val) #define SET_STR_RESULT(res, val) /* xx */ #define SET_TEXT_RESULT(res,val) /* xx */ #define SET_LOG_RESULT(res, val) /* xx */ #define SET_MSG_RESULT(res, val) /* xx */ /* xx: ALWAYS allocate new memory for val! * DON'T USE STATIC OR STACK MEMORY!!! */
our C module #include "module.h" /* return the version number of this module interface. */ int zbx_module_api_version(void) { return (ZBX_MODULE_API_VERSION); }
initialize /* agent start-up; Zabbix will fail to load if we return ZBX_MODULE_FAIL */ int zbx_module_init(void) { if ((db = db_open()) == NULL) { return (ZBX_MODULE_FAIL); } return (ZBX_MODULE_OK); } /* agent is shutting down */ int zbx_module_uninit(void) { db_close(db); return (ZBX_MODULE_OK); }
key to function map static ZBX_METRIC item_keys[] = { /* ITEM.KEY FLAG FUNCTION TEST PARAMETERS */ { "owntracks.ping", 0, owntracks_ping, NULL}, { "owntracks.lld", 0, owntracks_lld, NULL}, { "owntracks.vel", CF_HAVEPARAMS, owntracks_vel, "1234567890012345"}, { NULL } }; /* return list of item keys supported by this module */ ZBX_METRIC *zbx_module_item_list(void) { return (item_keys); }
owntracks.ping $ zabbix_get -s 127.0.0.1 -k owntracks.ping 42 /* 42. what else? */ static int owntracks_ping(AGENT_REQUEST *request, AGENT_RESULT *result) { SET_UI64_RESULT(result, 42); return (SYSINFO_RET_OK); }
frontend items If our module is loaded into a server, the item type must be Simple Check. If our module is loaded into an agent, the item type must be Zabbix agent or Zabbix agent (active).
programatically create item from zabbix_api import ZabbixAPI # https://pypi.org/project/zabbix-api/ import json zapi = ZabbixAPI(server="http://server/zabbix/") zapi.login("admin", "secret") res = zapi.item.create({ "name" : "The pong", "key_" : "owntracks.ping", "hostid" : 10268, "interfaceid" : "11", "type" : 0, # Zabbix agent "value_type" : 3, # numeric, unsigned "delay" : "60s", })
owntracks.vel static int owntracks_vel(AGENT_REQUEST *request, AGENT_RESULT *result) { unsigned long retval; if (request->nparam != 1) { SET_MSG_RESULT(result, strdup("Expecting 1 parameter with IMEI")); return (SYSINFO_RET_FAIL); } retval = db_get(db, get_rparam(request, 0)); SET_UI64_RESULT(result, retval); return (SYSINFO_RET_OK); }
low-level discovery
What is LLD? Low-Level Discovery, widely used, built-in methods* can create items, triggers, graphs, custom LLD, send JSON to Zabbix server LLD rules discover entities, not item values * network interfaces, CPUs, SNMP OIDs, JMX objects, ODBC queries, Windows services, file systems, host interfaces
how do we use LLD? (1)
how do we use LLD? (2) What does [process] do? • subscribe to MQTT • extract OwnTracks' payload • update device database (store values) => LLD • device not seen or dead, i.e.TTL expired?
 remove from database => LLD • device details (e.g. speed) add to database: owntracks.vel() reads it • spikes trigger Zabbix alerts • 120 lines of Python
how do we use LLD? (3) Why LMDB? • in-process • blindingly fast • single writer (our [process]) • multiple readers (functions in our loadable C module) • no external moving parts • multiple language bindings (C, Python, etc…)
OwnTracks payloads { "_type": "location", "tst": 1553542948, "lat": 59.373399, "lon": 17.94669, "cog": 280, "alt": 101, "vel": 15, "tid": "xq", "batt": 89, "name": "20-ER-20" } https://owntracks.org/booklet/tech/json/ owntracks/zbx/012549656802107 topic payload
defining LLD Define discovery rule.Text item receives JSON. Prototypes create items and triggers.
trapper item $ zabbix_sender -z 192.168.1.53 -s AirJP -k my.item -o '{ "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }'
LLD JSON format $ zabbix_get -s 127.0.0.1 -k owntracks.lld { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
owntracks.lld static int owntracks_lld(AGENT_REQUEST *request, AGENT_RESULT *result) { JsonNode *obj = json_mkobject(), *list = json_mkarray(), *o; char *js; while ((o = db_getnext(db)) != NULL) { json_append_element(list, o); } json_append_member(obj, "data", list); if ((js = json_stringify(obj, " ")) == NULL) { SET_MSG_RESULT(result, strdup("Cannot stringify JSON.")); return (SYSINFO_RET_FAIL); } SET_STR_RESULT(result, js); /* Zabbix will free() this */ json_delete(obj); return (SYSINFO_RET_OK); }
discovery rule
item prototypes { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
trigger prototypes {airjp:owntracks.vel[{#OT.IMEI}].last()}>90
LLD in action
latest data
stale data auto expiry
graph prototypes
history export (1) Server only, quite complicated, invoked at server start and then occasionally, badly documented, replaceable with real-time export into JSON files.
static void history_cb_integer(const ZBX_HISTORY_INTEGER *history, int history_count) { int n; itemid, val, clock; for (n = 0; n < history_count; n++) { itemid = history[n].itemid; clock = history[n].clock; val = history[n].value; ... } } ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) { static ZBX_HISTORY_WRITE_CBS my_callbacks = { NULL, /* my_history_float_cb */ history_cb_integer, /* my_history_integer_cb */ history_cb_string, /* my_history_string_cb */ NULL, /* my_history_text_cb */ NULL, /* my_history_log_cb */ }; return (my_callbacks); }
history export (2) { "host": "airjp", "groups": [ "Linux servers" ], "applications": [], "itemid": 28530, "name": "jjolie.cn", "clock": 1553636580, "ns": 518541363, "value": "Jane Jolie" } { "host": "Zabbix server", "groups": [ "Zabbix servers" ], "applications": [ "Filesystems" ], "itemid": 28506, "name": "Free disk space on / (percentage)", "clock": 1553636586, "ns": 523296311, "value": 70.684192 }
low-level recipe write and compile module, edit configuration, restart agent, server, or proxy, create items, test, enjoy
buywhy.gif C functions are fast. LLD is highly flexible. Together this is a powerful combination. Unfortunately the API has no support for reading zabbix.conf
 Documentation is minimal.
zabbix.com @zabbix
Have you heard of
 MQTT?
MQTT is MQTT is a standard, a transport, for PUB/SUB messaging, designed for unreliable networks, binary payloads up to 256MB, (+2 bytes), fast, lightweight, ideal for low-bandwith, high- latency networks, TLS, authentication, ACLs,TLS-PSK, (payload encryption), keepalive, last will & testament, UTF-8 hierarchical topics, wildcards
the landscape
PUB/SUB cauldron
topic names UTF-8, hierarchical, wildcards home/ground-floor/kitchen/kettle finance/eur/rate finance/+/rate 14dfa2e2-d580-4574-88ff-dcc120330482 cellar/stairlamp/cmd cellar/stairlamp/status owntracks/jpm/5s/event
 owntracks/jpm/# openhab/homie/5ccf7faac88e/$stats/uptime
MQTT brokers the server bit of MQTT
Mosquitto C, fast, lightweight, ACLs (plugin),TLS,TLS-PSK, bridge, logging via $SYS http://mosquitto.org
VerneMQ Erlang, Websockets, clustering, file, SQL & Redis authentication, Lua plugins, Webhooks http://vernemq.com
more brokers RSMB, Mosca, Apollo, HiveMQ, (RabbitMQ)
bridging
CLI utilities mosquitto_sub [-h localhost] [-p 1883] [--cafile file] [--cert file --key file] [-u username [-P password]]
 -v -t 'topic/#' subscribe publish mosquitto_pub ... [-r]
 -t topic -m message
Language bindings C, C++, Clojure, Dart, Delphi, Erlang, Elixir, Go, Haskell, Java, JavaScript, LotusScript, Lua, .NET, Objective-C, OCaml, Perl, PHP, Python, REXX, Ruby, Smalltalk, Swift, Tcl, …
 
 COBOL
Python API: PUB #!/usr/bin/env python import paho.mqtt.publish as mqtt mqtt.single('conf/hello', 'Hello MQTT') $ mosquitto_sub -h localhost -v -t 'conf/#' conf/hello Hello MQTT payloadtopic
Python API: SUB callbacks #!/usr/bin/env python import paho.mqtt.client as paho def on_connect(mosq, userdata, flags, rc): mqttc.subscribe("conf/+", 0) def on_message(mosq, userdata, msg): print "%s %s" % (msg.topic, str(msg.payload)) mqttc = paho.Client(userdata=None) mqttc.on_connect = on_connect mqttc.on_message = on_message mqttc.connect("localhost", 1883, 60) mqttc.loop_forever()
Python API: SUB $ mosquitto_pub -t 'conf/thirsty' -m ‘Beer time?' $ mosquitto_pub -t 'conf/catering' -m 'Coffee is ready' $ ./sub.py conf/thirsty Beer time? conf/catering Coffee is ready
libmosquitto #include <stdio.h> #include <string.h> #include <mosquitto.h> #define MESSAGE "Goodbye, cruel world" int main(int argc, char *argv[]) { struct mosquitto *mosq; mosquitto_lib_init(); if ((mosq = mosquitto_new(NULL, true, NULL)) == NULL) { return fprintf(stderr, "Error: Out of memory.n"); } if (mosquitto_connect(mosq, "192.168.1.130", 1883, 60) != 0) { return fprintf(stderr, "Unable to connect to MQTT brokern"); } mosquitto_publish(mosq, NULL, /* mid */ "message/adieu", /* topic */ strlen(MESSAGE), /* payload length */ MESSAGE, /* payload */ 1, /* qos */ false); /* retain */ mosquitto_loop(mosq, -1, 1); mosquitto_disconnect(mosq); mosquitto_destroy(mosq); mosquitto_lib_cleanup(); return (0); }
job monitor, reporting https://gist.github.com/jpmens/7101170 $ mosquitto_sub -v -t 'processes/#' processes/run.sh Starting processes/monitor/spec1 Starting processes/run.sh Still going strong at Tue Oct 22 15:49:07 CEST 2013 processes/run.sh That's it, folks! #!/bin/sh
 topic="processes/$(basename $0)" mqtt_opts="--quiet -h 192.168.1.130 -p 1883" mqtt() { mosquitto_pub ${mqtt_opts} -t "${topic}" -m "$*" || true } mqtt "Starting"
tracking logins (1) https://jpmens.net/2018/03/25/alerting-on-ssh-logins/
tracking logins (2) https://jpmens.net/2018/03/25/alerting-on-ssh-logins/ $ mosquitto_sub -v -t 'logging/#' -F '%I %J' 2019-03-14T10:19:54+0000 { "tst": 1552558794, "topic": "logging/hare", "qos": 0, "retain": 0, "payloadlen": 130, "payload": { "hostname": "canfb12", "remote": "192.168.33.123", "rhost": "192.168.33.1", "service": "sshd", "tst": 1552562392, "tty": null, "user": "jane" } }
tracking logins (3) https://dan.langille.org/2018/04/15/using-mtqq-to-create-a-notification-network-mosquitto-mqttwarn-hare-and-hared/ Date: Thu, 14 Mar 2019 11:19:54 +0100 From: MQTTwarn <jpm@localhost> Subject: SSH login on canfb12 X-Mailer: mqttwarn login via sshd by jane on canfb12 from 192.168.33.1 at 2019-03-14 12:19:52
For the sysadmin
Your things speak MQTT
Last Will & Testament #!/usr/bin/env python # -*- coding: utf-8 -*- import paho.mqtt.subscribe as subscribe import os def on_message(client, userdata, m): print("%s %s" % (m.topic, m.payload)) lwt = { "topic" : "clients/{0}".format(os.path.basename(__file__)), "payload": "I am no longer" } subscribe.callback(on_message, "test/+", hostname="localhost", will=lwt)
MQTT in the wild Graylog, beaver, Ansible, RabbitMQ, collectd, openHAB, Github, Wireshark, Flukso, RemakeElectric, Jenkins, Diamond, OwnTracks, Telegraf
mqtt.org @mqttorg

More Related Content

PPTX
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
PROIDEA
 
PDF
FalsyValues. Dmitry Soshnikov - ECMAScript 6
Dmitry Soshnikov
 
PDF
Compose Async with RxJS
Kyung Yeol Kim
 
ODP
Intravert Server side processing for Cassandra
Edward Capriolo
 
PDF
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
Sages
 
KEY
連邦の白いヤツ 「Objective-C」
matuura_core
 
PDF
Introdução ao Desenvolvimento Android com Kotlin
Nelson Glauber Leal
 
PDF
Cascadia.js: Don't Cross the Streams
mattpodwysocki
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
PROIDEA
 
FalsyValues. Dmitry Soshnikov - ECMAScript 6
Dmitry Soshnikov
 
Compose Async with RxJS
Kyung Yeol Kim
 
Intravert Server side processing for Cassandra
Edward Capriolo
 
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
Sages
 
連邦の白いヤツ 「Objective-C」
matuura_core
 
Introdução ao Desenvolvimento Android com Kotlin
Nelson Glauber Leal
 
Cascadia.js: Don't Cross the Streams
mattpodwysocki
 

What's hot (19)

PPTX
ES6 is Nigh
Domenic Denicola
 
PDF
Think Async: Asynchronous Patterns in NodeJS
Adam L Barrett
 
PPTX
C#을 이용한 task 병렬화와 비동기 패턴
명신 김
 
PDF
Construire une application JavaFX 8 avec gradle
Thierry Wasylczenko
 
PDF
Wprowadzenie do technologi Big Data i Apache Hadoop
Sages
 
PPTX
Down to Stack Traces, up from Heap Dumps
Andrei Pangin
 
PDF
Новый InterSystems: open-source, митапы, хакатоны
Timur Safin
 
PDF
Wprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
Sages
 
PDF
Profiling Ruby
Ian Pointer
 
PDF
Lean React - Patterns for High Performance [ploneconf2017]
Devon Bernard
 
PDF
Node 관계형 데이터베이스_바인딩
HyeonSeok Choi
 
PPTX
The zen of async: Best practices for best performance
Microsoft Developer Network (MSDN) - Belgium and Luxembourg
 
PDF
What is nodejs
JeongHun Byeon
 
PPTX
The State of JavaScript (2015)
Domenic Denicola
 
PDF
Nginx-lua
Дэв Тим Афс
 
PDF
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
PDF
rsyslog v8: more than just syslog!
Yury Bushmelev
 
PDF
Grails 1.2 探検隊 -新たな聖杯をもとめて・・・-
Tsuyoshi Yamamoto
 
PPTX
JPoint 2016 - Валеев Тагир - Странности Stream API
tvaleev
 
ES6 is Nigh
Domenic Denicola
 
Think Async: Asynchronous Patterns in NodeJS
Adam L Barrett
 
C#을 이용한 task 병렬화와 비동기 패턴
명신 김
 
Construire une application JavaFX 8 avec gradle
Thierry Wasylczenko
 
Wprowadzenie do technologi Big Data i Apache Hadoop
Sages
 
Down to Stack Traces, up from Heap Dumps
Andrei Pangin
 
Новый InterSystems: open-source, митапы, хакатоны
Timur Safin
 
Wprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
Sages
 
Profiling Ruby
Ian Pointer
 
Lean React - Patterns for High Performance [ploneconf2017]
Devon Bernard
 
Node 관계형 데이터베이스_바인딩
HyeonSeok Choi
 
The zen of async: Best practices for best performance
Microsoft Developer Network (MSDN) - Belgium and Luxembourg
 
What is nodejs
JeongHun Byeon
 
The State of JavaScript (2015)
Domenic Denicola
 
Nginx-lua
Дэв Тим Афс
 
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
rsyslog v8: more than just syslog!
Yury Bushmelev
 
Grails 1.2 探検隊 -新たな聖杯をもとめて・・・-
Tsuyoshi Yamamoto
 
JPoint 2016 - Валеев Тагир - Странности Stream API
tvaleev
 
Ad

Similar to Zabbix LLD from a C Module by Jan-Piet Mens (20)

ODP
Writing MySQL UDFs
Roland Bouman
 
PDF
Server Side JavaScript - You ain't seen nothing yet
Tom Croucher
 
PDF
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
PDF
CouchDB Mobile - From Couch to 5K in 1 Hour
Peter Friese
 
PPTX
Introduction to Apache Mesos
Joe Stein
 
PDF
ESCMAScript 6: Get Ready For The Future. Now
Krzysztof Szafranek
 
PDF
Saltstack - Orchestration & Application Deployment
inovex GmbH
 
PPTX
Hadoop cluster performance profiler
Ihor Bobak
 
PDF
Fun Teaching MongoDB New Tricks
MongoDB
 
KEY
Introducing the Seneca MVP framework for Node.js
Richard Rodger
 
KEY
20120816 nodejsdublin
Richard Rodger
 
PDF
10 Key MongoDB Performance Indicators
iammutex
 
ODP
Slickdemo
Knoldus Inc.
 
PPTX
How Secure Are Docker Containers?
Ben Hall
 
KEY
JavaScript Growing Up
David Padbury
 
PDF
External Language Stored Procedures for MySQL
Antony T Curtis
 
PDF
Presto anatomy
Dongmin Yu
 
PDF
Applying profilers to my sql (fosdem 2017)
Valeriy Kravchuk
 
PDF
soft-shake.ch - Hands on Node.js
soft-shake.ch
 
PDF
Catalyst MVC
Sheeju Alex
 
Writing MySQL UDFs
Roland Bouman
 
Server Side JavaScript - You ain't seen nothing yet
Tom Croucher
 
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
CouchDB Mobile - From Couch to 5K in 1 Hour
Peter Friese
 
Introduction to Apache Mesos
Joe Stein
 
ESCMAScript 6: Get Ready For The Future. Now
Krzysztof Szafranek
 
Saltstack - Orchestration & Application Deployment
inovex GmbH
 
Hadoop cluster performance profiler
Ihor Bobak
 
Fun Teaching MongoDB New Tricks
MongoDB
 
Introducing the Seneca MVP framework for Node.js
Richard Rodger
 
20120816 nodejsdublin
Richard Rodger
 
10 Key MongoDB Performance Indicators
iammutex
 
Slickdemo
Knoldus Inc.
 
How Secure Are Docker Containers?
Ben Hall
 
JavaScript Growing Up
David Padbury
 
External Language Stored Procedures for MySQL
Antony T Curtis
 
Presto anatomy
Dongmin Yu
 
Applying profilers to my sql (fosdem 2017)
Valeriy Kravchuk
 
soft-shake.ch - Hands on Node.js
soft-shake.ch
 
Catalyst MVC
Sheeju Alex
 
Ad

Recently uploaded (20)

PPTX
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PPTX
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PPTX
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PDF
System and Network Administration Chapter 2
jaramharo
 
PDF
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
PDF
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
Introduction Database Management System for Course Database
ReinertYosua
 
System and Network Administration Chapter 2
jaramharo
 
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 

Zabbix LLD from a C Module by Jan-Piet Mens

  • 1. Zabbix loadable C modules and Low-Level Discovery (LLD) Jan-Piet Mens OSMC, November 2019 @jpmens
  • 2. @jpmens: consultant, part-time admin, trainer, small-scale fiddler, loves plain text, and things which work. Contributes to Ansible, dreamed up OwnTracks, and chases bugs in open source DNS servers.
  • 4. Zabbix cannot, out of the box, support every possible thing you may want to monitor; let's extend it. extensible
  • 5. extending Zabbix system.run[] User Parameters External checks Loadable modules Beware fork(), invalid results: unsupported items
  • 6. system.run[] # agent EnableRemoteCommands=1 LogRemoteCommands=1 76775:20190326:173402.315 Executing command '/usr/local/bin/zdisc.sh one two three' #!/bin/sh echo $# Runs on agent
  • 7. user parameters #!/bin/sh echo 16 exit 0 # agent
 UserParameter=jjolie.cn[*],echo "$1 Jolie" UserParameter=jjolie.uid,/usr/local/bin/uidnumber.sh Runs on agent
  • 8. external checks Runs on server # server
 ExternalScripts=/usr/lib/zabbix/externalscripts
  • 9. loadable modules Written in the C language, compiled, loaded dynamically by server, proxy, and/or agent
  • 10. what can modules do? Basically anything: sockets, SystemV IPC, files, databases, IoT, sensors, … Beware permissions, context, errors
  • 11. custom module # server, proxy, agent LoadModulePath=/usr/lib/zabbix/agent
 LoadModule=owntracks-module.so
 LoadModule=two.so
  • 12. make install Build inside / outside source tree owntracks-module.so: owntracks-module.c gcc -fPIC -shared -o owntracks-module.so owntracks-module.c install owntracks-module.so /usr/lib/zabbix/agent/
  • 13. module.h --- zabbix-4.0.5/include/module.h 2019-02-25 10:36:39.000000000 +0100 +++ module.h 2019-03-26 15:03:59.000000000 +0100 @@ -20,7 +20,8 @@ #ifndef ZABBIX_MODULE_H #define ZABBIX_MODULE_H -#include "zbxtypes.h" +#include <inttypes.h> +#define zbx_uint64_t uint64_t #define ZBX_MODULE_OK 0 #define ZBX_MODULE_FAIL -1
  • 14. the API int zbx_module_api_version(void) int zbx_module_init(void) ZBX_METRIC *zbx_module_item_list(void) void zbx_module_item_timeout(int timeout) ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) int zbx_module_uninit(void)
  • 15. returning a value #define SET_UI64_RESULT(res,val) #define SET_DBL_RESULT(res, val) #define SET_STR_RESULT(res, val) /* xx */ #define SET_TEXT_RESULT(res,val) /* xx */ #define SET_LOG_RESULT(res, val) /* xx */ #define SET_MSG_RESULT(res, val) /* xx */ /* xx: ALWAYS allocate new memory for val! * DON'T USE STATIC OR STACK MEMORY!!! */
  • 16. our C module #include "module.h" /* return the version number of this module interface. */ int zbx_module_api_version(void) { return (ZBX_MODULE_API_VERSION); }
  • 17. initialize /* agent start-up; Zabbix will fail to load if we return ZBX_MODULE_FAIL */ int zbx_module_init(void) { if ((db = db_open()) == NULL) { return (ZBX_MODULE_FAIL); } return (ZBX_MODULE_OK); } /* agent is shutting down */ int zbx_module_uninit(void) { db_close(db); return (ZBX_MODULE_OK); }
  • 18. key to function map static ZBX_METRIC item_keys[] = { /* ITEM.KEY FLAG FUNCTION TEST PARAMETERS */ { "owntracks.ping", 0, owntracks_ping, NULL}, { "owntracks.lld", 0, owntracks_lld, NULL}, { "owntracks.vel", CF_HAVEPARAMS, owntracks_vel, "1234567890012345"}, { NULL } }; /* return list of item keys supported by this module */ ZBX_METRIC *zbx_module_item_list(void) { return (item_keys); }
  • 19. owntracks.ping $ zabbix_get -s 127.0.0.1 -k owntracks.ping 42 /* 42. what else? */ static int owntracks_ping(AGENT_REQUEST *request, AGENT_RESULT *result) { SET_UI64_RESULT(result, 42); return (SYSINFO_RET_OK); }
  • 20. frontend items If our module is loaded into a server, the item type must be Simple Check. If our module is loaded into an agent, the item type must be Zabbix agent or Zabbix agent (active).
  • 21. programatically create item from zabbix_api import ZabbixAPI # https://pypi.org/project/zabbix-api/ import json zapi = ZabbixAPI(server="http://server/zabbix/") zapi.login("admin", "secret") res = zapi.item.create({ "name" : "The pong", "key_" : "owntracks.ping", "hostid" : 10268, "interfaceid" : "11", "type" : 0, # Zabbix agent "value_type" : 3, # numeric, unsigned "delay" : "60s", })
  • 22. owntracks.vel static int owntracks_vel(AGENT_REQUEST *request, AGENT_RESULT *result) { unsigned long retval; if (request->nparam != 1) { SET_MSG_RESULT(result, strdup("Expecting 1 parameter with IMEI")); return (SYSINFO_RET_FAIL); } retval = db_get(db, get_rparam(request, 0)); SET_UI64_RESULT(result, retval); return (SYSINFO_RET_OK); }
  • 24. What is LLD? Low-Level Discovery, widely used, built-in methods* can create items, triggers, graphs, custom LLD, send JSON to Zabbix server LLD rules discover entities, not item values * network interfaces, CPUs, SNMP OIDs, JMX objects, ODBC queries, Windows services, file systems, host interfaces
  • 25. how do we use LLD? (1)
  • 26. how do we use LLD? (2) What does [process] do? • subscribe to MQTT • extract OwnTracks' payload • update device database (store values) => LLD • device not seen or dead, i.e.TTL expired?
 remove from database => LLD • device details (e.g. speed) add to database: owntracks.vel() reads it • spikes trigger Zabbix alerts • 120 lines of Python
  • 27. how do we use LLD? (3) Why LMDB? • in-process • blindingly fast • single writer (our [process]) • multiple readers (functions in our loadable C module) • no external moving parts • multiple language bindings (C, Python, etc…)
  • 28. OwnTracks payloads { "_type": "location", "tst": 1553542948, "lat": 59.373399, "lon": 17.94669, "cog": 280, "alt": 101, "vel": 15, "tid": "xq", "batt": 89, "name": "20-ER-20" } https://owntracks.org/booklet/tech/json/ owntracks/zbx/012549656802107 topic payload
  • 29. defining LLD Define discovery rule.Text item receives JSON. Prototypes create items and triggers.
  • 30. trapper item $ zabbix_sender -z 192.168.1.53 -s AirJP -k my.item -o '{ "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }'
  • 31. LLD JSON format $ zabbix_get -s 127.0.0.1 -k owntracks.lld { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
  • 32. owntracks.lld static int owntracks_lld(AGENT_REQUEST *request, AGENT_RESULT *result) { JsonNode *obj = json_mkobject(), *list = json_mkarray(), *o; char *js; while ((o = db_getnext(db)) != NULL) { json_append_element(list, o); } json_append_member(obj, "data", list); if ((js = json_stringify(obj, " ")) == NULL) { SET_MSG_RESULT(result, strdup("Cannot stringify JSON.")); return (SYSINFO_RET_FAIL); } SET_STR_RESULT(result, js); /* Zabbix will free() this */ json_delete(obj); return (SYSINFO_RET_OK); }
  • 34. item prototypes { "data": [ { "{#OT.IMEI}": "012549656802107", "{#OT.PLATE}": "20-ER-20" } ] }
  • 40. history export (1) Server only, quite complicated, invoked at server start and then occasionally, badly documented, replaceable with real-time export into JSON files.
  • 41. static void history_cb_integer(const ZBX_HISTORY_INTEGER *history, int history_count) { int n; itemid, val, clock; for (n = 0; n < history_count; n++) { itemid = history[n].itemid; clock = history[n].clock; val = history[n].value; ... } } ZBX_HISTORY_WRITE_CBS zbx_module_history_write_cbs(void) { static ZBX_HISTORY_WRITE_CBS my_callbacks = { NULL, /* my_history_float_cb */ history_cb_integer, /* my_history_integer_cb */ history_cb_string, /* my_history_string_cb */ NULL, /* my_history_text_cb */ NULL, /* my_history_log_cb */ }; return (my_callbacks); }
  • 42. history export (2) { "host": "airjp", "groups": [ "Linux servers" ], "applications": [], "itemid": 28530, "name": "jjolie.cn", "clock": 1553636580, "ns": 518541363, "value": "Jane Jolie" } { "host": "Zabbix server", "groups": [ "Zabbix servers" ], "applications": [ "Filesystems" ], "itemid": 28506, "name": "Free disk space on / (percentage)", "clock": 1553636586, "ns": 523296311, "value": 70.684192 }
  • 43. low-level recipe write and compile module, edit configuration, restart agent, server, or proxy, create items, test, enjoy
  • 44. buywhy.gif C functions are fast. LLD is highly flexible. Together this is a powerful combination. Unfortunately the API has no support for reading zabbix.conf
 Documentation is minimal.
  • 46. Have you heard of
 MQTT?
  • 47. MQTT is MQTT is a standard, a transport, for PUB/SUB messaging, designed for unreliable networks, binary payloads up to 256MB, (+2 bytes), fast, lightweight, ideal for low-bandwith, high- latency networks, TLS, authentication, ACLs,TLS-PSK, (payload encryption), keepalive, last will & testament, UTF-8 hierarchical topics, wildcards
  • 50. topic names UTF-8, hierarchical, wildcards home/ground-floor/kitchen/kettle finance/eur/rate finance/+/rate 14dfa2e2-d580-4574-88ff-dcc120330482 cellar/stairlamp/cmd cellar/stairlamp/status owntracks/jpm/5s/event
 owntracks/jpm/# openhab/homie/5ccf7faac88e/$stats/uptime
  • 51. MQTT brokers the server bit of MQTT
  • 52. Mosquitto C, fast, lightweight, ACLs (plugin),TLS,TLS-PSK, bridge, logging via $SYS http://mosquitto.org
  • 53. VerneMQ Erlang, Websockets, clustering, file, SQL & Redis authentication, Lua plugins, Webhooks http://vernemq.com
  • 54. more brokers RSMB, Mosca, Apollo, HiveMQ, (RabbitMQ)
  • 56. CLI utilities mosquitto_sub [-h localhost] [-p 1883] [--cafile file] [--cert file --key file] [-u username [-P password]]
 -v -t 'topic/#' subscribe publish mosquitto_pub ... [-r]
 -t topic -m message
  • 57. Language bindings C, C++, Clojure, Dart, Delphi, Erlang, Elixir, Go, Haskell, Java, JavaScript, LotusScript, Lua, .NET, Objective-C, OCaml, Perl, PHP, Python, REXX, Ruby, Smalltalk, Swift, Tcl, …
 
 COBOL
  • 58. Python API: PUB #!/usr/bin/env python import paho.mqtt.publish as mqtt mqtt.single('conf/hello', 'Hello MQTT') $ mosquitto_sub -h localhost -v -t 'conf/#' conf/hello Hello MQTT payloadtopic
  • 59. Python API: SUB callbacks #!/usr/bin/env python import paho.mqtt.client as paho def on_connect(mosq, userdata, flags, rc): mqttc.subscribe("conf/+", 0) def on_message(mosq, userdata, msg): print "%s %s" % (msg.topic, str(msg.payload)) mqttc = paho.Client(userdata=None) mqttc.on_connect = on_connect mqttc.on_message = on_message mqttc.connect("localhost", 1883, 60) mqttc.loop_forever()
  • 60. Python API: SUB $ mosquitto_pub -t 'conf/thirsty' -m ‘Beer time?' $ mosquitto_pub -t 'conf/catering' -m 'Coffee is ready' $ ./sub.py conf/thirsty Beer time? conf/catering Coffee is ready
  • 61. libmosquitto #include <stdio.h> #include <string.h> #include <mosquitto.h> #define MESSAGE "Goodbye, cruel world" int main(int argc, char *argv[]) { struct mosquitto *mosq; mosquitto_lib_init(); if ((mosq = mosquitto_new(NULL, true, NULL)) == NULL) { return fprintf(stderr, "Error: Out of memory.n"); } if (mosquitto_connect(mosq, "192.168.1.130", 1883, 60) != 0) { return fprintf(stderr, "Unable to connect to MQTT brokern"); } mosquitto_publish(mosq, NULL, /* mid */ "message/adieu", /* topic */ strlen(MESSAGE), /* payload length */ MESSAGE, /* payload */ 1, /* qos */ false); /* retain */ mosquitto_loop(mosq, -1, 1); mosquitto_disconnect(mosq); mosquitto_destroy(mosq); mosquitto_lib_cleanup(); return (0); }
  • 62. job monitor, reporting https://gist.github.com/jpmens/7101170 $ mosquitto_sub -v -t 'processes/#' processes/run.sh Starting processes/monitor/spec1 Starting processes/run.sh Still going strong at Tue Oct 22 15:49:07 CEST 2013 processes/run.sh That's it, folks! #!/bin/sh
 topic="processes/$(basename $0)" mqtt_opts="--quiet -h 192.168.1.130 -p 1883" mqtt() { mosquitto_pub ${mqtt_opts} -t "${topic}" -m "$*" || true } mqtt "Starting"
  • 64. tracking logins (2) https://jpmens.net/2018/03/25/alerting-on-ssh-logins/ $ mosquitto_sub -v -t 'logging/#' -F '%I %J' 2019-03-14T10:19:54+0000 { "tst": 1552558794, "topic": "logging/hare", "qos": 0, "retain": 0, "payloadlen": 130, "payload": { "hostname": "canfb12", "remote": "192.168.33.123", "rhost": "192.168.33.1", "service": "sshd", "tst": 1552562392, "tty": null, "user": "jane" } }
  • 65. tracking logins (3) https://dan.langille.org/2018/04/15/using-mtqq-to-create-a-notification-network-mosquitto-mqttwarn-hare-and-hared/ Date: Thu, 14 Mar 2019 11:19:54 +0100 From: MQTTwarn <jpm@localhost> Subject: SSH login on canfb12 X-Mailer: mqttwarn login via sshd by jane on canfb12 from 192.168.33.1 at 2019-03-14 12:19:52
  • 68. Last Will & Testament #!/usr/bin/env python # -*- coding: utf-8 -*- import paho.mqtt.subscribe as subscribe import os def on_message(client, userdata, m): print("%s %s" % (m.topic, m.payload)) lwt = { "topic" : "clients/{0}".format(os.path.basename(__file__)), "payload": "I am no longer" } subscribe.callback(on_message, "test/+", hostname="localhost", will=lwt)
  • 69. MQTT in the wild Graylog, beaver, Ansible, RabbitMQ, collectd, openHAB, Github, Wireshark, Flukso, RemakeElectric, Jenkins, Diamond, OwnTracks, Telegraf