Writing External Rsyslog Plugins
6 likes16,054 views
The document explains how to write external output plugins for rsyslog, allowing connections to various log destinations using different programming languages. It outlines the design goals, communication methods, threading considerations, and provides examples such as plugin skeletons in Python. The document encourages users to create plugins easily, using provided skeletons or simple stdin handling.
Writing External Rsyslog Plugins
- 1. Writing external output plugins for rsyslog Rainer Gerhards
- 2. Output plugins in rsyslog • Output plugins permit to connect to a variety of log destinations • Pre-v8, plugins needed to be internal modules ▫ Loaded into rsyslog process space ▫ Must be written in C (or link to it) ▫ Deeply tied into rsyslog infrastructure • with v8, we officially support external plugins ▫ Written in any language (Python, Perl, Java, ...) ▫ Have their own process ▫ Relatively de-coupled Rainer Gerhards, http://blog.gerhards.net
- 3. Interface Overview process border Internal plugin rsyslog core engine perl plugin external plugin connector python plugin Rainer Gerhards, http://blog.gerhards.net
- 4. External Interface Design Goals • Keep it stupid simple ▫ Must support almost any language ▫ Dumb easy to use even for novice programmer ▫ Do not require explicit threading • Speed is NOT the most important goal ▫ Don't make it unnecessarily slow ▫ Many real-world log destinations are slow in any case (like when you connect via http...) ▫ Focus on “enable to build solution” ▫ If necessary, conversion to internal module can be done later Rainer Gerhards, http://blog.gerhards.net
- 5. Interface Details: communication • use pipes • stdin ▫ one message per line ▫ format can be customized via rsyslog templates ▫ multi-line messags via JSON • stdout/stderr ▫ Must NOT be written in initial version ▫ Will later convey back state information via plain text (e.g. “ERR”, “ERRMSG:xxx”, ...) • JSON as input format is recommended Rainer Gerhards, http://blog.gerhards.net
- 6. Interface Details: Threading • Do NOT care about threading • Write app according to single-thread paradigm • rsyslog will spawn multiple instances of your plugin if there is need to do so ▫ ▫ ▫ ▫ Happens based on config in busy cases Works well in most cases (e.g. http connects) Can be disabled if necessary If your program can run in multiple ter-minal sessions concurrently, it can also be run as multiple rsyslog action instances. Rainer Gerhards, http://blog.gerhards.net
- 7. Startup & Termination • • • • • • rsyslog will startup the plugin automatically Plugin needs to read stdin until EOF Do NOT terminate before EOF is reached On EOF, cleanup and terminate If the plugin dies, rsyslog restarts a new instance Some signals (like sigint) are blocked and should remain so Rainer Gerhards, http://blog.gerhards.net
- 8. Skeletons • The rsyslog project provides sample plugin skeletons • Available in ./plugins/external/skeletons • These contain ▫ the necessary plumbing ▫ often a kind of abstraction layer to make writing plugins even easier ▫ often performance-enhancement features • Can simply be copied to create your own plugins, don't care about the (minimal) plumbing! Rainer Gerhards, http://blog.gerhards.net
- 9. Example: Python Skeleton • Handles all interface plumbing • Uses “eventHandlers” to call the acutal app coding • Is used as a basis, for example, for the Solr output plugin • Available at https://github.com/rsyslog/rsyslog/blob/master/plugin Rainer Gerhards, http://blog.gerhards.net
- 10. onInit() Handler • Called when the plugin is initially loaded • Ready outbound connections, files, etc... def onInit(): """ Do everything that is needed to initialize processing (e.g. open files, create handles, connect to systems...) """ global outfile outfile = open("/tmp/logfile", "w") Rainer Gerhards, http://blog.gerhards.net
- 11. onReceive() • Called when new messages arrive • Receives one or more message via Python list object --> e.g. place them into one transaction, HTTP request, ... def onReceive(msgs): """This is the entry point where actual work needs to be done. It receives a list with all messages pulled from rsyslog. The list is of variable length, but contains all messages that are currently available. It is suggest NOT to use any further buffering, as we do not know when the next message will arrive. It may be in a nanosecond from now, but it may also be in three hours... """ global outfile for msg in msgs: outfile.write(msg) Rainer Gerhards, http://blog.gerhards.net
- 12. onExit() • Called immediately before the plugin terminates • Gurantees that no more messages arrive • Used to cleanup connections, write some final records, ... def onExit(): """ Do everything that is needed to finish processing (e.g. close files, handles, disconnect from systems...). This is being called immediately before exiting. """ global outfile outfile.close() Rainer Gerhards, http://blog.gerhards.net
- 13. Call to Action • If you need to send logs to a destination that is not yet supported, you can quickly write an external plugin – in any language you know! • Writing rsyslog plugins is easy ▫ If there is already a skeleton for your language, copy it and add your app-specific code ▫ If not ... no problem, the interface is dumb easy If you can write a script that reads stdin and does something useful with it, you can also write a rsyslog plugin! Rainer Gerhards, http://blog.gerhards.net