Finding OOMS in Legacy Systems with the Syslog Telegraf Plugin
0 likes821 views
The document discusses the use of Telegraf, an open-source server agent for collecting and reporting metrics, in conjunction with a tick stack layout for log data management. It covers configurations for syslog and Telegraf, techniques for sanitizing log data, and using Kapacitor for advanced data querying and analysis. The presentation also delves into methods for optimizing Telegraf against memory management issues, specifically addressing the OOM killer in server environments.
![Armouring Telegraf Against the OOM Killer
[Service]
OOMScoreAdjust=-600
/etc/systemd/system/telegraf.service.d/oom_score_adj.conf
value between -1000 and +1000/proc/$PID/oom_score_adj
/proc/$PID/oom__adj value between -17 and +15](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-13-320.jpg)
![Telegraf Config
[[inputs.syslog]]
server = "tcp://:6514"
[[outputs.influxdb]]
urls = [ "http://influxdb-syslog.service.consul:8086" ]
database = "syslog"
retention_policy = "autogen"
precision = "ns"
user_agent = "telegraf"
namepass = ["syslog"]](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-14-320.jpg)
![Enable Counters
# Create per-minute counts of syslog data
[[aggregators.basicstats]]
period = "1m"
drop_original = false
stats = ["count"]
name_suffix = "_counts"
# filtering
namepass = ["syslog"]](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-20-320.jpg)
![Adding More Tags From Your Log Data
[[processors.parser]]
parse_fields = ["message"]
drop_original = false
merge = "override"
data_format = "json"
tag_keys = ["method","uri"]
namepass = ["syslog"]
[processors.parser.tagpass]
appname = ["crazy-cool-api"]
{
"hostname": "ab704bf3336f" ,
"ip": "10.0.0.30",
"method": "GET",
"msg": "request completed" ,
"start": "2018-10-23T07:00:03Z" ,
"status": 200,
"time": "2018-10-23T07:00:03.942327Z" ,
"uri": "/health",
"uri_query": "",
"user_agent": "Consul Health Check"
}
Log Message Telegraf Config](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-23-320.jpg)
![var ipv4_address = /d{1,3}.d{1,3}.d{1,3}.d{1,3}/
|eval(lambda: regexReplace(ipv4_address, "message", '<ipv4-address>'))
.as('message')
.keep()
Rewriting your data with regexReplace
var email_address = /w+([-+.']w+)*@w+([-.]w+)*.w+([-.]w+)*/
|eval(lambda: regexReplace(email_address, "message", '<email-address>'))
.as('message')
.keep(
Email Addresses
IP Addresses](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-26-320.jpg)
![Email Address Regex (RFC 5322)
(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[
x01-x08x0bx0cx0e-x1fx21x23-x5bx5d-x7f]|[x01-x09x0bx0cx0e
-x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*
[a-z0-9])?|[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|
2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[x01-x08x0bx0cx0e
-x1fx21-x5ax53-x7f]|[x01-x09x0bx0cx0e-x7f])+)])](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-27-320.jpg)
![Email Address Regex / Pain Amplifier
(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?
[t])*))*>(?:(?:rn)?[t])*)|(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:
rn)?[t])*)*:(?:(?:rn)?[t])*(?:(?:(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"
(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]
))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn
)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*|
(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)*<(?:(?:rn)?[t]
)*(?:@(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^(
)<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*(?:,@(?:(?:rn)?[t])*(?:[^()<>@,;:"
.[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-0
31]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*)*:(?:(?:rn)?[t])*)?(?:[^()<>@,;:".[]000-031]+(?:(?:
(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?
:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+
(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:r
n)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*>(?:(?:rn)?[t])*)(?:,s*(?:(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)
?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:
rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?
:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])
+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*|(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|
"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)*<(?:(?:rn)?[t])*(?:@(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[
]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^
[]r]|.)*](?:(?:rn)?[t])*))*(?:,@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|
.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:
rn)?[t])*))*)*:(?:(?:rn)?[t])*)?(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"
(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]
))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn
)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*
>(?:(?:rn)?[t])*))*)?;s*)](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-28-320.jpg)
![Tick Template
var where_filter = lambda: TRUE
var appname_lambda lambda
var event_type string
var data = stream
|from()
.database('syslog')
.retentionPolicy('autogen')
.measurement('syslog')
|where(lambda: "facility" == 'kern')
|where(where_filter)
|eval(appname_lambda)
.as('appname')
.tags('appname')
.keep()
|eval(lambda: regexReplace(/w+([-+.']w+)*@w+([-.]w+)*.w+([-.]w+)*/, "message", '<email-address>'))
.as('message')
.keep()
|eval(lambda: unixNano("time") - "timestamp")
.as('timestamp_lag_ns')
.keep()
|log()
/etc/kapacitor/load/templates/template-syslog_kern_service_events.tick](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-30-320.jpg)
![Task Definition : OOM
template-id: template-syslog_kern_service_events
vars:
Where_filter:
type: lambda
value: >
"message" =~ /.+ Memory cgroup out of memory: Kill process .+/
appname_lambda:
type: lambda
value: >
regexReplace(/^[.+] Memory cgroup out of memory: Kill process .+? ((.+?)) score .+/, "message", '$1')
event_type:
type: string
value: "OOM"
/etc/kapacitor/load/tasks/syslog_kern_service_events-OOM.yaml](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-31-320.jpg)
![Task Definition : Segfault
/etc/kapacitor/load/tasks/syslog_kern_service_events-segfault.yaml
template-id: template-syslog_kern_service_events
vars:
Where_filter:
type: lambda
value: >
"message" =~ /.+: segfault at .+/
appname_lambda:
type: lambda
value: >
regexReplace(/^[.+] (.+?)[.+]: segfault at .*/, "message", '$1')
event_type:
type: string
value: "segfault"](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-32-320.jpg)
![metric_relabel_configs:
- source_labels: ['name']
regex: '([[:ascii:]]{1,15}).*)'
target_label: 'name_short'
replacement: '$1'
Linkage Workarounds
Prometheus
[[processors.regex]]
[[processors.regex.tags]]
key = "<tag>"
pattern = "^([[:ascii:]]{1,15}).*$"
replacement = "${1}"
result_key = "short_<tag>"
Telegraf](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-34-320.jpg)
![Group By Using A Custom Variable
GROUP BY time(1m), "[[group]]" fill(0)](https://image.slidesharecdn.com/findingoomswithtelegraf-final-181110202538/85/Finding-OOMS-in-Legacy-Systems-with-the-Syslog-Telegraf-Plugin-38-320.jpg)
Finding OOMS in Legacy Systems with the Syslog Telegraf Plugin
- 1. Finding OOMs with Telegraf Dylan Ferreira @dylanferreira
- 3. Agenda ● A whole bunch of dry stuff ● Maybe snacks?
- 4. Agenda.real ● Our TICK stack layout ● Configuring Rsyslog & Telegraf ● Doing more with Telegraf plugins ● Using Kapacitor to search & sanitize log data ● Exploring this data with Grafana
- 5. How did we get here? ● Multi-dimensional TSDB. ● Very high performance. ● All data is aggregated on ingest. ● Can store raw metrics. ● Multi-dimensional TSDB. ● Complex aggregations (CQs). ● Multiple fields! ● Can store raw metrics. ● Flat structure with metadata stored in the metric name (pre-v1). ● Fixed per-series aggregations
- 6. What is Telegraf? “Telegraf is InfluxData's open source plugin-driven server agent for collecting and reporting metrics.” ● Inputs ● Processors ● Aggregators ● Outputs ● Stream Buffer ●● Metrics Router
- 8. Applications Host Setup Applications sending log data to syslog Syslog relaying log data to Telegraf
- 12. The OOM Killer /** * oom_badness - heuristic function to determine which candidate task to kill * @p: task struct of which task we should calculate * @totalpages: total present RAM allowed for page allocation * @memcg: task's memory controller, if constrained * @nodemask: nodemask passed to page allocator for mempolicy ooms * * The heuristic for determining which task to kill is made to be as simple and * predictable as possible. The goal is to return the highest value for the * task consuming the most memory to avoid subsequent oom failures. */
- 13. Armouring Telegraf Against the OOM Killer [Service] OOMScoreAdjust=-600 /etc/systemd/system/telegraf.service.d/oom_score_adj.conf value between -1000 and +1000/proc/$PID/oom_score_adj /proc/$PID/oom__adj value between -17 and +15
- 14. Telegraf Config [[inputs.syslog]] server = "tcp://:6514" [[outputs.influxdb]] urls = [ "http://influxdb-syslog.service.consul:8086" ] database = "syslog" retention_policy = "autogen" precision = "ns" user_agent = "telegraf" namepass = ["syslog"]
- 15. Rsyslog Config $ActionQueueType LinkedList $ActionQueueFileName telegraf $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on # forward over tcp with octet framing according to RFC 5425 *.* @@(o)localhost:6514;RSYSLOG_SyslogProtocol23Format # all logs that contain the string "**WARNING**" if ($msg contains '**WARNING**') then @@(o)localhost:6514;RSYSLOG_SyslogProtocol23Format
- 16. What do you get?
- 17. Telegraf Output ● syslog ○ tags ■ severity (string) ■ facility (string) ■ hostname (string) ■ appname (string) ○ fields ■ version (integer) ■ severity_code (integer) ■ facility_code (integer) ■ timestamp (integer): the time recorded in the syslog message ■ procid (string) ■ msgid (string) ■ sdid (bool) ■ Structured Data (string) ○ timestamp: the time the messages was received Original timestamp Timestamp given by Telegraf
- 18. Timestamps & Ingest Latency var data = stream |from() .database('syslog') .retentionPolicy('autogen') .measurement('syslog') |groupBy('hostname') |window() .align() .period(1m) .every(1m) |eval(lambda: unixNano("time") - "timestamp") .as('timestamp_lag_ns') var mean_latency = data |mean('timestamp_lag_ns') .as('val') mean_latency |log() Typical Latency: under 1ms
- 19. Getting more from Telegraf
- 20. Enable Counters # Create per-minute counts of syslog data [[aggregators.basicstats]] period = "1m" drop_original = false stats = ["count"] name_suffix = "_counts" # filtering namepass = ["syslog"]
- 23. Adding More Tags From Your Log Data [[processors.parser]] parse_fields = ["message"] drop_original = false merge = "override" data_format = "json" tag_keys = ["method","uri"] namepass = ["syslog"] [processors.parser.tagpass] appname = ["crazy-cool-api"] { "hostname": "ab704bf3336f" , "ip": "10.0.0.30", "method": "GET", "msg": "request completed" , "start": "2018-10-23T07:00:03Z" , "status": 200, "time": "2018-10-23T07:00:03.942327Z" , "uri": "/health", "uri_query": "", "user_agent": "Consul Health Check" } Log Message Telegraf Config
- 26. var ipv4_address = /d{1,3}.d{1,3}.d{1,3}.d{1,3}/ |eval(lambda: regexReplace(ipv4_address, "message", '<ipv4-address>')) .as('message') .keep() Rewriting your data with regexReplace var email_address = /w+([-+.']w+)*@w+([-.]w+)*.w+([-.]w+)*/ |eval(lambda: regexReplace(email_address, "message", '<email-address>')) .as('message') .keep( Email Addresses IP Addresses
- 27. Email Address Regex (RFC 5322) (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[ x01-x08x0bx0cx0e-x1fx21x23-x5bx5d-x7f]|[x01-x09x0bx0cx0e -x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]* [a-z0-9])?|[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]| 2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[x01-x08x0bx0cx0e -x1fx21-x5ax53-x7f]|[x01-x09x0bx0cx0e-x7f])+)])
- 28. Email Address Regex / Pain Amplifier (?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)? [t])*))*>(?:(?:rn)?[t])*)|(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?: rn)?[t])*)*:(?:(?:rn)?[t])*(?:(?:(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*" (?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t] ))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn )?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*| (?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)*<(?:(?:rn)?[t] )*(?:@(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^( )<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*(?:,@(?:(?:rn)?[t])*(?:[^()<>@,;:" .[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-0 31]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*)*:(?:(?:rn)?[t])*)?(?:[^()<>@,;:".[]000-031]+(?:(?: (?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(? :(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+ (?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:r n)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*>(?:(?:rn)?[t])*)(?:,s*(?:(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn) ?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?: rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(? :rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t]) +|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))*|(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))| "(?:[^"r]|.|(?:(?:rn)?[t]))*"(?:(?:rn)?[t])*)*<(?:(?:rn)?[t])*(?:@(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[ ]]))|[([^[]r]|.)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^ []r]|.)*](?:(?:rn)?[t])*))*(?:,@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]| .)*](?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?: rn)?[t])*))*)*:(?:(?:rn)?[t])*)?(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t]))*" (?:(?:rn)?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|"(?:[^"r]|.|(?:(?:rn)?[t] ))*"(?:(?:rn)?[t])*))*@(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn )?[t])*)(?:.(?:(?:rn)?[t])*(?:[^()<>@,;:".[]000-031]+(?:(?:(?:rn)?[t])+|Z|(?=[["()<>@,;:".[]]))|[([^[]r]|.)*](?:(?:rn)?[t])*))* >(?:(?:rn)?[t])*))*)?;s*)
- 30. Tick Template var where_filter = lambda: TRUE var appname_lambda lambda var event_type string var data = stream |from() .database('syslog') .retentionPolicy('autogen') .measurement('syslog') |where(lambda: "facility" == 'kern') |where(where_filter) |eval(appname_lambda) .as('appname') .tags('appname') .keep() |eval(lambda: regexReplace(/w+([-+.']w+)*@w+([-.]w+)*.w+([-.]w+)*/, "message", '<email-address>')) .as('message') .keep() |eval(lambda: unixNano("time") - "timestamp") .as('timestamp_lag_ns') .keep() |log() /etc/kapacitor/load/templates/template-syslog_kern_service_events.tick
- 31. Task Definition : OOM template-id: template-syslog_kern_service_events vars: Where_filter: type: lambda value: > "message" =~ /.+ Memory cgroup out of memory: Kill process .+/ appname_lambda: type: lambda value: > regexReplace(/^[.+] Memory cgroup out of memory: Kill process .+? ((.+?)) score .+/, "message", '$1') event_type: type: string value: "OOM" /etc/kapacitor/load/tasks/syslog_kern_service_events-OOM.yaml
- 32. Task Definition : Segfault /etc/kapacitor/load/tasks/syslog_kern_service_events-segfault.yaml template-id: template-syslog_kern_service_events vars: Where_filter: type: lambda value: > "message" =~ /.+: segfault at .+/ appname_lambda: type: lambda value: > regexReplace(/^[.+] (.+?)[.+]: segfault at .*/, "message", '$1') event_type: type: string value: "segfault"
- 33. Memory cgroup out of memory: Kill process 3056 ( upstart-socket-) score 1057 or sacrifice childn The kernel truncates process names down to 15 chars (16 chars -1 NUL) and stores this in /proc/<PID>/comm include/linux/sched.h /* Task command name length: */ #define TASK_COMM_LEN 16 e.g. upstart-socket-bridge becomes upstart-socket- The Kernel & Process Names
- 34. metric_relabel_configs: - source_labels: ['name'] regex: '([[:ascii:]]{1,15}).*)' target_label: 'name_short' replacement: '$1' Linkage Workarounds Prometheus [[processors.regex]] [[processors.regex.tags]] key = "<tag>" pattern = "^([[:ascii:]]{1,15}).*$" replacement = "${1}" result_key = "short_<tag>" Telegraf
- 35. Dashboards
- 37. Ad-hoc Variable
- 38. Group By Using A Custom Variable GROUP BY time(1m), "[[group]]" fill(0)
- 39. Regex Search Using A Constant WHERE ("message" =~ / $search/)
- 40. Annotations