![Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services
To find just your search heads
Find Search Heads
index=_internal sourcetype=splunkd_remote_searches | rex field=server
"(?<searchHead>[^.]+)" | dedup searchHead](https://image.slidesharecdn.com/internalsofinternal-190227033510/85/Internals-of-internal-10-320.jpg)
![Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services
Maybe it was skipped
index=_internal sourcetype=scheduler (status!=success status!=continued) |
rex "status=w+,s(?<reasonB>[^,]+)" | eval reason=coalesce(reason,
reasonB) | table _time savedsearch_name search_type user app status reason
Why Did My Search Not Run?](https://image.slidesharecdn.com/internalsofinternal-190227033510/85/Internals-of-internal-15-320.jpg)
![Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services
index=_internal sourcetype=splunkd "sendemail" "ERROR"
| rex field=message "ssname=(?<searchName>[^"]+)"
| rex field=message "(?J)ERROR:w+:((d+,s'(?<msg>[^']+)|(?<msg>[^:]+))"
| rex field=message "sending mail to:s(?<recips>.*)"
| table _time searchName msg recips
Why Did I Not Get Emailed Alerts?](https://image.slidesharecdn.com/internalsofinternal-190227033510/85/Internals-of-internal-18-320.jpg)
![Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services
• Dashboards with concurrent users should be optimized first
Dashboards with Concurrent Users
index=_internal user="*" sourcetype="splunk_web_access" uri_path="/en-US/app/*" NOT
(user ="-")
| rex field=uri_path "/app/(?<splunk_app>[^/]+)/(?<splunk_view>[^/]+)"
| search NOT (splunk_app="launcher") NOT (splunk_view="search" OR
splunk_view="show_source" OR splunk_view="alert*" OR splunk_view="dashboards" OR
splunk_view="field_extractor" OR splunk_view="job_manager")
| eval splunk_page =splunk_app+"-"+splunk_view
| bucket _time span=5m
| stats dc(user) as user_count by _time splunk_page
| search user_count>1
| timechart max(user_count) as user_count by splunk_page useother=f limit=25](https://image.slidesharecdn.com/internalsofinternal-190227033510/85/Internals-of-internal-24-320.jpg)
![Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services
_audit doesn’t always provide user name
Who Deleted the Dashboard?
index=_internal sourcetype=splunkd_ui_access method=DELETE views | rex
field=uri_path "/[-
w]+/w+/[_w]+/w+/(?<User>[^/]+)/(?<App>[^/]+)/data/ui/views/(?<D
ashboard>[^?]+)" | table _time App Dashboard User](https://image.slidesharecdn.com/internalsofinternal-190227033510/85/Internals-of-internal-25-320.jpg)
Internals of internal
- 1. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Internals of _internal Rich Galloway RTP Technology Corp.
- 2. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services • Me • Splunk user since 2012 • Splunk Enterprise Certified Architect • SplunkTrust member since 2016 • Splunk consultant since 2017 • RTP • VAR and integrator of technology products and solutions • Splunk Elite Partner About…
- 3. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services • Index for many of Splunk’s own logs • By default, viewable only by admin role • Does not count against license quota What is _internal?
- 4. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services What’s Inside Logs that are part of _internal include: splunkd splunkd_ui_access splunkd_web_access license_usage scheduler and many more
- 5. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Logs not part of _internal: audittrail (_audit) resource_usage (_introspection) kvstore (_introspection) http_event_collector_metrics (_introspection) disk_objects (_introspection) license_usage_summary (_introspection) What’s Not Inside
- 6. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Large systems may produce a lot of results Filter using host= or source!=“*forwarder*” When did Splunk Start? index=_internal source=*splunkd.log "Splunkd starting (build"
- 7. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Maybe you’d rather know when Splunk stopped index=_internal source=*splunkd.log component=ShutdownHandler "Shutting down splunkd" OR "Shutdown complete" When Did Splunk Stop?
- 8. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services To see the full shutdown sequence: index=_internal source=*splunkd.log | transaction startswith="Shutting down splunkd" endswith="Shutdown complete" When Did Splunk Stop? (2)
- 9. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services index=_internal component=ServerRoles sourcetype=splunkd role!=“deployment_client.” | stats values(role) as Roles by host Server Roles • Run this when you inherit a system • You may be surprised by what you find
- 10. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services To find just your search heads Find Search Heads index=_internal sourcetype=splunkd_remote_searches | rex field=server "(?<searchHead>[^.]+)" | dedup searchHead
- 11. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services To list forwarders: List Forwarders index=_internal group=tcpin_connections | stats latest(version) as version latest(arch) as arch latest(os) as os latest(fwdType) as fwdType by hostname
- 12. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Why are deployment clients not getting updated? Phone Home index=_internal component=DC:*
- 13. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Timestamp errors can make data hard to find index=_internal component=DateParserVerbose Where Is My Data?
- 14. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Good for troubleshooting stale data in dashboards Replace “*” with search name to find a specific search When Did My Saved Search Last Run? index=_internal savedsearch_name="*" status="success" | stats max(_time) as lastrun by savedsearch_name | eval lastrun = strftime(lastrun, "%c" ) | table savedsearch_name lastrun
- 15. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Maybe it was skipped index=_internal sourcetype=scheduler (status!=success status!=continued) | rex "status=w+,s(?<reasonB>[^,]+)" | eval reason=coalesce(reason, reasonB) | table _time savedsearch_name search_type user app status reason Why Did My Search Not Run?
- 16. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Maybe there was a scheduler error index=_internal source=*scheduler.log NOT INFO | cluster Why Did My Search Not Run? (2)
- 17. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Some searches only produce results when an anomaly is found index=_internal sourcetype=scheduler result_count earliest=-30d@d | stats max(result_count) as results by savedsearch_name | where results = 0 | fields - results Did My Search Produce Results?
- 18. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services index=_internal sourcetype=splunkd "sendemail" "ERROR" | rex field=message "ssname=(?<searchName>[^"]+)" | rex field=message "(?J)ERROR:w+:((d+,s'(?<msg>[^']+)|(?<msg>[^:]+))" | rex field=message "sending mail to:s(?<recips>.*)" | table _time searchName msg recips Why Did I Not Get Emailed Alerts?
- 19. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services More flexible than MC How Much Am I Indexing? index=_internal source=*metrics group=per_index_thruput series=wineventlog | eval GB=kb/1024/1024 | timechart span=1h sum(GB) as GB by series | eval GB=round(GB,2)
- 20. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services “series” = index index=_internal source=*metrics.log sourcetype=splunkd group=per_index_thruput | stats sum(kb) by series | addcoltotals How much am I writing to disk?
- 21. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Substitute ‘idx’ or ‘host’ for ‘st’ to see usage by index or host License Usage by Sourcetype index=_internal component=LicenseUsage | timechart span=1h sum(eval(round(b/1024/1024/1024,5))) as GBytes by st
- 22. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services index=_internal sourcetype=scheduler | eval _time = _time - run_time | concurrency duration=run_time | timechart max(concurrency) Search Concurrency Show how many searches have overlapping run times
- 23. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services index=_internal source=*scheduler.log* app=* run_time=* | stats min(run_time) as "Min runtime (seconds)", median(run_time) as “Median runtime”, max(run_time) as “Max runtime” by app, savedsearch_name Long-running searches Find candidates for improvement
- 24. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services • Dashboards with concurrent users should be optimized first Dashboards with Concurrent Users index=_internal user="*" sourcetype="splunk_web_access" uri_path="/en-US/app/*" NOT (user ="-") | rex field=uri_path "/app/(?<splunk_app>[^/]+)/(?<splunk_view>[^/]+)" | search NOT (splunk_app="launcher") NOT (splunk_view="search" OR splunk_view="show_source" OR splunk_view="alert*" OR splunk_view="dashboards" OR splunk_view="field_extractor" OR splunk_view="job_manager") | eval splunk_page =splunk_app+"-"+splunk_view | bucket _time span=5m | stats dc(user) as user_count by _time splunk_page | search user_count>1 | timechart max(user_count) as user_count by splunk_page useother=f limit=25
- 25. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services _audit doesn’t always provide user name Who Deleted the Dashboard? index=_internal sourcetype=splunkd_ui_access method=DELETE views | rex field=uri_path "/[- w]+/w+/[_w]+/w+/(?<User>[^/]+)/(?<App>[^/]+)/data/ui/views/(?<D ashboard>[^?]+)" | table _time App Dashboard User
- 26. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Where did users login from? May not be accurate if users are load-balanced index=_internal clientip=* NOT (user="-" OR user="splunk-system-user") | eval timestamp=strftime(_time, "%c") | table timestamp user clientip What about logins?
- 27. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services • Logins are in _audit, which could be a separate presentation • However… What about logins? (2) index=_audit action=login*
- 28. Professional & Managed Services Advisory Services Re-sale and Supply Chain Services Global Field Deployment Services Thank you rgalloway@rtptech.com https://www.splunk.com/en_us/community.html https://www.youtube.com/channel/UCC4GSZIocBGf7R61eDGb4cA
Editor's Notes
- #18: Consider adding the names of searches that must have results to a lookup file and check the lookup as part of the query.