SplunkLive! Developer Breakout
1 like1,893 views
This document provides an overview of using Splunk for application development. It discusses using the Splunk developer platform, APIs, and SDKs to index, search, manage and visualize data in Splunk from other applications. The agenda covers assumptions, using Splunk for development, an overview of the Splunk developer platform including the REST API and SDKs for Java, JavaScript, Python, and PHP. Code samples are provided for logging events and performing searches using the Java SDK. Support resources for developers are also listed.
SplunkLive! Developer Breakout
- 1. Copyright © 2012 Splunk, Inc. Splunk for Developers Paul Sanford Director, Developer Platform Jon Rooney Director, Developer Marketing
- 2. Agenda Assumptions Using Splunk for Application Development The Splunk Developer Platform Splunk By Example – the Java SDK Support and Community 2
- 3. Assumptions
- 4. You Are in This Session Because…. • You are an experienced Splunk user (search, dashboards, sourcetyping, extracting fields) • You are a developer and want to use your development skills to customize and extend your Splunk experience • You love REST and love developing with APIs • You are interested in using SDKs to index, search, manage and visualize data in Splunk • You have http://dev.splunk.com bookmarked 4
- 6. Using Splunk for Application Development Accelerate Dev & Test – Every developer should use Splunk to find and fix bugs, trace transactions in real time & build intelligence into your apps without defining a schema with semantic logging Integrate date from Splunk into other applications – Search, manage and visualize data in other applications with the REST API and SDKs for Java, Python, JavaScript and PHP Build Real-time Big Data Applications – Collection, storage, query language, visualization “out-of-the-box” – Real-time insights: clickstream analysis, IT early-warning systems, security and fraud protection 6
- 7. Splunk is Hackable! 1st Splunk Hackathon: .conf2012 • 50 hackers • 10 teams • 5 hours • 3 winners 7
- 8. What You Need • Splunk • Data • Text Editor or IDE • Documentation (dev.splunk.com) • SDKs on GitHub 8
- 10. The Splunk Platform Inputs, Apps, Other Operational Intelligence Platform Content UI SDK Content REST API User and Developer Interfaces Core Functions Core Engine Search Processing Language Indexing Collection 10
- 11. What can you do with the APIs and SDKs? Index – Log directly to Splunk (TCP, UDP, HTTP) Search – Including saved searches – Extract data from Splunk Visualize – Integrate search results with third-party reporting tools, portals and other custom applications Manage – Add/remove users and roles – Create inputs 11
- 12. The Splunk REST API Exposes an API method for every feature in the product – Whatever you can do in the UI – you can do through the API – Run searches – Manage Splunk configurations API is RESTful – Endpoints are served by splunkd – Requests are GET, POST, and DELETE HTTP methods – Responses are Atom XML Feeds – JSON coming in 5.0 – Versioning coming in 5.0 – Search results can be output in CSV/JSON/XML 12
- 13. Spring Integration Splunk Inbound Adaptor • Blocking, Non Blocking, Saved & Realtime Searches • Exporting 13
- 14. Spring Integration Splunk Outbound Adaptor • HTTP REST Input • TCP Input 14
- 15. Demos
- 17. SDK Design Concepts • Stay true to the semantics of the particular language • E.g. Keep Python “pythonic” • Provide implementation that feels to the developer • E.g. Project, build, IDE (where applicable) support • Cover REST API endpoints based on use cases of language • E.g. Java SDK has most comprehensive coverage. JavaScript has fewer management facilities • Initially stay true to REST API semantics and abstract based on feedback • Namespaces • owner: splunk username (defaults to current user) • app: app context (defaults to default app) • sharing: user | app | global | system • Service Class • Instantiate an object to connect and login • Entry point for REST API calls 17
- 18. Java SDK • Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • refresh() : to get changes from splunkd • Getting Started - http://dev.splunk.com/view/java-sdk/SP-CAAAECN • Open sourced under the Apache v2.0 License • Current release status is “beta” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-java.git • Project level support for Eclipse and IntelliJ (git plugins available) • Pre-requisites • Splunk installed • JRE 6+ • Ant (test, build, generate javadocs) • Run the unit tests and examples • Set up a “.splunkrc” file in your user’s home directory • Ant (build, test, generate javadocs) 18
- 19. JavaScript SDK • 2 main components • Data SDK – Manage Splunk objects, input and search data etc. • UI SDK – Includes Splunk UI components like Charting and Timeline controls • Use of native JavaScript objects • Resource, Entity and Collection objects provide the necessary abstraction • Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • fetch() : to get changes from splunkd • Getting Started - http://dev.splunk.com/view/javascript-sdk/SP-CAAAECM • Open sourced under the Apache v2.0 License • Current release status is “beta” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-javascript.git • Pre-requisites • Splunk installed • Node.js for server side scripting, building and running tests and examples • Run the unit tests and examples using node. 19
- 20. Python SDK • 4 main modules • binding: Provides thin abstraction over raw HTTP. • client: Provides an abstraction layer over REST APIs. • results: Provides a Splunk specific streaming XML reader. • data: Converts Splunk’s Atom feed response into Pythonic structure – directory or list • Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • refresh() : to get changes from splunkd • Getting Started - http://dev.splunk.com/view/python-sdk/SP-CAAAEBB • Open sourced under the Apache v2.0 License • Current release status is “beta” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-python.git • Pre-requisites • Splunk installed • Python 2.6+ • easy_install or pip • Run the unit tests and examples • Set up a “.splunkrc” file in your user’s home directory 20
- 21. PHP SDK • Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • fetch() : to get changes from splunkd • Getting Started - http://dev.splunk.com/view/php-sdk/SP-CAAAEJM • Open sourced under the Apache v2.0 License • Current release status is “preview” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-php.git • Pre-requisites • Splunk installed • PHP 5.2.11+ • Web Server that supports PHP (e.g. MAMP) – for running examples • PHPUnit 3.6+ - for running the unit tests • Run the unit tests and examples • Set up a “settings.default.php” file in the examples and tests directory 21
- 23. Connecting / Authenticating 23
- 24. Namespaces 24
- 25. Logging Events via HTTP REST Uses receivers/simple endpoint Uses receivers/stream endpoint 25
- 26. Logging Events via Raw TCP If you don’t already have a TCP port listening, simply create one via the REST API Setup Log to Splunk Teardown 26
- 27. Synchronous Search 27
- 28. Asynchronous Search 28
- 29. Paginating Results • “maxresultrows” in Splunk config default 50K • Not recommended to change this • If result set > 50K , then page through results 29
- 30. Real-time Search 30
- 31. Saved Search 31
- 32. Processing CSV/JSON/XML results Results put into Hashmap 32
- 34. The Splunk Developer Community Splunkbase • Over 1,000 unique visitors/week to dev.spunk.com • Over 650 followers of @splunkdev 34
- 35. Where to Go for More Info Portal – http://dev.splunk.com/ GitHub – https://github.com/splunk/ Twitter – https://twitter.com/splunkdev Blog – http://blogs.splunk.com/dev/ Support 35
- 36. Thank you
Editor's Notes
- #7: We’re extending Splunk so it’s easier for your to leverage it’s capabilities using technologies you’re familiar with. We’re delivering SDKs on top of our REST API to help you integrate Splunk data with other applications. Splunk is a fully-integrated platform that delivers rapid “time-to-value” to developers. Many of our customers are building robust applications on Splunk today that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at a scale that their businesses demand.
- #11: Whatdoes this platform look like?The platform consists of 2 layer:A core engine and an interface layerOn top of the platform you can’t run a broad spectrum of content that supports use casesUse cases range from application mgmt. and IT operations, to ES and PCI compliance, to web analyticsThe core engine provides the basic services for real time data input, indexing and search as well alerting, large scale distributed processing and role based accessThe Interface layer consist of the basic UI for search, reporting and visualization– it contains developer interfaces, the REST API and SDKsThe SDKs provide a convenient access to core engine services in a variety of programing language environments. These programmatic interfaces allow you to eithe:r:extend Splunkintegrate Splunk with other applicationsbuild completely new applications from scratch that require OI or analytical services that Splunk provides
- #29: There is code in the develop branch (which we should probably push into main before .conf) that obviates the need for job.refresh()isDone() and isReady() refresh behind your back.
- #30: In order to get all events, you have to use the export endpoint. But the export endpoint has different behavior than a normal job. An export cannot be "restarted" when getting events if the network hiccups. A search job can just do another getResults() with the appropriate offset — this is because the export endpoint doesn't save the results like a search job does. But a search job has a limited number of events it will store on the server — which can be affected by status_buckets — but there is no way to guarantee the upper limit. With the default status_buckets we can get to 500K events. Itay and I experimented with hundreds of stratus_buckets but were only to get up to about 1M events, out of 13M available events.