Experian Customer Presentation
1 like1,144 views
The document discusses migrating log ingestion from Splunk's S3 connector to using AWS Kinesis and Lambda functions to send logs directly to Splunk's HTTP Event Collector. It describes setting up Kinesis, configuring Lambda functions to batch and send logs to the HTTP Event Collector, and tuning various parameters like Lambda memory, batch size, and HTTP Event Collector limits to reduce latency from 15 minutes to under 5 seconds. Metrics and dashboards are used to measure the progress of reducing latency.
![17
HTTP Event Collector Scaling
Limits.conf
[http_input]
max_content_length = 1000000 (bytes)
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf
Increase the max_content_length = 5,000,000 bytes (~5MB)
Batch size = 5000, memory for the Lambda at 512MB](https://image.slidesharecdn.com/splunklivelongbeachexperian-161202175545/85/Experian-Customer-Presentation-17-320.jpg)
Experian Customer Presentation
- 3. 3 About Me • Mike Sclimenti, Senior Systems Engineer • Experian Consumer Services – IT Systems Administration/Engineering for 20+ years ê Highly Scalable Infrastructure Deployments & Disaster Recovery ê Large Scale VMware & Symantec (Veritas) NetBackup Environments ê Application Deployments, Systems Management, Active Directory, etc. – Monitoring Systems 2+ years • Splunk customer – User for 8 years – Admin for 2 years (Splunk 6.1, 6.3) • Favorite Splunk tee-shirt: “Because ninjas are too busy”
- 4. 4 Agenda • Architecture & Lessons Learned deploying Splunk Cloud: – S3 via the Splunk App for AWS – Kinesis – Lambda Functions – The HTTP Event Collector • How we went from 15 minutes of latency on production dashboards to… – Sub-5 seconds of latency sending logs directly from Kinesis (via Lambda) to the HTTP Event Collector
- 6. 6 Splunk’s S3 Connector The S3 Connector is efficient for: ⏤ CloudFront ⏤ ELB (Elastic Load Balancer) ⏤ CloudWatch & CloudWatch Logs ⏤ Cloudtrail ⏤ Billing
- 7. 7 The S3 Connector Was Working, But… • Then I went to .conf 2015 • So, as I was sitting in the Keynote session on Day 1, I thought: – Could I go directly to the HTTP Event Collector from the application? ê No more Universal Forwarders to install or update ê Fewer agents running on the EC2 instances – Would logging to Kinesis and then to the HTTP Event Collector be more efficient? Amazon EC2 Amazon Kinesis Amazon Lambda
- 8. 8 The HTTP Event Collector Applications IoT Devices Agentless, direct data onboarding via a standard developer API curl -k https://<host>:8080/services/collector -H ‘Authorization: Splunk <token>’ -d ‘{”event”:”Hello Event Collector”}’
- 9. 9 The HTTP Event Collector (cont.) • Got back to the office, began doing further research • Started planning migration from S3 Connector to the HTTP Event Collector • Began seeing latency issues w/ the ingest from S3 while running some load tests • Timeline for migration accelerated due to latency of 15 minutes ingesting logs from S3 But then… I realized HOUSTON WE HAVE A PROBLEM!
- 10. 10 The HTTP Event Collector (cont.) 1 • We were running Splunk Cloud version 6.2 • The HTTP Event Collector did not exist in Splunk Cloud version 6.2 • Installed the HTTP Event Collector on a Heavy Forwarder running the Splunk Enterprise 6.3. Amazon EC2 Amazon Kinesis Amazon Lambda Splunk Enterprise 6.3 Heavy Forwarder SplunkCloud 6.2
- 11. 11 The HTTP Event Collector (cont.) • Everything was running great until until we cranked up our traffic… • Luckily Splunk Cloud made version 6.3 available for production • Splunk Cloud 6.2 was upgraded to 6.3 • HTTP Event Collector was enabled on indexers • Lambda functions updated • Tuning began...
- 15. 15 Lambda Batch Size • Batch size is the max number of events that sent for single invocation of the Lambda function • Increased it from 100 to 1000 to 5000 to 10000 then back to 5000 • 646 bytes average event size but then HTTP event collector started to error sometimes because of the default max_content_length = 1,000,000 bytes • 1,000,000 / 646 = 1548 events in batch sourcetype=applogs host=http-inputs.splunkcloud.com earliest=-24h latest=now | eval event_size=len(_raw) | stats avg(event_size
- 18. 18 Lambda Tuning • Make sure you use https/SSL between Lambda and HTTP Event Collector • Set an appropriate batch size! “1000” is better than “100” • Set Lambda Function to “Latest” NOT “Trim Horizon” • Give your Lambda function the right amount of memory • Change the timeout from “10” to “30”
- 19. 19 AWS Kinesis Shards • Each shard can support: – Up to 5 transactions per second for reads – Up to a max total data read rate of 2MB/sec – Up to 1K records per second for writes – Up to a max total data write rate of 1MB/sec ê 2MB/sec per shard ê Plan for peaks • Make sure you split Kinesis into enough shards so that it can handle: – Inbound streams from your application – Outbound streams to S3 and/or the HTTP Event Collector
- 20. 20 Measuring Our Progress • Latency Search sourcetype=applogs host=http-inputs.splunkcloud.com earliest=-2m latest=now | eval latency_in_seconds=(_indextime - _time) | stats perc80(latency_in_seconds) as 80th_percentile_latency_in_seconds
- 22. 22 Things to Remember • S3 works but the HTTP Event Collector is faster • You must be using Splunk Cloud OR Splunk Enterprise 6.3 (or higher) • Tune your Lambda function (may impact your function $$$) • Scale up your HTTP Event Collector • Make sure you have enough Kinesis shards (may impact your Kinesis $$$) • Measure your progress through dashboards and alerts
- 23. 23 Resources • .conf2015 “The Great Shake Off” – http://www.ustream.tv/recorded/73893599 (starts at the 22min mark) • Splunk’s HTTP Event Collector – http://dev.splunk.com/view/event-collector/SP-CAAAE6M • AWS Lambda – http://docs.aws.amazon.com/lambda/latest/dg/welcome.html • AWS Kinesis Shard Limits – http://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
- 24. Thank You