Elastic Stack @ Swisscom Application Cloud
Download as PPTX, PDF1 like274 views
The document discusses the use of the Elastic Stack at Swisscom Application Cloud, highlighting its components like Elasticsearch, Logstash, and Kibana for log management and visualization. It details the methodology for processing logs within application instances, emphasizing best practices and the significance of logging in cloud-native applications. Additionally, it introduces tools and community contributions aimed at enhancing Logstash configurations and testing integrity.
![Logstash Filter Verifier testsuite file:
{
"fields": {},
"codec": "line"
"ignore": [ "@version", "host" ],
"testcases": [ {
"input": [
"2017/06/12 08:12:58 WARN message e361827a-990e-
4237-8ea3-047f292f1d14 (1534 bytes) from <mind-blowing-
musa@dagger.com> to <epic_williams@centaur.com> could not
be sent, will retry"
],
"expected": [ {
"@timestamp": "2017-06-12T08:12:58.000Z",
"severity": "WARN",
"from": "mind-blowing-musa@dagger.com",
"to": "epic_williams@centaur.com",
"message": "could not be sent, will retry",
"size": 1534
} ]
} ] }
08.06.17
18
BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR
ElasticStack@SwisscomAppCloud.pptxC1-
Public
Example
Additional fields, provided by the
source or added by the input plugin
Codec to decode input data (usually
one of line or json_lines)
Fields to be ignored, when the result
is compared
Testcases:
• provided input
• expected log event provided by Logstash](https://image.slidesharecdn.com/elasticstackswisscomapplicationcloudclean-170614055007/85/Elastic-Stack-Swisscom-Application-Cloud-18-320.jpg)
Elastic Stack @ Swisscom Application Cloud
- 1. Elastic Stack @ Swisscom Application Cloud Swisscom (Schweiz) AG Bremgartner Lucas 13.06.2017 C1 - Public
- 2. > Introduction > What is Swisscom Application Cloud / What is the Elastic Stack > Use of Elastic Stack @ Swisscom Application Cloud > Process Logs with Logstash @ Swisscom Application Cloud > Testing growing Logstash Configurations 2Agenda BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public 08.06.17
- 3. Home of Cloud Native Applications
- 4. 08.06.17 4 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public What is Swisscom Application Cloud / What is the Elastic Stack Kibana User Interface ElasticsearchStore, Index, & Analyze Ingest Logstash Beats Elastic Stack Swisscom Developer Portal
- 6. > Lucas Bremgartner, Cloud Developer @ Swisscom Application Cloud Quick notes: > Elasticsearch user since version 0.9.x. > My current «goto» programming language is Go Open Source: > Logstash Community Maintainer > Contributor to logstash-filter-verifier (LFV) > Maintainer of pigeon (PEG grammar parser generator for Go) > Author of logstash-config (parser for Logstash configuration, written in Go) 08.06.17 6 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Introduction
- 7. > ELK as a Service – Available in marketplace, containing Elasticsearch, Logstash and Kibana – Intended use-case: collect logs from apps running in Application Cloud and visualize them with Kibana > Elasticsearch Enterprise – Currently under development – Intended use-case: scalable Elasticsearch clusters as a service – Open for all Elasticsearch use cases (classical full-text search, log management, geo location search, etc.) > Elastic Stack for Log Management of the Infrastructure – Classical pipeline with Filebeat, Logstash, Elasticsearch and Kibana 08.06.17 7 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Use of Elastic Stack @ Swisscom Application Cloud
- 8. 14.06.2017 8 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Process Logs with Logstash @ Swisscom Application Cloud
- 9. > Application instances in Cloud Foundry are ephemeral, storing logs on local disk is not a good idea > With multiple instances of the app running in parallel, an aggregated log stream is needed > The 12 factor apps methodology defines for log data: – «A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout» > Cloud Foundry collects and ships the log events of the application and makes the log events available through the API: cf logs <app> > Cloud Foundry also allow to stream the logs to a customer provided service (syslog or https) 08.06.17 9 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Application Logs in Swisscom Application Cloud
- 10. App deployed by Customer App deployed by Customer Service by Swisscom AppCloud 05.09.16 10 BremgartnerLucas,ENT-NTC-PHC-PFD-ELR ELKEnterprise.pptxC2-Internal Stream Application Logs in Cloud Foundry Logstash KibanaElasticsearch ES Dashboards (e.g. Cerebro, Kopf) Logstash Logstash House- Keeping (e.g. curator) App App logs to stdout, CF log facility forwards via customer provided service to Logstash App App
- 11. > Buildpacks provide framework and runtime support for your applications. > Buildpacks typically examine user-provided artifacts to determine what dependencies to download and how to configure applications to communicate with bound services. > This is done by three entrypoints: – bin/detect: determines whether or not to apply the buildpack to an app. – bin/compile: builds a droplet by packaging the app dependencies, assuring that the app has all the necessary components needed to run. – bin/release: provides feedback metadata to Cloud Foundry indicating how the app should be executed. 08.06.17 11 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public What is a Cloud Foundry Buildpack
- 13. 14.06.2017 13 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Testing growing Logstash Configurations
- 14. Elasticsearch KibanaRabbitMQLogstash (Shipper to RabbitMQ) Filebeat on Edge Nodes 08.06.17 14 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Log Management @ Swisscom Application Cloud Logstash (Filter)
- 15. > Every application/service/daemon has its own log format, which needs to be tackled with a specific set of Logstash filters. > While adding more and more log formats, the complexity increases and changes to the configuration become more and more delicate. > With new software versions (lifecycle), also changed log patterns may occur, which need to be processed in parallel to the old one. > Integrate the testing of the Logstash configuration into the CI pipeline. > Additionally to the Logstash configuration, also the Elasticsearch mapping needs to be maintained. > The Elasticsearch mapping could become a quite large (JSON file), which is a pain to update (unhandy, error prone, etc.). > Undocumented Elasticsearch mappings are harder to understand and to maintain (especially if this is not done on a regular bases) 08.06.17 15 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Challenges
- 16. > Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” > Logstash follows the classical input–process–output (IPO) pattern, the process stage is called «filter». > A long list of different input, filter and output plugins is available, which allow to adopt Logstash to a wide variety of use cases. > A Logstash configuration is like a program which is applied to every log event. 08.06.17 16 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Logstash
- 17. > LFV provides unit test kind of functionality for Logstash filter configurations > Run test input against a given Logstash configuration and compare the result with the expected value 08.06.17 17 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Logstash Filter Verifier Logstash filter config LFV Logstash Test cases Kudos to @magnusbaeck for developing and maintaining Logstash Filter Verifier (LFV) «If you get something wrong (… in the Logstash config …) you might have millions of incorrectly parsed events before you realize your mistake. » – Magnus Bäck
- 18. Logstash Filter Verifier testsuite file: { "fields": {}, "codec": "line" "ignore": [ "@version", "host" ], "testcases": [ { "input": [ "2017/06/12 08:12:58 WARN message e361827a-990e- 4237-8ea3-047f292f1d14 (1534 bytes) from <mind-blowing- musa@dagger.com> to <epic_williams@centaur.com> could not be sent, will retry" ], "expected": [ { "@timestamp": "2017-06-12T08:12:58.000Z", "severity": "WARN", "from": "mind-blowing-musa@dagger.com", "to": "epic_williams@centaur.com", "message": "could not be sent, will retry", "size": 1534 } ] } ] } 08.06.17 18 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Example Additional fields, provided by the source or added by the input plugin Codec to decode input data (usually one of line or json_lines) Fields to be ignored, when the result is compared Testcases: • provided input • expected log event provided by Logstash
- 21. Software & Tools: > Logstash Buildpack for Swisscom Application Cloud https://github.com/swisscom/cf-buildpack-logstash > Kibana Buildpack for Swisscom Application Cloud https://github.com/swisscom/cf-buildpack-kibana > Logstash Filter Verifier (LFV) https://github.com/magnusbaeck/logstash-filter-verifier > Logstash Config Check https://github.com/breml/logstash-config Additional Links: > 12 Factor Apps: https://12factor.net/ > Grok Debugger: https://grokdebug.herokuapp.com/ > ./jq: https://stedolan.github.io/jq/ > jsondiff: https://github.com/yudai/gojsondiff/ > dockerize: https://github.com/jwilder/dockerize 08.06.17 21 BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR ElasticStack@SwisscomAppCloud.pptxC1- Public Links