Esctl in action elastic user group presentation aug 25 2020
- 1. 1 Jeff Moore 08/25/20, Systems Engineer, Bandwidth esctl in Action or How I Built a CLI tool to Interact with Elasticsearch
- 2. 2 About this presentation It was fairly difficult deciding between catering towards Devs or Ops. In the end, I decided to side with the Ops side of the house. Based on audience request, I will go into the code in depth - or that can be the topic of a later talk. Feel free to stop me at any point* so we can discuss this utility - however, suggestions are encouraged to be held until the end of the slide deck
- 4. 4 Hopeful Takeaways - An extensible CLI tool that may impact both yourself and your Org - Lessons learned from over a year of becoming a stronger Golang developer
- 5. 5 Caveats - This is not complete. - Work has been focused on getting good boilerplate - Not all endpoints have been implemented (templates, indices, etc) - This is not intended to replace the SDKs
- 6. 6 Why esctl? Devs have a lot of SDKs, but Ops doesn’t appear to have anything
- 7. 77 We needed an easier way to interact with Elasticsearch as operators of the cluster
- 8. 8 Intent of esctl 1. Standard endpoints are mocked, with basic config options exposed as flags 2. Extra commands are added based on functions that could be necessary (i.e admin, util)
- 10. 10
- 11. 11
- 12. 12 escli and esctl A brief history of two cli tools
- 13. 13 escli vs esctl Comparison of metrics Metric escli esctl Language Bash Golang Function implementation Bash functions Subcommands # of commands (Aug 2020) 117 44 New functions Easy to add Tedious Extensibility Difficult Easy Refactorability Difficult Not as difficult Barrier to entry Easy Difficult
- 14. 14 Getting Started Demo Overview of Demo Environment 1 2 3 4 Overview of local cluster init script Overview of Kubernetes manifests Demo of commands and features
- 16. 16
- 17. 17 On Command Structure - Elasticsearch api structure is very different to that of Kubernetes - Format in v0.1.4 is esctl <endpoint/resource> <function> - Format in Kubectl is kubectl <function> <endpoint/resource>
- 18. 18 Command Structure Comparison pt. 1 kubectl vs. esctl - Similarities CLI Tool Command Function Resource kubectl kubectl get nodes Get Nodes esctl esctl cat nodes Get Nodes
- 19. 19 Command Structure Comparison pt. 2 kubectl vs. esctl - Differences CLI Tool Command Function Resource kubectl kubectl get ns Get Namespaces esctl esctl index get Get Indices (Yes, I am comparing namespaces to shards. Feel free to discuss offline)
- 20. 20 Command Structure Comparison pt. 3 kubectl vs. esctl - the Justification The endpoint behind commands like esctl cat nodes is /_cat/nodes. By mocking functions after their API syntax, there is no mental translation tax needed to move to/from esctl and the REST API
- 21. 21 Proudly Built on go-elasticsearch SDK Elasticsearch SDK, mostly written by Karel Minarik from Elastic Defines api endpoint options as struct fields Allows client initialization Located at https://github.com/elastic/go-elasticsearch
- 22. 22
- 23. 23 esctl Extensibility Examples - Showing Inactive Watchers Problem: Watchers not currently active (read: disabled) need to be listed Solution: - Get all watchers (SearchRequest on .watches index) - Filter to desired activity state - Return output (esutil helper function)
- 24. 24 esctl Extensibility Examples - Getting Search Metrics Problem: User queries need to be validated before a DeleteByQuery is run Solution: • Take a query • Merge a known aggregation definition to that query • Execute that query (SearchRequest) • Return output (esutil helper function)
- 25. 25 esctl Extensibility Examples - Removing Empty Indices Problem: Empty indices created by ILM policies cause disk utilization inbalances Solution: - Get a list of all indices (IndicesGetRequest) - Filter to indices containing 0 documents - Delete those indices (IndicesDeleteRequest)
- 26. 2626 Still Not Sold? esctl api <HTTP verb> endpoint
- 27. 27 Lessons Learned - Writing a CLI tool isn’t easy - Design should come before writing code to avoid a large number of refactors - Reading documentation thoroughly before using imported code is extremely valuable - This CLI tool continues to be a great learning opportunity for me
- 28. 28 Overall, building this CLI tool was a painful experience; but, if it was painless, I would be a Java dev instead of a Golang dev Me, just now
- 29. 29 Future Improvements - Add documentation - Discuss command structure - Need a logo - Codegen?
- 30. 30 What I Need from YOU! - Use it - File bugs and RFEs
- 31. 3131 Questions?
- 33. 3333 Thanks!