Writing Docker monitoring agent with Go
3 likes2,268 views
Fune is a Go program that acts as a Docker monitoring agent. It listens to Docker events through the Docker events API and emits actions in response to events, such as notifying Slack or updating a service registry. The agent communicates with the Docker daemon via its socket. It allows emitting pluggable actions when container events like create, start, stop and die are received.
![Run go as Docker container
• Go single binary with "SCRATCH"
• You can run a go stuff inside extremely simple container!
FROM
scratch
EXPOSE
8080
COPY
your-‐golang-‐app
/app/your-‐golang-‐app
ENV
PATH=/app:$PATH
ENTRYPOINT
["/app/your-‐golang-‐app"]
CMD
["-‐some-‐option=haha"]](https://image.slidesharecdn.com/150826kyobashi-go-150827030509-lva1-app6891/85/Writing-Docker-monitoring-agent-with-Go-23-320.jpg)
Writing Docker monitoring agent with Go
- 1. Writing Docker monitoring agent with Go ainoya
- 2. About me • Naoki Ainoya( ainoya) • iOS(new!)/Server/Infra engineer • Swift/Scala/Go/Docker • Recruit Marketing Partners Co, Ltd.
- 3. A tiny deployment pipeline tool Written in Go walter-cd/walter Open source project
- 4. Introduction of Fune • An event emitter triggered by Docker Event API • The agent process (fune-agent) communicates with docker daemon via its socket • https://github.com/ainoya/fune Docker daemon Funeagent ConttainerA ConttainerB ConttainerC Nginx/Redis vulcand(etcd) Slack notification etc.. Listens docker events API notifies each containers are created/died/stopped/started Emits pluggable actions Your arbitrary docker ecosystem ECS/k8s/fleet/etc.. Docker Host Deploy containers
- 5. Overview of Fune • Docker Events API provides a lifecycle of container as JSON format • create/start/die/stop
- 6. Overview of Fune fune-agent listens these events Docker daemon Funeagent ConttainerA ConttainerB ConttainerC Listens docker events API notifies each containers are created/died/stopped/started Your arbitrary docker ecosystem ECS/k8s/fleet/etc.. Docker Host Deploy containers
- 7. Overview of Fune fune-agent emits Action when received Docker events from event Docker daemon Funeagent ConttainerA ConttainerB Nginx/Redis vulcand(etcd) Slack notification etc.. Listens docker events API notifies each containers are created/died/stopped/started Emits pluggable actions Your arbitrary docker ecosystem ECS/k8s/fleet/etc.. Docker Host Deploy containers
- 8. Overview of Fune Docker daemon Funeagent ConttainerA ConttainerB Nginx/Redis vulcand(etcd) Slack notification etc.. Listens docker events API notifies each containers are created/died/stopped/started Emits pluggable actions Your arbitrary docker ecosystem ECS/k8s/fleet/etc.. Docker Host Deploy containers • Action, for example; • Set/Del a container information to Redis as FQDN/IP:Port pair for proxy • Slack notification • More details are next time :)
- 9. Today I talk about: • Some tips I learned with writing fune • Studied a lot of stuffs from coreos/etcd
- 10. Setup project • Standard structure for working with Godeps correctly • /src/github.com/ainoya/fune ./${GOPATH}/src !"" 9fans.net # %"" go !"" code.google.com # %"" p !"" github.com # !"" ainoya/fune # !"" barakmich # !"" coreos etc...
- 11. Godeps for the package dependencies • Using Godeps once, all dependencies copied into directory and rewrite import statement • Then you need Godeps command only if package dependency is added import "github.com/fsouza/go-dockerclient" // after `Godep save -r ./…` import "github.com/ainoya/fune/Godeps/_workspace/ src/github.com/fsouza/go-dockerclient"
- 12. Godeps for the package dependencies • All dependencies are included in git repos • But it makes dirty commit log!! • Keep off its dirtiness by split commits :)
- 13. How about vendor feature in go1.5? • go1.5 vendor feature have some problems yet • ex) go tool "./…" doesn't works well • https://github.com/golang/go/issues/11659 • See also ) http://deeeet.com/writing/2015/06/26/golang-dependency- vendoring/
- 14. Write tests • Rerun automatically if files are changed • rerun is awesome! (but ruby tool gem) rerun -x -c -d directory1, directory2,... —pattern '**/*.{go}' -- go test ./...
- 15. Be careful about race condition • Test with "-‐race" option if you often use goroutine in your code • "-‐cpu N" option may reveal race condition you didn't expected • I use "go test ./… -‐race -‐cpu 1,2,4" inside test script WARNING: DATA RACE Write by goroutine 8: sync.raceWrite() /usr/local/Cellar/go/1.5/libexec/src/sync/race.go:41 +0x2e sync.(*WaitGroup).Wait() /usr/local/Cellar/go/1.5/libexec/src/sync/waitgroup.go:124 +0xf9 github.com/ainoya/fune/emitter.TestBroadCast() /Users/ainoya/.go/src/github.com/ainoya/fune/gopath/src/github.com/ ainoya/fune/emitter/emitter_test.go:57 +0x444 testing.tRunner() /usr/local/Cellar/go/1.5/libexec/src/testing/testing.go:456 +0xdc
- 16. Is Go1.5 compilation time slower than 1.4? • "go test" obviously takes time much longer • It's probably because of GoGo compiler performance Ref) Performance section of https://golang.org/doc/go1.5 • It should be improved with Go1.6
- 17. Channel tips • coreos/etcd implements a shutdown feature with using close(channel) type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} } func main() { done := make(chan struct{}) stop := make(chan struct{}) s := &Server{done: done, stop: stop, hoge: hoge} go s.run() s.channel <-‐ struct{}{} osutil.RegisterInterrputHandler(s.Stop) <-‐s.done } func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } } } func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
- 18. Channel tips • coreos/etcd implements a shutdown feature with using close(channel) type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} } func main() { done := make(chan struct{}) stop := make(chan struct{}) s := &Server{done: done, stop: stop, hoge: hoge} go s.run() s.channel <-‐ struct{}{} osutil.RegisterInterrputHandler(s.Stop) <-‐s.done } func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } } } func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
- 19. Channel tips • coreos/etcd implements a shutdown feature with using close(channel) type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} } func main() { done := make(chan struct{}) stop := make(chan struct{}) s := &Server{done: done, stop: stop, hoge: hoge} go s.run() s.channel <-‐ struct{}{} osutil.RegisterInterrputHandler(s.stop) <-‐s.done } func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } } } func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
- 20. Channel tips • coreos/etcd implements a shutdown feature with using close(channel) type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} } func main() { done := make(chan struct{}) stop := make(chan struct{}) s := &Server{done: done, stop: stop, hoge: hoge} go s.run() s.channel <-‐ struct{}{} osutil.RegisterInterrputHandler(s.Stop) <-‐s.done } func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } } } func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
- 21. Channel tips • coreos/etcd implements a shutdown feature with using close(channel) type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} } func main() { done := make(chan struct{}) stop := make(chan struct{}) s := &Server{done: done, stop: stop, hoge: hoge} go s.run() s.channel <-‐ struct{}{} osutil.RegisterInterrputHandler(s.Stop) <-‐s.done } func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } } } func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
- 22. Test goroutine easily "s.done" channel makes testing goroutine easily // https://github.com/coreos/etcd/blob/master/etcdserver/ server_test.go#L1031 // TestPublishRetry tests that publish will keep retry until success. func TestPublishRetry(t *testing.T) { n := &nodeRecorder{} srv := &EtcdServer{ cfg: &ServerConfig{TickMs: 1}, r: raftNode{Node: n}, w: &waitRecorder{}, done: make(chan struct{}), reqIDGen: idutil.NewGenerator(0, time.Time{}), } time.AfterFunc(500*time.Microsecond, func() { close(srv.done) }) srv.publish(10 * time.Nanosecond)
- 23. Run go as Docker container • Go single binary with "SCRATCH" • You can run a go stuff inside extremely simple container! FROM scratch EXPOSE 8080 COPY your-‐golang-‐app /app/your-‐golang-‐app ENV PATH=/app:$PATH ENTRYPOINT ["/app/your-‐golang-‐app"] CMD ["-‐some-‐option=haha"]
- 24. Round up • Learned a lot of stuffs from well-knowned products (coreos/etcd) • project structure, test, golang-way
- 25. See you next kyobashi.*! • 9/7: kyobashi.dex • 9/16: potatotips • ???: kyobashi.???