![Pipelines
• produce a channel of stuff
• creates at least one goroutine
• source -> [middle] -> [middle] -> sink](https://image.slidesharecdn.com/gogeneratepipeliner-150423072752-conversion-gate01/85/go-generate-pipeliner-4-320.jpg)
![Go Generate
• since go1.4
• “comments” in your go project
• //go:generate [command] [args…]](https://image.slidesharecdn.com/gogeneratepipeliner-150423072752-conversion-gate01/85/go-generate-pipeliner-19-320.jpg)
![My common pipelines
• batch(<-chan Event) (<-chan []Event)
• flatMap(<-chan gzipfile) (<-chan Event)
• from([]Event) (<-chan Event)](https://image.slidesharecdn.com/gogeneratepipeliner-150423072752-conversion-gate01/85/go-generate-pipeliner-25-320.jpg)
go generate pipeliner
- 2. Pipelines?
- 3. Pipelines • https://blog.golang.org/pipelines • concurrency pattern • “stages connected by channels"
- 4. Pipelines • produce a channel of stuff • creates at least one goroutine • source -> [middle] -> [middle] -> sink
- 6. Pattern • directional channels • goroutine • close what you create • for/range to completion (previous stage closes)
- 7. Concurrency • increase the concurrency by stage • easy to add • https://play.golang.org/p/_qQVZ0Rh73
- 8. Generators vs Pipeline • generators create values when you ask for them • pipeline produce values before you ask for them • https://play.golang.org/p/R_XHMu1jBz
- 9. Real Program Concerns • leaking goroutines • cancelation • errors
- 10. Leaking Goroutines • every goroutine must terminate! • pipelines create many goroutines • https://play.golang.org/p/jUZIv6H4l2
- 11. Cancelation Strategies • exit program • close source stage let downstream drain naturally • close all stages
- 12. Exit program • easy • really really easy
- 13. Close source & drain downstream • source closing trickles down to consumers • all goroutines can exit • https://play.golang.org/p/pRpNspMG2z
- 14. Close all stages • `out <-` must be interruptible everywhere • every channel producing function provided “done” • https://play.golang.org/p/Rd-hEVic0O
- 15. Errors • can happen anywhere • second output channel • https://play.golang.org/p/aUv5MrD7Bj
- 16. Errors Take 2 • https://godoc.org/golang.org/x/net/context • need a “pipeline context” • signal done and place to write errors • WIP idea • https://play.golang.org/p/-sn3QGpc7X
- 17. Pipelines • `out <-` always interruptible by `done` • close what you create • error channel/pipeline context • often repetitive…
- 18. Go Generate
- 19. Go Generate • since go1.4 • “comments” in your go project • //go:generate [command] [args…]
- 20. Just a command • generate say hi (for fun) • generate rm -fr / (for evil) • generate printenv (for understanding)
- 21. Not a shell… • go:generate echo $EDITOR > /tmp/nope • go:generate cat /tmp/file | grep something
- 22. But shellish… • $GOFILE • $GOPACKAGE • $GOARCH • $GOOS • go:generate something -out=$GOARCH_something.go
- 23. Meant to write files • stderr for communicating to the user • silent in proper operation • side effect (file) then committed to repository • can also generate _test.go file
- 24. Pipeliner
- 25. My common pipelines • batch(<-chan Event) (<-chan []Event) • flatMap(<-chan gzipfile) (<-chan Event) • from([]Event) (<-chan Event)
- 26. Pipeliner • side-project • saves me time • may annoy you (bad error messages) • github.com/supershabam/pipeliner