Simple ETL in Python 3.5+ - PolyConf Paris 2017 - Lightning Talk (10 minutes)
0 likes416 views
The document discusses Bonobo, a lightweight ETL (Extract, Transform, Load) framework in Python, highlighting its features, installation process, and basic usage examples. It emphasizes the simplicity and effectiveness of Bonobo for data processing without focusing on big data analytics. Bonobo is open-source, has a comprehensive standard library, and aims to streamline data operations for users.
![Extract Transform Load
• Not new. Popular concept in the 1970s [1] [2]
• Everywhere. Commerce, websites, marketing, finance, …
[1] https://en.wikipedia.org/wiki/Extract,_transform,_load
[2] https://www.sas.com/en_us/insights/data-management/what-is-etl.html](https://image.slidesharecdn.com/polyconf-bonoboetl-170708093211/85/Simple-ETL-in-Python-3-5-PolyConf-Paris-2017-Lightning-Talk-10-minutes-4-320.jpg)
![Code
import bonobo
input_data = (
'Gérard Chamayou, La Géode, sa sphère miroir, in: Le Paris des Centraliens, pp
'Armelle Lavalou (2000), La Villette, Paris: Éditions du Patrimoine, ISBN 2858
'Jean Marie Pérouse de Montclos (1994), Le guide du Patrimoine: Paris, Ministè
)
def transform(line):
return line.split(',')[0]
graph = bonobo.Graph(
input_data,
transform,
print
)](https://image.slidesharecdn.com/polyconf-bonoboetl-170708093211/85/Simple-ETL-in-Python-3-5-PolyConf-Paris-2017-Lightning-Talk-10-minutes-9-320.jpg)
![Docker Extension
$ pip install bonobo[docker]
$ bonobo runc myjob.py
PREVIEW](https://image.slidesharecdn.com/polyconf-bonoboetl-170708093211/85/Simple-ETL-in-Python-3-5-PolyConf-Paris-2017-Lightning-Talk-10-minutes-30-320.jpg)
Simple ETL in Python 3.5+ - PolyConf Paris 2017 - Lightning Talk (10 minutes)
- 1. bonobo Simple ETL in Python 3.5+
- 2. Romain Dorgueil @rdorgueil CTO/Hacker in Residence Technical Co-founder (Solo) Founder Eng. Manager Developer L’Atelier BNP Paribas WeAreTheShops RDC Dist. Agency Sensio/SensioLabs AffiliationWizard Felt too young in a Linux Cauldron Dismantler of Atari computers Basic literacy using a Minitel Guitars & accordions Off by one baby Inception
- 3. Once upon a time…
- 4. Extract Transform Load • Not new. Popular concept in the 1970s [1] [2] • Everywhere. Commerce, websites, marketing, finance, … [1] https://en.wikipedia.org/wiki/Extract,_transform,_load [2] https://www.sas.com/en_us/insights/data-management/what-is-etl.html
- 5. Extract Transform Load foo bar baz Extract Transform Load
- 6. Extract Transform Load foo bar baz Extract Transform Load Transform more Join DB HTTP POST log?
- 7. Let’s see
- 8. Init ~ $ pip install bonobo ~ $ bonobo init polyconf
- 9. Code import bonobo input_data = ( 'Gérard Chamayou, La Géode, sa sphère miroir, in: Le Paris des Centraliens, pp 'Armelle Lavalou (2000), La Villette, Paris: Éditions du Patrimoine, ISBN 2858 'Jean Marie Pérouse de Montclos (1994), Le guide du Patrimoine: Paris, Ministè ) def transform(line): return line.split(',')[0] graph = bonobo.Graph( input_data, transform, print )
- 10. Run $ bonobo run polyconf/ Gérard Chamayou Armelle Lavalou (2000) Jean Marie Pérouse de Montclos (1994) - tuple in=1 out=3 - transform in=3 out=3 - print in=3
- 13. bonobo.run(graph) or in a shell… $ bonobo run main.py
- 16. Transformations ? a.k.a nodes in the graph
- 17. Functions def get_more_infos(api, **row): more = api.query(row.get('id')) return { **row, **(more or {}), }
- 18. Generators def join_orders(order_api, **row): for order in order_api.get(row.get('customer_id')): yield { **row, **order, }
- 19. Iterators extract = ( 'foo', 'bar', 'baz', ) extract = range(0, 1001, 7)
- 20. Classes from bonobo.config import Configurable, Option, Service class QueryDatabase(Configurable): table_name = Option(str, default='customers') database = Service('database.default') def call(self, database, **row): customer = database.query(self.table_name, customer_id=row['clientId' return { **row, 'is_customer': bool(customer), }
- 21. Services ?
- 22. Define as names class QueryDatabase(Configurable): database = Service('database.default') def call(self, database, **row): return { … }
- 23. Runtime injection import bonobo graph = bonobo.Graph(...) def get_services(): return { ‘database.default’: MyDatabaseImpl() }
- 24. Candies
- 25. Library bonobo.FileReader(…) bonobo.CsvReader(…) bonobo.JsonReader(…) bonobo.PickleReader(…) bonobo.ExcelReader(…) bonobo.XMLReader(…) … more to come bonobo.FileWriter(…) bonobo.CsvWriter(…) bonobo.JsonWriter(…) bonobo.PickleWriter(…) bonobo.ExcelWriter(…) bonobo.XMLWriter(…) … more to come
- 27. Console Plugin
- 28. Jupyter Plugin
- 30. Docker Extension $ pip install bonobo[docker] $ bonobo runc myjob.py PREVIEW
- 31. Wrap up
- 32. Young • First commit : December 2016 • 23 releases, ~420 commits, 4 contributors • Current « stable » 0.4.3 • Target : 1.0 early 2018
- 33. 1.0 • 100% Open-Source. • Light & Focused. • Very few dependencies. • Comprehensive standard library. • The rest goes to plugins and extensions.
- 34. Small scale • 1 minute to install • Easy to deploy • NOT : Big Data, Statistics, Analytics … • IS : Lean manufacturing for data
- 35. www.bonobo-project.org github.com/python-bonobo Let me know what you think! Data Processing for Humans
- 37. bonobo @monkcage