Simple ETL in python 3.5+ with Bonobo, Romain Dorgueil
1 like1,717 views
This document provides an overview of the bonobo Python library for simple ETL (extract, transform, load) workflows. It discusses the history and concepts of ETL, demonstrates basic usage of bonobo to define extract, transform, and load functions in a graph, and outlines the library's future plans, which include adding support for scheduling, monitoring, and more execution strategies. The goal of bonobo is to provide a Pythonic way to write ETL jobs using code as configuration in a simple and testable way.
![Extract Transform Load
• Not new. Popular concept in the 1970s [1] [2]
• Everywhere. Commerce, websites, marketing, finance, …
• Update secondary data-stores from a master data-store.
• Insert initial data somewhere
• Bi-directional API calls
• Every day, compute time reports, edit invoices if threshold then send e-mail.
• …
[1] https://en.wikipedia.org/wiki/Extract,_transform,_load
[2] https://www.sas.com/en_us/insights/data-management/what-is-etl.html](https://image.slidesharecdn.com/7-170615113258/85/Simple-ETL-in-python-3-5-with-Bonobo-Romain-Dorgueil-8-320.jpg)
![Classes!
from bonobo.config import Configurable, Option, Service
class QueryDatabase(Configurable):
# Simple string option
table_name = Option(str, default='customers')
# External dependency
database = Service('database.default')
def call(self, database, **row):
customer = database.query(self.table_name, customer_id=row['clientId'])
return {
**row,
'is_customer': bool(customer),
}](https://image.slidesharecdn.com/7-170615113258/85/Simple-ETL-in-python-3-5-with-Bonobo-Romain-Dorgueil-54-320.jpg)
![Docker Extension
$ pip install bonobo[docker]
$ bonobo runc myjob.py
PREVIEW](https://image.slidesharecdn.com/7-170615113258/85/Simple-ETL-in-python-3-5-with-Bonobo-Romain-Dorgueil-71-320.jpg)
Simple ETL in python 3.5+ with Bonobo, Romain Dorgueil
- 1. bonobo Simple ETL in Python 3.5+
- 2. Romain Dorgueil @rdorgueil CTO/Hacker in Residence Technical Co-founder (Solo) Founder Eng. Manager Developer L’Atelier BNP Paribas WeAreTheShops RDC Dist. Agency Sensio/SensioLabs AffiliationWizard Felt too young in a Linux Cauldron Dismantler of Atari computers Basic literacy using a Minitel Guitars & accordions Off by one baby Inception
- 4. STARTUP ACCELERATION PROGRAMS NO HYPE, JUST BUSINESS launchpad.atelier.net
- 5. bonobo Simple ETL in Python 3.5+
- 6. • History & Market • Extract, transform, load • Basics – Bonobo One-O-One • Concepts & Candies • The Future – State & Plans
- 7. Once upon a time…
- 8. Extract Transform Load • Not new. Popular concept in the 1970s [1] [2] • Everywhere. Commerce, websites, marketing, finance, … • Update secondary data-stores from a master data-store. • Insert initial data somewhere • Bi-directional API calls • Every day, compute time reports, edit invoices if threshold then send e-mail. • … [1] https://en.wikipedia.org/wiki/Extract,_transform,_load [2] https://www.sas.com/en_us/insights/data-management/what-is-etl.html
- 9. Small Automation Tools • Mostly aimed at simple recurring tasks. • Cloud / SaaS only.
- 10. Data Integration Tools • Pentaho Data Integration (IDE/Java) • Talend Open Studio (IDE/Java) • CloverETL (IDE/Java)
- 14. Data Integration Tools • Java + IDE based, for most of them • Data transformations are blocks • IO flow managed by connections • Execution
- 15. In the Python world … • Bubbles (https://github.com/stiivi/bubbles) • PETL (https://github.com/alimanfoo/petl) • and now… Bonobo (https://www.bonobo-project.org/) You can also use amazing libraries including Joblib, Dask, Pandas, Toolz, but ETL is not their main focus.
- 16. Big Data Tools • Can do anything. And probably more. Fast. • Either needs an infrastructure, or cloud based.
- 17. Story time
- 18. Partner 1 Data Integration
- 19. WE GOT DEALS !!!
- 20. Partner 1 Partner 2 Partner 3 Partner 4 Partner 5 Partner 6 Partner 7 Partner 8 Partner 9 …
- 21. Tiny bug there… Can you fix it ?
- 23. I want… • A data integration / ETL tool using code as configuration. • Preferably Python code. • Something that can be tested (I mean, by a machine). • Something that can use inheritance. • Fast install on laptop, thought to run on servers too.
- 25. It is … • A framework to write ETL jobs in Python 3 (3.5+) • Using the same concepts as the old ETLs • But for coders! • You can use OOP!
- 26. It is NOT … • Pandas / R Dataframes • Dask • Luigi / Airflow • Hadoop / Big Data • A monkey
- 27. Bonobo - One - O - One
- 28. Let’s create a project
- 29. Let’s create a project ~ $ pip install bonobo cookiecutter
- 30. Let’s create a project ~ $ pip install bonobo cookiecutter ~ $ bonobo init pyparis
- 31. Let’s create a project ~ $ pip install bonobo cookiecutter ~ $ bonobo init pyparis ~ $ cd pyparis
- 32. Let’s create a project ~ $ pip install bonobo cookiecutter ~ $ bonobo init pyparis ~ $ cd pyparis ~/pyparis $ ls -lsah total 32 0 drwxr-xr-x 7 rd staff 238B 30 mai 18:02 . 0 drwxr-xr-x 4 rd staff 136B 30 mai 18:03 .. 0 -rw-r--r-- 1 rd staff 0B 30 mai 18:02 .env 8 -rw-r--r-- 1 rd staff 6B 30 mai 18:02 .gitignore 8 -rw-r--r-- 1 rd staff 333B 30 mai 18:02 main.py 8 -rw-r--r-- 1 rd staff 140B 30 mai 18:02 _services.py 8 -rw-r--r-- 1 rd staff 7B 30 mai 18:02 requirements.txt
- 33. It works! $ bonobo run . 1 3 5 ... 37 39 41 - range in=1 out=42 - Filter in=42 out=21 - print in=21
- 35. Let’s rewrite main.py import bonobo
- 36. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i
- 37. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n
- 38. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n)
- 39. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, )
- 40. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph)
- 41. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42)
- 42. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42) bonobo.Filter(lambda n: n % 2)
- 43. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42) bonobo.Filter(lambda n: n % 2) print
- 44. Let’s rewrite main.py import bonobo def extract(): for i in range(42): yield i def transform(n): if n % 2: yield n def load(n): print(n) graph = bonobo.Graph( extract, transform, load, ) if __name__ == '__main__': bonobo.run(graph) range(42) bonobo.Filter(lambda n: n % 2) print graph = bonobo.Graph( range(42), bonobo.Filter(filter=lambda x: x % 2), print, ) if __name__ == '__main__': bonobo.run(graph)
- 47. bonobo.run(graph) or in a shell… $ bonobo run main.py
- 50. Transformations ? a.k.a nodes in the graph
- 51. Functions! def get_more_infos(api, **row): more = api.query(row.get('id')) return { **row, **(more or {}), }
- 52. Generators! def join_orders(order_api, **row): for order in order_api.get(row.get('customer_id')): yield { **row, **order, }
- 53. Iterators! extract = ( 'foo', 'bar', 'baz', ) extract = range(0, 1001, 7)
- 54. Classes! from bonobo.config import Configurable, Option, Service class QueryDatabase(Configurable): # Simple string option table_name = Option(str, default='customers') # External dependency database = Service('database.default') def call(self, database, **row): customer = database.query(self.table_name, customer_id=row['clientId']) return { **row, 'is_customer': bool(customer), }
- 55. Services ?
- 56. get_services() COINBASE_API_KEY, COINBASE_API_SECRET = environ.get('...'), environ.get('...') KRAKEN_API_KEY, KRAKEN_API_SECRET = environ.get('...'), environ.get('...') POLONIEX_API_KEY, POLONIEX_API_SECRET = environ.get('...'), environ.get('...') def get_services(): return { 'fs': bonobo.open_fs( path.join(path.dirname(__file__), './data') ), 'coinbase.client': coinbase.wallet.client.Client( COINBASE_API_KEY, COINBASE_API_SECRET ), 'kraken.client': krakenex.API( KRAKEN_API_KEY, KRAKEN_API_SECRET ), 'poloniex.client': poloniex.Poloniex( apikey=POLONIEX_API_KEY, secret=POLONIEX_API_SECRET ), 'google.client': get_google_api_client(), }
- 57. run() will handle injection import bonobo from bonobo.commands.run import get_default_services graph = bonobo.Graph() if __name__ == '__main__': bonobo.run( graph, services=get_default_services(__file__) )
- 58. … class QueryDatabase(Configurable): # External dependency database = Service('database.default') def call(self, database, **row): return { … }
- 59. Bananas!
- 60. Files bonobo.FileReader( path: str, *, eol: str, encoding: str, ioformat, mode: str ) bonobo.FileWriter( path: str, *, eol: str, encoding: str, ioformat, mode: str )
- 61. CSV bonobo.CsvReader( path: str, *, eol: str, encoding: str, ioformat, mode: str, delimiter: str, quotechar: str, headers: tuple, skip: int ) bonobo.CsvWriter( path: str, *, eol: str, encoding: str, ioformat, mode: str, delimiter: str, quotechar: str, headers: tuple )
- 62. JSON bonobo.JsonReader( path: str, *, eol: str, encoding: str, ioformat, mode: str ) bonobo.JsonWriter( path: str, *, eol: str, encoding: str, ioformat, mode: str )
- 63. Pickle bonobo.PickleReader( path: str, *, eol: str, encoding: str, ioformat, item_names: tuple, mode: str ) bonobo.PickleWriter( path: str, *, eol: str, encoding: str, ioformat, item_names: tuple, mode: str )
- 66. Filters @bonobo.Filter def NoneShallPass(username): return username == 'romain' bonobo.graph( # ... NoneShallPass(), # ... ) bonobo.Filter(lambda username: username == 'romain')
- 67. More bananas!
- 69. Jupyter Notebook
- 71. Docker Extension $ pip install bonobo[docker] $ bonobo runc myjob.py PREVIEW
- 72. State & Plans
- 73. RIP rdc.etl • 2013-2015, python 2.7. • Brings most of the execution algorithms. • Different context. • Learn from mistakes.
- 74. Bonobo is young • First commit : December 2016 • 21 releases, ~400 commits, 4 contributors • Current stable 0.4.1, released yesterday
- 75. __future__ • Stay 100% Open-Source • Light & Focused : Graphs and Executions • The rest goes to Extensions.
- 76. Pre 1.0 Roadmap • Towards stable API • More tests, more documentation • Developer experience first Target: 1.0 before 2018
- 77. Post 1.0 Roadmap • Scheduling • Monitoring • More execution strategies • Optimizations
- 78. Hacker News is crazy
- 79. Data Processing for Humans
- 80. www.bonobo-project.org docs.bonobo-project.org bonobo-slack.herokuapp.com github.com/python-bonobo Let me know what you think!
- 82. bonobo @monkcage