OpenERP data integration in an entreprise context: a war story
13 likes6,516 views
The document discusses various methods for integrating OpenERP as an enterprise app platform, focusing on techniques such as importing CSV files and using ORM APIs. It highlights the use of import_odbc for ETL processes, as well as challenges associated with data synchronization and the importance of efficient data handling. Additionally, it suggests considering alternative solutions like Pentaho Kettle for complex transformations and discusses potential improvements in the integration process.
![ORM API: load()
product_categ.load(cr, uid, # import_data() on v6.1
[id , name ], # field names
[ ['product_categ_X', 'Categ X'], # data rows
['product_categ_Y', 'Categ Y'],
]
product_product.load(cr, uid,
[id , name , ref , categ_id.id ],
[ ['product_product_aa', 'Prod A', 'AA', 'product_categ_X'],
['product_product_bb', 'Prod B', 'BB', 'product_categ_X'],
['product_product_cc', 'Prod C', 'CC', 'product_categ_Y'],
]
product_categ.load(cr, uid, # import_data() on v6.1
[id , name ], # field names
[ ['product_categ_X', 'Categ X'], # data rows
['product_categ_Y', 'Categ Y'],
]
product_product.load(cr, uid,
[id , name , ref , categ_id.id ],
[ ['product_product_aa', 'Prod A', 'AA', 'product_categ_X'],
['product_product_bb', 'Prod B', 'BB', 'product_categ_X'],
['product_product_cc', 'Prod C', 'CC', 'product_categ_Y'],
]](https://image.slidesharecdn.com/openerpdataintegration20130704-130704175021-phpapp02/85/OpenERP-data-integration-in-an-entreprise-context-a-war-story-6-320.jpg)
OpenERP data integration in an entreprise context: a war story
- 1. OpenERP OpenERP Days 2013OpenERP Days 2013 OpenERP enterprise integration: A war story
- 2. Daniel Reis IT Applications Manager Securitas Portugal @reis_pt daniel.reis@securitas.pt https://launchpad.net/~dreis-pt
- 3. OpenERP as an Enterprise App platform OpenERP Corporate ERP Customers Contracts Analytic accounts HRMS Employees Departments Op Systems Serviced sites Serviced systems
- 4. The basic method: import CSV file id, name, ref, categ_id product_product_aa, Prod A, AA, product_categ_X product_product_bb, Prod B, BB, product_categ_X product_product_cc, Prod C, CC, product_categ_Y
- 5. External Identifiers aka XML IDs XML ID Model Database ID product_product_aa product.product 879 product_product_bb product.product 880 product_product_cc product.product 881 product_product_AAA product.product 879 product_categ_X product.categ 45 product_categ_y product.categ 56
- 6. ORM API: load() product_categ.load(cr, uid, # import_data() on v6.1 [id , name ], # field names [ ['product_categ_X', 'Categ X'], # data rows ['product_categ_Y', 'Categ Y'], ] product_product.load(cr, uid, [id , name , ref , categ_id.id ], [ ['product_product_aa', 'Prod A', 'AA', 'product_categ_X'], ['product_product_bb', 'Prod B', 'BB', 'product_categ_X'], ['product_product_cc', 'Prod C', 'CC', 'product_categ_Y'], ] product_categ.load(cr, uid, # import_data() on v6.1 [id , name ], # field names [ ['product_categ_X', 'Categ X'], # data rows ['product_categ_Y', 'Categ Y'], ] product_product.load(cr, uid, [id , name , ref , categ_id.id ], [ ['product_product_aa', 'Prod A', 'AA', 'product_categ_X'], ['product_product_bb', 'Prod B', 'BB', 'product_categ_X'], ['product_product_cc', 'Prod C', 'CC', 'product_categ_Y'], ]
- 7. Import CSV file summary Official method Uses ORM External identifiers Many-to-one support
- 8. Automate ?
- 9. import_odbc
- 10. SQL import module solution OpenOpenERPERP 1. Query external database 2. Build import data structure 3. Call load()
- 11. Build "CSV file" from SQL id, name, ref, categ_id product_product_AA, Prod A, AA, product_categ_X product_product_BB, Prod B, BB, product_categ_X product_product_CC, Prod C, CC, product_categ_Y SELECT product.id as “None”, -- generates id column product.name as “name”, product.id as “ref”, 'product_categ_' + product.category_id as “categ_id” FROM product WHERE change_timestamp >= ? -- last sync date
- 13. SQL import module summary OpenOpenERPERP lp:openobject-extension (https://code.launchpad.net/~extra-addons-commiter/openobject-extension ) Concerns Runs in OE server DB middleware installation SQL sources only SQL transforms only
- 15. Another class of solutions: ETL & Pentaho Kettle OpenOpenERPERP
- 16. ETL allows for complex transformations and validations...
- 17. The naive way: direct table loading. (Don't do this!)
- 18. Pentaho Kettle ETL summary http://kettle.pentaho.com Concerns Learning curve OpenOpenERPERP
- 20. OpenOpenERPERP import_sqlpush: push and load solution 1. ETL push data a. Get data b. Build import line c. Write to queue table 2. OE scheduled import a. Read queue table b. Run import_data() c. Update queue status
- 21. An example: import categorized products
- 23. Import line built by the "data" column
- 25. import_sqlpush: push and load summary https://code.launchpad.net/~dreis-pt/reis-openerp-addons/trunk Concerns Two step latency Process in OpenERP server Data rows building logic OpenOpenERPERP
- 27. OpenOpenERPERP OpenERP output step solution ETL Steps 1. Get data 2. Transformations 3. OpenERP output
- 29. An example: Customers and Contracts.
- 31. OpenOpenERPERP OpenERP output step issues: roundtrips & no XML Ids 1. search() all model ids 2. read() all lookup fields 3. Find ids for input rows 4. Call import_data()
- 32. Pentaho Kettle ETL summary http://tinyurl.com/openerpstep https://bugs.launchpad.net/openobject-server/+bug/1186977 Concerns Slow (roundtrips) No XML ID support Needs server patch on v7 OpenOpenERPERP
- 33. import_odbc etl table loading import_sqlpush openerp steps ... better steps...
- 34. Concerns OpenOpenERPERP OpenERP output step: evolution is in the works ... ? http://jira.pentaho.com/browse/PDI-9974 http://jira.pentaho.com/browse/PDI-9975
- 35. Conclusions: things we should pay attention to ... ● How fast? ● Uses OpenERP ORM? ● How easy to use? ● Interferes with other OE server processes? ● What data source types can be used? ● What type of transformations can be used?
- 36. Integration techniques wrap up ● import_odbc: simple, maybe too simple. ● Table loading: don't do this at home kids... ● import_sqlpush: good, but could be simpler. ● OpenERP Output Step: great, but slow ...
- 37. http://www.mulesoft.com/mule-esb-features Going further: why not use an ESB?
- 38. Also have a look at other options ● TerminatOOR: Kettle Openerp step using Ruby script https://github.com/akretion/ooor ● Talend Data Integration: ETL task code generator (Java) http://www.talend.com/products/data-integration