Eran Rom, IBM - Bringing Compute to Openstack Swift, OpenStack Israel 2015
Download as PPTX, PDF1 like851 views
The document discusses IBM's Storlet engine, which extends the OpenStack Swift object storage system. The Storlet engine allows computations to be pushed down to storage nodes rather than bringing data to computation nodes. It uses Docker containers to run user-defined storlet programs on Swift storage nodes. Storlets can be invoked on GET and PUT requests to objects in Swift to transform or analyze data during storage operations. The Storlet engine is open source and available on GitHub for others to experiment with and integrate into applications.
Eran Rom, IBM - Bringing Compute to Openstack Swift, OpenStack Israel 2015
- 1. © 2015 IBM Corporation Bringing Compute to Openstack Swift Eran Rom, System and Storage Researcher, IBM Research - Haifa
- 2. © 2015 IBM Corporation2 The Storlet engine extends Openstack Swift by enabling the push down of filtering, transforming and analysis tasks to Swift instead of bringing the data to the computation.
- 3. © 2015 IBM Corporation3 Proxy Tier Storage Tier Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff
- 4. © 2015 IBM Corporation4 Containers Powered Swift Cluster Proxy Tier Storage Tier
- 5. © 2015 IBM Corporation5 Image Registry Proxy Tier Storage Tier Containers Powered Swift Cluster Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff
- 6. © 2015 IBM Corporation6 Containers Powered Swift Cluster Image Registry Proxy Tier Storage Tier Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff Ubuntu 14.04 Storlet Stuff
- 7. © 2015 IBM Corporation7 Proxy Tier Storage Tier Developer Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Writing and Deploying a Storlet @Override public void invoke(ArrayList<StorletInputStream> inStreams, ArrayList<StorletOutputStream> outStreams, Map<String, String> parameters, StorletLogger logger) throws StorletException { … }
- 8. © 2015 IBM Corporation8 Proxy Tier Storage Tier Developer Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Writing and Deploying a Storlet PUT http://my.swift.co.il/v1/AUTH_111/Storlet/watermarkstorlet-10.jar
- 9. © 2015 IBM Corporation9 Proxy Tier Storage Tier Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Invoking a Storlet on GET GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webm X-Run-Storlet : watermarkstorlet-10.jar
- 10. © 2015 IBM Corporation10 Proxy Tier Storage Tier Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Invoking a Storlet on GET Forward Request to an Object Server holding a Replica GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webm X-Run-Storlet : watermarkstorlet-10.jar
- 11. © 2015 IBM Corporation11 Invoking a Storlet on GET Ubuntu 14.04 Storlet Stuff ffmpeg Intercept Request Invoke Deamon, and Establish Connection with the Account’s Docker Container Pass Input and Output FDs Send back the Object’s Metadata and the Object’s Data Stream Continue the GET Flow, where the Storelt’s response is sent back to the Proxy Object Server
- 12. © 2015 IBM Corporation12 Proxy Tier Storage Tier Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Ubuntu 14.04 Storlet Stuff ffmpeg Invoking a Storlet on PUT PUT http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webm X-Run-Storlet : MetadataExtractorStorlet-10.jar
- 13. © 2015 IBM Corporation13 Invoking a Storlet on PUT Ubuntu 14.04 Storlet Stuff ffmpeg Intercept Request Invoke Deamon, and Establish Connection with the Account’s Docker Container Pass Input and Output FDs Send back the Object’s Metadata and the Object’s Data Stream Continue the Proxy PUT Flow with the returned Metadata and data stream Proxy Server
- 14. © 2015 IBM Corporation14 Demo http://159.8.49.36:8080/raimediabridge/login.html
- 15. © 2015 IBM Corporation15 The Storlet Engine is in the Open https://github.com/Open-I-Beam/swift-storlets Comprehensive Documentation • Installation guide • How to write, deploy and invoke storlets guide • How to tailor your own image for running storlets S2AIO • Install swift and storlets in a single VM for experimentation Public users will be able to experiment with the Storlet engine thanks to FiWare – An EU research project delivering a novel service infrastructure for the future internet (http://www.fiware.org/)
- 16. © 2015 IBM Corporation16 Media Workflow at NAB (with RAI) Storlets run as material is uploaded and downloaded Metadata Enrichment Storlet Swift Object Store Upload Enriched Metadata Data myvideo.mxf Metadata Data myvideo.mxf Data Loudness Calculation Storlet <Loudness,-23> myvideo.mxf Fix Loudness Storlet
- 17. © 2015 IBM Corporation17 Acknowledgements The Storlets project is partially funded by the following EU research projects Digital Preservation Inspired by Human Memory IoT for Smart Cities