OpenStack APIs: Present and Future (Beta Talk)
6 likes1,868 views
OpenStack APIs: Present and Future by H. Wade Minter provides an overview of OpenStack storage (Swift) APIs and how to use them. The document discusses how OpenStack is an open source cloud computing platform producing interrelated projects for cloud infrastructure. It then summarizes the Swift storage component and demonstrates making API calls to authenticate, list containers and objects, upload/download objects, and set metadata using CURL examples. Language bindings like Ruby gems simplify using the OpenStack Storage APIs.
![CURL Example - List Containers
$ curl
-H 'X-Auth-Token: 6312661f-1a18-41c8-a823-
d28e44b6c2f9'
-X GET https://
storage101.dfw1.clouddrive.com/v1/
MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0
e7f
.CDN_ACCESS_LOGS
Books
CWX
Disk Images
Raw Video
[...]](https://image.slidesharecdn.com/openstackapispresentandfuture-110621204733-phpapp01/85/OpenStack-APIs-Present-and-Future-Beta-Talk-11-320.jpg)
![CURL Example - List Containers (JSON)
$ curl
-H 'X-Auth-Token: 6312661f-1a18-41c8-a823-
d28e44b6c2f9'
-X GET
https://storage101.dfw1.clouddrive.com/v1/
MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0
e7f?format=json
[{"name":".CDN_ACCESS_LOGS","count":
49,"bytes":31871},{"name":"Books","count":
9,"bytes":48055492},{"name":"CWX","count":
14,"bytes":37933163},{"name":"Disk
Images","count":4,"bytes":6062775740},
{"name":"Raw Video","count":399,"bytes":
29976077454},[....]]](https://image.slidesharecdn.com/openstackapispresentandfuture-110621204733-phpapp01/85/OpenStack-APIs-Present-and-Future-Beta-Talk-12-320.jpg)
![Ruby - Containers
irb(main):003:0> cf.containers
=> [".CDN_ACCESS_LOGS", "Books", "CWX", "Disk
Images", "Raw Video", "a_test_container",...]
irb(main):004:0> c =
cf.create_container('another_container')
=> #<CloudFiles::Container:0x101924120
@name="another_container"....>
irb(main):006:0> cf.containers
=> [".CDN_ACCESS_LOGS", "Books", "CWX", "Disk
Images", "Raw Video", "a_test_container",
"another_container",...]](https://image.slidesharecdn.com/openstackapispresentandfuture-110621204733-phpapp01/85/OpenStack-APIs-Present-and-Future-Beta-Talk-19-320.jpg)
OpenStack APIs: Present and Future (Beta Talk)
- 1. OpenStack APIs: Present and Future OSCON Beta Talk H. Wade Minter Chief Architect, TeamSnap Inc. @minter
- 2. What Is OpenStack? OpenStack is a global collaboration of developers and cloud computing technologists producing the ubiquitous open source cloud computing platform for public and private clouds. The project aims to deliver solutions for all types of clouds by being simple to implement, massively scalable, and feature rich. The technology consists of a series of interrelated projects delivering various components for a cloud infrastructure solution.
- 3. Community Members And Sponsors • Rackspace Hosting (Storage) • NASA (Compute) • Citrix • Dell • Peer1 • Over 70 others
- 4. Why Does OpenStack Matter? "All of the code for OpenStack is freely available under the Apache 2.0 license. Anyone can run it, build on it, or submit changes back to the project. We strongly believe that an open development model is the only way to foster badly-needed cloud standards, remove the fear of proprietary lock-in for cloud customers, and create a large ecosystem that spans cloud providers." Run your own API-driven internal cloud and get access to a wealth of community resources already built on top of it.
- 6. The OpenStack Universe • Storage (Swift) • Compute (Nebula) • Image Service (Glance) • Authentication (Keystone)
- 7. OpenStack Storage "Swift"
- 8. OpenStack Storage • Terminology: "Containers" and "Storage Objects" • Scalable back-end storage for files • Replicates objects in software to multiple drives • Multi-tenant with integrated reseller support • Scales to multi-petabytes and billions of files • Stores objects of unlimited size (in 5GB chunks) • Software written in Python • API-driven access: Native API or S3-Compatible (under development)
- 9. OpenStack Storage Native API • REST API • Accepts data in either XML or JSON • Returns data in either XML or JSON • Authenticates with Username and API Key, returns access token. • Well-documented.
- 10. CURL Example - Authentication $ curl -H 'X-Auth-User: minter' -H 'X-Auth-Key: XXXX' -I -X GET https://auth.api.rackspacecloud.com/v1.0 HTTP/1.1 204 No Content Server: Apache/2.2.13 (Red Hat) X-Storage-Url: https://storage101.dfw1.clouddrive.com/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0e7f Content-Type: application/octet-stream Date: Sat, 18 Jun 2011 18:10:07 GMT X-Auth-Token: 6312661f-1a18-41c8-a823-d28e44b6c2f9 X-Storage-Token: 6312661f-1a18-41c8-a823-d28e44b6c2f9 X-Server-Management-Url: https://servers.api.rackspacecloud.com/ v1.0/361291 Connection: Keep-Alive X-CDN-Management-Url: https://cdn.clouddrive.com/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0e7f Content-Length: 0
- 11. CURL Example - List Containers $ curl -H 'X-Auth-Token: 6312661f-1a18-41c8-a823- d28e44b6c2f9' -X GET https:// storage101.dfw1.clouddrive.com/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0 e7f .CDN_ACCESS_LOGS Books CWX Disk Images Raw Video [...]
- 12. CURL Example - List Containers (JSON) $ curl -H 'X-Auth-Token: 6312661f-1a18-41c8-a823- d28e44b6c2f9' -X GET https://storage101.dfw1.clouddrive.com/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0 e7f?format=json [{"name":".CDN_ACCESS_LOGS","count": 49,"bytes":31871},{"name":"Books","count": 9,"bytes":48055492},{"name":"CWX","count": 14,"bytes":37933163},{"name":"Disk Images","count":4,"bytes":6062775740}, {"name":"Raw Video","count":399,"bytes": 29976077454},[....]]
- 13. CURL - Create Container $ curl -I -H 'X-Auth-Token: 6312661f-1a18-41c8-a823- d28e44b6c2f9' -X PUT https://storage101.dfw1.clouddrive.com/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0 e7f/a_test_container HTTP/1.1 201 Created Content-Length: 18 Content-Type: text/html; charset=UTF-8 X-Trans-Id: txaa7b23c0fb53437cb29b3720102c0528 Date: Sat, 18 Jun 2011 18:23:47 GMT
- 14. CURL - Create Object With Data $ curl -T - -I -H 'Content-Length: 28' -H 'Content-Type: text/plain' -H 'X-Auth-Token: d17f67b1-13da-4352-b769-b9dfe9f06458' -X PUT https://storage101.dfw1.clouddrive.com/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0e7f/ a_test_container/first_object.txt HTTP/1.1 100 Continue This is the body of my file HTTP/1.1 201 Created Content-Length: 118 Content-Type: text/html; charset=UTF-8 Etag: 4ca503d05d7e613d963fac927ac4ce3e Last-Modified: Sat, 18 Jun 2011 20:02:59 GMT X-Trans-Id: tx55427a97e6264ecc9c51edfa7bcf096c Date: Sat, 18 Jun 2011 20:03:21 GMT
- 15. But There Has To Be An Easier Way Language bindings for Rackspace Cloud Files are available. Since OpenStack Storage is Cloud Files minus the CDN, you can use those. Official bindings (github.com/rackspace): • Ruby • Python • PHP • C# • Java
- 16. Ruby Cloud Files Gem https://github.com/rackspace/ruby-cloudfiles gem install cloudfiles require 'cloudfiles' Patches welcome!
- 17. Ruby - OpenStack Authentication irb(main):002:0> require 'cloudfiles' => true irb(main):003:0> cf = CloudFiles::Connection.new(:auth_url => 'http://swift.chmouel.com:8080/auth/v1.0', :username => 'test:tester', :api_key => 'XXXX') => #<CloudFiles::Connection:0x10194d700 @storagescheme="http", @authok=true, @retry_auth=true, @authuser="test:tester", @storageport=8080, @proxy_host=nil, @authtoken="AUTH_tkcb32965fb22c4108b0d8510bcbb86854", @storagepath="/v1/AUTH_01835552-7bf4-4b00-9b90- ab9b4e82e359", @proxy_port=nil, @auth_url="http:// swift.chmouel.com:8080/auth/v1.0", @http={}, @storagehost="127.0.0.1", @snet=nil, @authkey="XXX">
- 18. Ruby - Cloud Files Authentication irb(main):001:0> require 'cloudfiles' => true irb(main):002:0> cf = CloudFiles::Connection.new(:username => 'minter', :api_key => 'XXXX') => #<CloudFiles::Connection:0x10196d9b0 @http={}, @storagehost="storage101.dfw1.clouddrive.com", @cdnmgmtscheme="https", @snet=nil, @cdn_available=true, @proxy_port=nil, @authkey="XXXX", @storagescheme="https", @authok=true, @cdnmgmtport=443, @retry_auth=true, @authuser="minter", @storageport=443, @proxy_host=nil, @cdnmgmtpath="/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0e7f", @authtoken="3f9a3610-beaf-488f-8270-5f54f42619af", @storagepath="/v1/ MossoCloudFS_012559c7-030d-4bbd-8538-7468896b0e7f", @cdnmgmthost="cdn.clouddrive.com", @auth_url="https:// auth.api.rackspacecloud.com/v1.0">
- 19. Ruby - Containers irb(main):003:0> cf.containers => [".CDN_ACCESS_LOGS", "Books", "CWX", "Disk Images", "Raw Video", "a_test_container",...] irb(main):004:0> c = cf.create_container('another_container') => #<CloudFiles::Container:0x101924120 @name="another_container"....> irb(main):006:0> cf.containers => [".CDN_ACCESS_LOGS", "Books", "CWX", "Disk Images", "Raw Video", "a_test_container", "another_container",...]
- 20. Ruby - Objects irb(main):007:0> c = cf.container('another_container') => #<CloudFiles::Container:0x1018be668 @name="another_container", ...> irb(main):008:0> o = c.create_object('myfile.txt') => #<CloudFiles::StorageObject:0x1018ac170 @make_path=false, @name="myfile.txt",...> irb(main):009:0> o.write("O HAI WORLD") => true irb(main):010:0> o.data => "O HAI WORLD"
- 21. Ruby - Convenience Methods irb(main):012:0> o = c.create_object("Code Monkey.mp3") => #<CloudFiles::StorageObject:0x101849570 @make_path=false, @name="Code Monkey.mp3",...> irb(main):014:0> o.load_from_filename('/Users/ minter/Music/iTunes /iTunes Music/Jonathan Coulton/JoCo Looks Back/01 Code Monkey.mp3') => true irb(main):015:0> o.bytes => "7636175"
- 22. Other Notable API Operations • Set metadata for containers and objects • Server-side copy of objects • Create pseudo-hierarchical directory support • Offset/marker support for large collections • Fetching object data via byte offset and range • Chunked encoding for uploads • Manifest support for objects >5G
- 23. Changing The Content-Type $ curl -I -H "X-Auth-Token: XXXXX" http:// swift.chmouel.com:8080/v1/AUTH_test/krack/H. %20Wade%20Minter%20%2D%20RADIO%20H%20%2D %20KRACK%20Live%20%281%2D15%2D2010%29.mp3 HTTP/1.1 200 OK Last-Modified: Tue, 21 Jun 2011 14:56:05 GMT Etag: 608cc90b0a516fed3e10b869e5c2cb6a X-Object-Meta-Foo: bar Accept-Ranges: bytes Content-Length: 88434590 Content-Type: application/octet-stream Date: Tue, 21 Jun 2011 15:02:04 GMT
- 24. Do A Copy To Itself $ curl -I -X PUT -H "X-Auth-Token: XXX" -H "X-Copy-From: /krack/H.%20Wade%20Minter%20%2D%20RADIO %20H%20%2D%20KRACK%20Live%20%281%2D15%2D2010%29.mp3" -H "Content-Type: audio/mpeg" -H "Content-Length: 0" http://swift.chmouel.com:8080/v1/AUTH_test/krack/H.%20Wade %20Minter%20%2D%20RADIO%20H%20%2D%20KRACK%20Live %20%281%2D15%2D2010%29.mp3 HTTP/1.1 201 Created Content-Length: 118 Content-Type: text/html; charset=UTF-8 Etag: 608cc90b0a516fed3e10b869e5c2cb6a X-Copied-From: krack/H.%20Wade%20Minter%20-%20RADIO%20H%20- %20KRACK%20Live%20%281-15-2010%29.mp3 X-Copied-From-Last-Modified: Tue, 21 Jun 2011 14:56:05 GMT X-Object-Meta-Foo: bar Last-Modified: Tue, 21 Jun 2011 15:05:40 GMT Date: Tue, 21 Jun 2011 15:05:42 GMT
- 25. Check The Results $ curl -I -H "X-Auth-Token: XXXX" http://swift.chmouel.com:8080/v1/AUTH_test/ krack/H.%20Wade%20Minter%20%2D%20RADIO%20H %20%2D%20KRACK%20Live20%281%2D15%2D2010%29.mp3 HTTP/1.1 200 OK Last-Modified: Tue, 21 Jun 2011 15:05:40 GMT Etag: 608cc90b0a516fed3e10b869e5c2cb6a X-Object-Meta-Foo: bar Accept-Ranges: bytes Content-Length: 88434590 Content-Type: audio/mpeg Date: Tue, 21 Jun 2011 15:05:53 GMT
- 26. Or Do It The Easy Way (git HEAD) irb(main):012:0> o = c.object(c.objects.first) => #<CloudFiles::StorageObject:0x10174f4a8 @name="H. Wade Minter - RADIO H - KRACK Live (1-15-2010).mp3", ...> irb(main):013:0> o.content_type => "application/octet-stream" irb(main):014:0> o.content_type = "audio/mpeg" => "audio/mpeg" irb(main):015:0> o.refresh => true irb(main):016:0> o.content_type => "audio/mpeg" irb(main):017:0>
- 27. Second-Level Tools • Fog - Cloud abstraction library that supports both Cloud Files and Cloud Servers • paperclip-cloudfiles - Fork of Thoughtbot's paperclip with support for Cloud Files and OpenStack • Carrierwave - Rack/Rails asset storage • Wordpress CDN Tools - Store assets on Cloud Files • Safe - backup tool for local or cloud storage
- 28. Example Time Simple Sinatra app interfacing with Cloud Files
- 29. OpenStack Compute Under Construction
- 30. OpenStack Compute API 1.0 (Cloud Servers) • In production now on Rackspace Cloud Servers • Based on Slicehost technology and API • Integrated server image support • Hooks into Cloud Files for backup storage • IPV4 Only
- 31. OpenStack Compute API 2.0 (NASA Nebula) Enhancements: • IPv6 Support • Returns resources in full URLS (for accessing resources provided elsewhere) Differences: • No support for shared IP groups • No support for backup schedules • Image service soon to move to its own project (Glance) • Adjustable limits (rates, personality files, etc)
- 32. References OpenStack: http://openstack.org/ Rackspace Cloud APIs: http://docs.rackspace.com/api/ Rackspace Github: https://github.com/rackspace/ Me: • Twitter: @minter • Github: https://github.com/minter/ • Email: minter@lunenburg.org • My Startup: http://teamsnap.com/