YQL & Yahoo! APIs - Open Hack India 2011
6 likes1,899 views
YQL allows users to easily access and manipulate data from web services and APIs by treating their data as rows in database tables. Users can select, filter, and manipulate data from services like Flickr, YouTube, Twitter and more using simple SQL-like queries. YQL also allows users to publish their own custom data tables for others to query. This makes it simple to build apps and prototypes by combining data from different sources.
![URL STRUCTURE
http://query.yahooapis.com/v1/yql?[params]
http://query.yahooapis.com/v1/public/yql?[params]
Params
• q = [ YQL QUERY ]
• format = [ XML / JSON ]
• diagnostics = [ true / false ]
• debug = [ true / false ]
• callback = [ function name ]](https://image.slidesharecdn.com/yqlopenhack2011-110729151146-phpapp01/85/YQL-Yahoo-APIs-Open-Hack-India-2011-26-320.jpg)
![QUERY EXAMPLES
Scrape news from Yahoo! Finance:
select * from html where url="http://finance.yahoo.com/q?
s=yhoo" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a’
Select, filter data from google spreadsheets:
select * from csv where url="https://
spreadsheets.google.com/pub?key=0ArYndzim-
lbrdF8wc3A5QWl1ZGRpdkxRZk80SU9zUXc&output=csv"
and col5 like 'Bangalore%’ ;](https://image.slidesharecdn.com/yqlopenhack2011-110729151146-phpapp01/85/YQL-Yahoo-APIs-Open-Hack-India-2011-30-320.jpg)
YQL & Yahoo! APIs - Open Hack India 2011
- 2. YQL AND YAHOO! APIS Saurabh Sahni YDN Developer, Hacker, Evangelist Twitter: @saurabhsahni
- 3. Hacking together systems in 24 hours is lot of fun
- 4. Data manipulation Hacks = + Data visualization
- 5. The web has lot of data around
- 6. ProgrammableWeb.Com – 3517 APIs
- 7. Yahoo! has opened up its data
- 9. THE TROUBLE WITH DATA • You need to find data API • Get Access – Signup for key • Find data endpoint • Read docs to learn what parameters you have • Get data in obscure format • Use data after converting and filtering • More APIs you use, more is your annoyance
- 10. To make data access easy on the web, Yahoo! created YQL
- 11. YQL turns web services and data on the web into databases.
- 12. select {what} from {where} where {conditions}
- 13. You can select, filter, sort and limit
- 14. You can even insert, update and delete from it.
- 15. FINDING VIDEOS ABOUT BANGALORE SELECT * FROM youtube.search where query='bangalore'
- 16. SELECTING PHOTOS OF HACKDAY SELECT * FROM flickr.photos.search where text="hackday"
- 17. INSERTING DATA INSERT INTO bitly.shorten (login, apiKey, longUrl) VALUES ('ME', 'API_KEY', 'http://yahoo.com')
- 18. UPDATING DATA UPDATE social.profile.status SET status="Using YQL UPDATE” WHERE guid="NJFIDHVPVVISDX7UKED2WHU"
- 19. RETRIEVING MY CONTACTS SELECT * FROM social.contacts WHERE guid=me
- 20. ACCESSING PRIVATE DATA http://query.yahooapis.com/v1/yql Uses OAuth 1.0 for authorization OAuth is complicated – use one of our SDKs at https://github.com/yahoo
- 21. The easiest way to start with YQL is to use the console http://developer.yahoo.com/yql/console
- 26. URL STRUCTURE http://query.yahooapis.com/v1/yql?[params] http://query.yahooapis.com/v1/public/yql?[params] Params • q = [ YQL QUERY ] • format = [ XML / JSON ] • diagnostics = [ true / false ] • debug = [ true / false ] • callback = [ function name ]
- 27. LET’S SEE IT
- 28. QUERY EXAMPLES Find hackday tweets: SELECT * FROM twitter.search where q='hackday’ Search Yahoo! Answers for resolved questions about cars: select * from answers.search where query="cars" and type="resolved” Translate tweets into Hindi: select * from google.translate where q in (select text from twitter.search where q="hackday") and target='hi'; Extract important terms from top stories on Yahoo! news: select * from search.termextract where context in (select description from rss where url='http://rss.news.yahoo.com/rss/topstories')
- 29. QUERY EXAMPLES Find distance between Bangalore and Mumbai: select * from geo.distance where place1="bangalore" and place2="mumbai” Get stock quotes for Yahoo!, Apple, Google & Microsoft in one query: select * from yahoo.finance.quotes where symbol in ("YHOO","AAPL","GOOG","MSFT") Shorten a URL: insert into yahoo.y.ahoo.it (url, keysize) values ('http:// www.javarants.com', 5) Search apartments in criagslist: select * from craigslist.search where location="bangalore" and type="apa" and query="indiranagar”
- 30. QUERY EXAMPLES Scrape news from Yahoo! Finance: select * from html where url="http://finance.yahoo.com/q? s=yhoo" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a’ Select, filter data from google spreadsheets: select * from csv where url="https:// spreadsheets.google.com/pub?key=0ArYndzim- lbrdF8wc3A5QWl1ZGRpdkxRZk80SU9zUXc&output=csv" and col5 like 'Bangalore%’ ;
- 31. Let’s find openhackindia photos on flickr
- 33. MAKING REQUESTS: FLICKR URLS <photo farm="3" id="5708163920" isfamily="0" isfriend="0" ispublic="1" owner="31832337@N04" secret="0075137487" server="2496" title="San Francisco"/>
- 34. MAKING REQUESTS: FLICKR URLS Photo URL http://farm{$farm}.static.flickr.com/{$server}/ {$id}_{$secret}.jpg Photo Page URL http://www.flickr.com/photos/{$owner}/{$id} Photo Owner Profile URL http://www.flickr.com/photos/{$owner}
- 37. Finding Recent Photos from flickr
- 38. Some YQL Hacks
- 39. ChromYQLip is a chrome extension for page scraping via YQL Open Hack Bangalore 2010 Winner
- 41. VIDEO CLIP
- 43. WEBMEME.IN Fetch multiple feeds in different formats like atom, RSS and transform them into consistent RSS format Select * from rss where url in (‘http://feeds.feedburner.com/ pluggd’, ‘http://quatrainman.blogspot.com/atom.xml’, ‘…’) Filter news containing “india” from multiple feeds: select * from rss where url in ('http://feeds.feedburner.com/ TechCrunch', 'http://www.readwriteweb.com/rss.xml','http:// gigaom.com/feed/') and description like '%india%’
- 44. YQL is open – you can get your data tables in our system
- 45. All you need to do is write an XML schema and put it on Github.
- 47. Here is the craigslist search table https://github.com/yql/yql-tables/tree/master/ craigslist/craigslist.search.xml
- 49. USE INSTANTLY BY UPLOADING ON YOUR SITE USE 'http://www.mysite.com/my_table.xml' AS mytable; SELECT * FROM mytable WHERE user='saurabh'
- 50. You can even write server side javascript to build a webservice or augment one. h#p://developer.yahoo.com/yql/guide/yql-‐execute-‐chapter.html
- 51. There are lot of things you can do with YQL.
- 53. RESOURCES All Yahoo! APIs and Services http://developer.yahoo.com/everything.html YQL Documentation http://developer.yahoo.com/yql YQL Console http://developer.yahoo.com/yql/console YQL Github Account (Contribute Tables) http://github.com/yql/yql-tables
- 54. ANY QUESTIONS? http://www.slideshare.net/ saurabhsahni Saurabh Sahni Twitter: @saurabhsahni Github: http://github.com/saurabhsahni Web: http://www.saurabhsahni.com