Intro to-technologies-Green-City-Hackathon-Athens
0 likes516 views
This document provides an introduction to technologies that can be used to build real services using open agro-biodiversity data. It discusses technologies like cloud infrastructure, REST, data formats like JSON, sensors, big data technologies, natural language processing, image processing, machine learning, analytics, and frameworks like MVC. It also covers sharing data through metadata aggregation and linking data using semantic technologies. The goal is to explain how these technologies can be leveraged to get value from open agro-biodiversity data and build useful applications and services.
Intro to-technologies-Green-City-Hackathon-Athens
- 1. Turning agro-‐biodiversity open data to real services: an intro to technologies Giannis Stoitsis Agro-‐Know Technologies
- 2. AIM OF THE COURSE
- 3. How to get as much as possible from data? Which technologies and how they can be used?
- 4. Course syllabus • Intro • Technologies • Sharing data • Linking data • Architectural aspects • Building a simple case • Some suggesFons
- 5. ARE WE READY TO BUILD REAL SERVICES USING AGRO AND BIODIVERSITY OPEN DATA?
- 6. What we need data technology and of course a good idea
- 7. As for the data Thanks Nikos “Intro to green data ecosystems: why is a data-‐ powered tech start up for agriculture & biodiversity going to be profitable?”
- 8. TECHNOLOGIES: THERE IS REALLY GREAT STAFF OUT THERE
- 9. Infrastructure • We have the cloud • Cloud is geJng cheaper, beLer and more development friendly • OpFons – Amazon – Rackspace – Linode – MicrosoQ – Google – Okeanos – Other Private or even yours (e.g. OpenStack, OpenNebula)
- 10. Storage and Processing Monitoring/Management/Allocation layer Virtualization of Infrastructure Layer Virtual Machines Virtualization of Infrastructure LayerVirtualized Infrastractures Management Layer GUI tools and APIs Cloud provider A Cloud provider B Cloud provider B
- 11. What does this mean? Quick prototyping Get your running instance in minutes
- 12. REST • AbstracFng the architectural elements • Close to the way we are working in the web • Standard hLp methods – GET – PUT – POST – DELETE • easy to be built especially with MVC frameworks e.g. Rails • easy to be consumed with any language GET hLp://www.example.com/User/1234 POST hLp://www.example .com/User/ ‘{“user”:”{“”id”:”55”, “name”:”John”}’ DELETE hLp://www.example.com/User/1234
- 13. REST clients • easy to be developed with any language • check before start developing
- 14. Data exchange formats • XML, RDF or JSON? • They all work but we always have preferences • Why we love JSON? – can be parsed by any front end (web, mobile) – simple – can be used to represent data models – beLer in processing e.g. transformaFon – used widely by all document based repositories (e.g. MongoDB, elasFcsearch)
- 15. Sensors • Are everywhere • Two categories – The one that we hold i.e. from smartphones today from watch tomorrow – The one installed at sites e.g. fields • We can get and use data from both
- 16. get data from a smartphone • sensing data – MoFon sensor/accelerometer – Proximity – gyroscope (for gaming) • GPS: Core LocaFon Framework to get the coordinates and tracking user’s locaFon
- 17. GeJng real Fme weather data
- 18. GeJng real Fme weather data h:p://api.openweathermap.org/data/2.5/weather?q=London,uk
- 19. Big data technologies • A trend? • Some of them – NoSQL dbs e.g. MongoDB, CouchDB – elasFcsearch – MapReduce framework e.g. run processes over a big amount of data – Hadoop – Cassandra – and more ….
- 20. Natural Language Processing • Dbpedia Spotlight • EnFty recogniFon (idenFfy names and places in a text) • DisambiguaFon (apple can be a company, a fruit or a river)
- 21. Image processing APIs for mobile apps Open Source Libraries
- 22. how image processing can be used? • Similar to how Google Goggles works but trained for your data
- 23. Machine Learning
- 24. how machine learning can be used? Provide a recommendaFon service Mine and Cluster Data
- 25. JavaScript • can do almost everything • even image processing e.g. PixasFc for image processing • powerful libraries e.g. D3 for real Fme data visualizaFon
- 26. AnalyFcs • Track the users and events of your applicaFon – ElasFc Search, MongoDB – Google AnalyFcs
- 27. Follow a well accepted framework • Model View Controller (MVC) • There are many nice MVC based frameworks – Rails – Spring – Zend – Yii – Laravel – and more …
- 28. SHARING DATA
- 30. the value is sFll in metadata • Metadata connect us to the real data – Search metadata records not raw data • Metadata gives the context – themaFc classificaFon – target users • Well defined standards (e.g. DC, IEEE LOM, Darwin Core, ABCD) • RDF, XML and JSON binding
- 32. … the power is at the back end CulQvaQon HarvesQng Blossom Unorganized content in local and remote sites Organized and structured content in local and remote databases EducaQonal Bibliographic EducaQonal Geographical Bibliographic Agricultural Data PlaVorm Aggregate data from diverse sources Work with different type of data Prepare data for meaningful services IngesQon Enrichment TranslaQon Publishing Data discovery services Widgets Authoring services AnalyQcs services
- 33. Why to aggregate? • Enrich metadata • Provide developer friendly APIs that can be used to build good services • Data providers rarely have APIs
- 34. LINKING DATA
- 35. Why we sFll have data silos? • CompeFng metadata standards (e.g. DC, IEEE LOM) • Diversity of web interfaces (e.g. REST, OAI-‐PMH, SOAP, SPI, SQI) • Different exchange format (e.g. XML, RDF, JSON) • Fragmented use of texonomies LD for educational data/resource sharing Overview Approaches for LD in educational data sharing On the-fly/automated integration of heterogeneous APIs and data (http://www.meducator.net) Dataset (transformation and) cataloging (http://linkedup-project.eu) ? We are sFll here … … and not here …
- 36. What is Linked Data? • A set of principles and technologies for the Web of Data • Principles – Put the data online with permanent address (URIs) – Describe the data with a standard representaFon (RDF) – Link to other data through published taxonomies • Technologies – Triple stores – SPARQL – RDF, OWL – SWRL
- 37. Seman&c Web / Linked Data Technologies by Mathieu d'Aquin on Sep 11, 2013 ApplicaFons
- 38. Mash up and Linked Data: FAO case
- 39. How it works? • Based on AGROVOC • Using AGROVOC mappings to other ontologies e.g. GeopoliFcal and DBpedia <hLp://aims.fao.org/aos/agrovoc/c_690> <hLp://www.w3.org/2004/02/skos/core#closeMatch> <hLp://dbpedia.org/resource/Salmon> <hLp://dbpedia.org/page/Salmon> replace resource with page and get rdf
- 40. Why to rely on linked data • Easy to be processed e.g. microdata, rdf • Get more out of data through linking e.g. get the staFsFcs for a country through GeoPoliFcal Ontology Web resource about Greece annotated with Agrovoc term “Greece” hLp://www.fao.org/countryprofiles/geoinfo/geopoliFcal/data/Greece Country GDP
- 41. and even more … • Data that you can get from World Bank based on ISO3 country code – Agricultural irrigated land (% of total agricultural land) – Cereal yield (kg per hectare) – Rural populaFon (% of total populaFon) – Poverty headcount raFo at rural poverty line (% of rural populaFon) – Agricultural machinery, tractors per 100 sq. km of arable land – Arable land (% of land area) – CO2 emissions (kt) – Fish species, threatened
- 42. DECOUPLING FRONT END FROM BACK END
- 43. how it works Template customizaQon html, css, Ajax, JS Data mash up Search API Cloud Edu resources Data processing and enrichment e.g. semanQc enrichment or clustering Search API Cultural resources from Museums
- 44. how it works Template customizaQon html, css, Ajax, JS widget in Facebook page Data mash up Search API Cloud Edu resources Data processing and enrichment e.g. semanQc enrichment or clustering Search API Cultural resources from Museums
- 45. AUTHENTICATION
- 47. NATIVE OR WEB APP?
- 48. NaFve vs Web app • Depends on the business case • Responsive web app are cross plaworm • Responsive design is great but it is not so friendly as naFve app • NaFve app has nice workflow • NaFve app can use device sensors • In many data-‐powered cases you need both
- 49. BUILDING A BASIC APP IN HOURS
- 50. THE IDEA
- 51. A mobile app for discovering green learning resources • Similar apps – OER commons – iMarine • Similar web app – hLp://www.greenlearningnetwork.org/ organicedunet/
- 52. Which data • Learning Resources about environment, agriculture and biodiversity • API end point hLp://83.212.96.219:8080/ glnRepo/api/ariadne/restp?json=Χ&callback=Y
- 53. Recipe • Mobile app SDK e.g. XCODE • Mobile simulator • Some reading • Similar examples e.g. Searching in twiLer • A workflow • JSON parser • No beauFes at this stage
- 55. Defining the workflow is a game
- 56. ConnecFng to the data Use NSJSONSerializaFon Class to get the data
- 57. and finally
- 58. SuggesFons • focus on the idea and business value • re-‐use the workflow of successful apps • processing gives the value • do not create unnecessary procedures e.g. RegistraFon page • re-‐use exisFng data • put emphasis on the value of your app and not on the complexity of the technology • always check first if there is open source code • design maLers • go beyond trivial and benefit from innovaFon
- 59. thank you! stoitsis@agroknow.gr