Let your data shine... with OpenRefine
1 like1,162 views
- OpenRefine is a free, open source tool for cleaning and transforming messy data. - It allows users to work with data in a visual, interactive way through facets, clustering, and scripting capabilities like GREL. - Common tasks include cleaning data, transforming formats, reconciling concepts with external sources, and cross-referencing data.
![● Add columns by fetching url
■ find and replace
■ string parsing & splitting
■ add column based on column”straat” (value+”%20”+cells[‘huisnummer’].value)
■ Call google API (or openstreetmap or….) ("https://maps.googleapis.
com/maps/api/geocode/json?address="+value+ cells["huisnummer"].
value&key=AIzaSyDY2Z6wehbIqIPrHIb9ljC62pwRqEHOous")
■ Parse JSON (value.parseJson()["results"][0]["geometry"]["location"]["lng"])
Hands on: georeferencing](https://image.slidesharecdn.com/3-160307123921/85/Let-your-data-shine-with-OpenRefine-16-320.jpg)
![● Merge data from the two projects by creating a new column from values from
an existing column within one project that are used to index into a similar
column in the other project
○ cell.cross("datasetname.csv","scientificName").cells["order"].value[0]
Hands on: cross referencing](https://image.slidesharecdn.com/3-160307123921/85/Let-your-data-shine-with-OpenRefine-19-320.jpg)
Let your data shine... with OpenRefine
- 1. Let your data shine… with OpenRefine Open Belgium 2016 OpenRefine workshop Brosens - Desmet
- 2. What people say: tweets @bartox: "Damn! Wish I had this 5 years ago! RT @swiertz nice tools ! Format & clean your data with Google Refine http: //goo.gl/UniR6 #cleanup #tools" view tweet @Musebrarian: "YIPEEEE! Google Refine works with OAI-PMH XML out of the box. This is going to make my life much easier." view tweet @kb: "It’s kind of ridiculous how exciting I find this: https://code.google.com/p/google-refine/" view tweet @litcritter: "I rarely feel the desire to kiss a corporation on the mouth, but Google Refine is making me come close http: //goo.gl/8pvKB #datageek" view tweet
- 3. @LearonDalby: "I'm sold on #Google #Refine used it most of the day with "messy" data and managed to clean nearly all of it." view tweet @roolio: "Today google #refine saved my afternoon. Every #data #hacker should try it" view tweet @Salesient: "Google refine is awesome. Never before have I been home this early." view tweet @Mayin: "Not only will it clean your data, Google Refine will slice, dice and put bows on your hairdo!http://bit.ly/cPGn1E Rocks data exploration." view tweet @marklabedz: "Google Refine: Making interns unneccesary since 2010." view tweet @naterkane: "i'm completely in love with Google Refine. fo' reals." view tweet @LearonDalby: "Using #Google #Refine makes me happy. Even for the easy stuff." view tweet @loranstefani: "Google Refine: love at first click" view tweet @tracystan: "Google Refine is gonna change my life" view tweet What people say: tweets
- 4. "Google Refine isn’t going to solve the problem of poor data availability, but for those who manage to gain access to existing records, it can be a powerful tool for transparency." Rebekah Heacock, co-director of the Technology for Transparency Network and a Project Coordinator at Harvard’s Berkman Center for Internet and Society - Sunlight Foundation, Tools for transparency: Google Refine. "Google Refine is an immensely powerful tool for dealing with "messy" data, and it sports a myriad of advanced features for massaging and analyzing complex data sets" Dmitri Popov (Linux Magazine) - Use Google Refine to Massage Your Data "For anyone who’s ever had to sort through messy data to try to turn up a meaningful treatment, and who hasn’t, this tool is a godsend." Michael Lines, SLAW - Google Refine 2.0 "Google Refine 2.0 will serve an excellent back-end for data visualization services. It has been well received by the Chicago Tribune and open-government data communities. Along with Google Squared, Refine 2.0 can create a powerful research tool." Chinmoy Kanjilal, Techie Buzz - Google Refine 2.0: Power Tools for Working With Data What people say: blogs
- 5. ● Formerly known as Google Refine, now OpenRefine ● Site: http://openrefine.org ● Github: https://github.com/OpenRefine ● Used for ○ Data cleaning (detect and correct anomalies) ○ Transform data (change format, change datatype) ○ “Pimp” & “link” data (harvest & connect data from online databases) ● More powerful than a worksheet ● More visual than scripting A free, open source, powerful tool for working with messy data
- 6. ● Supported by a large community (lots of tutorials and plugins) ● Works quite well up to 100.000 rows of data ● Supports several file formats ● The original file is unaffected ● OpenRefine runs in a modern browser, but does not require an internet connection (except when you connect to services) A free, open source, powerful tool for working with messy data
- 7. Other tools OpenRefine Worksheet focus on cells focus on rows and columns focus on import data & calculations focus on exploring and transforming existing data Scripting data → script → output all steps are visualized focus on transformation of data Databases focus on queries looks like a worksheet you should know the data data is always visible, facets shows you choices OpenRefine vs other tools
- 8. Distribution Description Authors LODRefine LODRefine is actually OpenRefine with integrated extensions that make transition from tabular data to Linked Data a bit easier. Integrated extensions are: RDF extension, DBpedia extension, Crowdsourcing extension, Stats extension Sparkica OpenDataRise Tool to cleanse and semantify datasets from CKAN repositories. Based on OpenRefine. Open Data in Trentino p3-batchrefine BatchRefine adds batch processing capabilities to OpenRefine and support multiple back end including spark SpazioDati SparkonRefine RefineOnSpark is a driver program to run OpenRefine jobs on the Spark cluster SpazioDati Reconciliation-and-Matching- Framework A framework to allow the matching of string entities using customised sets of transformations and matchers, plus a tool to produce the necessary configurations and another to expose them as OpenRefine reconciliation services. RBGKew Tools working with OpenRefine
- 9. ● Download Google Refine on: http://openrefine.org/download.html ● Launch Google Refine ● Create a project ● Choose the file you want to clean (Example Dataset: Onderwijsaanbod in Vlaanderen (http://opendata.vlaanderen.be/dataset/onderwijsaanbod) Hands on: install OpenRefine
- 10. ● Check the preview and define parsing ○ Set character encoding (UTF8) ○ Choose delimiter (/t ; , …) ○ Parse data as (csv) ○ Parse first line as column header, ignore first … line(s).... Hands on: importing data
- 11. ● Accessing information organized according to a faceted classification system ○ Creating an overview of the data ○ Allows targeted editing of your data ○ Allows specific filtering ○ Facet choices as tab separated values (like pivot tables in Excel) Hands on: faceting
- 12. ● Clustering allows to automatically group and edit different but similar values Hands on: clustering
- 13. ● Common transforms: ○ to number ○ trim leading and trailing whitespace ○ to title case; to date; to number ● Split & Join multi valued cells Hands on: edit cells
- 14. ● Split columns (by separator or field length) ● Add columns (by fetching urls or based on column) (use GREL) ● Move columns ● Remove columns ● Rename columns Hands on: edit columns
- 15. ● GREL (google refine expression language) ○ add columns based on other column ■ basic string modification ■ find and replace ■ string parsing and splitting ■ calling web services ○ Result are always visible in the Preview Hands on: scripting using GREL
- 16. ● Add columns by fetching url ■ find and replace ■ string parsing & splitting ■ add column based on column”straat” (value+”%20”+cells[‘huisnummer’].value) ■ Call google API (or openstreetmap or….) ("https://maps.googleapis. com/maps/api/geocode/json?address="+value+ cells["huisnummer"]. value&key=AIzaSyDY2Z6wehbIqIPrHIb9ljC62pwRqEHOous") ■ Parse JSON (value.parseJson()["results"][0]["geometry"]["location"]["lng"]) Hands on: georeferencing
- 17. ● Grouping concepts with an external service, eg taxonomic reconciliation ○ Example from the natural environment (biodiversity data) ■ add a reconciliation service (reconcile, start reconciling) ■ Let’s use Encyclopedia of Life ■ Select Matches (Facet, Quick actions…) Hands on: reconciling
- 18. ● Grouping concepts with an external service, eg taxonomic reconciliation ○ Example from the natural environment (biodiversity data) ■ add ID EOL ID column (GREL) cell.recon.match.id ■ create url based on EOL ID ■ http://eol.org/pages/3465521 Hands on: reconciling
- 19. ● Merge data from the two projects by creating a new column from values from an existing column within one project that are used to index into a similar column in the other project ○ cell.cross("datasetname.csv","scientificName").cells["order"].value[0] Hands on: cross referencing
- 20. ● Extract and save parts of your operation history as JSON that you can apply to this or other projects in the future. Hands on: Extract operation history
- 21. ● https://github.com/OpenRefine/OpenRefine/wiki ● https://github.com/OpenRefine/OpenRefine/wiki/Recipes ● http://enipedia.tudelft.nl/wiki/OpenRefine_Tutorial ● ... Hands on: further reading