OpenRefine Class Tutorial
- 1. Advances in Data Science Fall 2016 TUTORIAL
- 3. WHAT IS … ?? • Formerly known as Google Refine OpenRefine is a power tool for working with messy data, primarily for • detecting and fixing inconsistencies • transforming data from one structure or format to another • extending it with web services and external data • connecting names within your data to name registries (databases) Use OpenRefine when you need something ... • more powerful than a spreadsheet • more interactive and visual than scripting • more provisional / exploratory / experimental / . playful than a database
- 4. • Import data in various formats (Ex: TSV, CSV,Excel (.xls, xlsx),XML,RDF as XML,JSON) • Explore datasets in a matter of seconds • Apply basic and advanced cell transformations • Deal with cells that contain multiple values • Create instantaneous links between datasets • Filter and partition your data easily with regular expressions • Use named-entity extraction on full-text fields to automatically identify topics • Perform advanced data operations with the General Refine Expression Language IMPORTANT FEATURES:
- 5. The LendingClub data contains complete loan data for all loans issued through the time period stated, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information LENDING CLUB LOAN STATS DATA Our aim is to perform exploratory analysis on given financial data
- 6. • Getting the data • Looking at the data • Cleansing • Transforming • Creating visualizations STEPS
- 7. 1 – Getting started with OpenRefine 2 – Analyzing and Fixing Data 3 – Advanced Data Operations 4 – Linking Datasets 5 – Regular Expressions and GREL TUTORIAL
- 8. • Requirements • Java JRE installed • Download • OpenRefine is a desktop application. Here’s the link: Google OpenRefine • Unlike most other desktop applications, it runs as a small web server on your own computer • You point your web browser at that web server in order to use Refine. So, think of Refine as a personal and private web application HOW TO INSTALL
- 9. • Install: • Once you have downloaded the .zip file, uncompress it into a folder wherever you want (such as in C:Google-Refine). • Run: • Run the .exe file in that folder. You should see the Command window in which OpenRefine runs. By default, the Command window has a black background and text in monospace font in it. • Shut down: • When you need to shut down OpenRefine, switch to that Command window, and press Ctrl- C. Wait until there's a message that says the shutdown is complete. That window might close automatically, or you can close it yourself. If you get asked, "Terminate all batch processes? Y/N", just press Y. INSTALLATION: WINDOWS
- 10. • Install: • Once you have downloaded the .dmg file, open it, and drag the OpenRefine icon into the Applications folder icon (just like you would normally install Mac applications). • Run: • To launch OpenRefine, go to the Applications folder and double click the OpenRefine app. You'll see the OpenRefine app appear in your dock. • Shut down: • You can switch to the OpenRefine app (clicking on its icon in the dock) and invoke its Quit command. • If you use Yosemite you will need to install Java for OS X 2014-001 first. INSTALLATION: MAC
- 11. • Install / Run: Once you have downloaded the tar.gz file, open a shell and type • tar xzf google-refine.tar.gz • cd google-refine • ./refine • This will start OpenRefine and open your browser to its starting page. • Shut down: Press Ctrl-C in the shell. INSTALLATION: LINUX
- 12. RUN OPENREFINE • To increase memory: refine.bat /m 4096m
- 13. IMPORT DATA
- 14. EXPLORING DATA
- 16. USING THE PROJECT HISTORY
- 18. ANALYZING AND FIXING DATA
- 19. WORKING ON THE DATA • sorting data • faceting data • detecting duplicates • applying a text filter • using simple cell transformations • removing matching rows • splitting data across columns • adding derived columns
- 20. SPECIAL FEATURE • Regular Expressions and GREL • Can use Python, Clojure
- 21. ADDING A RECONCILIATION SERVICE AND RECONCILING WITH LINKED DATA
- 22. ADVANCED DATA OPERATIONS • handling multi-valued cells • alternating between rows and records mode • clustering similar cells • transforming cell values • adding derived columns • transposing rows and columns • installing extensions
- 23. • Documentation: • https://github.com/OpenRefine/OpenRefine/wiki • Youtube Tutorial: • https://www.youtube.com/playlist?list=PL737054C67FCC0741 REFERENCES: