Data Quality: Towards a Common Validator
- 1. Data quality Towards a common validator Christian Gendreau, Anne Bruneau, David Shorthouse Université de Montréal, Biodiversity Centre
- 2. What is data quality? ● Relative ● Fitness for use o Coordinate precision for distribution o Hierarchy not provided ● What –When –Where
- 3. Examples ● 2008 VI 13, 2008-06-13, 13-06-2008, June 13 2008, 13 junho 2008 ● Canada, Québec, Montréal, -73.55399 45.508669 ● Narwalus microcephalus => Monodon monoceros Public Domain: Freshwater and Marine Image Bank, University of Washington Libraries Digital Collections
- 4. Data Quality Information Chain Courtesy of Arthur D. Chapman
- 5. Brief History ● First Canadensys Explorer/Harvester (2012) ● narwhal-processor (2013) ● TDWG 2013 o DQ Interest Group o Presentation about our plan o Discussion with GBIF
- 6. Why do we need a validator? ● Identify and quantify potential issues ● DarwinCore is permissive o DarwinCore itself can change ● Records and technologies will always evolve
- 7. What should a validator allow? ● Define a validation scope o at the source (e.g. collection) o national node o aggregator o GBIF
- 8. Validator - Expected design ● Modular, scalable, reusable ● Customizable o per configuration/extension o use user defined dictionary ● Validation Chain
- 9. Current Options ● GBIF validator ● CRIA tools ● ALA tools Probably all organisations have their own tools.
- 10. dwca-validator ● Starting from previous GBIF validator ● Building a community project ● Provide framework for Biodiversity Data Quality Interest Group (TDWG) https://github.com/gbif/dwca-validator
- 11. Vision • Library o Core module, reusable (e.g. IPT) • Web o Send archive, view report • narwhal-processor o Suggest interpreted value • Extensions o Domain knowledge / Quality index
- 12. Validation chain ● Chain element o Self contained (never relies on another chain element) o Ordering independent ● Composed chain element (narwhal and extensions) o Wrap chain elements under a new element o Ordering possible between wrapped element
- 14. Validation types ● Structure o metadata o organization of data ● Rows o dates, coordinates, ... ● Columns o ID uniqueness
- 15. Result Accumulator ● Records validation result as they occur o ID/Validator/Context/ValidationType/Result/Message ● Allows different views of result o Web view o Feed another application
- 16. Current Status ● Library with CLI (command line interface) ● Basic evaluators and rules ● Ready for contributions
- 17. Demo • Darwin Core Archive, Taxon Checklist o Invalid characters o Broken link synonym accepted taxon Lynx Canadensis, http://www.animalgalleries.org/
- 18. Future validations ● Use semantic web (e.g. GeoNames) ● Use external resolver (e.g. CoL) ● Use more complex validation (e.g. climate layer)
- 19. Future validations Accomodate localisation vs misspellings ● Brésil (fr) ● Brazil (en) ● Brasil (pt) ● Brasilien (se) ● Brézil (??)
- 20. Questions? Public Domain: robynm
- 21. Acknowledgements