Data quality in Real Estate
1 like680 views
The document discusses the importance of data quality in real estate, focusing on mapping buildings globally and providing a quality score based on various metrics. It highlights the complexities of real estate data, including addressing constraints and the use of ontologies for data integration. The authors emphasize automation in quality validation and integrating these processes within CI/CD pipelines to ensure data integrity.
![Possible constraints on addresses?
● An address will start with, or at least include, a building number.
● When there is a building number, it will be all-numeric.
● No buildings are numbered zero
● Well, at the very least no buildings have negative numbers
● A building number will only be used once per street
● A building will only have one number
● A building name won't also be a number
● [...] https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses](https://image.slidesharecdn.com/dataqualitygeophy1-170917080239/85/Data-quality-in-Real-Estate-4-320.jpg)
![Geophy [set of] ontologies
● 13 ontologies (+ 9 external)
● 125 Classes
○ Buildings
○ Addresses
○ Companies
○ [...]
● 720 properties
○ 500 datatype
○ 160 relation properties
● Growing...](https://image.slidesharecdn.com/dataqualitygeophy1-170917080239/85/Data-quality-in-Real-Estate-5-320.jpg)
Data quality in Real Estate
- 1. Data Quality In Real Estate Dimitris Kontokostas, Andy van der Hoeven, Samur Araujo Amsterdam, Sep 14th 2017, LDQ Workshop, SEMANTiCS Conference
- 2. About Geophy ● Goal to map all buildings in the world ● Provide a quality score for each building ○ Based on location, building status, history, environmental metrics, etc ● Semantic platform ○ RDF eases the data integration process ● Team of 45 with aim to double by next year
- 3. Real Estate is a very complex domain Really!
- 4. Possible constraints on addresses? ● An address will start with, or at least include, a building number. ● When there is a building number, it will be all-numeric. ● No buildings are numbered zero ● Well, at the very least no buildings have negative numbers ● A building number will only be used once per street ● A building will only have one number ● A building name won't also be a number ● [...] https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses
- 5. Geophy [set of] ontologies ● 13 ontologies (+ 9 external) ● 125 Classes ○ Buildings ○ Addresses ○ Companies ○ [...] ● 720 properties ○ 500 datatype ○ 160 relation properties ● Growing...
- 6. Quality is expensive ● Quality of source data ○ Free, open, closed data sources, etc. ● Data clean up process ○ Violations, deduplication, precision, etc. ○ How much time and effort can one afford? How much quality is good enough? Fitness for use
- 7. Quality of ... ● Source data ○ Accuracy of the source ● Translation of source data ○ RDF mappings, rml, d2rq, scripts etc. ● Model design ○ Modelling quality ○ Data fitting on schema ● Model definition ○ Mapping of model on RDFS, OWL, ShEx|SHACL Shapes, etc ○ Semantics i.e RDFS, OWL DL/RL/FULL, etc
- 8. Evolution & quality Data evolves so do ontologies so do RDF mappings so does code so do SPARQL queries so do constraints http://aligned-project.eu
- 9. Scaling quality ... ● Thousands of triples ● Millions of triples ● Billions of triples ● ? Try to move validation in the K range (when possible)
- 10. Validate closer to the source Validate the model Validate the RDF mappings Validate RDF mapping excerpts Validate instance data
- 11. Automate, automate & automate Can you spot the error? rdfs:label ⇒ rdf:langString :foo rdfs:label ″foo @en″ .
- 12. Automate, automate & automate Can you spot the error? rdfs:label ⇒ rdf:langString :foo rdfs:label ″foo @en″ . :foo rdfs:label ″foo″@en .
- 13. CI/CD is your buddy ● Integrate validation with your CI/CD ○ Choose tools & technologies wisely ○ Jenkins, Travis, Gitlab, TeamCity ● Fail the build until data issues are fixed ● Data integration validation checks ○ Standalone datasets can pass CI
- 14. Thank you for your attention Questions?