Daurum: Introduction
2 likes603 views
This document discusses deduplication and data fusion software. It introduces the benefits of identifying duplicated records across databases and merging data from different sources. The process section outlines how the software allows configuring input formats, similarity computations, filters, and validation steps. Successful stories provide examples of how a health service and beer manufacturer used the software to clean databases and identify incorrect deliveries. A demo is available to see the software in action.
Daurum: Introduction
- 3. Process
- 6. Process
- 8. DemoIntroductionBenefitsIdentification of suspected duplicated records inside a databaseMerging of data belonging to several databases with different formats detecting duplicated recordsValidation tools for the detected similarities
- 10. IntroductionDeduplicationConfigurationAutomatic executionValidation of resultsPersonalized export
- 11. IntroductionDeduplicationConfigurationAutomatic executionValidation of resultsPersonalized export
- 13. IntroductionFusionConfigurationAutomatic executionValidation of resultsPersonalized export
- 14. IntroductionFusionConfigurationAutomatic executionValidation of resultsPersonalized export
- 17. Process
- 19. DemoProcessConfigurationsInput data file format: CSV
- 20. Select relevant columns to link registers
- 21. Relation between columns from different data sources (only when merging)
- 22. Assign types to columns to help using the most adequate automatic filtersCSVConfigurationsExecutionValidationExportationExcelPDFXMLCSV
- 23. ProcessConfigurationsComparative type: exact value, estimation by text, numerical estimation
- 24. Percentage of the importance of each column for the similarity computationCSVConfigurationsExecutionValidationExportation30%35%35% 100% =ExcelPDFXMLCSV
- 25. ProcessConfigurationsSpecific percentage for registers with null valued columns
- 26. Use filters to make values standard
- 27. Available automatic and specific filters for values such as name, dates, address, etc…CSVConfigurationsExecutionValidationExportationExcelPDFXMLCSV
- 28. ProcessConfigurationsEdit filters (create new filters, delete or update existing ones)
- 29. Use of dictionaries: name-converter dictionary (I.e.: Pepe Jose)
- 30. Similarity computation algorithm configuration:
- 31. Size for the sliding window: number of registers compared at the same time
- 32. Order by column
- 33. Threshold of accepted similarityCSVConfigurationsExecutionValidationExportationExcelPDFXMLCSV
- 34. ProcessExecution Order by Surname 1
- 35. Sliding window = 2CSVConfigurationsExecutionWindow = 2ValidationExportationExcelPDFXMLCSV
- 36. ProcessExecution Similarities detectedCSVConfigurationsExecutionWindow = 2ValidationSimilarities ExportationSimilarity degreeExcelPDFXMLCSV
- 37. ProcessExecutionSimilarities detectedCSVwindow = 2ConfigurationsSimilaritiesExecutionValidationExportationSimilaritydegreeExcelPDFXMLCSV
- 38. ProcessExecutionList of detected similaritiesCSVConfigurationsExecutionValidationExportationExcelPDFXMLCSV
- 39. ProcessExecutionList of detected similarities with percentage bigger than threshold 50% CSVConfigurations> 50%ExecutionValidationExportationExcelPDFXMLCSV
- 40. ProcessValidationValidation of results (including only those above the threshold)
- 41. Visualize by similarity/by group
- 43. Share validation between several supervisorsCSVConfigurationsExecutionValidationExportationExcelPDFXMLCSV
- 44. ProcessExportationCSVSelect output formatConfigurationsExecutionValidationExportationExcelPDFXMLCSV
- 46. Process
- 48. DemoSuccessful storiesHealth ServiceWho? Health ServiceObjective Detect repeated health id cardsSolution Detect repeated registers in the database and delete themDeduplicaction with DAURUMResult Health id cards database cleaned of repetitions
- 49. Successful storiesBeer ManufacturerWho? Beer manufacturerObjective Detect dealers that deliver to not previously assigned centers Solution Identify duplicates in each dealer’s delivery database and delete themDeduplication with DAURUM Detect deliveries to centers shared between different dealersFusion with DAURUMResult Master database clean of repetitions and detection of dealers with wrong deliveries
- 51. Process
- 53. DemoDemo
- 54. Thanks for your attentionAny questions?Pere Baleta FerrerCEOpbaleta@sparsity-technologies.comJosep Lluís Larriba PeyFounderlarri@sparsity-technologies.comSPARSITY-TECHNOLOGIESJordi Girona, 1-3, Edifici K2M 08034 Barcelonainfo@sparsity-technologies.comhttp://www.sparsity-technologies.com