Clustering of Similar Values, in Spanish, for the Improvement of Search Systems

Notas del editor

• #2: {"27":"This method achieves good results in all experiments done, but it does not eliminate the need to review the obtained clusters.\nThe expansion of abbreviations improves on the results.\n","16":"The similarity between two strings must be evaluated. We use the EDIT DISTANCE and the INVARIANT DISTANCE FROM WORD POSITION. \nThe edit distance of two strings is a measure of similarity that is given by the minimal number of simple edit operations needed to transform an string into the other.\nThe simple editing operations considered are: the insertion of a character, the deletion of a character and the substitution of one character with another.\n","5":"For example, if we consult a database that stores information about university researches, we may easily find that there are different values for the same university:\n- Universidad Alicante (without the preposition)\n- Unibersidad de Alicante (with misspelling)\n- Universitat d’Alacant (in Catalan)\n- and even University of Alicante (in English)\nIf a database suffers inconsistency, a search using a given value will not provide all the available information about the term.\n","22":"As we can see, the clusters of file B are more complex than those of file.\nIn both files, the FCI is reduced when expanding the abbreviations.\n","11":"- Different denominations.\n- Punctuation marks: hyphens, commas, semi-colons, brackets, exclamation marks and so on.\n","17":"These distances speed up the clustering. The expensive computation of LD and IDWP can be avoided. \n","6":"The problem of the inconsistency in the values stored in databases may have two origins:\n- Different people may insert the same term with different values in a database.\n- When we try to integrate different databases, they may use different values for representing the same term.\n","23":"We have evaluated the clusters obtained by using four measures that are obtained by comparing the clusters produced with the optimal clusters (handcrafted).\n","12":"- Errors: misspelling, typing or printing errors.\n- Use of different languages.\n","1":"Perhaps we should begin.\nGood morning everyone. Thanks for coming. I am a member of the Deparment of Software and Computing Systems at the University of Alicante in Spain.\nThe work I’m going to present is Clustering of Similar Values, in Spanish, for the Improvement of Search Systems.\n","7":"We present an automatic method for reducing on the inconsistency found in existing databases, and thus, improving data quality. \nAll the values that refer to a same term are clustered by measuring their degree of similarity.\nThe clustered values can be assigned to a common value which, in principle, could be substituted for the original values.\n","24":"From the previous measures, we obtain Precision: NCC divided by ONC and Error: NIC divided by ONC.\n","13":"The method we propose can be divided into six steps.\n","2":"Perhaps we should begin.\nGood morning everyone. Thanks for coming. I am a member of the Deparment of Software and Computing Systems at the University of Alicante in Spain.\nThe work I’m going to present is Clustering of Similar Values, in Spanish, for the Improvement of Search Systems.\n","8":"After analysing several databases with information both in Spanish, we have noticed that the different values that appear for a given term are due to a combination of the following causes.\n","25":"In both files, the expansion of abbreviations produces improvements: it increases the precision and reduces the error.\nFor file A, a maximum precision of 70.7% and 84.8% is obtained without and with expansion of abbreviations.\nFor file B, a maximum precision of 67.4% and 72.8% is obtained without and with expansion of abbreviations.\n","14":"1. Preparation. It may be necessary to prepare the strings before applying the clustering algorithm.\n2. Reading. The following process is repeated for each of the strings contained in the input file: Read a string, Expand abbreviations and acronyms, Remove accents, Shift string to lower-case, Store the string.\n3. Sorting. The strings are sorted, in descending order, by frequency of appearance.\n","3":"First of all, I would like to outline the main points of my talk. \nI have structured my talk into six sections. Firstly, I will give an introduction to my research and I will make a few observations about the inconsistency problem. Then I will go on to explain the main causes of the problem. Next, I will talk about our proposed method for reducing inconsistency in databases. And then, I will present our clustering algorithm and the distance metrics it uses. Finally, I will highlight the main results of our method and the conclusions of our research.\nLet’s move on to the first part of the presentation.\n","9":"- The omission or inclusion of the written accent.\n- The use of lower-case and upper-case letters.\n","15":"The standard method of detecting exact duplicates in a table is to sort the table and then to check if neighboring tuples are identical. This approach can be extended to detect approximate duplicates.\n","4":"Existing information systems provide rapid and precise access to information stored in databases.\nOne of the main uses of databases is find information.\nIf a database has a bad design, it is very likely that it suffers inconsistency: the very same term may have different values, due to misspelling, a permuted word order, spelling variants and so on.\n","21":"We have used two files for evaluating our method. They contain data from two databases with inconsistency problems.\nWe have developed a coefficient named CONSISTENCY INDEX that permits the evaluation of the complexity of a cluster: the greater the value of the coefficient, the more different the strings that form the cluster are.\nThe FILE CONSISTENCY INDEX is defined as the average of the consistency indexes of all the existing clusters in the file.\n","10":"- The use of abbreviations and acronyms.\n- Different word order.\n"}