Automatic Key Term Extraction from Spoken Course Lectures

Editor's Notes

• #2: Hello, everybody. I am Vivian Chen, coming from National Taiwan University. Today I’m going to present my work about automatic key term extraction from spoken course lectures using branching entropy and prosodic/semantic features.
• #4: First I will define what is key term. A key term is a term that has higher term frequency and includes core content. There are two types of key terms. One of them is phrase, we call it key phrase. For example, “language model” is a key phrase. Another type is single word, we call it keyword, like “entropy”. Then there are two advantages about key term extraction. They can help us index and retrieve. We also can construct the relationships between key terms and segment documents. Here’s an example.
• #5: We can show some key terms related to acoustic model. If the key term and acoustic model co-occurs in the same document, they are relevant, so that we can show them for users.
• #6: Then we can construct the key term graph to represent the relationships between these key terms like this.
• #7: Similarly, we can also construct the relation between language model and other terms. Then we can show the whole graph to know the organization of key terms.
• #9: Here’s the flow chart. Now there are a lot of spoken documents.
• #10: First, with speech recognition system, we can get ASR transcriptions.
• #11: The input of our work includes transcriptions and speech signals.
• #12: The first part is to identify phrases. We propose branching entropy to finish this part.
• #13: Then we can enter key term extraction part. We first extract some features to represent a candidate term, and use machine learning methods to decide the key terms.
• #14: First we explain how to do phrase identification.
• #15: The target of this work is to decide the boundary of a phrase, but where’s the boundary? We can observe some characteristics first. Hidden is almost always followed by Markov.
• #16: Similarly, hidden Markov is almost always followed by model,
• #17: But at this position, we find that hidden Markov model is followed by some different words. The position is more likely to be boundary. Then we define branching entropy by the concept.
• #18: First, I will define what is right branching entropy. We assume that X is hidden and x_i is the children of X, representing hidden Markov. We define p of x_i to be the frequency of x_i over the frequency of X, which is probability of children x_i for X. Then we define right branching entropy as this equation, H_r of X, which represents the entropy of X’s chidren.
• #19: The idea is that the branching entropy at the boundary is higher. So we can find the right boundary where branching entropy is higher like hidden Markov model.
• #20: Similarly, left branching entropy has the same attribute.
• #21: The approach called branching entropy is to find the boundaries of key phrases, but where’s the boundary. The idea is that the word entropy at the boundary is higher. There are four parts in this approach. We can present each part in detail.
• #22: The approach called branching entropy is to find the boundaries of key phrases, but where’s the boundary. The idea is that the word entropy at the boundary is higher. There are four parts in this approach. We can present each part in detail.
• #23: We also decide the left boundary by H_l of X bar, X bar is the reverse pattern like model Markov hidden. Branching entropy can be implemented by PAT tree.
• #24: Then in the PAT tree, we can compute right branching entropy for each node. We can take an example to explain p(xi). X is the node representing a phrase hidden Markov, x_1 is X’s child hidden Markov model, x_2 is X’s another child hidden Markov chain. Previously described p of x_i is shown as this one, and then compute right branching entropy of this node. We compute H_r of X for all X in PAT tree and H_l of X bar for all X bar in the reverse PAT tree. We can co
• #25: With all candidate terms, including words and phrases, we can extract prosodic, lexical, semantic features for each term.
• #26: For each word, we also compute some prosodic features. First, we believe that lecturer would use longer duration to emphasize key terms. For the candidate term first appearing, we compute the duration for each phone in each candidate term. Then we normalize the duration of specific phone by the average duration of this phone. In this feature, we only use four values to represent this term. We can compute maximum, minimum, mean and range over all phones in a single term to be the features.
• #27: We believe that higher pitch may represent important information. So we can extract the pitch contour like this.
• #28: The method is like duration, but the segment unit is changed to single frame. We also use these four values to represent the features.
• #29: Similarly, we think that higher energy may represent important information. We also can extract energy for each frame in a candidate term.
• #30: The features are like pitch. The first set of features is shown in this.
• #31: The second set of features is lexical features. These features are well-known, which may indicate the importance of term. We just use these features to represent each term.
• #32: Third set of features is semantic features. The assumption is that key terms tend to focus on limited topics. We use PLSA to compute some semantic features. We can get the probability of each topic given a candidate term.
• #33: Here are distributions of two terms. Notice that the first term has similar probability for most topics, and it may be a function word, so that this candidate term may be non-key term. In other one, the term focuses on less topics, so it is more likely to be key term. In order to describe the distribution, for the distribution, we compute mean, variance and standard deviation to be the features of this term.
• #34: Similarly, we can compute latent topic significance, which is within-topic to out-of-topic ratio like this equation. This is the significance of topic T_k in term t_i. This one is within-topic frequency, where n of t_i and d_j is the number of t_i in document d_j, and this one is out-of-topic frequency.
• #35: Because the significance of key terms would focus on limited topics, we also compute this three values to represent a distribution.
• #36: Then, we also can compute latent topic entropy. Because this feature also can describe the difference between these two distributions. Non-key term may have higher LTE, and key term would have lower one.
• #38: Finally, we use some machine learning methods to extract key terms.
• #39: The first one is an unsupervised learning, K-means Examplar. First, we can transform each word into a vector in latent topic significance space, as this equation. With these vectors, we run K-means. The terms in the same cluster focus on a single topic. So we can extract the centroid of each cluster to be the key term, because the terms in the same group are related to this key term.
• #40: We also use two supervised learning, adaptive boosting and neural network to automatically adjust the weights of features to produce a classifier.
• #42: Then we do some experiments to evaluate our approach. The corpus is NTU lecture, which includes Mandarin Chinese and some English words, like this example. This sentence is ~~, which means our solution is viterbi algorithm, The lecture is from a single speaker, and total corpus is about 45 hours.
• #43: In the ASR system, we train two acoustic models for chinese and english, and use some data to adapt, finally getting a bilingual acoustic model. Language model is trigram interpolation of out-of-domain corpora and some in-domain corpus. Here’s ASR accuracy.
• #44: To evaluate our result, we need to generate reference key term list. The reference key terms are from students’ annotations, and these students have taken the course. Then we sort all terms and decide top N to be key terms. N is the average numbers of key terms extracted by students, and N is 154. The reference key term list includes 59 key phrases and 95 keywords.
• #45: Finally we evaluate the results. For unsupervised learning, we set the number of key terms to be N. And using 3-fold cross validation evaluates supervised learning.
• #46: At this experiment, we are going to see feature effectiveness. The results only include keywords, and it is from neural network using ASR transcriptions. Row a, b, c perform F1 measure from 20% to 42%. Then row d shows that prosodic and lexical features are additive. Row e proves that adding semantic features can further improve the performance so that three sets of features are all useful.
• #47: Finally, we show the overall performance for manual and ASR transcriptions. These are baseline, conventional TFIDF without extracting phrases using branching entropy. From the better performance, we can see that branching entropy is very useful. This proves our assumption that the term with higher branching entropy is more likely to be key term. And the best results are from supervised learning using neural network, achieving F1 measure of 67 and 62.
• #48: Finally, we show the overall performance for manual and ASR transcriptions. These are baseline, conventional TFIDF without extracting phrases using branching entropy. From the better performance, we can see that branching entropy is very useful. This proves our assumption that the term with higher branching entropy is more likely to be key term. And the best results are from supervised learning using neural network, achieving F1 measure of 67 and 62.
• #50: From the above experiments, the conclusion is that we proposed new approach to extract key terms efficiently. The performance can be improved by two ideas, using branching entropy to extract key phrases and using three sets of features.