Technical Development Workshop - Text Analytics with Python

Editor's Notes

• #2: Nhi
• #5: Michelle
• #6: M
• #7: Python is an interpreted, high-level, general-purpose programming language. Python supports the use of modules and packages, which means that programs can be designed in a modular style and code can be reused across a variety of projects. Once you've developed a module or package you need, it can be scaled for use in other projects, and it's easy to import or export these modules.
• #10: What is Anaconda? Anaconda is a free and open-source distribution of the Python and R programming languages for scientific computing, that aims to simplify package management and deployment. In this workshop, we will use Anaconda to Install Python and Jupyter Notebook as Anaconda also includes other commonly used packages for scientific computing and data science (and in this case, for text analytics!)
• #11: M What is Jupyter Notebook? The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Jupyter Notebooks are a powerful way to write and iterate on your Python code for data analysis. Rather than writing and re-writing an entire program, you can write lines of code and run them one at a time.
• #12: N
• #13: NLP enables the computer to interact with humans in a natural manner. It helps the computer to understand the human language and derive meaning from it. Analyzing movie review is one of the classic examples to demonstrate a simple NLP Bag-of-words model, on movie reviews.
• #14: N NLTK is a powerful Python package that provides a set of diverse natural languages algorithms. It is free, opensource, easy to use, large community, and well documented. NLTK helps the computer to analysis, preprocess, and understand the written text. Going back to the phase slide, NLTK consists of the most common algorithms such as tokenizing, part-of-speech tagging, stemming, sentiment analysis, topic segmentation, and named entity recognition.
• #16: M
• #17: Talk about package —> module —> class (draw venn diagram on white board maybe?)
• #19: # Frequency Distribution Plot import matplotlib.pyplot as plt fdist.plot(30,cumulative=False) plt.show() https://matplotlib.org/tutorials/introductory/pyplot.html#pyplot-tutorial
• #22: M
• #24: N
• #25: Lexicon normalization considers another type of noise in the text. For example, connection, connected, connecting word reduce to a common word "connect". It reduces derivationally related forms of a word to a common root word.
• #26: Stemming process of linguistic normalization reduces words to their word root word or chops off the derivational affixes. Lemmatization more sophisticated than stemming. reduces words to their base word, which is linguistically correct lemmas. A lemma is a word that stands at the head of a definition in a dictionary. All the head words in a dictionary are lemmas. Technically, it is "a base word and its inflections" Stemmer works on an individual word without knowledge of the context. For example, The word "better" has "good" as its lemma. This thing will miss by stemming because it requires a dictionary look-up.
• #27: N
• #28: The primary target of Part-of-Speech(POS) tagging is to identify the grammatical group of a given word. Whether it is a NOUN, PRONOUN, ADJECTIVE, VERB, ADVERBS, etc. based on the context. POS Tagging looks for relationships within the sentence and assigns a corresponding tag to the word. List of POS tags: https://medium.com/@gianpaul.r/tokenization-and-parts-of-speech-pos-tagging-in-pythons-nltk-library-2d30f70af13b
• #31: Text classification is one of the important tasks of text mining. Identifying category or class of given text such as a blog, book, web page, news articles, and tweets. It has various application in today's computer world such as spam detection, task categorization in CRM services, categorizing products on E-retailer websites, classifying the content of websites for a search engine, sentiments of customer feedback, etc.
• #33: What users and the general public think about the latest feature? You can quantify such information with reasonable accuracy using sentiment analysis. Quantifying users content, idea, belief, and opinion is known as sentiment analysis. Human communication is just not limited to words, it is more than words. Sentiments are combination words, tone, and writing style. Two approaches Lexicon-based: Count a number of positive and negative words in given text and the larger count will be the sentiment of text. Machine learning based approach: Develop a classification model, which is trained using the pre-labeled dataset of positive, negative, and neutral. In this Tutorial, you will use the second approach(Machine learning based approach). This is how you learn sentiment and text classification with a single example.
• #35: Break down paragraphs into smaller chunks like sentences or words. Remove punctuation and stopwords to increase the accuracy of our analysis. Use Stemming or Lemmatization to reduce words to their base words. Understand Part-of-Speech tagging. Create simple graphs in Python. Scratch a bit of the surface of Sentiment Text Analysis!