AI_08_NLP.pptx

LECTURE 8
Natural Language Processing
NLP
Instructor : Yousef Aburawi
Cs411 -Artificial Intelligence
Misurata University
Faculty of Information Technology
Spring 2022/2023

Natural Language Processing (NLP)
 NLP stands for Natural Language Processing, which is a part
of Computer Science, Human language, and Artificial
Intelligence. It is the technology that is used by machines to
understand, analyze, manipulate, and interpret human's
languages.
 Getting computers to do reasonably intelligent things with
human language is the domain of Natural Language
Processing (or Computational Linguistics or Human
Language Technology)

Natural Language Processing (NLP)
1-3

Language Analysis: The Science (Linguistics)
 Language is a multi-layered phenomenon
 To some useful extent these layers can be studied independently
 There need to be interfaces between layers.
 There are areas of overlap between layers.

The Language Layers
 L1: Phonology: The noises you make and understand.
 L2: Morphology: What you know about the structure of the words in
your language, including their derivational and inflectional behavior.
 L3: Syntax: What you know about the order and constituency of the
utterances you spout.

The Language Layers
 L4: Semantics: (Meaning), What is the connection between language
and the world (Truth/Knowledge).
 L5: Pragmatics: How language is used by speakers, as opposed to
what things mean.
 L6: Discourse: Dealing with larger chunks of language and language in
context.
1-6

Components of NLP
 Natural Language Understanding (NLU):
 Natural Language Understanding (NLU) helps the machine to understand
and analyze human language by extracting the metadata from content such
as concepts, entities, keywords, emotion, relations, and semantic roles.
 NLU mainly used in Business applications to understand the customer's
problem in both spoken and written language.
 Natural Language Generation (NLG):
 Natural Language Generation (NLG) acts as a translator that converts the
computerized data into natural language representation. It mainly involves
Text planning, Sentence planning, and Text Realization.
1-7

Components of NLP
NLU NLG
NLU is the process of reading and
interpreting language.
NLG is the process of writing or
generating language.
It produces non-linguistic outputs
from natural language inputs.
It produces constructing natural
language outputs from non-
linguistic inputs.
1-8

Applications of NLP
 Applications of NLP can be include:
 Machine translation.  Information extraction.
 Automatic summarization.  Named Entity Recognition (NER).
 Relationship extraction.  Topic segmentation.
 Question answering.  Speech recognition.
 Spelling correction.  OCR.
 Grammar checkers.

NLP Applications: Question Answering
 Question Answering focuses on building systems that automatically
answer the questions asked by humans in a natural language.
1-10

NLP Applications: Spam Detection
 Spam detection is used to detect unwanted e-mails getting to a
user's inbox.
1-11

NLP Applications: Sentiment Analysis
 Sentiment Analysis is also known as opinion mining. It is used on the
web to analyze the attitude, behavior, and emotional state of the
sender. This application is implemented through a combination of
NLP (Natural Language Processing) and statistics by assigning the
values to the text (positive, negative, or natural), identify the mood
of the context (happy, sad, angry, etc.)
1-12

NLP Applications: Machine Translation
 Machine translation is used to translate text or speech from one
natural language to another natural language.
1-13

NLP Applications: Spelling correction
 Microsoft Corporation provides word processor software like MS-
word, PowerPoint for the spelling correction.
1-14

NLP Applications: Chatbot
 Implementing the Chatbot is one of the important applications of
NLP. It is used by many companies to provide the customer's chat
services.
1-15

How to build an NLP pipeline
 Step1: Sentence Segmentation:
 Sentence Segment is the first step for building the NLP pipeline. It breaks
the paragraph into separate sentences.
 Step2: Word Tokenization:
 Word Tokenizer is used to break the sentence into separate words or tokens.
 Example:
 JavaTpoint offers Corporate Training, Summer Training, Online Training, and Winter
Training.
 Word Tokenizer generates the following result:
"JavaTpoint", "offers", "Corporate", "Training", "Summer", "Training", "Online",
"Training", "and", "Winter", "Training", "."
1-17

 Step3: Stemming:
 Stemming is used to normalize words into its base form or root form. For
example, celebrates, celebrated and celebrating, all these words are
originated with a single root word "celebrate." The big problem with
stemming is that sometimes it produces the root word which may not have
any meaning.
 For Example, intelligence, intelligent, and intelligently, all these words are
originated with a single root word "intelligen." In English, the word
"intelligen" do not have any meaning.
1-18

 Step 4: Lemmatization:
 Lemmatization is quite similar to the Stemming. It is used to group
different inflected forms of the word, called Lemma. The main difference
between Stemming and lemmatization is that it produces the root word,
which has a meaning.
 For example: In lemmatization, the words intelligence, intelligent, and
intelligently has a root word intelligent, which has a meaning.
1-19

 Step 5: Identifying Stop Words:
 In English, there are a lot of words that appear very frequently like "is",
"and", "the", and "a". NLP pipelines will flag these words as stop
words. Stop words might be filtered out before doing any statistical
analysis.
 Example: He is a good boy.
 Step 6: Dependency Parsing:
 Dependency Parsing is used to find that how all the words in the sentence
are related to each other.
1-20

 Step 7: POS tags:
 POS stands for parts of speech, which includes Noun, verb, adverb, and
Adjective. It indicates that how a word functions with its meaning as well
as grammatically within the sentences. A word has one or more parts of
speech based on the context in which it is used.
 Example: "Google" something on the Internet.
In the above example, Google is used as a verb, although it is a proper noun.
1-21

 Step 8: Named Entity Recognition (NER)
 Named Entity Recognition (NER) is the process of detecting the named
entity such as person name, movie name, organization name, or location.
 Example: Steve Jobs introduced iPhone at the Macworld Conference
in San Francisco, California.
1-22

Why NLP is difficult?
 NLP is difficult because Ambiguity and Uncertainty exist in the
language.
 There are the following three ambiguity -
1. Lexical Ambiguity
 Lexical Ambiguity exists in the presence of two or more possible meanings of the
sentence within a single word.
2. Syntactic Ambiguity
 Syntactic Ambiguity exists in the presence of two or more possible meanings
within the sentence.
 Example:
 I saw the girl with the binocular.
 In the above example, did I have the binoculars? Or did the girl have the binoculars?
1-23

Why NLP is difficult?
3. Referential Ambiguity
 Referential Ambiguity exists when you are referring to something using the
pronoun.
 Example: Kiran went to Sunita. She said, "I am hungry."
1-24

Readings
 Chapters 23 of Textbox.
1-25

AI_08_NLP.pptx

More Related Content

Similar to AI_08_NLP.pptx (20)

More from Yousef Aburawi (7)

Recently uploaded (20)

AI_08_NLP.pptx