Indexing and Retrieval of Audio

Indexing and Retrieval of
Audio
Rachmat Wahid Saleh Insani, S.Kom
Multimedia Database Management System - Chapter 5

Introduction
• Audio is classified into three types: speech, music,
and noise.
• Different audio types are processed and indexed in
different ways.
• Query audio pieces are similarly classified, processed,
and indexed.
• Audio pieces are retrieved based on similarity between
the query index and the audio index in the database.

Objectives
• Main audio properties and features.
• Audio classification.
• Main speech recognition techniques.
• General approach in indexing and retrieval.
• Temporal and content relationship between media
types.

Main Audio Properties and
Features
• Time domain
• Frequency domain

Features Derives in the
Time Domain
A signal is represented as amplitude varying with
time.

Features Derives in the
Time Domain
• Average energy
• Zero crossing rate
• Silence ratio
E =
NΣ
N−1 Σ
N
x(n)2
n=0
ZC =
| sgn x(n) − sgn x(n −1)
n=1
2N

Features Derived from
the Frequency Domain
• Sound spectrum

Features Derived from
the Frequency Domain
• Bandwidth
• Energy Distribution
• Harmonicity
• Pitch

Timbre
• Quality of a sound.

Audio Classification
Why audio classification is important?
- Different audio types require different processing and indexing
retrieval techniques.
- Different audio types have different significance to different
applications.
- Speech is important audio types which is successful speech
recognition techniques available.
- Audio types is very useful to some applications.
- The search space after classification is reduced to a particular
audio class during the retrieval process.

• There are two types of sound: speech and music.

Main Characteristics
Music
• Music has frequency range
from 16-20,000 Hz.
• Music has low silence ratio.
• Music has regular beats.
Speech
• Speech frequency
range from 100-7,000
Hz.
• Speech has high
silence ratio.
• No regular beats.

Frameworks
• Step by Step Classification
• Feature Vector Based Audio Classification

Step by Step
Classification

Feature Vector Based
Audio pieces of the same class are located close to
each other in the feature space and audio pieces of
different classes are located far apart in the feature
space.

Speech Recognition
and Retrieval

Automatic
Speech Recognition
ASR system collect models or feature vectors for all
possible speech units. Speech unit e.g., phoneme,
word, and phrases.

Automatic Speech
Recognition Factors
• A phoneme spoken by different speakers or the same
speaker at different times produces different features in
term of duration, amplitude, and frequency
components.
• The above differences are exacerbated by the
background or environmental noise.
• Normal speech is continuous and difficult to separate
into individual phonemes.
• Phonemes vary with their location in a word.

General ASR System

Speech Recognition
Performance
Speech recognition performance is normally measured by
recognition error rate. The lower the error rate, the higher the
performance.
The performance are affected by following factors:
- Subject matter: this may vary from a set of digits, a
newspaper article, to general news.
- Types of speech: read or spontaneous conversation.
- Size of the vocabulary: it ranges from dozens to a few
thousand words.

Music Indexing and
Retrieval

Indexing and Retrieval of Structured
Music and Sound Effects
• Structured music are represented by a set of
commands.
• The most common structured music is MIDI.
• A new standard of structured audio is MPEG-4
Structured Audio.
• The formats contains structure and notes
description.

Indexing and Retrieval of Structured
Music and Sound Effects

Indexing and Retrieval of
Sample Based Music
• Based on extracted sound features.
• Based on pitches of music notes.

Music Retrieval Based on a
set of Features

Music Retrieval Based on
Pitch

Multimedia Information IR Using
Relationships between Audio and Other
Media

Indexing and Retrieval of Audio

More Related Content

What's hot (20)

Similar to Indexing and Retrieval of Audio (20)

More from Rachmat Wahid Saleh Insani (9)

Recently uploaded (20)

Indexing and Retrieval of Audio