A short introduction to multimedia forensics the science discovering the history of multimedia contents

A short introduction to Multimedia
Forensics: the science
discovering the history of
multimedia contents
Sebastiano Battiato
Dipartimento di Matematica e Informatica,
Università di Catania
Image Processing LAB – http://iplab.dmi.unict.it
battiato@dmi.unict.it

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015
Forensic science (often shortened to forensics) is the practical
application of science to matters of the law. Use of scientific
methods for gaining probative facts (from physical/analog or
digital evidences )

History of Digital Forensics
• Earliest notion of digital forensics came when the Federal
Rules (US) of evidence first started to discuss digital
evidence in the 1970s
• Real digital forensics investigations started in the mid-to late
1980s when federal agents had to start figuring out ways to
search computers for digital evidence
• This “home-grown”, bottom-up approach continued until the
late 1990s when security researchers at universities and
labs started to figure out that this problem was big enough to
warrant investigation.
• First research groups started around 2000 or 2001.

The first Digital Forensics Research
Workshop (DFRWS) was held in Utica, NY in
August 2001.

Multimedia
Forensics
- Source identification
- Integrity verification/tampering detection
Techniques from multimedia forensics merely provide a way to
test for the authenticity and source of digital sensor data. In this
sense is not about analyzing the semantics of digital or
digitized media objects.

Multimedia Forensics (in academic
literature)
• The terms appeared in early 2000
• K. J. Ray Liu, on “Multimedia Forensics: Where
Sherlock Holmes Meets Signal Processing” Invited
talk @ ICME 2006
• Multimedia Forensics is not Computer Forensics
(2009, Bohem et al.)
• Recent Surveys on the field (Stamm 2013, Piva 2013)
• WIFS, TIFS, Dedicated workshop at major conference
(ACM MM, ICME, ICIAP), Special issues on Int. Journal

Multimedia Forensics (in practice)
• Source Identification
• Integrity/Authenticity
• Enhancement/Restoration
• Interpretation and Content Analysis
– Plate Recognition
– Dynamic Reconstruction (car crashes, etc.)
– Antropomethric issues
– …

“Forensics Image (Video) analysis is
the application of IMAGE SCIENCE
and DOMAIN EXPERTISE to interpret
the content of an image or the image
itself in legal matters” (SWGIT –
www.fbi.gov)

Recent documents:
• 2015-09-29 SWGDE Best Practices for Photographic Comparison for All
Disciplines
• 2015-09-29 SWGDE Image Processing Guidelines
• 2015-09-29 SWGDE Proposed Techniques for Advanced Data Recovery
from Security Digital Video Recorders v1-1
• 2015-09-29 SWGDE Training Guidelines for Video Analysis, Image
Analysis and Photography V1-1
https://www.swgde.org/

Fantasy/Fiction

CSI

Reality

Fantasy
• We cannot introduce new information
(there is no data..)
• But we can enhance and extract only
something already present

• What is the minimum quality for video? “Minimum quality”
doesn’t exist.
• The success of the enhancement depends on several
factors:
Main goal (video captured with an HD camera but the
license plate we need to extract is too far away)
Technical related details: Resolution of the area of
interest, Level of compression, Presence of blur /
focus, Number of available frames, Noise / brightness
and contrast
• It’s important to understand which defects are
present in order to apply the proper tools.
Understand When It’s Possible To Get
Something

Perspective Correction
Geometrical transformation

Multimedia Forensics is based on the idea
that inherent traces (like digital fingerprints)
are left behind in a digital media during both
the creation phase and any other
successively process.

• Example:
• Forensic analysis of a smartphone: which pictures have been generated
on the device and which ones have been generated by other devices
and sent by messaging application or saved from the internet
• We can identify:
• Type of device
• Maker and model
• Specific exemplar
Camera Ballistics
Which Device Has Created This Picture?

Source Identification Noise Based
Sensor output carries not only pure signal
but also various noise components. Sensor
noise model could be used as a
representative feature for cameras.

Sensor Identification Using
Pattern Noise

Sensor Identification Using
Pattern Noise
This method provide good results, and is
quite reliable also using:
–images with different level of JPEG
compression (low, medium and high)
–images processed using point-wise operator
such as brightness/contrast adjustment or
gamma correction.
–images acquired by two cameras of the same
brand and model.

Integrity: What is a Forgery?
• “Forgery” is a
subjective word.
• An image can
become a forgery
based upon the
context in which
it is used.
• An image altered for fun or someone who has taken an bad
photo, but has been altered to improve its appearance
cannot be considered a forgery even though it has been
altered from its original capture.

What is a Forgery?
• The other side of forgery are those who perpetuate a
forgery for gain and prestige
• They create an image in which to dupe the recipient into
believing the image is real and from this be able to gain
payment and fame
• Three type of forgery can be identified:
•An image that is created using graphical software
•An image where the content has been altered
•An image where the context has been altered

The content has been altered
Duping the recipient into believing that the objects in an
image are something else from what they really are!
November 1997: After 58 tourists were killed in a terrorist attack at the
temple of Hatshepsut in Luxor Egypt, the Swiss tabloid Blick digitally altered
a puddle of water to appear as blood flowing from the temple.

The context has been altered
• Objects are be removed or added,
for example, a person can be
added or removed
• The easiest way is to cut an
object from one image and insert
it into another image – image
editing software makes this a
simple task
• An example is this altered image
which could be used to influence
events in foreign countries which
are not aware of manipulation.

Altering Images
The concepts have moved
into the digital world by
virtue of digital cameras
and the availability of
digital image editing
software
The ease of use of digital image editing software, which does
not require any special skills, makes image manipulation easy
to achieve.
circa 1860: This nearly iconic portrait of U.S. President
Abraham Lincoln is a composite of Lincoln's head and the
Southern politician John Calhoun's body.
27

Off-side (February 2011)
Barcelona – Athletico Bilbao

Sydney conman
who photoshopped
pictures of himself
with famous people
gets 12 years’jail
http://www.smh.com.au
31

More (and more) examples
Photo Tampering through History
http://www.fourandsix.com/photo-tampering-history/
Photoshopdisaster
http://www.photoshopdisasters.com/

http://petapixel.com/2013/05/08/how-photographers-photoshopped-their-pictures-back-in-1946/

Image Editing (2)
 Malicious image editing alters the image semantic
content, mainly:
 Adding information
 Removing information
Piva 2013

Image Editing (2)
• Splicing (two images)
– Also called cut and paste, compositing
– Used to add information
• Cloning (single image)
– Also called copy and paste, copy move, region duplication
– Used to add or remove information
– Can be exact, or the clone can be resized, rotated…)
• Inpainting (kind of intelligent clone)
– Seam carving, content aware resize, content aware fill, content dependent crop
– Used to remove information
• Retouch (local editing)
– Dodge and burn, healing tool…
• Image enhancement/filtering
– Histogram equalization, contrast enhancement, median filtering, denoise, smooth…
• Image editing (geometric transformation)
– Resize, crop, zoom, shear

Analog Vs. Digital
• Analogue evidence (audio, photo, video)
– Has always an original (negative or positive)
– Copies are never an exact replica of the original
– Copies of copies will be further deteriorated
• Digital evidence
– Only a finite sequence of number
– Exact copies can be made
– No loss of information between generations
– Every copy of a digital data can be considered the original,
unless is tagged in order to be distinguished by copies
Best evidence clause (Federal Rules of Evidence 1003): if data is stored
in a computer or similar device, any printout readable by sight, shown to
reflect the data accurately, is an “original”.
36

Original File: Special Cases
• Recapture: create a fake and then take a
picture with the camera we want to
pretend the picture was taken with
• Staging: the image file is authentic, but
the content has been staged
In these cases an authentic file does not
imply an authentic content.
37

How To Authenticate An Image?
• Visual Inspection
• File Analysis
File Format and Structures
Metadata (EXIF)
Compression Parameters (Quantization
Tables)
• Global Analysis
Pixel and compressed data statistics
• Local Analysis
Finding inconsistencies of pixel statistics
across the image

Image Forensics Methods
Active Methods: Hiding a mark or a
message in a picture when it is created:
WATERMARKING
Drawbacks:
• limitate to specially equipped
(=expensive) digital cameras;
• not so robust.

Image Forensics Methods
Passive Methods: Using the alterations of
the underlying statistics produced by digital
forgeries on an image:
PHYSICS BASED
CAMERA BASED
PIXEL BASED
GEOMETRIC BASED
FORMAT BASED

INTERPOLATION
LENS CFA SENSOR
POST
PROCESSINGDIGITAL IMAGESTORAGE
Processing and
Storage
ORIGINAL
IMAGE
Acquisition

Types Of Analysis: Scene Level
• Use features of the scene, not of the image pixels and could work well also
on scanned photos
• Difficult to spoof
– Most forgeries contains slight errors not visible by human eye but detectable by
proper analysis
• Difficult to automatize the analysis
– Requires a lot of experience
• Better results on splicing, less successful on forgeries which are hiding
something
• Physics based:
– Lighting
• Geometry and perspective inconsistencies
– Principal point analysis
– Shadows
– Photogrammetry

Types Of Analysis: Signal Level
Based on statistical features of pixel values; need good quality image
• Clone detection
– Cloned image blocks
– Similar couples of key points
• Resampling detection
– For resize, rotate, but also when splicing or cloning
• Enhancement Detection
– Specific for algorithms (median, histogram equalization, color
adjustment)
• Seam carving detection
• General intrinsic footprints
• Inconsistencies from acquisition and coding fingerprints
– CFA, PRNU, DCT, ELA…

Format-Based
JPEG compression engine
(for both luminance and chrominance channels):
the input image is
partitioned into 8x8
non-overlapping blocks

A JPEG image (luminance component only)
Format-Based

Format-Based

Format-Based
JPEG compression engine
(for both luminance and chrominance channels):
the input image
is partitioned
into 8x8 non-
overlapping
blocks
a DCT transform
is applied to each
block

Format-Based
JPEG forgery engine

THE TYPICAL PIPELINE
FOR A COPY-PASTE
OPERATION
+
=
original image
QF(1) = q1
resulting image
QF(3) = q3
2nd image
QF(2) = q2
duplicating
resizing

Order of quantization
Top row: single quantized with steps 2 (left) and 3 (right).
Bottom row: double quantized with steps 3 followed by 2 (left), and 2
followed by 3 (right)

F. Galvan, G. Puglisi, A. R. Bruna, S. Battiato, First Quantization Matrix Estimation from Double Compressed JPEG
Images, IEEE Transactions on Information Forensics and Security, 2014.

Current Trends And
Challenges

Current Trends: Point&Shoot
and Share…

Social MF on Facebook
Preliminar studies involving:
- Different devices
- Dataset of images:
- Different Scene (outdoor artificial,
outdoor natural, indoor)
- Different Quality (resolution and
compression)
- Different upload setting
has proven that some invariance could be
guaranteed.
Some specific editing could be then traced
and used to retrieve useful info about
image before uploading.
Moltisanti, Paratore, Battiato, Saravo - Image Manipulation on Facebook for Forensics
Evidence – ICIAP 2015, LNCS 2015;

Social (Multimedia) Forensics
• Image and Video Phylogeny
ReVeal project

Video Fingerprint
• Photo DNA
(http://www.microsoftphotodna.com/ )
• VideoGenome
(http://v-nome.org/ )
• Videntifier
(http://www.eff2.net/ )
73

PRNU on Video (online)
Each video is re-encoded (compressed) after it is uploaded. This
compression is needed to reduce the bandwidth usage by reducing the file
size and thus increasing streaming speeds.
By re-encoding the video, the video is altered which makes it harder to
identify the source.
Related performances depends on the brand of the camera, the resolution
and the amount of compression.
The reliability of the PRNU pattern decreases exponentially if the
compression becomes higher.
Challenges:
- Video obtained by some post processing (e.g., transition effects, fusion of
more than one source, …)
- No reference available
- Various Encoders

Who Cares?
geopolitics
…
…and political propaganda

WorldPhotoPress 2012: faked?

When is an image fake, and when
is it merely enhanced?

WPP Report:The integrity of the
Image (Nov. 2014)
Current practices and accepted
standards relating to the
manipulation of still images in
photojournalism and documentary
photography.
Media organizations
prohibit the alteration of
images beyond
traditional darkroom
techniques.
It means – as first – that the
alteration of images – where
alteration means the digital
addition or subtraction of
elements is forbidden

Data Manipulation on Science: Images
Gilbert, N. “Science Journals Crack down on Image Manipulation.” Nature (2009):
doi:10.1038/news.2009.991.
Source: Office For Research Integrity (ORI)

Forgery on Biomedical Images
Corriere della Sera – Ottobre 2013

Plagiarized paper:
Fabrication of
Monodisperse Magnetic
Fe3O4-SiO2
Nanocomposites with Core-
Shell Structures Hua Fang,*
Chun-yang Ma, Tai-li Wan,
Mei Zhang, and Wei-hai Shi
J. Phys. Chem C 2007, 111,
1065-1070
Original Paper
Oriented Assembly of
Fe3O4 Nanoparticles into
Monodisperse Hollow
Single-Crystal Microspheres
Yu et al, J. Phys. Chem. B
2006, 110, 21667-21671
(Figure 3)
M. Rossner and K. M. Yamada, “What’s in a picture? The temptation of image manipulation.,”
J. Cell Biol., vol. 166, no. 1, pp. 11–5, Jul. 2004.
Forgery on Science

Forgery on Science
“What’s in a picture? The temptation of image manipulation.,” J. Cell Biol., vol. 166, no. 1, pp.
11–5, Jul. 2004.

Image Manipulation: Case “Mozzarella Blu”
Multimedia Truthfulness
Verification in LegalMultimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015

Multimedia Truthfulness
Verification in LegalMultimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015

Evidence on the web

http://revealproject.eu/
http://www.rewindproject.eu/
http://maven-project.eu/#_=_
https://s-five.eu/
The first public draft of the FIVE Best
Practice Manual is publically available
from Oct. 9, 2015 ("October/DIWG2015
version"): DRAFT_BPM_FIVE_20151009

Future of Imaging
Nikon

Sharing

Conclusions
• Multimedia Forensics is now a
consolidated field but new intriguing
challenges emerge every day.
• The various involved actors are expected:
– To disseminate «basic knowledge»
– To share «best practices» and results
– To identify new challenges
– ..

Survey
• Matthew C. Stamm, Min Wu and K. J. Ray Liu,
Information Forensics: An Overview of the First
Decade (2013), in: IEEE Access, 1(167-200)
• Alessandro Piva, An Overview on Image Forensics
(2013), in: ISRN Signal Processing, 2013 (Article ID
496701, 22 pages)
- C. Baron - Adobe Photoshop Forensics – Sleuths,
Thruts, and Fauxtography – Thomson Course
Tehcnology - 2009

On line Resources
Tutorial by Prof. Hany Farid - Digital Image Forensics:
lecture notes, exercises, and matlab code for a survey
course in digital image and video
forensics. http://www.cs.dartmouth.edu/farid/downloads/tutor
ials/digitalimageforensics.pdf
Special Issue on Multimedia in Forensics, Security and
Intelligence - IEEE Multimedia Magazine Vol. 19, Issue 1,
pp. 17-19, 2012
SOFTWARE: Amped5, Authenticate, Adroit, Four&Six, Izitru,
Ghiro, …

Credits
• Amped (www.amped.it)
• Undergraduate course: Computer
Forensics – University of Catania
(http://www.dmi.unict.it/~battiato/CF.html)

Prof. Sebastiano Battiato
Dipartimento di Matematica e Informatica
University of Catania, Italy
Image Processing LAB – http://iplab.dmi.unict.it

Main Contacts
Further Info
Image Processing Lab
Università di Catania
www.dmi.unict.it/~iplab
Email

A short introduction to multimedia forensics the science discovering the history of multimedia contents

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to A short introduction to multimedia forensics the science discovering the history of multimedia contents (20)

Recently uploaded (20)

A short introduction to multimedia forensics the science discovering the history of multimedia contents