The challenges of file formats

Ange Albertini
the challenges
of
file formats
sharing my perspectives
with the DigiPres community

USE
CRAFTDISSECt
My perspectives
PRESERVE
Malicious files Extreme files
Documents
Video Games
Load/save
share

tinkering with Computers
for 30+ years...
since MO5 (1984)
I draw many kinds of things...github.com/corkami/pics

...and document file formats.
github.com/corkami/docs

The first use of file formats:
"Save your progress"

Ever printed something
that looked nothing like
what you have on screen?
Story time:
A document without vowels

Enabling people to share information:
a Wonderful digital language.

Ever tried to import
someone else's
Word document ?
"Ouch !"

The developer relies on the specifications
to add support in their library.

Preserve...
- Insert rant here -
Web pages anyone ? ;)

The archivist wants to make sure that
their data will be re-usable much later.

Ever tried to re-import
an old Word document ?
R.I.P.
Formatting
(at best)

I'm also a Reverse Engineer
● Interested since 1989
● Video games preservation in 1999
Science & Vie Micro, November 1989
Instructions to manually remove a boot sector virus

PRofessionally
● 12 years of malware analysis
● Executables, documents...
Note: this talk reflects my own opinion, not my employer.

Attackers will try to take
advantages of weaknesses...

Black hats
Find (and sell) vulnerabilities
in software that can be exploited
to steal information or money,
or spy on people (and get them arrested…).
Goals: bypass security, hack into systems.

Find bugs to hack your target.
- Analyze code (source)
or disassembly (the binary itself)
- Blind fuzzing (flip bits randomly and see what happens)
- Generational fuzzing (according to the structure of the file format)
- Smart fuzzing (modify/instrument/analyse code)

...while defenders will try
to prevent this from happening.

Do the same as the bad guys
But disclose and fix vulnerabilities (see Bug Bounties)
- Remove the low-hanging fruits from the tree.
- Controlled burn to prevent huge fires.
=> improve practices
IMHO file formats should receive this treatment too.

Anti-malware industry
1- Analyze files
2- Determine if they're corrupted, or malicious
3- come up with ways to detect them
4- improve defenses

Malware analyst or DFIR are looking for clues.
(typically, I open files with a hex editor first)

Digital Forensics - Incident Response
determine if and how a file or system was:
- hacked
- tampered with (casino machines)
- If data was stolen (copyright infrigment)

One more thing...
So far, just InfoSec things:
nothing for the DigiPres world.

PRofessionally (next)
- I was fed up of being only retroactive
to attackers' "innovation"
- I started to experiment and
create my own files, from scratch.
And share them openly and freely.

A single file with multiple types:
It's not a gadget,
It's actually useful to hack people!
Used in the wild, in 2008!
https://en.wikipedia.org/wiki/Gifar
Polyglots Image
Java

a mini PDF (Adobe-only, 36 bytes)
"Too small to be
Suspicious!"
(or even considered valid)
%PDF-0trailer<</Root<</Pages<<>>>>>>

How do I do it ?
1. I study a simple file with the specs
2. I create my own script to reproduce it
(Standard tools usually don't give you total control)
3. I can now create my own files, with full control
4. I can then experiment,
and optionally, document and visualize.

The challenges of file formats

my collection of hand-made executablesand "documentation" (completely free).

Victor Frankeinstein
Initially, I started with simpler stuff
But… it excalated quickly :)
Dual headed cow
Loooooong !

a presentation slide deck viewing itself
(PDF viewer and PDF document)
PDF viewer
PDF slides

HTML JavaScript Java
Windows executable
PDF
2 standard infection chains
in a single file

1
3DES
Mixing binary and cryptography
AESK
AESK
JPG
JAR
(ZIP + CLASS)
PDF
FLV
PNG
2

a Java & JavaScript polyglot - at source level
unicode //

a Java & JavaScript polyglot - at binary level

=> Java = JavaScript
Yes, your management was right all along ;)

My own Resume is a PDF,
compatible Nintendo and Sega

https://www.w3.org/Graphics/JPEG/jfif3.pdf
I crafted the binary structure of the files
with the first SHA1 collision.
These files violate the specifications!
And yet they work everywhere.
same SHA-1

These files display
their own MD5!
(not by me)
PDF
GIF
Nintendo Rom

a JavaScript || GIF polyglot (useful to embed payload - also works with JPG or BMP)
image
JavaScript

"useless?"I remove potential traps by researching,
And I train myself with extreme files.
Also, some of these were used in the wild to attack people,
And I can reproduce key features into shareable files.
Isn't this all

My perspective:
1- What is a file?
A sequence of byte(s)
Any parser can give an incomplete perspective.
I open most files first with a hex editor (out of curiosity, at least)Yes, I'm a hex-addict ;)

2- What is a VALID file?
A file loaded successfully
by a parser/loader/processor.
A file in itself is nothing.

Specifications.
What we wished...

Specifications.
In reality, they are more complex,
often for no particular reason (see design by commitee)

Specifications.
Now comes a software that takes these files as input.
This software defines validity (the parser/loader), not the specifications.
READER

Specifications are irrelevant!
As long as the file 'works as intended'.

We'll launch...
Run this
command

size=0
create empty file
Let's create… an EMPTY file!

Is it valid?
Yes: Transient Commands are copied blindly
and execution started at offset zero.

Does it do anything?
Transient Memory Area is not
Cleared between executions,
so the previous command is re-executed.

works as intended
Under a commercial OS from 1985,
the empty file is valid, useful and reliable.
It was even sold as a commercial program for £5.

Takeaway
- it's old-school, obsolete…
- And yet this example is pure simplicity…
to prove that the software defines the rules!
- The specifications are volatile….
the software is the ground truth.

Text files
What could go wrong?
Just a bad encoding maybe?

This is a …
malicious Flash file !!No, not base64 - it's directly executable as is!
But, aren't Flash files...
pure binary!?
CWSMIKI0hCD0Up0IZUnnnnnnnnnnnnnnnnnnnUU5nnnnnn3Snn7iiudIbEAt333swW0ssG03
sDDtDDDt0333333Gt333swwv3wwwFPOHtoHHvwHHFhH3D0Up0IZUnnnnnnnnnnnnnnnnnnnU
U5nnnnnn3Snn7YNqdIbeUUUfV13333333333333333s03sDTVqefXAxooooD0CiudIbEAt33
swwEpt0GDG0GtDDDtwwGGGGGsGDt33333www033333GfBDTHHHHUhHHHeRjHHHhHHUccUSsg
SkKoE5D0Up0IZUnnnnnnnnnnnnnnnnnnnUU5nnnnnn3Snn7YNqdIbe13333333333sUUe133
333Wf03sDTVqefXA8oT50CiudIbEAtwEpDDG033sDDGtwGDtwwDwttDDDGwtwG33wwGt0w33
333sG03sDDdFPhHHHbWqHxHjHZNAqFzAHZYqqEHeYAHlqzfJzYyHqQdzEzHVMvnAEYzEVHMH
bBRrHyVQfDQflqzfHLTrHAqzfHIYqEqEmIVHaznQHzIIHDRRVEbYqItAzNyH7D0Up0IZUnnn
nnnnnnnnnnnnnnnnUU5nnnnnn3Snn7CiudIbEAt33swwEDt0GGDDDGptDtwwG0GGptDDww0G
DtDDDGGDDGDDtDD33333s03GdFPXHLHAZZOXHrhwXHLhAwXHLHgBHHhHDEHXsSHoHwXHLXAw
XHLxMZOXHWHwtHtHHHHLDUGhHxvwDHDxLdgbHHhHDEHXkKSHuHwXHLXAwXHLTMZOXHeHwtHt
HHHHLDUGhHxvwTHDxLtDXmwTHLLDxLXAwXHLTMwlHtxHHHDxLlCvm7D0Up0IZUnnnnnnnnnn
nnnnnnnnnUU5nnnnnn3Snn7CiudIbEAtuwt3sG33ww0sDtDt0333GDw0w33333www033GdFP
DHTLxXThnohHTXgotHdXHHHxXTlWf7D0Up0IZUnnnnnnnnnnnnnnnnnnnUU5nnnnnn3Snn7C
iudIbEAtwwWtD333wwG03www0GDGpt03wDDDGDDD33333s033GdFPhHHkoDHDHTLKwhHhzoD
HDHTlOLHHhHxeHXWgHZHoXHTHNo4D0Up0IZUnnnnnnnnnnnnnnnnnnnUU5nnnnnn3Snn7Ciu
dIbEAt33wwE03GDDGwGGDDGDwGtwDtwDDGGDDtGDwwGw0GDDw0w33333www033GdFPHLRDXt
hHHHLHqeeorHthHHHXDhtxHHHLravHQxQHHHOnHDHyMIuiCyIYEHWSsgHmHKcskHoXHLHwhH
HvoXHLhAotHthHHHLXAoXHLxUvH1D0Up0IZUnnnnnnnnnnnnnnnnnnnUU5nnnnnn3SnnwWNq
dIbe133333333333333333WfF03sTeqefXA888oooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooo888888880Nj0h
<= "Hello World" in Flash:
It's a lot of non-ASCII!

Flash can be compressed with ZIP's Deflate.
Which can use the Huffman algorithm,
in which case you supply a code dictionary.
You can craft such a dictionary to 'expand' your data,
But in return it's ascii-only.
>>> zlib.decompress('xf3Hxcdxc9xc9Wx08xcf/xcaIQxe4x02x00 x91x04H',-8)
'Hello World!n'
>>> zlib.decompress("D0Up0IZUnnnnnnnnnnnnnnnnnnnUU5nnnnnn3SUUnUUUwCiudIbEAtwwwEt3
33wwG0swGpDDGpDDwDDDGtD33333s033333GdFPkWwwOaGOowgQ4", -8)
'Hello World!n'
https://molnarg.github.io/ascii-flash/ascii-flash.pdf
https://github.com/molnarg/ascii-zip

Valid Flash files entirely in ASCII:
-> bypassed all filters
-> abused most websites
https://miki.it/blog/2014/7/8/abusing-jsonp-with-rosetta-flash/

Specifications are blurry.
There's always a corner case that is not clarified.

Specifications are imperfect.
I know no perfect specifications.except for the empty file? :)
I could only make specifications better
by finding problems before they were
finalized and set in stone.

The real problem
● Unlike laws, specifications are not enforced.
● Not updated either. Set in stone.
● The problem remains the same - and grows with time.
● Survival of the fittest software:
the Winner decides the rules. Specifications
Conformity
(Whatever that means)

Specifications.
Now comes a software that creates these files.
Their planned features and actual abilities of readers and writers may not overlap.
Writer
READER

Large Format Scanners:
Infinite "height" scans
-> image height fixed to 65535!
Tolerated by LibJPEG,
So valid everywhere!
Detected by Anti-Virus, because it was used to exploit MS04-028.

Specifications are not updated:
They become outdated and irrelevant.

File formatsSome specifications are even worse:
They're nothing but…
a"gentle introduction":
Almost useless from the start.
Data is
in the .data
section
SEEN
ON TV

Divergences
As specifications are not perfect,
They are interpreted by different people in different ways:
So one file may work on one reader, not on the other one.

%PDF
1 0 obj
<< /Pages
<< /Kids [
<< /Contents 2 0 R >>
] >>
>>
2 0 obj
<<>>
stream
95 Tf
20 400 Td
(Chrome) Tj
endstream
trailer <<
/Root 1 0 R
>>
%PDF-1.
1 0 obj
<< /Kids [<<
/Parent 1 0 R
/Resources <<>>
/Contents 2 0 R
>>]
>>
2 0 obj
<<>>
stream
BT
/F1 110 Tf
10 400 Td
(Adobe Reader) Tj
ET
endstream
endobj
trailer <<
/Root << /Pages 1 0 R >>
>>
working PDFs

%PDF-1.
1 0 obj
<< /Kids [<<
/Parent 1 0 R
/Resources <<>>
/Contents 2 0 R
>>]
>>
2 0 obj
<<>>
stream
BT
/F1 110 Tf
10 400 Td
(Adobe Reader) Tj
ET
endstream
endobj
trailer <<
/Root << /Pages 1 0 R >>
>>
truncated signature
direct /Kids
No /Type
No /Font
No /CountNo /Type
No /Length
No XREF
Direct /Root
No /Size
No /Type No startxref
No %%EOF
%PDF
1 0 obj
<< /Pages
<< /Kids [
<< /Contents 2 0 R >>
] >>
>>
2 0 obj
<<>>
stream
95 Tf
20 400 Td
(Chrome) Tj
endstream
trailer <<
/Root 1 0 R
>>
very truncated signature
direct /Kids
No /Type
No /Font
No /CountNo /Type
No /Resources
No endobj
No /Length
No XREF
Direct /Root
No /Size
No /Type No startxref
No %%EOF
No BT/ET No Font selection
INVALID?
INVALID?
No /Parent

I made extreme PDFs for each reader [by hand].

These extreme PDFs fail on any other reader.

Specifications.
If one of these software becomes standard, the other software will have to adapt to it.
Writer
READER

RECOVERY
Since there's no official 'direction',
other softwares may have
to be taken into consideration.

Take a standard PDF.
(it opens in Adobe
with no warnings)

If you modify
its XREF table,
it won't work correctly...

...or maybe even
not open at all!

But if you ERASE
the XREF table,
Adobe will fall back
to recovery mode,
and open the file
without any warning!

Schizophrenia
Different contents (clean & malicious) can be combined
In the same file, to bypass security or fool softwares.

Zip archives
3 different softwares will see
3 different archives
from the same file
This enabled a critical vulnerability
in all Android devices in 2013:
validate a content, execute a different one!
1
2
3

PDF
The trailer defines the start of the document tree.
See a different trailer -> see a completely different document!
Several trailers can co-exist in the same file, and
Parsers tolerate 'unused' objects (referenced by unseen trailers).

3 different documents as seen by 3 different readers in the same file
(such things even happen accidentally in the wild!)
commented line - seen by PDFium
missing trailer keyword, but seen by Poppler
Standard trailer - the only one seen by Adobe Reader

It used to work with PDF/A too
(OK for Adobe Reader, but not for Preflight)

What you see is not always what you print - when you use Layers [O ptional C ontent G roups]!
Fun fact: you can’t change the printing output with Adobe Reader ;)
Layers present

1 image (same data), 2 palettes

Is this just an infinite
vicious circle?
Do we need a great fire destroying our knowledge
so that we build file formats in a smarter way?
Or ultimately, we'll forget our roots and
just maintain stacks of emulation layers…?

Law enforcements
Like archivists, investigators rely on softwares
to determine if someone is guilty or not.
These softwares rely on the same blurry specs…
They're vulnerable to the same problems.
Same file, two different tabs

http://www.pdfa.org/2015/10/whats-unique-about-pdf/
- Flash is now dying, for security reasons.
- Adobe is out of the game for PDF.
- Looking at PDF 2.0, I'm very skeptical…
(many extra security risks)

- specs & software tolerated to encode
a (malicious) JavaScript as JPEG.
- This was used to bypass security scans.
- This 'feature' was killed to improve security
-> specifications were never updated -> they're now clearly outdated.
script == picture

PDF is great, but...
No major actor (Adobe) behind it anymore?
PDF 2.0 is too different?
Too permissive security-wise?
(Don't get me wrong, I really like to [ab]use PDF)

My point of view:
need to create better corpuses
To demonstrate the current state of things.
To determine the actual limits of file formats.
To make automation easier for everyone.
Atomic - Copyright-free - PII-free

Preservation
Content, or the files? And private data?
Preserving usually means "saving to a safer format"
This means we gave up on the original format…
But which one is safe?
We need to preserve interesting files as-is too.

We don't need 'safer' formats(whatever that means)
Yet another format?
Blurry specs,
Divergences,
Unclear corner cases...

We need a new process:
File formats should be alive!
Like cryptography: Updates, deprecations…
-> eventually uniform standardisation
With a date, a version number, a commit.
Up to date specifications and open validator.
Once things are properly documented, we could go back easily.

Why doesn't it happen?
Because we haven't proved enough yet how broken they are?
In cryptography, there are official competitions
to break the drafts before choosing the standard:
survival of the fittest before setting standards in stone.

- Files formats are awesome (Even PDF!)
- Except when they don't work as expected :)
- Specifications are not challenged enough.
- formats authorship is not a liability.
We need to develop our expertises
and share our knowledge.

Feedback?
Thank you for
reading that far!

Extra resourcesFunky files formats:
https://speakerdeck.com/ange/funky-file-formats-31c3
Schizophrenic files:
https://speakerdeck.com/ange/schizophrenic-files-v2
Abusing file formats:
https://archive.org/stream/pocorgtfo07#page/n17/mode/2up

The challenges of file formats

More Related Content

What's hot (8)

Similar to The challenges of file formats (20)

More from Ange Albertini (20)

Recently uploaded (20)

The challenges of file formats