Cleaning plain text books with Text::Perfide::BookCleaner

Cleaning plain text books with
Text::Perﬁde::BookCleaner
Andr´ Santos
e
andrefs@cpan.org

September 23, 2011

Introduction Per-Fide

1 Introduction
Per-Fide
Text alignment
Books

2 Text::Perﬁde::BookCleaner

3 Conclusions, wish list and future work

Andr´ Santos andrefs@cpan.org
e Cleaning plain text books with Text::Perﬁde::BookCleaner


Project Per-Fide

Joint venture between the Computer Science
Department and the School of Humanities of
the University of Minho



Project Per-Fide

Portuguese in parallel with six languages:
Espa˜ol, Russian, Fran¸ais, Italiano, Deutsch,
n c
English



Project Per-Fide

Portuguese in parallel with six languages:
Espa˜ol, Russian, Fran¸ais, Italiano, Deutsch,
n c
English
Build parallel corpora that will establish a
relation between Portuguese and the other 6
languages



[Parallel] Corpora

Corpora Collection of natural language texts
Parallel corpora Collection of nat. lang. bitexts
Bitext Pair formed by a text in a given
language and its translation in
another language, frequently aligned.
Alignment Mapping between the
sentences/paragraphs/words of one
text and the other.



Project Per-Fide

Original texts in the seven languages and their
translations



Project Per-Fide

translations
Two main genres: contemporary ﬁction
and non-ﬁction



Project Per-Fide

translations
and non-ﬁction
non-ﬁction: judicial, journalistic, religious,
technical, ...



Project Per-Fide

translations
and non-ﬁction
technical, ...
ﬁction: contemporary novels and short
stories



Project Per-Fide

translations
and non-ﬁction
technical, ...
ﬁction: contemporary novels and short
stories
per-fide.di.uminho.pt


Introduction Text alignment

Text alignment
Manual or automatic



Text alignment
Manual or automatic
Paragraph/sentence/word level



Text alignment
Manual or automatic
Automatic alignment tools/algorithms
generally fall into three categories:



Text alignment
Manual or automatic
length based: “when two sentences correspond, the
words in them also correspond”



Text alignment
Manual or automatic
lexical/dictionary based: relies on lexical
information or dictionaries to perform the
alignment



Text alignment
Manual or automatic
lexical/dictionary based: relies on lexical
information or dictionaries to perform the
alignment
partial similarity (cognates) based: relies on
occurrences of tokens graphically or
otherwise identical (cognates)


Text alignment – Example

Table: Extract of sentence-level alignment performed using
Portuguese and Russian subtitles from the movie Tron.


Introduction Books

Books


Introduction Books

Books

Obtained directly from publishers or, if in
public domain, from Project Gutenberg and
similar projects
Large variety of formats: PDF, MS Word,
HTML, ebook formats, ...


Introduction Books

Books

similar projects
If not already in plain text, they need to be
converted before the alignment


Introduction Books

Books

similar projects
If not already in plain text, they need to be
converted before the alignment
This is where all the trouble starts!


Introduction Books

Book alignment problems
pagination – page numbers, headers,
footers, . . .
previous text formatting – sub/superscript,
bold, italics, . . .
sections
paragraphs
translineations and transpaginations
footnotes
text encoding
...

Introduction Books

Book alignment problems – Example
(. . . )
gaiement. Sur le devant s<92>’ouvrait la porte
d<92>’entr´e, donnant acc`s dans la salle commune.
e e
Une l´g`re v´randa, qui en prot´-
e e e e

<96>- 86 <96>-
^L geait la partie ant´rieure contre l<92>’action
e
des rayons solaires, reposait sur de sveltes bambous.
Le tout ´tait peint d<92>’une fra^che
e ı
(. . . )

La Jangada, Jules Verne



1 Introduction
Per-Fide
Text alignment
Books





First approach
RegExp + Find & Replace



First approach

Well-intentioned but:
Too na¨ıve
Big mess
A more soﬁsticated approach was needed!



Architecture
Build a pipeline; each step handles a speciﬁc set of
problems.
1 pages
2 sections
3 paragraphs
4 footnotes
5 chars
6 ...



Architecture
Build a pipeline; each step handles a speciﬁc set of
problems.
1 pages
2 sections
3 paragraphs
4 footnotes
5 chars
6 ...
7 commit



Architecture



Architecture

whenever possible, use ontologies and DSLs
they help organizing stuﬀ
they allow to abstract from the code and
discuss details at a higher level (even with
people from other areas)



Pages

Goal
Identify and remove from text elements related to
book pagination:
page numbers
headers
footers
page breaks
These elements often lead to a bad performance of
the aligner.



Pages – Example

est vrai qu’il fallait etre assez chanceux pour
^
rencontrer le nabab, et assez audacieux pour
s’emparer de sa personne.

Page 3
^L La maison ` vapeur
a Jules Verne

Le faquir, - evidemment le seul entre tous
´
que ne surexcit^t pas l’espoir de gagner la
a
prime, - filait au milieu des groupes, s’arr^tant
e

La Maison ` Vapeur, Jules Verne
a



Pages – Algorithm
1 identify page breaks (e.g., ^L )
2 nearby: candidates to headers and footers
3 count the occurrences of each normalized
candidate
4 headers and footers are extracted from
candidates which occur more thant a threshold
value
5 replace everything with a custom mark
6 move all the necessary information to a
standoﬀ ﬁle


Pages – Example

est vrai qu’il fallait etre assez chanceux pour
^
rencontrer le nabab, et assez audacieux pour
s’emparer de sa personne. _pb2_

Le faquir, - evidemment le seul entre tous
´
que ne surexcit^t pas l’espoir de gagner la
a
prime, - filait au milieu des groupes, s’arr^tant
e

La Maison ` Vapeur, Jules Verne
a



Sections

Goal
Identify and normalize the divisions between the
several sections of a book (parts, chapters, acts,
scenes, epilogue, afterword, ...)
An ontology was created, containing types of
divisions and subdivisions, in several languages.



Sections – Ontology
Example
cap
PT cap´tulo, cap, capitulo
ı
FR chapitre, chap
EN chapter, chap
NT sec

PT fim
FR fin
EN the_end
BT _alone

This ontology is used to automatically generate a
parte of the code.


Sections – Example
PRIMEIRA PARTE

FANTINE

^L LIVRO PRIMEIRO

UM JUSTO

O abade Myriel

Em 1815, era bispo de Digne, o reverendo Carlos
Francisco Bemvindo Myriel, o qual contava setenta e
Os Miser´veis, Vitor Hugo
a


Sections – Algorithm

1 Search for potential sections divisions:
lines with keywords – cap´ıtulo, chapter, Chap.,
Appendix, Table des Mati´res, . . .
e
pages or lines containing only numbers
roman numbering
...
2 Insert a custom mark immediately before the
section identiﬁed



Sections – Example
_sec+O:PARTE=PRIMEIRA_
FANTINE

_sec+O:LIVRO=PRIMEIRO_

UM JUSTO

O abade Myriel

Em 1815, era bispo de Digne, o reverendo Carlos
Francisco Bemvindo Myriel, o qual contava setenta e

Os Miser´veis, Vitor Hugo
a



Sections

Identifying the diﬀerent parts within a bitext:
allows to subsequently compare the two
versions and remove parts which can only be
found in one of them
allows to perform a structural alignment1

1
Text::Perfide::BookSync


Paragraphs

Goal
Handles things related with identifying and
normalizing paragraph notation, direct speech, etc.



Paragraphs – Example

L’h^tesse prit la d´fense de son cur´:
o e e

- D’ailleurs, il en plierait quatre comme vous sur
son genou. Il a, l’ann´e derni`re, aid´ nos gens a
e e e `
rentrer la paille; il en portait jusqu’` six bottes
a
a la fois, tant il est fort!
`

- Bravo! dit le pharmacien. Envoyez donc vos filles
en confesse a des gaillards d’un temp´rament pareil!
` e
Moi, si j’´tais le gouvernement, je voudrais qu’on
e
saign^t les pr^tres une fois par mois.
a e



Paragraphs – Example

L’h^tesse prit la d´fense de son cur´:
o e e

"D’ailleurs, il en plierait quatre comme vous sur
son genou. Il a, l’ann´e derni`re, aid´ nos gens a
e e e `
rentrer la paille; il en portait jusqu’` six bottes
a
a la fois, tant il est fort! "
`

"Bravo!" dit le pharmacien. "Envoyez donc vos filles
en confesse a des gaillards d’un temp´rament pareil!
` e
Moi, si j’´tais le gouvernement, je voudrais qu’on
e
saign^t les pr^tres une fois par mois."
a e



Paragraphs – Algorithm

paragraph identiﬁcation is performed by
calculating metrics based on the number of
blank lines and indentation
identiﬁcation and normalization of direct
speech:
punctuation, paragraph, dash
text in quotes



Footnotes

Goal
Identify and remove footnote callmarks and
footnote expansions



Footnotes – Example
On fit un inventaire de son argent comptant, et on
le mena dans le ch^teau que fit construire le roi
a
Charles V, fils de Jean II, aupr`s de la rue
e
Saint-Antoine, a la porte des Tournelles[1].
`

[1] La Bastille, qui fut prise par le peuple de
Paris, le 14 juillet 1789, puis d´molie. B.
e

^L Quel etait en chemin l’´tonnement de l’Ing´nu!
´ e e
je vous le laisse a penser. Il crut d’abord
`
que c’´tait un r^ve.
e e

Oeuvres de Voltaire, Voltaire



Footnotes – Algorithm

1 Search for footnote expansions (lines beggining
with <<1>>, [2], ^3, . . . )
2 Replace with custom mark
3 Only footnote call marks left
4 Search again for the same patterns in the
middle of the text
5 Replace with custom mark



a
e
Saint-Antoine, a la porte des Tournelles[1].
`

[1] La Bastille, qui fut prise par le peuple de
Paris, le 14 juillet 1789, puis d´molie. B.
e

(fbox^LQuel ´tait en chemin l’´tonnement de l’Ing´nu!
e e e
`
e e





a
e
Saint-Antoine, a la porte des Tournelles_fnr29_.
`
_fne8_

^L Quel etait en chemin l’´tonnement de l’Ing´nu!
´ e e
`
e e




Words and characters

translineations
text encoding
...



Report

Previous steps produce a report
Summarizes what was found, what was
assumed and what was done
Main goal is to allow to make a diagnostic of
the program, allowing to manually emend what
is wrong



Report

livros/_FR_15.pdf.txt:
footers=[’( Page) = 241’]
headers=[
"(La maison x{e0} vapeur Jules Verne) = 241"]
ctrL=1;
pagnum_ctrL=241;

sectionsO=2;
sectionsN=30;

word_tr=58;
words=118036;



Commit

Final and irreversible step which removes all
the custom marks added by the previous steps
Outputs a cleaned copy of the document
This is the last stage before the alignment (or
any other further processing)


Conclusions, wish list and future work

1 Introduction
Per-Fide
Text alignment
Books





Conclusions and wish list

There is no de facto standard format for plain
text books (documents?)
Documents are way heterogeneous
(provenience, type and quantity, notation
formats, . . . )
Hurrah to regular expressions!
20/80 rule applies



Conclusions and wish list

Ontologies and DSLs lead to a better structure
Common pattern:
search text
calculate metrics
perform action accordingly
Report generated at the end should present a
smart summary of what was found and done



Related ongoing work
Text::Perﬁde::BookPairs Find repeated books and
pairs of books (same book in diﬀerent
languages) within a collection



Text::Perﬁde::BookSync Uses the section
delimitation made by T::P::BC to make a
structural alignment:



Text::Perﬁde::CorporaFlow Uses a DSL to guide the
corpora preparation workﬂow (to be
done)



Text::Perfide::CorporaFlow Uses a DSL to guide the
corpora preparation workflow (to be
done)
Text::Perfide::SciPaperCleaner Cleaner for scientific
papers (to be done)


Future work

Standoﬀ annotation – no changes in the
original ﬁle until commit
Export to ebook formats – .fb2, .epub, . . .
...



CPAN

Is it on CPAN yet?



CPAN

Is it on CPAN yet?
No, but it will be really, really soon!

Missing
More and better documentation
More and better tests



Questions

o/

Andr´ Santos
e
andrefs@cpan.org


Cleaning plain text books with Text::Perfide::BookCleaner

More Related Content

Viewers also liked (11)

More from andrefsantos (10)

Recently uploaded (20)

Cleaning plain text books with Text::Perfide::BookCleaner