Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Representations from Transformers

Introduction Backgrounds Methods Results Conclusion Appendix
11th
International Conference on Electrical, Electronics and
Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Track: Artiﬁcial Intelligence (session AII2, slot AII2.3, paper #9264)
Hall 3 15:15 Tuesday, 5 June 2024
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
1 10 100 1000 10000
ROUGE-1R
ROUGE-2R
ROUGE-LR
ROUGE-SU4R
Fitness (scaled)
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
1 10 100 1000 10000
ROUGE-1R
ROUGE-2R
ROUGE-LR
ROUGE-SU4R
Fitness (scaled)
Comparing Evolved
Extractive Text Summary
Scores of
Bidirectional Encoder
Representations from
Transformers
Aleš Zamuda1
, Jani Dugonik1
, Elena Lloret2
1
University of Maribor <ales.zamuda@um.si>
2
University of Alicante
Acknowledgement. This work is supported by project DAPHNE (Integrated Data Analysis Pipelines for Large-Scale Data Management, HPC and Machine Learning) funded by the European Union’s Horizon 2020
research and innovation programme under grant agreement No 957407and ARIS (Slovenian Research And Innovation Agency) programme P2-0041 (Computer Systems, Methodologies, and Intelligent Services).
Part of travel expense to present the paper is also supported through IEEE GRSS Inter Society Networking (ISN) Activities grant involving IEEE Slovenia GRSS chapter and IEEE Slovenia CIS chapter. Moreover, this
work is also conducted as part of the R&D project “CORTEX: Conscious Text Generation” (PID2021-123956OB-I00), funded by MCIN/ AEI/10.13039/501100011033/ and by “ERDF A way of making Europe”.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40 60 80 100 120 140 160 180
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) sim
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model1) F1
BERT(peer,model2) P
BERT(peer,model2) R
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40 60 80 100 120 140 160 180
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
1 10 100 1000 10000
ROUGE-1R
ROUGE-2R
ROUGE-LR
ROUGE-SU4R
Fitness (scaled)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40 60 80 100 120 140 160 180
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Aleš Zamuda 7@aleszamuda Comparing Evolved Extractive TS Scores of BERT @ IcETRAN, Niš, 3-6 June 2024 1/181

11th
Comparing Evolved Extractive Text
Summary Scores of Bidirectional
Encoder Representations from
Transformers
—
Introduction & Outline
—

Introduction & Outline: Aims of this Talk
1 (5 minutes) Part I: Background – Text Summarization
and Summary Evaluation
2 (5 minutes) Part II: Methods
3 (2 minutes) Part III: Results
4 (1 minutes) Part IV: Conclusion with Takeaways
5 (1 minute) Questions, Misc
6 (Appendix) AoB

11th
Transformers
—
I.a Background: Text
Summarization
—

HPC Application 1:
Text Summarization
• NLP and computational linguistics for Text Summarization:
• Multi-Document Text Summarization is a hard CI challenge.
• Basically, an evolutionary algorithm is applied for
summarization,
• it is a state-of-the-art topic of text summarization for NLP (part of
”Big Data”) and presented as a collaboration [JoCS2020],
acknowledging several efforts.
• we add: self-adaptation of optimization control parameters;
analysis through benchmarking using HPC, and
apply additional NLP tools.
• How it works: for the abstract, sentences from original text are
selected for full inclusion (extraction).
• To extract a combination of sentences:
• can be computationally demanding,
• we use heuristic optimization,
• the time to run optimization can be limited.

1 – Preprocessing (environment sensing, knowledge
representation) (1/2)
1) The ﬁles of documents are each taken through the following
process using NLP (Natural Language Processing) tools:
INPUT
CORPUS
NATURAL
LANGUAGE
PROCESSING
ANALYSIS
CONCEPTS
DISTRIBUTION
PER
SENTENCES
MATRIX OF
CONCEPTS
PROCESSING
CORPUS
PREPROCESSING
PHASE
OPTIMIZATION
TASKS
EXECUTION
PHASE
ASSEMBLE
TASK
DESCRIPTION
SUBMIT
TASKS
TO PARALLEL
EXECUTION
OPTIMIZER
+
TASK DATA
ROUGE
EVALUATION
2) For each document is D, sentences are indexed using NLP
tools.
• Terms across sentences are determined using a semantic
analysis using both:
• coreference resolution (using WordNet) and
• a Concept Matrix (from Freeling).

• 3) For each i-th term (wi), during indexing
• number of occurences in the text is gathered, and
• number of occurences (nk) of a term in some k-th
statement,
• 4) For each term wi in the document, inverse frequency:
isfw i = log(
n
nk
),
• where n denotes number of statements in the document,
and
• nk number of statements including a term wi.
• 5) To conclude preprocessing, for each term in the
document, a weight is calculated:
wi,k = tfi,kisfk,
where tfik is number of occurences (term frequency) of a
term wk in a statement si.

2 – Summary Optimization (1/3)
• Sentence combination (X) is optimized using jDE
algorithm:
• as a 0/1 knapsack problem, we want to include optimal
selection of statements in the final output
• an i-th sentence si is selected (xi = 1) or unselected
(xi = 0).
• a) Price of a knapsack (its fitness) should be maximized,
• the fitness represents a ratio between content coverage,
V(X), and redundancy, R(X):
f(X) =
V(X)
R(X)
,
• considering a constraint: the summary length is L ± ϵ
words.
• Constraint handling with solutions:
• each feasable solution is better than unfeasable,
• unfeasable compared by constraint value (lower better),
• feasable compared by fitness (higher better).

• b) Content coverage V(X) is computed as a double sum of
similarities (deﬁned at d)):
V(X) =
n−1
X
i=1
n
X
j=i+1
(sim(si, O) + sim(sj, O))xi,j,
• where xi,j denotes inclusion of both statement, si, and sj,
• xi,j is only 1 if xi = xj = 1, otherwise 0,
• and O is a vector of average term weights wi,k:
O = (o1, o2, ..., om) for all i = {1..m} different text terms:
oi =
Pn
j=1 wi,j
n
.
• c) Redundance R(X) is also measured as double similarity
(deﬁned at d)) sum for all statements:
R(X) =
n−1
X
i=1
n
X
j=i+1
sim(si, sj)xi,j,
• again, xi,j is only 1 if xi = xj = 1, otherwise 0.

• d) Similarity between statements si = [wi,1, wi,2, ..., wi,m]
and sj = [wj,1, wj,2, ..., wj,m] is computed:
sim(si, sj) =
m
X
k=1
wi,kwj,k
qPm
k=1 wi,kwi,k
Pm
k=1 wj,kwj,k
,
where wi,k is term weight (deﬁned in 5)) and m number of
all terms in text.
• e) When concluded:
• the selected statements from the best assessed
combination are printed,
• in order as they appear in the text, and
• the summary is stored.

Summary Optimization — Algorithm Pseudocode
The detailed new method called
CaBiSDETS is developed in the
HPC approach comprising of:
• a version of evolutionary
algorithm (Differential
Evolution, DE),
• self-adaptation, binarization,
constraint adjusting, and
some more pre-computation,
• optimizing the inputs to
deﬁne the summarization
optimization model.
Aleš Zamuda, Elena Lloret. Optimizing
Data-Driven Models for Summarization
as Parallel Tasks. Journal of
Computational Science, 2020, vol. 42, pp.
101101. DOI 10.1016/j.jocs.2020.101101.

Running the Tasks on HPC: ARC Job Preparation &
Submission, Results Retrieval & Merging [JoCS2020]
Through an HPC approach and by parallelization of tasks,
a data-driven summarization model optimization yields
improved benchmark metric results (drawn using gnuplot merge).

Results Published in Journal of Computational Science
The most interesting finding of the HPC study though is that
• the fitness of the NLP model keeps increasing with
prolonging the dedicated HPC resources (see below) and
that
• the fitness improvement correlates with ROUGE
evaluation in the benchmark, i.e. better summaries.
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
1 10 100 1000 10000
ROUGE-1R
ROUGE-2R
ROUGE-LR
ROUGE-SU4R
Fitness (scaled)
Hence, the use of HPC significantly contributes to
capability of this NLP challenge.

Summarization Implementation — Parallel
• In text summarization,
once the text model has been built,
• different lengths of summaries can be created with
Summarizer using CaBiSDETS algorithm in parallel.
• Generates large summaries
(more output tokens than single prebuilt LLMs)
& accepts long task inputs (no pre-clustering).
• Results are explainable, tracable (no hallucinated
content/citations), and automatic (no manual RL scoring).
• But moreover from HPC & Big-Data perspective,
• the input text can be preprocessed in parallel
• by computing the cosine sentence similarity pairs
in parallel using MPI in an integrated pipeline,
• before the generation of summaries commences and
• just before a complete parameterization of the summary
generation process.
• During summary optimization,
the ﬁtness function evaluations can be run in parallel.

Summarization Task Conﬁguration & Execution
• An example task for Summarizer computational architecture
• based on a recent lecturing material,
PDF −→ text (language), on:
• Optimization Algorithms and Autonomous Systems
• Las Palmas de Gran Canaria, March 2023.
• https://www.slideshare.net/AlesZamuda/ulpgc2023erasmuslecturesaleszamuda
systemstheoryintelligentautonomoussystemseswamllsgohpcdeugpppdf
• for Summarizer runs, the parameters were:
--GMAX 1000
--NP 319
--summarylength 500
--epsilonLengthSummary 20
• For the job execution, the conﬁguration for SLURM was:
srun --ntasks-per-node=16
--mpi=pmix
./summarizer.sif
./summarizer

Summarization Job Speedup Results
• Speedup comparison: SLURM --nodes parameter = 1, 2, 3, 4, or 5
• obtained timings on this scaling as seen on the graphs below.
150
200
250
300
350
400
450
500
550
16 32 48 64 80
Seconds
to
compute
a
workload
Number of tasks (equals 16 times the SLURM --nodes parameter)
Summarizer workload
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
16 32 48 64 80
Seconds
to
compute
a
workload
Summarizer workload
1
2
3
4
5
16 32 48 64 80
Workload
scaling
(wall
time)
Summarizer workload
Resources

Optimization Beginnings - Optimization is
”Everywhere”
• Time: optimizing distribution of what is matter and what is not
(anti-matter), what is energy and what is not (dark energy), etc.:
according to the function of Nature, the system is propelled through
optimizing its constituents dynamics.
• Organic systems combination and propulsion: life (optimization).
• Optimality and optimization modeling (human builds tools).
• Describing ways of acchieving optimality.
• Mathematical optimization procedure deﬁned (Kepler).
• Stepping towards optimum (Newton), gradient method (Lagrange).
• Multi-objective optimization (Pareto):
• meta-criterion (A ⪯ B): make criteria ordered by
dominance.
f′
(x) =
∆f(x)
∆x
,
f∗
(x) = f(x) + ∆xf′
(x).
1
2 2
f
x
x 1
f
( )
A
B
C
D
f x
f(B)
(A)
f
f(D)
0
0
E
f(E)
F
G f
(C)
f
f(F)
(G)

Introduction to Optimization Algorithms
and Mathematical Programming
• Global optimization, mathematical programming, digital computers.
• Computing Machines + Intelligence = Artiﬁcial Intelligence.
• Computational Intelligence.
• Simplistic numerical optimization algorithms:
hill climbing, Nelder-Mead, supervised random search,
simulated annealing, tabu search.
• Optimization: constrained, inseparable, multi-modal, multi-objective,
dynamic, noisy, high dimensional/large-scale/big-data, deceptive, etc.
• multi-objective: f(x)): Pareto optimal approximation set.

Evolutionary Computation and Algorithms
• Evolution theory: C. Darwin (1859), Weismann, Mendel.
• Popularization: darwinism (Huxley), neodarwinism
(Romanes).
• Generational: reproduction, mutation, competition,
selection.
• Evolutionary Computation: Evolutionary Algorithms (EAs)
• population generations (reproduction-based),
• mutation, crossover, selection (evolutionary operators),
• EAs comprised of different mechanisms.
• These algorithms share several common
mechanisms/operators,
• good conﬁgured DEs were prevalent at the winning
positions of all (CEC, including ICEC 1996) competitions on
optimization.

Evolutionary Computation and Algorithms: Given
Names
• Simulated Annealing (SA),
• Tabu Search (TS),
• Genetic Algorithms (GA),
• Genetic Programming (GP),
• Evolutionary Programming (EP),
• Memetic Algorithms (MA),
• Evolution Strategy (ES),
• Artificial Immune Systems (AIS),
• Cultural Algorithms (CA)
• Swarm Intelligence (SI),
• Particle Swarm Optimization
(PSO),
• Firefly Algorithm (FA),
• Ant Colony Optimization (ACO),
• Artificial Bee Colony (ABC),
• Cuckoo Search (CS),
• Artificial Weed Optimization (IWO),
• Bacterial Foraging
Optimization(BFO),
• Estimation of Distribution Alg. (EDA),
• Harmony Search (HS),
• Gravitational Search Algorithm
(GSA),
• Biogeography-based
Optimization(BBO),
• Differential Evolution (DE)
and its variants (jDE, L-SHADE, DISH),
• ... and many more, including
hybrids.

Range of Applications of the Optimization Algorithms
• Meta-heuristics algorithms, applicable to:
• (architectural) morphology (re)construction
(vivo/technical),
• artificial life:
• modeling ecosystem and environmental living conditions,
• e.g.: (automatic) procedural tree modeling,
interactive ecosystem breeding.
• pattern recognition, image processing, computer vision,
• language/documents understanding, speech processing,
• robotics, bioinformatics, chemical engineering,
manufacturing,
• oil search, nuclear plant safety, finance, electrical
engineering,
• energy, big data, data mining, security, ocean/space
research,
• systems of systems, ..., artificial intelligence.

Differential Evolution (DE)
• A ﬂoating point encoding EA for global optimization over
continuous spaces,
• through generations,
the evolution process improves population of vectors,
• iteratively by combining a parent individual and
several other individuals of the same population,
using evolutionary operators.
• We choose the strategy jDE/rand/1/bin
• mutation: vi,G+1 = xr1,G + F × (xr2,G − xr3,G),
• crossover:
ui,j,G+1 =
(
vi,j,G+1 if rand(0, 1) ≤ CR or j = jrand
xi,j,G otherwise
,
• selection: xi,G+1 =
(
ui,G+1 if f(ui,G+1) < f(xi,G)
xi,G otherwise
,

Algorithm DE
1: algorithm canonical algorithm DE/rand/1/bin (Storn,
1997)
Require: f(x) – fitness function; D, NP, G – DE control parameters
Ensure: xbest – includes optimized parameters for the fitness function
2: Uniform randomly initialize the population (xi,0, i = 1..NP);
3: for DE generation loop g (until g < G) do
4: for DE iteration loop i (for all vectors xi,g in current population) do
5: DE trial vector computation xi,g (mutation, crossover):
6: vi,g+1 = xr1,g + F × (xr2,g − xr3,g);
7: ui,j,g+1 =
(
vi,j,g+1 if rand(0, 1) ≤ CR or j = jrand
xi,j,g otherwise
;
8: DE selection using fitness evaluation f(ui,G+1):
9: xi,g+1 =
(
ui,g+1 if f(ui,g+1) < f(xi,g)
xi,g otherwise
;
10: end for
11: end for
12: return best obtained vector (xbest);

Control Parameters Self-Adaptation
• Through more suitable values of control parameters the
search process exhibits a better convergence,
• therefore the search converges faster to better solutions,
which survive with greater probability and they create
more offspring and propagate their control parameters
• Recent study with cca. 10 million runs of SPSRDEMMS:
A. Zamuda, J. Brest. Self-adaptive control parameters’
randomization frequency and propagations in differential
evolution. Swarm and Evolutionary Computation, 2015,
vol. 25C, pp. 72-99.
DOI 10.1016/j.swevo.2015.10.007.
– SWEVO 2015 RAMONA / SNIP 5.220

Self-adaptive control parameters’ randomization
frequency and propagations in differential evolution
– Overview
• Randomization frequency
inﬂuences performance
(SPSRDEMMS on right)
• Suggesting values for
different problems
• 0.1 to 0.8 for τF,
0.05 to 0.25 for τCR
• Empirical insight into
operation of the
randomization mechanism

Listing Some More DE-Family Algorithms Proposed
• My algorithms (CEC – world championships on EAs):
• SA-DE (CEC 2005: SO) – book chapter JCR,
• MOjDE (CEC 2007: MO) – vs. DEMO 40/57 IR, 39/57 IH,
• DEMOwSA (CEC 2007: MO) – rank #3, 53 citations,
• DEwSAcc (CEC 2008: LSGO) – 63 citations,
• DEMOwSA-SQP (CEC 2009: CMO) – rank #2, 47 citations,
• DECMOSA-SQP (CEC 2009: CMO) – rank #3 at 2 functions,
• jDENP,MM (CEC 2011: RWIC) – LNCS SIDE 2012,
• SPSRDEMMS (CEC 2013: RPSOO); Large-scale @SWEVO.
• DISH (SWEVO 2019) – best CEC 2015 & 2017 results.
• Performance assessment of the algorithms at world EA
championships: several times best on some criteria
(also won CEC 2009 dynamic optimization competition).
• Performance assessment on several industry challenges
• procedural tree models reconstruction (ASOC 2011, INS
2013),
• underwater glider path planning (ASOC 2014),
• hydro-thermal energy scheduling (APEN 2015),
• RWIC (Real World Industry Challenges) - CEC 2011; 2013, ...

SPSRDEMMS: Example of Optimization Mechanisms
• SPSRDEMMS = Structured Population Size Reduction
Differential Evolution with Multiple Mutation Strategies
• canonical DE, upgraded with: mechanism of F and CR
control parameters self-adaptation, mutation strategy
ensembles, population structuring (distributed islands),
and population size reduction.
• is an extension of the jDENP,MM variant (Zamuda and
Brest, SIDE 2012) and was published at CEC 2013
(competition).
• The SPSRDEMMS, for a ﬁxed part of the population (NPbest
number of individuals at end of the entire population),
executes only the best/1 strategy.
• This part of population (which might be seen as a
sub-population) has a separate best vector index, xbest bestpop.
• The ﬁrst part of the population (mainpop) operates on target
vectors xi ∈ {x1...xNP−NPbest} and second part (bestpop)
operates on target vectors xi = {xNP−NPbest+1...xNP}.
• Both strategies generate mutation vectors using all vectors of
the population x1...xNP, i.e. r1, r2, r3 ∈ {1..NP}.

– Methods
• G. Karafotias, M. Hoogendoorn, A. Eiben,
Parameter control in evolutionary algorithms:
trends and challenges, IEEE Trans. Evolut.
Comput. 19 (2) (2015) 167–187.
• A. Zamuda, J. Brest, E. Mezura-Montes,
Structured population size reduction
differential evolution with multiple mutation
strategies on CEC 2013 real parameter
optimization, in: Proceedings of the 2013 IEEE
Congress on Evolutionary Computation (CEC),
vol. 1, 2013, pp. 1925–1931.
• J. Brest, S. Greiner, B. Bošković, M. Mernik, V.
Žumer, Self-adapting control parameters in
differential evolution: a comparative study on
numerical benchmark problems, IEEE Trans.
Evolut. Comput. 10 (6) (2006) 646–657.
• Parameter control study
• Systematic approach to
answering questions about the
control parameters
mechanism
• For certain interesting
functions, deeper insight is
shown

Other Enhancements / Improvements / Mechanisms
in DE
DE, jDE, SaDE, ODE, DEGL, JADE, EPSDE; ϵ-DE, DDE, CDE; PDE,
GDE, DEMO, MOEA/D, ...
• Swagatam Das and Ponnuthurai Nagaratnam Suganthan.
”Differential evolution: a survey of the
state-of-the-art.” IEEE Transactions on Evolutionary
Computation 15(1), 2011: 4-31. DOI:
10.1109/TEVC.2010.2059031.
CoDE, Compact DE, L-SHADE, Binary DE,
Successful-Parent-Selecting Framework DE, ...
• Swagatam Das, Sankha Subhra Mullick, and Ponnuthurai
Nagaratnam Suganthan.
”Recent Advances in Differential Evolution –
An Updated Survey.”
Swarm and Evolutionary Computation, Volume 27, April
2016, Pages 1-30, 2016.
DOI: 10.1016/j.swevo.2016.01.004.
Several hybridizations, improvements, and general
mechanisms.

11th
Transformers
—
I.b Background: Summary
Evaluation
—

Summary Evaluation — DUC Corpora
• The existing DUC collection of English newswire documents from the Document
Understanding Conferences (DUC) ﬁts very well for experimenting within the
text summarization scenarios1.
• On the one hand, it provides pairs of documents-summary or cluster-summary
for diverse summarization types (extractive, single-document, multi-document,
etc.), and, on the other hand, there is a wide number of previous summarization
systems to be compared with, and determine to what extent our approach is
effective. In particular, the DUC 2002 dataset, contains 567 documents grouped
in 59 clusters (denoted as d061 to d120), where each cluster represents a set of
topic-related documents (the average number of documents per cluster is 10).
• Besides this dataset, a dataset, i.e., CNN/DailyMail has also been made
available for the research community2. This dataset contains more than
300,000 documents, and it provides summaries of about 50 words.
1
Aleš Zamuda and Elena Lloret. “Optimizing Data-Driven Models for Summarization as Parallel Tasks”. In:
Journal of Computational Science 42 (2020), p. 101101.
2
Karl Moritz Hermann et al. “Teaching Machines to Read and Comprehend”. In: Proceedings of the 28th
International Conference on Neural Information Processing Systems - Volume 1. NIPS’15. Montreal, Canada: MIT
Press, 2015, pp. 1693–1701.

Summary Evaluation — ROUGE Metric
• Concerning the evaluation of summaries, ROUGE3
is one of the common standard, and most used tools.
• The idea behind ROUGE is that, if two texts have a similar meaning, they must also share similar
words or phrases. As a consequence, it relies on n-gram co-occurrence, and the idea behind it is
to compare the content of a peer summary with one or more model summaries, and compute the
number of n-gram of words they all have in common.
• Different types of n-grams can be obtained, such as unigrams (ROUGE-1), bigrams (ROUGE-2), the
longest common subsequence (ROUGE-L), or bigrams with a maximum distance of four words
in-between (ROUGE-SU4), and, based on them, values for recall, precision and F-measure can be
obtained, thus determining the summary accuracy in terms of content (the higher recall, precision
and F-measure values, better).
• Among all the metrices, recall values are then usually reported for peer evaluation of
generated model summaries.
• Other metrices also exist, like AutoSummENG4
which is an automatic character n-gram based evaluation
method with high correlation with human judgments, or SummTriver5
, which does not need to have
human summaries for the evaluation; however, they are not often used by the research community, thus
it is difﬁcult to ﬁnd results for comparison purposes.
• Other ways to evaluate the summaries would be to consider standard similarity metrices, such as
Simmetric (https://github.com/Simmetrics/simmetrics).
3
Chin-Yew Lin. “ROUGE: A package for automatic evaluation of summaries”. In: Proceedings of the ACL-04
Workshop on Text Summarization Branches Out. Barcelona, Spain, 2004, pp. 74–81.
4
George Giannakopoulos et al. “Summarization System Evaluation Revisited: N-gram Graphs”. In: ACM Trans.
Speech Lang. Process. 5.3 (2008), 5:1–5:39.
5
Luis Adrián Cabrera-Diego and Juan-Manuel Torres-Moreno. “SummTriver: A new trivergent model to
evaluate summaries automatically without human references”. In: Data Knowl. Eng. 113 (2018), pp. 184–197.

Summary Evaluation — BERTScore Metric
• In NLP, sentence embeddings relate to methodologies that encapsulate the semantic essence of complete
sentences, portraying them as condensed numerical vectors. This approach enables a diverse array of
subsequent tasks, including but not limited to sentence similarity assessment, paraphrase detection, and
text classiﬁcation.
• The embeddings derived from BERT capture contextual information, as it computes token similarity using
contextual embeddings.
• Multilingual BERT (mBERT)6
in text summarization proves to be a transformative approach for enhancing
the efﬁciency and accuracy of the summarization process across diverse languages.
• As an extension of the original BERT model, mBERT enables the understanding of the meaning and
context of words and sentences across multiple languages, having been trained on an extensive corpus
with over 100 different languages.
• Work with mBERT has been applied also in7
, embedding source sentences and translations into a shared
vector space for machine translation.
• BERTScore8
has been therefore proposed as an evaluation metric derived from BERT models. A
comprehensive literature review on summarization evaluation methods and metrices can be found in9
.
6
Telmo Pires, Eva Schlinger, and Dan Garrette. “How Multilingual is Multilingual BERT?”. In: Proceedings of
the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational
Linguistics, 2019, pp. 4996–5001.
7
Jani Dugonik et al. “Reduction of Neural Machine Translation Failures by Incorporating Statistical Machine
Translation”. In: Mathematics 11.11 (2023), pp. 1–22.
8
Tianyi Zhang et al. “BERTScore: Evaluating Text Generation with BERT”. In: Proceedings of the ICLR 2020.
2020.
9
Cabrera-Diego and Torres-Moreno, “SummTriver: A new trivergent model to evaluate summaries
automatically without human references”.

11th
Transformers
—
III. Methods
—

Methods
• To compare the values from both metrices
ROUGE and BERTScore, we use the initial
twelve document instances d061j to d072f
from DUC 2002 corpus to execute the runs of
summarization.
• The summarization system employed was an
adapted version of CaBiSDETS
(Constraint-adjusting Binary Self-adaptive
Differential Evolution for Data-Driven Models
Extractive)a, which instead of preprocessing
to concepts used just splits of sentences with
lematization, but still used sum of cosine
similarities to evolve fitness.
• During the run, the summarization system
improves current fitness and for viable
summaries, their assessments are reported
as (Fitness).
a
Zamuda and Lloret, “Optimizing Data-Driven Models for
Summarization as Parallel Tasks”.
• A Fitness is a fitness value of the
represented summary which the
optimization algorithm extracted and is
scoring it using its fitness function
during the optimization.
• For ROUGE and BERTScore metrices, that
evolved summary (peer) is then
compared to one of the two model
summaries (model1 or model2) being
provided by the DUC corpus of
corresponding summaries by humans
who wrote the model summaries.
• The BERTScore metric is configured using
the code snippet seen on next page,
where a base model from BERT is used
for scoring, while a multilingual BERT
model is used for the remaining values.
• Then, the evaluation through contextual
embeddings is also computed, where for
a generated evolved summary in text file
text0, two human summaries are used
as input text files text1 and text2 to
print the BERTScore metric values.

Methods: Computational Coding
1 — Conﬁguring the BERT models:
model name = ’bert−base−multilingual−uncased’
tokenizer = BertTokenizer.from pretrained(model name)
model = BertModel.from pretrained(model name)
scorer = BERTScorer(model type=’bert−base−uncased’)
2 — Scoring summary ﬁles using BERTScore and human input:
inputs0 = tokenizer(text0, return tensors=’pt’, padding=True, truncation=True)
outputs0 = model(**inputs0)
embeddings0 = outputs0.last hidden state.mean(dim=1).detach().numpy()
# BERT: metric 0 to 1
similarity = np.dot(embeddings0, embeddings1.T) / (np.linalg.norm(embeddings0)
* np.linalg.norm(embeddings1))
P, R, F1 = scorer.score([text0], [text1])
sim, P, R, F1 = similarity[0][0], P.mean(), R.mean(), F1.mean()
print(f’BERTScore sim/P/R/F1@1: {sim:.4f} {P:.4f} {R:.4f} {F1:.4f} ’, end=’ ’)
# BERT: metric 0 to 2
similarity = np.dot(embeddings0, embeddings2.T) / (np.linalg.norm(embeddings0)
* np.linalg.norm(embeddings2))
P, R, F1 = scorer.score([text0], [text2])
sim, P, R, F1 = similarity[0][0], P.mean(), R.mean(), F1.mean()
print(f’sim/P/R/F1@2: {sim:.4f} {P:.4f} {R:.4f} {F1:.4f} ’)

11th
Transformers
—
IV. Results
—

Results — Experimental Setup
• The text summarization metrices shown are
Fitness, ROUGE, and BERTScore (BERT).
• The Fitness metric is the evaluation of the generated evolved summary
which the optimization algorithm extracted
and is scoring it using its ﬁtness function during the optimization.
• The values of Fitness are plotted by normalizing the values between 0 and 1 to plot.
• For ROUGE and BERTScore,
an observed evolved summary (peer) is compared to
one of the two model summaries (model1 or model2)
being provided by the DUC corpus of
corresponding summaries
by a human who wrote a corresponding model summary.
• The ROUGE metric values reported are the ROUGE 1, ROUGE 2, ROUGE 4,
and ROUGE SU.
• The BERTScore values reported are the similarity score (sim) or
BERTScore precision (P), recall (R), and F1 value (F1), respectively.

Results — Plots for Documents 1–6
• Evaluations for documents from DUC corpus, for documents 1–6.
• The figures (a) to (f) plot the text summarization metrices (normalized value of Fitness, then values of
ROUGE 1/2/4/SU, respectively, and finally the individual comparisons of values from BERTScore) on
documents d061j to d066j.
• Their value scales are represented on the vertical axis, while on the horizontal axis the current sequential
generation number (g) through evolution are shown.
• Only the feasible values and where there is an improvement in the fitness are plotted.
• The values from BERTScore are in eight combinations, where each time, the currently generated evolved
summary text (peer) is compared to one of the two model summaries (model1 or model2), reported
though the similarity score (sim) or BERTScore precision (P), recall (R), and F1 value (F1), respectively.
Document 1 (d061).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 2 (d062).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 3 (d063).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 4 (d064).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 100 200 300 400 500 600 700
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 5 (d065).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000 7000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 6 (d066).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R

Results — Plots for Documents 7–12
• Evaluations for documents from DUC corpus, for documents 7–12.
• The figures (a) to (f) plot the text summarization metrices (normalized value of Fitness, then values of
ROUGE 1/2/4/SU, respectively, and finally the individual comparisons of values from BERTScore) on
documents d067f to d072f.
• Their value scales are represented on the vertical axis, while on the horizontal axis the current sequential
generation number (g) through evolution are shown.
• Only the feasible values and where there is an improvement in the fitness are plotted.
• The values from BERTScore are in eight combinations, where each time, the currently generated evolved
summary text (peer) is compared to one of the two model summaries (model1 or model2), reported
though the similarity score (sim) or BERTScore precision (P), recall (R), and F1 value (F1), respectively.
Document 7 (d067f).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 8 (d068f).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000 7000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 9 (d069f).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
Document 10 (d070f).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 500 1000 1500 2000 2500 3000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40 60 80 100 120 140 160 180
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R

Results — Discussion
• Fitness improves by increasing in its
value,
• with each jump the assessment metric
values from ROUGE and BERTScore also
change,
• hence we can deduce from these
observations that the metrices do
relate a change in response to a
different text summary.
• Also, while the curves of ROUGE values
are seen as all lower than numbers from
BERTScore, the curves from values of
BERTScore similarity score (sim) are
highest,
• hence from the experiment, we
can expect to obtain values of
BERTScore being higher as those
from ROUGE.
• Also, the matching values from
BERTScore in evaluation on the two
corresponding human summaries are
closely correlated, i.e.,
• the generated extractive summary
always scored closely related on
both corresponding human
models:
• merely in the case of d068f, the points
in lines at generations g = 1000 and
g = 3000 interweave slightly.
• From these observations on our
summarization system, the main
takeaways are that
• the metric BERTScore provided
higher values than ROUGE and
that
• both metrices evaluated a
summary closely responsive to a
corresponding human model
summary.

11th
Transformers
—
IV: Conclusion Summary
with Takeaways
—

Conclusion
• This paper presented: a comparison of a set of two
evaluation metrices for a summarization system,
• where a summarization system based on algorithm
CaBiSDETS was applied.
A summarization algorithm was executed on DUC
documents and then evaluated on two metrices,
• ROUGE that was the original metric from DUC, and
• of the more recent BERTScore using BERT from Google AI.
The observed results have shown close correlation on
same matching summarization texts.
• In future work:
• research on further use of BERTScore and similar LLMs
metrics on DUC corpora summarizers could be conducted,
but also
• improving and analysing summarization algorithms,
• as well as developing new metrices for text summarization
and understanding, or
• deployment of trained ML systems to different modalities
like video and animation.

Summary Conclusion with Takeaways
Conclusion: BERT vs. ROUGE in CaBiSDETS — plots.
Takeaways: text summarization; EAs; HPC; explainable AI
Thanks!
Acknowledgement: this work is supported by DAPHNE, funded by the European Union’s Horizon 2020
research and innovation programme under grant agreement No 957407.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R
10000
1x106
1x108
1x1010
1x1012
0 1 2 3 4 5 6 7 8 9
FES
Digits
f1
f2
f3
f4
f5
f6
f7
f8
f9
f10
-50
-40
-30
-20
-10
0
10
0 50 100 150 200 250 300
Fitness, run 1
Fitness, run 2
Fitness, run 3
Fitness, run 4
Fitness, run 5
Fitness, run 6
Fitness, run 7
Fitness, run 8
Fitness, run 9
0 1 2 3 4 5 6 7 8 9
Combined power from 110 MW to 975 MW, step 0.01 MW#104
0
100
200
300
400
500
600
700
Individual
output
(power
[MW]
or
unit
total
cost
[$])
Cost, TC / 3
Powerplant P1 power
Powerplant P2 power
Powerplant P3 power
150
200
250
300
350
400
450
500
550
16 32 48 64 80
Seconds
to
compute
a
workload
Summarizer workload
Real examples: science and HPC
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Fitness
ROUGE 1,
ROUGE 2,
ROUGE 4,
ROUGE SU
BERT(peer,model1) P
BERT(peer,model1) R
BERT(peer,model2) P
BERT(peer,model2) R

11th
Transformers
—
Final Slide: Questions, Misc
Acknowledgement: this work is supported by EU project no. 957407 (DAPHNE).
—

Method Results AI Challenges Shortlist HPC Initiatives EuroHPC Vega &,Deploying DAPHNE HPC & GenAI Language Video Machine Power Opportunities References
11th
Transformers
—
APPENDIX with Backgrounds
& Marketing Materials
—

11th
Transformers
—
I. Background: Optimization,
100-Digit Challenge
—