A Stream-Based Approach to Intrusion Detection

S. Hallé
Sylvain Hallé
Université du Québec à Chicoutimi
CANADA
A Stream-Based
Approach to Intrusion
Detection
CRSNG
NSERC
CHAPTER 9
November 26th, 2023

S. Hallé
$
$$$$$
$$$$$
$$$$$
update of a stock price
sensor reading
link
state of a video game
user visi�ng a web page
parcel being delivered
/
move in a chess game
Events
The expected behavior of an information system
can be evaluated through the observation and
analysis of events.

S. Hallé
A stream is an ordered sequence of events of a
given type.
Streams
...
...
Formally, if Σ is a set of events, a stream is a
sequence σ ∈ Σ*.
A stream processing calculation is a function:
π : (Σ1* × ... ×Σm*) → (Σ1* × ... ×Σn*)

Z
Z
Z
S. Hallé
expected
normal
rou�ne
nominal
"ok"
surprising
out of the ordinary
strange
diﬀerent
"not ok"
Anomaly / intrusion detection

S. Hallé
Approaches
A taxonomy of existing approaches classiﬁes
detection systems into three broad categories:
Anomaly-based approaches: detect uncommon
events, outliers, etc.
Multi-agent-based approaches: cooperating
units observe and interact with an
environment
Knowledge-based approaches: a priori
information about expected behavior is
compared with observations
1.
2.
3.

S. Hallé
Dealing with volume
Common issue with many approaches: high volume
of alarms and related information

S. Hallé
Dealing with volume
Two possible ways to reduce volume:
1. Decrease false positive rate
2. Identify relevant information about the
occurrence of an alarm
1
2
3
4
5
1 2 3 4 5
...
+
+
+
+

S. Hallé
Dealing with volume
Two possible ways to reduce volume:
1. Decrease false positive rate
2. Identify relevant information about the
occurrence of an alarm
1
2
3
4
5
1 2 3 4 5
...
+
+
+
+
this
chapter

S. Hallé
Enter RV
Proposed approach: intrusion/attack detection as a
form of Runtime Veriﬁcation

S. Hallé
Enter RV
System

S. Hallé
Enter RV
System
Instrumentation

S. Hallé
Enter RV
System
Trace
Events
Instrumentation

S. Hallé
Enter RV
System
Monitor
Trace
Events
Instrumentation

S. Hallé
Enter RV
System
Monitor
Trace
Events
Instrumentation
Formal
spec.

S. Hallé
Pattern monitor
The monitor is a process that updates a 3-valued
verdict:
⊤ if the input stream
contains the pattern
⊥ if the input stream can
never* contain the pattern
? otherwise
Example: "a eventually followed by b, then c"
c a b a b a c b c
? ? ? ? ? ? ⊤ ⊤
⊤

S. Hallé
Shortcomings
c a b a b a c b c
? ? ? ? ? ? ⊤ ⊤
⊤

S. Hallé
Shortcomings
c a b a b a c b c
? ? ? ? ? ? ⊤ ⊤
⊤
A monitor "stops" at the ﬁrst occurrence of the
pattern (i.e. cannot detect multiple instances)
found!
!

S. Hallé
Shortcomings
c a b a b a c b c
? ? ? ? ? ? ⊤ ⊤
⊤
A monitor "stops" at the ﬁrst occurrence of the
pattern (i.e. cannot detect multiple instances)
found!
!
Limited feedback: location of last element of the
pattern + everything that precedes
!
all this may be relevant

S. Hallé
Solution
Run one instance of on every suﬀix of the
input

S. Hallé
Solution
input
c a b a b a c b c

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤
2 ? ? ? ? ? ⊤ ⊤
⊤

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤
2 ? ? ? ? ? ⊤ ⊤
⊤
3 ? ? ? ? ⊤ ⊤
⊤

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤
2 ? ? ? ? ? ⊤ ⊤
⊤
3 ? ? ? ? ⊤ ⊤
⊤
4 ? ? ? ⊤ ⊤
⊤

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤
2 ? ? ? ? ? ⊤ ⊤
⊤
3 ? ? ? ? ⊤ ⊤
⊤
4 ? ? ? ⊤ ⊤
⊤
5 ? ? ⊤
? ?

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤
2 ? ? ? ? ? ⊤ ⊤
⊤
3 ? ? ? ? ⊤ ⊤
⊤
4 ? ? ? ⊤ ⊤
⊤
5 ? ? ⊤
? ?
6
7
8
9
? ⊤
? ?
? ? ?
? ?
?

S. Hallé
Solution
input
c a b a b a c b c
1 ? ? ? ? ? ? ⊤ ⊤
⊤
2 ? ? ? ? ? ⊤ ⊤
⊤
3 ? ? ? ? ⊤ ⊤
⊤
4 ? ? ? ⊤ ⊤
⊤
5 ? ? ⊤
? ?
6
7
8
9
? ⊤
? ?
? ? ?
? ?
?
Multiple matches for
what is essentially
the "same" pattern

S. Hallé
Monitor state
Access to the monitor's internal state opens the
door to pruning tactics.
*
*
*
*
a b c
? ? ? ⊤

S. Hallé
First step
Pruning tactic #1
Discard monitors that
remain in their initial state
after consuming the ﬁrst
event
c a b a b a c b c
1
*
*
*
*
a b c
? ? ? ⊤
? ? ? ? ? ? ⊤ ⊤
⊤

S. Hallé
First step
Pruning tactic #1
event
c a b a b a c b c
1
*
*
*
*
a b c
? ? ? ⊤

S. Hallé
First step
Pruning tactic #1
event
c a b a b a c b c
1
*
*
*
*
a b c
? ? ? ⊤
Intuition: if the monitor does not change state
after reading σ, then a stream σ is a match if
and only if σ · σ is a match

S. Hallé
Unique state
Pruning tactic #2
Discard monitors that reach
the same state at the same
step
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤
4
? ? ? ? ? ⊤ ⊤
⊤
? ? ? ⊤ ⊤
⊤

S. Hallé
Unique state
Pruning tactic #2
step
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤
4

S. Hallé
Unique state
Pruning tactic #2
step
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤
4
Intuition: if monitor i is in state s after reading
σ · σ', and monitor j is in state s after reading
σ', then σ' is a match iﬀ σ · σ' is a match

S. Hallé
Progressing subsequence
Pruning tactic #3
Retain only the progressing
sub-sequence of the input
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤
⇒ all loops in the state space are removed
? ? ? ? ? ⊤ ⊤
⊤

S. Hallé
Pruning tactic #3
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤

S. Hallé
Pruning tactic #3
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤
*
?

S. Hallé
Pruning tactic #3
c a b a b a c b c
2
*
*
*
*
a b c
? ? ? ⊤
Intuition: if σ is a match, the progressing sub-
sequence is the shortest subset of σ visiting
each new state in the same order

S. Hallé
Comparison
c a b a b a c b c
1
2
3
4
5
6
7
8
9
? ? ? ? ? ? ⊤ ⊤
⊤
? ? ? ? ? ⊤ ⊤
⊤
? ? ? ? ⊤ ⊤
⊤
? ? ? ⊤ ⊤
⊤
? ? ⊤
? ?
? ⊤
? ?
? ? ?
? ?
?

S. Hallé
Comparison
c a b a b a c b c
1
2
3
4
5
6
7
8
9

S. Hallé
Comparison
c a b a b a c b c
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
ﬁrst-step

S. Hallé
Comparison
c a b a b a c b c
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
ﬁrst-step same state
+

S. Hallé
Comparison
c a b a b a c b c
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
ﬁrst-step same state progressing
+ +

S. Hallé
Pros and cons
Pros
Expects to be a (ﬁnite) state machine
Limiting in terms of possible patterns that can be
expressed
Cons
Pattern to look for can be any monitor
Drastically reduces number of matches (and events
retained in each match)
Produces results that correspond to intuition

S. Hallé
Pros and cons
Pros
Expects to be a (ﬁnite) state machine
Limiting in terms of possible patterns that can be
expressed
Cons
Pattern to look for can be any monitor
Drastically reduces number of matches (and events
retained in each match)
Produces results that correspond to intuition
does it?

S. Hallé
Monitor state (revisited)
A "state" can be deduced even for processes that
are not expressed as state machines.
Given a monitor 𝜋 : Σ* → Σ'*, a function 𝜄𝜋 : Σ* → 𝑆 is
said to be a state function if, for every 𝜎1, 𝜎2 ∈ Σ*:
𝜄𝜋(𝜎1) = 𝜄𝜋(𝜎2) implies 𝜋(𝜎1 · 𝜎') = 𝜋(𝜎2 · 𝜎')
for every 𝜎' ∈ Σ*.

S. Hallé
Monitor state (revisited)
A "state" can be deduced even for processes that
are not expressed as state machines.
Given a monitor 𝜋 : Σ* → Σ'*, a function 𝜄𝜋 : Σ* → 𝑆 is
said to be a state function if, for every 𝜎1, 𝜎2 ∈ Σ*:
𝜄𝜋(𝜎1) = 𝜄𝜋(𝜎2) implies 𝜋(𝜎1 · 𝜎') = 𝜋(𝜎2 · 𝜎')
for every 𝜎' ∈ Σ*.
Intuition: two instances of 𝜋 are in the "same
state" if they have identical behavior for any
future events they read

S. Hallé
Introducing
BeepBeep is an open source library written in Java
and developed at UQAC since 2015.
Centered on the notion of processor, a computation
unit that ingests input events and produces output
events:
h�ps://liﬂab.github.io/beepbeep-3
P
processor
input pipe
event
output pipe

S. Hallé
Introducing
Complex calculations can be achieved through
composition (i.e. passing the output of a processor
to the input of another).
Doing so creates processor chains or "pipelines":

S. Hallé
,φ
ψ
f
π
⊻
⊻
n
{
π
π
Sequence Eventual
disjunction
Eventual
conjunction
Eventual
occurrence
Existential
window
Existential
slice
Monitors in
BeepBeep has been extended with a set of
processors acting as monitors.
Recognize an
elementary pattern of
events
Provide a state
function
Can identify their
progressing
sub-sequence

S. Hallé
Monitors in
,φ
ψ
,φ
ψ
f
B
1 =?
f
C
1 =?
f
A
1 =?
2
1
2
3
4
6
7
5
"a eventually followed by b, then c"
Complex monitors can be obtained by freely
composing these processors

S. Hallé
Explainability in
BeepBeep allows processors to deﬁne associations
between output events and input events —a feature
called explainability.*
P
1 P
2
S. Hallé. (2020). Explainable Queries over Event Logs. Proc. EDOC 2020.
IEEE Computer Society. DOI: 10.1109/EDOC49727.2020.00029
*
Monitors associate their verdict to the progressing
sub-sequence of their input.

S. Hallé
Monitors in
,φ
ψ
,φ
ψ
f
B
1 =?
f
C
1 =?
f
A
1 =?
2
1
2
3
4
6
7
5
"a eventually followed by b, then c"
The progressing sub-sequence of each processor
can be propagated upstream
AbABbC
AbABbC
AbABbC
AbABbC
⊤⊤⊤⊤⊤⊤
???⊤⊤⊤
?????⊤
???⊤⊤⊤
AabcAaBabbCa
?????⊤

S. Hallé
Usage
Sample script in Java:
CountDecimate d =
new CountDecimate(2);
Fork f = new Fork(3);
SomeEventually a = new SomeEventually(
new ApplyFunction(new Equals("a")));
SomeEventually b = new SomeEventually(
new ApplyFunction(new Equals("b")));
SomeEventually c = new SomeEventually(
new ApplyFunction(new Equals("c")));
Sequence s1 = new Sequence();
Sequence s2 = new Sequence();
EventTracker t = new IndexEventTracker();
Connector con = new Connector(t);
con.connect(d, 0, f, 0).connect(f, 0, a, 0)
.connect(f, 1, b, 0).connect(f, 2, c, 0)
.connect(a, 0, s1, 0).connect(b, 0, s1, 1)
.connect(s1, 0, s2, 0).connect(c, 0, s2, 1);
,φ
ψ
,φ
ψ
f
B
1 =?
f
C
1 =?
f
A
1 =?
2
1
2
3
4
6
7
5

S. Hallé
Usage
FindPattern fp = new FindPattern(g);
Connector.connect(fp, new Print());
for (char e : "AabcAaBabbCa".toCharArray())
fp.getPushableInput().push(e);
,φ
ψ
,φ
ψ
f
B
1 =?
f
C
1 =?
f
A
1 =?
2
1
2
3
4
6
7
5
Detecting the pattern and producing
the witness:
Produces the output:
{(1,1), (1,7), (1,11)}

S. Hallé
Examples
Pattern monitors have been implemented to detect
"distilled" versions of real-world attacks and
intrusions

S. Hallé
Examples
intrusions
Linear sequence
Declare a match when n
successive events are
observed
ptrace exploit

S. Hallé
,φ
ψ
,φ
ψ
f
B
1 =?
f
C
1 =?
f
A
1 =?
2
1
2
3
4
6
7
5
Linear sequence

S. Hallé
Examples
intrusions
Linear sequence
observed
Combined
other patterns have been
observed
GandCrab attack
ptrace exploit

S. Hallé
f
B
1 =?
f
A
1 =?
⊻
,φ
ψ
f
D
1 =?
f
C
1 =?
,φ
ψ
f
F
1 =?
f
E
1 =?
,φ
ψ
⊻
Combined

S. Hallé
Examples
intrusions
Incomplete
Declare a match when more
than n instances of a sequential
pattern are incomplete
SYN ﬂooding
Linear sequence
observed
Combined
observed
GandCrab attack
ptrace exploit

S. Hallé
f
A
=?
,φ
ψ
f
f
=?
1 ?
0
1 f
B
B
=?
B
A
1
f
>
1 k
4
↑
Σ
⊻
?
2
3
5
Incomplete

S. Hallé
Examples
intrusions
Incomplete
than n instances of a sequential
pattern are incomplete
Threshold
than n instances of a pattern
have been observed
Remote Access Trojan
SYN ﬂooding
Linear sequence
observed
Combined
observed
GandCrab attack
ptrace exploit

S. Hallé
A
f f
#
>
1
k
Σ
B
{ }
∪
1
2 3 4 5
∨
Σ
Threshold

S. Hallé
Results
Do the pruning strategies reduce the number of
matches?

S. Hallé
Results
matches?
Linear sequence Combined Incomplete Threshold
0
50
100
150
200
250
300
None
First step
Distinct states
Progressing
Matches

S. Hallé
Results
events included as witnesses?

S. Hallé
Results
events included as witnesses?
0
0.2
0.4
0.6
0.8
1
1.2
None
First step
Distinct states
Progressing
Fraction
of
events
included

S. Hallé
Results
What is the impact of the pruning strategies on
processing time?

S. Hallé
Results
What is the impact of the pruning strategies on
processing time?
0
200
400
600
800
1000
1200
1400
None
First step
Distinct states
Progressing
Time
(ms)

S. Hallé
Take-home points
Attack/intrusion patterns can be formalized as
runtime monitors
Stream processing pipelines allow the expression of
complex monitors
Reducing the volume of data produced by pattern
matches can be done using simple state-based
pruning strategies on these monitors
A proof-of-concept implementation of these notions
in a stream processing engine shows that intuitive
incident reports can be produced automatically

S. Hallé
For more information
https://liﬂab.ca
https://liﬂab.github.io/beepbeep-3

A Stream-Based Approach to Intrusion Detection

More Related Content

Similar to A Stream-Based Approach to Intrusion Detection (20)

More from Sylvain Hallé (20)

Recently uploaded (20)

A Stream-Based Approach to Intrusion Detection