Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt

© 2017 VMware Inc. All rights reserved.
Understanding SCHED_DEADLINE
Controlling CPU Bandwidth
Steven Rostedt
26/9/2017

What is SCHED_DEADLINE?
●
A new scheduling class (Well since v3.14)
– others are: SCHED_OTHER/NORMAL, SCHED_FIFO, SCHED_RR
●
SCHED_IDLE, SCHED_BATCH (out of scope for today)
●
Constant Bandwidth Scheduler
●
Earliest Deadline First

Other Schedulers
●
SCHED_OTHER / SCHED_NORMAL
– Completely Fair Scheduler (CFS)
– Uses “nice” priority
– Each task gets a fair share of the CPU bandwidth
●
SCHED_FIFO
– First in, first out
– Each task runs till it gives up the CPU or a higher priority task preempts it
●
SCHED_RR
– Like SCHED_FIFO but same prio tasks get slices of CPU

Priorities
●
You have two programs running on the same CPU
– One runs a nuclear power plant
●
Requires 1/2 second out of every second of the CPU (50% of the CPU)
– The other runs a washing machine
●
Requires 50 millisecond out of every 200 milliseconds (25% of the CPU)
– Which one gets the higher priority?

Priorities
½ sec
½ sec
1 sec
1 sec

Priorities Nuke > Washing Machine
½ sec
½ sec
1 sec
1 sec

Priorities Nuke < Washing Machine
½ sec
½ sec
1 sec
1 sec

Rate Monotonic Scheduling (RMS)
●
Computational time vs Period
●
Can be implemented by SCHED_FIFO
●
Smallest period gets highest priority
●
Compute computation time (C)
●
Compute period time (T)
U =∑
i=1
n
Ci
Ti

●
Add a Dishwasher to the mix...
●
Nuclear Power Plant : C = 500ms T=1000ms (50% of the CPU)
●
Dishwasher: C = 300ms T = 900ms (33.3333% of the CPU)
●
Washing Machine: C = 100ms T = 800ms (12.5% of the CPU)
U =
500
1000
+
300
900
+
100
800
=.958333

0 1 12111098765432 13

0 1 12111098765432 13
FAILED!

●
Computational time vs Period
●
Can be implemented by SCHED_FIFO
●
Smallest period gets highest priority
●
Compute computation time (C)
●
Compute period time (T)
U =∑
i=1
n
Ci
Ti
≤n(
n
√2−1)

●
Add a Dishwasher to the mix...
●
Nuclear Power Plant : C = 500ms T=1000ms (50% of the CPU)
●
Dishwasher: C = 300ms T = 900ms (33.3333% of the CPU)
●
Washing Machine: C = 100ms T = 800ms (12.5% of the CPU)
U =
500
1000
+
300
900
+
100
800
=.958333
U ≤n(
n
√2−1)=3(
3
√2−1)=0.77976

U =∑
i=1
n
Ci
Ti
≤n(
n
√2−1)
lim
n→ ∞
n(
n
√2−1)=ln 2≈0.693147

SCHED_DEADLINE
●
Utilizes Earliest Deadline First (EDF)
●
Dynamic priority
●
The task with next deadline has highest priority
U =∑
i=1
n
Ci
Ti
=1

Earliest Deadline First (EDF)
0 1 12111098765432 13

Earliest Deadline First (EDF)
0 1 12111098765432 13
:) HAPPY :)

Setting a RT priority
sched_setscheduler(pid_t pid, int policy, struct sched_param *param)
struct sched_param {
u32 sched_priority;
};

Implementing SCHED_DEADLINE in Linux
30
Two new syscalls
sched_getattr(pid_t pid, struct sched_attr *attr, unsigned int size, unsigned int flags)
(Similar to sched_getparam(pid_t pid, struct sched_param *param)
sched_setattr(pid_t pid, struct sched_attr *attr, unsigned int flags)
(Similar to sched_setparam(pid_t pid, struct sched_param *param)

Implementing SCHED_DEADLINE
struct sched_attr {
u32 size; /* Size of this structure */
u32 sched_policy; /* Policy (SCHED_*) */
u64 sched_flags; /* Flags */
s32 sched_nice; /* Nice value (SCHED_OTHER,
SCHED_BATCH) */
u32 sched_priority; /* Static priority (SCHED_FIFO,
SCHED_RR) */
/* Remaining fields are for SCHED_DEADLINE */
u64 sched_runtime;
u64 sched_deadline;
u64 sched_period;
};

Implementing SCHED_DEADLINE
struct sched_attr attr;
ret = sched_getattr(0, &attr, sizeof(attr), 0);
if (ret < 0)
error();
attr.sched_policy = SCHED_DEADLINE;
attr.sched_runtime = runtime_ns;
attr.sched_deadline = deadline_ns;
ret = sched_setattr(0, &attr, 0);
if (ret < 0)
error();

sched_yield()
●
Most use cases are buggy
– Most tasks will not give up the CPU
●
SCHED_OTHER
– Gives up current CPU time slice
●
SCHED_FIFO / SCHED_RR
– Gives up the CPU to a task of the SAME PRIORITY
– Voluntary scheduling among same priority tasks

sched_yield()
●
Buggy code!
again:
pthread_mutex_lock(&mutex_A);
B = A->B;
if (pthread_mutex_trylock(&B->mutex_B)) {
pthread_mutex_unlock(&mutex_A);
sched_yield();
goto again;
}

sched_yield()
●
What you want for SCHED_DEADLINE!
●
Tells the kernel the task is done with current period
●
Used to relinquish the rest of the runtime budget

Constant Bandwidth Server
36
scheduling deadline = current time + deadline
remaining runtime = runtime
remaining runtime
scheduling deadline−current time
>
runtime
period

Self sleeping tasks
Courtesy of Daniel Bristot de Oliveira
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 3/9

Self sleeping tasks
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 3/9
●
Remainder = 2/3 > 3/9

Self sleeping tasks
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 3/9
●
Deadline = current time + new deadline

Self sleeping tasks
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 3/9
●
Remaining Runtime = Runtime (3 units)

Self sleeping tasks
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 3/9
●
Another Deadline task?

Self sleeping tasks
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 3/9
●
Only ran 2 units in the original 9
Original Deadline

Deadline vs Period
●
Can't have offset holes in our donuts
●
Have a specific deadline to make within a period
runtime <= deadline <= period
●
But is this too constrained?
U =∑
i=1
n
Ci
Di
=1

Self sleeping constrained tasks
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 2/4/10

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 2/4/10
●
1/1 > 2/4

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 2/4/10
●
Move deadline from 4 to 7 (period from 10 to 13)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 2/4/10
●
Runs for 1 and sleeps again

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 2/4/10
●
Wakes up again with 1 to go (moves deadline to 10, period to 16)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
U = 2/4/10
●
4 out of 10! Instead of 2 out 4 in 10

Multi processors!
It's all fun and games until someone throws another processor into
your eye

●
M CPUs
●
M+1 tasks
●
One task with runtime 999ms out of 1000ms
●
M tasks of runtime of 10ms out of 999ms
●
All start at the same time
●
The M tasks have a shorted deadline
●
All M tasks run on all CPUs for 10ms
●
That one task now only has 990 ms left to run 999ms.
Multi processors! (Dhall's Effect)
999
1000
+ M (
10
999
)=0.999+.01001 M <M
M=2;
999
1000
+2(
10
999
)=0.999+2∗0.01001=1.01902<2

2 tasks: 2/9; 1 task 9/10; U = 2/9 + 2/9 + 9/10 = 1.34444 < 2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
2/9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
2/9

2 tasks: 2/9; 1 task 9/10; U = 2/9 + 2/9 + 9/10 = 1.34444 < 2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
2/9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
2/9 9/10

Multi processors!
●
EDF can not give you better than U = 1
– No matter how many processors you have
– Full utilization should be U = N CPUs
●
Two methods
– Partitioning (Bind each task to a CPU)
– Global (let all tasks migrate wherever)
– Neither give better than U = 1 guarantees

Multi processors!
●
EDF partitioned
Can not always be used:
●
U_t1 = .6
●
U_t2 = .6
●
U_t3 = .5
●
The above would need special scheduling to work anyway
To figure out the best utilization is the bin packing problem
●
Sorry folks, it's NP complete
●
Don't even bother trying

Multi processors!
●
Global Earliest Deadline First (gEDF)
●
Can not guarantee deadlines of U > 1 for all cases
●
But special cases can be satisfied for U > 1
D_i = P_i
U_max = max{C_i/P_i}
U =∑
i=1
n
Ci
Pi
≤M −( M−1)∗U max

Multi processors!
●
M = 8
●
U_max = 0.5
U =∑
i=1
n
Ci
Pi
≤M −( M−1)∗U max
U =∑
i=1
n
Ci
Pi
≤8−(7)∗.5=4.5

Multi processors!
●
M = 2
●
U_max = 999/1000
U =∑
i=1
n
Ci
Pi
≤M −( M−1)∗U max
U =∑
i=1
n
Ci
Pi
≤2−(1)∗0.999=1.001

The limits of SCHED_DEADLINE
●
Runs on all CPUS (well sorta)
No limited sched affinity allowed
Global EDF is the default
Must account for sched migration overheads
●
Can not have children (no forking)
Your SCHED_DEADLINE tasks have been fixed
●
Calculating Worse Case Execution Time (WCET)
If you get it wrong, SCHED_DEADLINE may throttle your task before it finishes

Giving SCHED_DEADLINE Affinity
Setting task affinity on SCHED_DEADLINE is not allowed
But you can limit them by creating new sched domains
CPU sets
Implementing Partitioned EDF

cd /sys/fs/cgroup/cpuset

mkdir my_set

mkdir my_set
mkdir other_set

mkdir my_set
mkdir other_set
echo 0-2 > other_set/cpuset.cpus

mkdir my_set
mkdir other_set
echo 0 > other_set/cpuset.mems

mkdir my_set
mkdir other_set
echo 1 > other_set/cpuset.sched_load_balance

mkdir my_set
mkdir other_set
echo 1 > other_set/cpuset.cpu_exclusive

mkdir my_set
mkdir other_set
echo 3 > my_set/cpuset.cpus

mkdir my_set
mkdir other_set
echo 0 > my_set/cpuset.mems

mkdir my_set
mkdir other_set
echo 1 > my_set/cpuset.sched_load_balance

mkdir my_set
mkdir other_set
echo 1 > my_set/cpuset.cpu_exclusive

mkdir my_set
mkdir other_set
echo 0 > cpuset.sched_load_balance

mkdir my_set
mkdir other_set
echo 0 > cpuset.sched_load_balance
That’s a lot!

cat tasks | while read task; do
echo $task > other_set/tasks
done
echo $sched_deadline_task > my_set/tasks

Calculating WCET
●
Today's hardware is extremely unpredictable
●
Worse Case Execution Time is impossible to know
●
Allocate too much bandwidth instead
●
Need something between RMS and CBS

GRUB (not the boot loader)
●
Greedy Reclaim of Unused Bandwidth
●
Allows for SCHED_DEADLINE tasks to use up the unused utilization
of the CPU (or part of it)
●
Allows for tasks to handle WCET of a bit more than calculated.
●
Just went into mainline (v4.13)

New Work
●
Revisiting affinity
●
Semi-partitioning (dynamic deadlines to boot)

2 tasks: 2/9; 1 task 9/10; U = 2/9 + 2/9 + 9/10 = 1.34444 < 2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
2/9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
2/9 2/9/10
7/8/10

Links
Documentation/scheduler/sched_deadline.txt
http://disi.unitn.it/~abeni/reclaiming/rtlws14-grub.pdf
http://www.evidence.eu.com/sched_deadline.html
http://www.atc.uniovi.es/rsa/starts/documents/Lopez_2004_rts.pdf
https://cs.unc.edu/~anderson/papers/rtj06a.pdf

Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt

More Related Content

Similar to Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt (20)

More from Anne Nicolas (20)

Recently uploaded (20)

Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt