SlideShare a Scribd company logo

Structured Interactive Scores

M
Mauricio Toro-Bermudez, PhD

This document discusses structured interactive scores, which provide a formalism for interactive multimedia. It presents examples of multimedia interaction in domains like contemporary dance and interactive installations. It describes problems with most existing multimedia tools, such as a lack of formal semantics and unrelated time models. The document proposes interactive scores as a solution and discusses their history and development at LaBRI. It outlines existing tools for interactive scores, related formalisms, and the Virage software implementation.

Science
1 of 167
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra “If I cannot draw it, I do not understand it” –Anonymous. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 1
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structured interactive scores: From a structural description of a multimedia scenario to a real-time capable implementation with formal semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. Ph.D defense Supervised by Myriam Desainte-Catherine and Camilo Rueda September 25th 2012 Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 2
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Some domains shaped by the arose of technology Tape Music Electroacoustic Music Installations Interactive Multimedia Experimental Music Non-linear Multimedia Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 3
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 4
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Contemporary Dance ./videos/theatre.mov Minchoul synchronizes human gestes, video and sound in real-time in cleaning (2009). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 5
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive multimedia installations ./videos/installation.mov Torres uses the participation of the audience to co-author the meaning of the installation in pollen (2008). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 6
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Meta-instruments for physically disabled people ./videos/baopao.mov The Bao-Pao1 is a meta-instrument. As an example, this meta-instrument can control the start and end of notes. 1 http://www.bao-pao.com/index.php Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 7
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 8
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 9
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 10
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Cue-Lists model As an example, consider theater cue-list manager Qlab 2 . 2 http://figure53.com/qlab/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 11
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timeline model As an example, consider sequencer Pro Tools 3 . 3 http://www.avid.com/US/resources/digi-orientation Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 12
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time models are unrelated: Cue Lists Vs. Timeline Vs. It is argued in [DCAA12] that having both time models temporally related will bring new possibilities for the creation of interactive multimedia. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 13
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] 3. Schedulers are not appropriate for real-time [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 14
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Scheduler problem: aggravated under high cpu load • Most tools do not allow the users to interact with them without letting them experience noticeable delays • Karplus-Strong is a well-known digital signal processing algorithm to simulate the sound of metallic strings • A Karplus-Strong arpeggio implementation in Pure Data4 . ./sounds/barry-loaded.mov • A Karplus-Strong arpeggio implementation using Faust ([OFL04]) and interactive scores [TDCC12]. ./sounds/faust-loaded.mov Results from [TDCC12]. 4 Colin Barry’s implementation found at www.loomer.co.uk Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 15
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] 3. Schedulers are not appropriate for real-time [TDCC12] 4. No hierarchy (e.g., movements, parts, measures, motives and notes) [Vic04] 5. No unified model for conditional branching and temporal relations [Vic03, Ran09, All09, Vic11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 16
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 17
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solution: Interactive Scores Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time A formalism for interactive multimedia developed at LaBRI: [BDC01, DCB03, ADC05, AADCR06, AADC07, ADCLA08, AADC08, Ran09, All09, OR09, BAM+ 09, Sar08, ABM+ 10], [TDCB10, TDC10, Tor10], [MADC11], [ADCT11, TDCC12, TDCR12, Tor12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 18
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 19
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra History of interactive scores • Boxes: A software to model a hierarchy and temporal constraints [BDC01] • A model of interactive scores [ADC05] • Hierarchical Time Stream Petri nets (hts petri nets) semantics for interactive scores [AADC07] • An extension of interactive scores with temporal reductions [ADCLA08] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 20
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra History of interactive scores • First non-deterministic timed concurrent constraint (ntcc) model of interactive scores [AADCR06] • A model change the hierarchy of the temporal objects during execution [OR09] • A model to represent conditional branching and temporal relations separately [Ran09, All09] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 21
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 22
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Existing tools for interactive scores • Boxes [BDC01] • Iscore developed for OpenMusic [AADC08] • Acousmouscribe • Stand-alone i-score (currently maintained by LaBRI) • Prototypes using Ntccrt, a real-time interpreter for ntcc [TDCB10, TDC10, TDCC12] • Virage [BAM+ 09, ABM+ 10, MADC11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 23
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Virage: An implementation of interactive scores ./videos/virage.mov Software Virage developed under Virage project [BAM+ 09]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 24
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 25
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related formalisms to interactive scores • Globally Asynchronous, Locally Synchronous (gals) [TGL07] • Hierarchical time stream Petri nets (hts petri nets) [SSSW95] • Temporal constraints: qualitative [All83], quantitative [Gen98] and combination of both [Mei96] • Concurrent constraint programming (ccp) calculi [Sar92, SJG94, Sar08, OV08]. • Non-deterministic timed concurrent constraint programming (ntcc) [NPV02] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 26
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-deterministic Timed Concurrent Constraint Programming (ntcc) • Process calculus to model reactive systems with discrete time, non-determinism, partial information and asynchrony [NPV02, Val02] • Includes a temporal logic to verify properties [NPV02] • A logic formula can be translated into a process and the strongest post-condition5 of a process into a B¨uchi automaton6 [Val05] 5 Sequences that the process can output under any environment. 6 A B¨uchi automaton recognizes sequences of infinite length. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 27
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-deterministic Timed Concurrent Constraint Programming (ntcc) • Has been used for the verification and simulation of other interactive multimedia systems [RV02, RV05, RV04] • Has been used for interactive scores [AADCR06, Sar08, OR09] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 28
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: Interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalism to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 29
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related tools to interactive scores • Score following (e.g., Antescofo [Con08, ECGJ11]) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 30
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra An example of score following ./videos/antescofo.mov Example of the score following system Antescofo. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 31
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related tools to interactive scores • Score Following (e.g., Antescofo [Con08, ECGJ11]) • Asynchronous Dataflow Languages (e.g., Max/msp [PAZ98] and Pure Data[Puc96]) • Synchronous Dataflow Languages (e.g., CSound, Faust [OFL04], Esterel, Signal [GLGB87] and Lustre) • Machine Improvisation (e.g., OMax [ABC+ 06] and Continuator [Pac02]) • Computer Assisted Composition (e.g., OpenMusic’s Maquettes [BAA11]) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 32
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. We saw some examples of multimedia interaction 2. There are problems with most existing tools 3. Our solution is interactive scores (is) 4. is is a mature formalism 5. There are several tools for is 6. Score following and Maquettes are related to is 7. gals, htspn and ntcc are related to is Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 33
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 34
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition [TDCR12] 2. Abstract semantics [TDCR12] 3. Operational semantics [TDCR12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 35
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Temporal objects Green (g) Red (r) Sound (u) time Temporal objects are aligned on a timeline, thus they have nominal start time and nominal duration. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 36
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Hierarchy of temporal objects Green (g) Red (r) Sound (u) Lights (l) Scenario (c) A temporal object can contain other objects to constraint the execution start and end times of its children. The hierarchical model has been previously discussed in [BDC01, All09]. We presented a new formalization in [TDCR12]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 37
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Interactive objects Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d Interactive objects (gray circles) allow to control the start or end of other temporal objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 38
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Temporal relations Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time Temporal relations constraint the set of possible execution times and durations of the temporal objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 39
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d ∆r ∆g ∆u ∆l ∆c {0} {0} {0} {0} {0} [0, ∞) [0, ∞) [0, ∞) Durations and temporal relations can be represented by point-to-point temporal relations labeled by integer intervals [Mei96]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 40
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 41
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) • Langerak’s timed event structures is a mathematical model to represent systems with non-determinism, real-time and concurrency [BKL98]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 42
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e Figure : Circles are events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 43
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e [0,7] [0,5] [4,4] Figure : Circles are events. Full-line arrows represent event delays. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 44
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e [0,7] [0,5] [4,4] Figure : Circles are events. Full-line arrows represent event delays. Pointed-line arrows represent conflicts; when they have no direction, the conflict is symmetrical. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 45
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal objects sc su sb sl sr sg sa sd ec eu er eg el STRUCTURAL DEFINITION tes semantics for interactive scores [TDCR12]. Colored circles represent events of static temporal objects. Gray circles represent events of interactive objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 46
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Duration of temporal objects ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg sa sd ec eu er eg el STRUCTURAL DEFINITION Durations of the temporal objects are represent by event delays. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 47
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal relations 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) sc su sb sl sr sg 0 sa sd ec eu er eg STRUCTURAL DEFINITION Red arrows are event delays of the form ei [0,∞) → ej , derived from the hierarchy. Black arrows are event delays derived from the explicit temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 48
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Interactive scores 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg STRUCTURAL DEFINITION tes semantics is obtained by combining the events, the event delays from the duration of temporal objects, and the event delays from the temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 49
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 50
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Operational semantics: Intuition sc su sb sl sr sg sa sd ec eu er eg 0 0 0 0 (0 ,∞ )(0,∞) (0 , ∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg CONSTRAINT STORESTRUCTURAL DEFINITION x Agent that represents the behavior of point x • Ntcc semantics for interactive scores were introduced in [AADCR06] and later formalized in [TDCR12]. • Operational semantics respects the event structures semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 51
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. The structural definition is composed by temporal objects and point-to-point temporal relations 2. Abstract semantics are defined in timed event structures 3. Operational semantics are defined in ntcc Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 52
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 53
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition [TDCB10, TDC10, ADCT11] 2. Abstract semantics 3. Operational semantics [TDCB10, TDC10, ADCT11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 54
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model: Structural definition Verse 1 (v) Chorus (ch) Verse 3 (x) Verse 2 (w) Verse 4 (y) Chorus (c) ∆v ∆w∆ch ∆c ∆ = {0} ∆ = {0} ∆ = {0} ∆ = [2, 5] ∆ = [1, 4] ϕ > 0 ∨ ψ ≤ 3 φ > 0 ∧ ψ ≤ 5 d ∆x ∆y ∆a ∆b V = {φ, ψ, ϕ} main (m) ∆ = {0} ϕ > 0 ∨ ψ ≤ 3 V = {φ, ψ, ϕ} Timed conditional relation duration condition Variables Interactive object Choice The model was introduced in [TDCB10, TDC10, ADCT11]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 55
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 56
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: “Visible”-action events sa sb ea ecscsv ev d Structural Definition Colored circles are the “visible”-action events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 57
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: “Invisible”-action events sa sb eb ea ecscsv ev d s a s b Structural Definition Black circles are the “invisible”-action events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 58
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal relations 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Temporal relations are translated to event delays. Event delays are represented as arrows. For simplicity, the arrows have no labels. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 59
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Conflicts 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Dashed lines represent symmetrical conflicts: It means that two events are mutually exclusive. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 60
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: More conflicts 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Dashed-line arrows represent asymmetrical conflicts: It means that one event must happen before another or it will not happen. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 61
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Why visible and invisible events? • Because it is a simple way to represent conditional branching in a single bi-dimensional plane. A b s t r a c t i o n Tim e (Im plication) Values Paralelism (and) Alternative (or) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 62
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra David Janin’s dimensions A b s t r a c t i o n Tim e (Im plication) Values Paralelism (and) Alternative (or) There are several dimensions in multimedia interaction7 . 7 http://www.labri.fr/perso/janin/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 63
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 64
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model: Operational semantics b a c when finish unless finish ∆b = 3 ∆c = 2 d ∆d = 1 ∆ = 0 ∆ = 1 ∆ = 1 ∆ = 1 ∆ = 1 ∆ = 0 V = {finish} Operational semantics for the conditional branching extension with loops were introduced in [TDCB10, TDC10, ADCT11]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 65
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition is given for scores with and without loops 2. Abstract semantics are defined in timed event structures for scores without loops 3. Operational semantics are defined in ntcc, support loops, but are not related to the abstract semantics, as it was the case in the hierarchical model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 66
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 67
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive scores with signal processing 1. Structural definition [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 68
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores with signal processing Karplus (k1) Karplus (k2) Karplus (k3) Output An extension in which ntcc and the synchronous programming language with formal semantics Faust [OFL04] interact. Introduced in [TDCC12] and it is inspired from [All09]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 69
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Temporal objects Karplus (k1) Karplus (k2) Karplus (k3) ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Output Temporal objects may contain a Faust audio process, before they used to control external audio processes. Input8 and Output streams are also temporal objects. 8 In this score there is no audio input. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 70
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Temporal relations Karplus (k1) Karplus (k2) Karplus (k3) 100smp [2s, 4s] [0s, 0s] [0s, 0s] Output In addition to the usual temporal relations, dashed-line arrows represent high-precision temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 71
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Dataflow relations Karplus (k1) Karplus (k2) Karplus (k3) Output White-headed arrows represent dataflow relations: To describe how the sound is transferred from one object to another. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 72
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Interactive objects Karplus (k1) Karplus (k2) Karplus (k3)a b Output Interactive objects are represented (and behave) as usual. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 73
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 74
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 75
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation of Interactive Scores 1. Ntccrt [TAAR09] 2. Results on the conditional branching model [TDC10] 3. Results on the signal processing extension [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 76
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation: Interpreter for ntcc OpenMusic interface Ntccrt compiler Pure Data external Max/Msp external Common Lisp interface C++ interface Stand-alone program User Programmer Ntccrt9 is a simulation tool for ntcc introduced in [TAAR09]. Experience on the usage of Ntccrt is described in [ORS+ 11]. 9 http://sourceforge.net/projects/ntccrt/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 77
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation: Example using Pure Data Structural Definition “Bang” objects are controlled by the Ntccrt plugin [TDC10]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 78
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Ntccrt 2. Results on the conditional branching model 3. Results on the signal processing extension Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 79
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the conditional branching extension 0
 5
 10
 15
 20
 25
 30
 35
 40
 45
 50
 0
 200
 400
 600
 800
 1000
 1200
 1400
 1600
 1800
 Average
dura*on
of
a
*me
unit
(ms)
 Number
of
points
in
the
Interac*ve
Score
 Stress test to compute the average duration of a discrete time unit of the simulation of a conditional branching score using Ntccrt [TDC10]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 80
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Ntccrt 2. Results on the conditional branching model 3. Results on the signal processing extension Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 81
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension Average relative jitter: The average time difference between the expected starting time of each string, with respect to the first string of an arpeggio, and the time obtained during execution. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 82
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension Implementation 85% cpu load 3% cpu load Using object block˜ 1 in Pd 7991 ms 1 ms Using object z˜ 1 in Pd 9231 ms 2 ms Using Faust object in Pd 0.5 ms 0.5 ms Table : Average relative jitter of each string of an arpeggio with respect to the beep. All implementations use the Karplus-Strong algorithm to simulate the sound of the strings [TDCC12]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 83
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension In some cases –such as programs with feedback loops– our interactive scores implementation using Faust outperforms Pure Data under high-cpu load. Pure data’s performance is unacceptable, in such cases, for real-time interaction. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 84
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Our models are simulated using Ntccrt, a real-time capable interpreter for ntcc 2. Results on the conditional branching model show that we can execute up to 400 temporal objects 3. Results on the signal processing extension show that interactive scores outperforms Pure Data under high-cpu load Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 85
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 86
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) [NPV02] 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 87
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification of interactive scores • Existing semantics for interactive scores were given in hts petri nets. • Existing verification tools for time Petri nets, such as Rom´eo [GLMR05] and Tapaal [BJS09], cannot be used to verify hts petri nets [SSSW95] because hts petri nets cannot be translated to time Petri nets [BD99]. • Existing verification tools for linear-time logic, such as : Uppaal [BDL+ 01], Kronos [BDM+ 98] and Spin [Hol97], cannot be used directly for ntcc. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 88
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification of interactive scores • Results on bisimilarity for ccp [Ari10, ABPV12] are quite promising, but not yet applicable to ntcc. • Strongest post-condition of a process can be translated into a B¨uchi automaton [Val05], but using classic Schimpf’s algorithm [SMS09] is intractable because the encoding heavily relies on the complement of such automata [Saf88]. • We proposed a bounded-time finite-state-automata-based model checker for ntcc, ntccmc10 . 10 http://sourceforge.net/projects/ntccmc/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 89
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 90
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra cltl properties Examples of constraint linear-time logic (cltl) [NPV02] formulae are • “always the constraint pitch = 60 can be deduced from the output store”, namely (pitch = 60); and • “eventually, both object a and object b are launched at the same time”, namely ♦(launcha ˙∧launchb). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 91
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 92
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Is the language empty? Bounded-time automata-based model checking algorithm for ntcc A. intersection P → Pt F → Ft Ft Pt Pt → A Pt → A Ft → Pt Qt A A BF P t A B B Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Adaptation of classic model checking algorithm for linear-time logic [SMS09] to ntcc, based on the encoding of ntcc to B¨uchi automata given by Valencia et al. [Val05]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 93
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc Bounded-time automata-based model checking algorithm for ntcc ntcc Process CLTL formulaF P t Number of discrete time units Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 94
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Does a ntcc process P satisfy a CLTL formula F for a bounded time t ? Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → FtF P t Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 95
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process F P t Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 96
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A BF P t Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 97
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 98
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc A. intersection P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t A B B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 99
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Is the language empty? Bounded-time automata-based model checking algorithm for ntcc A. intersection P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t A B B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 100
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Existing tools cannot be used directly to verify interactive scores and we want to prove properties of interactive scores automatically 2. In particular, constraint linear-time temporal logic (cltl) properties 3. We presented our bounded-time automata-based model checking algorithm for ntcc, ntccMC, for cltl properties Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 101
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 102
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 103
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Summary • A Formal model for multimedia interaction that combines • timeline model • cue-list model [TDCR12] • An extension for conditional branching [TDCB10, TDC10, ADCT11] • An extension for audio processing [Tor10, TDCC12] • A simulation tool [TAAR09, ORS+ 11] • A verification tool [Tor10] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 104
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Contributions • 2 book chapters in “Constraint Programming in Music” [ADCT11, ORS+ 11] • 1 journal article in “Mathematics and Music” [TDCR12] • 2 papers in “Sound and Music Computing” conference [TDC10, TDCC12] • 1 extended abstract in “International Conference on Logic Programming” [Tor10] • 1 paper in “Journ´ees d’Informatique Musicale” [TDCB10] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 105
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 106
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • Time models are unrelated • In interactive scores, time models are temporally related Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 107
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • Schedulers are not appropriate for real-time • Using interactive scores and Faust, it is possible to have real-time interaction even under high cpu load Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 108
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No hierarchy • Interactive scores provide a hierarchy of temporal objects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 109
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No formal semantics • We defined abstract and operational semantics for interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 110
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No unified model for conditional branching and temporal relations • We provide a unified model for conditional branching and temporal relations in which all branches have the same set of possible durations Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 111
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 112
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Comparison w.r.t previous models of interactive scores • Model to change the hierarchy during performance [OR09] • Previous model developed at LaBRI [All09] • Our models [TDC10, TDCB10, TDCR12, TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 113
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Model to change the hierarchy during performance • Provides an elegant solution to change the structure of the hierarchy during performance • Has not been implemented Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 114
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Previous model developed at LaBRI • Supports different mechanisms of temporal reductions • Average duration of a discrete time unit outperforms our simulations • Durations are intervals are limited to {0}, (0, ∞] and [0, ∞] • Conditional branching and temporal relations are presented separately • Processes associated are meant to be handled outside the formalism; in the implementation as external programs Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 115
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Our models • Abstract semantics to describe the execution traces • Arbitrary integer intervals in the operational semantics • Unified model for conditional branching and temporal relations • Automatic verification • Does not allow temporal reductions • Simulation tool is a prototype; it is not user friendly • Average duration of time units is slower than in the implementation of previous model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 116
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 117
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) • Qlab and Ableton Live do not allow to model delays of 100 samples • Protools does not allow interaction In all the programs mentioned above, it is very hard to synchronize processes whose durations are integer intervals such as duration ∈ [2, 4]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 118
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 119
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Future work directions • Event structures semantics for scores with loops • Model interactive theater performances • Model interactive museum exhibitions • Verify scores in which ntcc and Faust interact • Model music pieces combining the timeline and cue-list model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 120
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Music pieces combining both time models Baltazar et al. used i-score to compose miniature # 1 (2012). I-score controls the smoke, fan, sound and lights. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 121
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Music pieces combining both time models ./videos/pascal.mp4 Baltazar et al. needed to have objects with flexible duration because the behavior of the smoke is unpredictable. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 122
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 123
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra M E R C I. Avez-vous des questions? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 125
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions 7. Extra Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Antoine Allombert, G´erard Assayag, and Myriam Desainte-Catherine. A system of interactive scores based on petri nets. In Proc. of SMC ’07, Athens, Greece, 2007. Antoine Allombert, G´erard Assayag, and Myriam Desainte-Catherine. Iscore: a system for writing interaction. In Proc. of 3rd International Conference on Digital Interactive Media in Entertainment and Arts (DIMEA) ’08, pages 360–367, New York, NY, USA, 2008. ACM. Antoine Allombert, G´erard Assayag, M. Desainte-Catherine, and Camilo Rueda. Concurrent constraint models for interactive scores. In Proc. of Sound and Music Computing (SMC) ’06, Marseille, France, 2006. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra G´erard Assayag, Georges Bloch, Marc Chemillier, Arshia Cont, and Shlomo Dubnov. Omax brothers: a dynamic topology of agents for improvization learning. In Proc. of the 1st ACM workshop on Audio and music computing multimedia (AMCMM) ’06, pages 125–132, New York, NY, USA, 2006. ACM. Antoine Allombert, Pascal Baltazar, Rapha¨el Marczak, Myriam Desainte-Catherine, and Laurent Garnier. Designing an interactive intermedia sequencer from users requirements and theoretical background. In Proc. of International Computer Music Conference (ICMC) ’10, 2010. A. Aristizabal, F. Bonchi, L. Pino, and F. Valencia. Partition refinement for bisimilarity in ccp. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of the 27th ACM Symposium On Applied Computing (SAC 2012), 2012. Antoine Allombert and Myriam Desainte-Catherine. Interactive scores: A model for specifying temporal relations between interactive and static events. In Journal of New Music Research, 2005. Antoine Allombert, Myriam Desainte-Catherine, J. Larralde, and G´erard Assayag. A system of interactive scores based on qualitative and quantitative temporal constraints. In Proc. of 4th International Conference on Digital Arts (Artech) ’08, Porto, Portugal, 2008. The Artech International Association. Antoine Allombert, Myriam Desainte-Catherine, and Mauricio Toro. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Modeling temporal constrains for a system of interactive score. In G´erard Assayag and Charlotte Truchet, editors, Constraint Programming in Music, chapter 1, pages 1–23. Wiley, 2011. James F. Allen. Maintaining knowledge about temporal intervals. Communication of ACM, 26, 1983. Antoine Allombert. Aspects temporels d’un syst`eme de partitions num`eriques interactives pour la composition et l’interpr´etation. PhD thesis, Universit´e de Bordeaux, November 2009. Andr´es Aristiz´abal. Bisimilarity in concurrent constraint programming. In 26th International Conference on Logic Programming (ICLP) 2010, 2010. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Jean Bresson, Carlos Agon, and G´erard Assayag. Openmusic: visual programming environment for music composition, analysis and research. In Proceedings of the 19th ACM international conference on Multimedia, MM ’11, pages 743–746, New York, NY, USA, 2011. ACM. P. Baltazar, A. Allombert, R. Marczak, J.M. Couturier, M. Roy, A. S`edes, and M. Desainte-Catherine. Virage : Une reflexion pluridisciplinaire autour du temps dans la creation numerique. In in Proc. of JIM, 2009. Marc Boyer and Michel Diaz. Non equivalence between time petri nets and time stream petri nets. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of the The 8th International Workshop on Petri Nets and Performance Models, number 98 in 1, Washington, DC, USA, 1999. IEEE Computer Society. Anthony Beuriv´e and Myriam Desainte-Catherine. Representing musical hierarchies with constraints. In 7th International Conference on Principles and Practice of Constraint Programming, Musical Constraints Workshop, Paphos, 2001. Gerd Behrmann, Alexandre David, Kim G. Larsen, Oliver M¨oller, Paul Pettersson, and Wang Yi. Uppaal - present and future. In Proc. of 40th IEEE Conference on Decision and Control. IEEE Computer Society Press, 2001. Marius Bozga, Conrado Daws, Oded Maler, Alfredo Olivero, Stavros Tripakis, and Sergio Yovine. Kronos: A model-checking tool for real-time systems. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Anders Ravn and Hans Rischel, editors, Formal Techniques in Real-Time and Fault-Tolerant Systems, volume 1486 of Lecture Notes in Computer Science, pages 298–302. Springer Berlin / Heidelberg, 1998. Joakim Byg, Kenneth Yrke Jørgensen, and Jiˇr´ı Srba. Tapaal: Editor, simulator and verifier of timed-arc petri nets. In Proceedings of the 7th International Symposium on Automated Technology for Verification and Analysis (ATVA) ’09, pages 84–89, Berlin, Heidelberg, 2009. Springer-Verlag. Christel Baier, Joost-Pieter Katoen, and Diego Latella. Metric semantics for true concurrent real time. In Proc. of International Conference on Automata, Languages and Programming (ICALP) ’98, Berlin, Germany, 1998. Springer. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Arshia Cont. Antescofo: Anticipatory synchronization and control of interactive parameters in computer music. In Proc. of ICMC ’08, 2008. Myriam Desainte-Catherine, Antoine Allombert, and G´erard Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Computer Music Journal, To appear in fall, 2012. M. Desainte-Catherine and N. Brousse. Towards a specification of musical interactive pieces. In Proc. of the XIX Colloquium on Musical Informatics (CIM), Firenze, Italy, 2003. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Jos´e Echeveste, Arshia Cont, Jean-Louis Giavitto, and Florent Jacquemard. Formalisation des relations temporelles entre une partition et une performance musicale dans un contexte d’accompagnement automatique : Accompagnement musical automatique. In Journal Europ´een des Syst`emes Automatis´es, 2011. Rosella Gennari. Temporal resoning and constraint programming - a survey. CWI Quaterly, 11:3–163, 1998. Thierry Gautier, Paul Le Guernic, and L¨oic Besnard. Signal: A declarative language for synchronous programming of real-time systems. In Proc. of FPCA ’87, 1987. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Guillaume Gardey, Didier Lime, Morgan Magnin, and Olivier (H.) Roux. Rom´eo: A tool for analyzing time Petri nets. In 17th International Conference on Computer Aided Verification (CAV’05), volume 3576 of Lecture Notes in Computer Science, pages 418–423, Edinburgh, Scotland, UK, 2005. Springer. Gerard J. Holzmann. The model checker spin. Software Engineering, 23(5):279–295, 1997. Rapha¨el Marczak, Antoine Allombert, and Myriam Desainte-Catherine. Real-time temporal control of musical processes. In Proc. of the International Conferences on Advances in Multimedia (MMEDIA) ’11, 2011. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Itay Meiri. Combining qualitative and quantitative constraints in temporal reasoning. Artificial Inteligence, 87(1-2):343–385, 1996. M. Nielsen, C. Palamidessi, and F. Valencia. Temporal concurrent constraint programming: Denotation, logic and applications. Nordic Journal of Computing, 1(9):145–188, 2002. Yann Orlarey, Dominique Fober, and Stephane Letz. Syntactical and semantical aspects of faust. Soft Comput., 8(9):623–632, 2004. Carlos Olarte and Camilo Rueda. A Declarative Language for Dynamic Multimedia Interaction Systems. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of Mathematics and Computation in Music, volume 38, Berlin, Germany, july 2009. Springer. Carlos Olarte, Camilo Rueda, Gerardo Sarria, Mauricio Toro, and Frank Valencia. Concurrent constraints models of music interaction. In G´erard Assayag and Charlotte Truchet, editors, Constraint Programming in Music, chapter 6, pages 133–153. Wiley, Hoboken, NJ, USA., 2011. Carlos Olarte and Frank D. Valencia. The expressivity of universal timed ccp: undecidability of monoadic fltl and closure operators for security. In PPDP ’08: Proceedings of the 10th international ACM SIGPLAN conference on Principles and practice of declarative programming, pages 8–19. ACM, 2008. Fran¸cois Pachet. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Playing with virtual musicians: the continuator in practice. IEEE Multimedia, 9:77–82, 2002. Miller Puckette, Theodore Apel, and David Zicarelli. Real-time audio analysis tools for pd and max/msp. In Proc. of ICMC ’98, Ann Arbor, USA, 1998. M. Puckette. Pure data. In Proceedings of the International Computer Music Conference. San Francisco 1996, 1996. Nirina Ranaivoson. R´eflexion sur la mise en place des structures logiques dans un logiciel de partitions musicales interactives. Master’s thesis, Universit´e de Bordeaux, 2009. Camilo Rueda and Frank Valencia. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Proving musical properties using a temporal concurrent constraint calculus. In Proc. of the 28th International Computer Music Conference (ICMC) ’02, 2002. C. Rueda and F. Valencia. On validity in modelization of musical problems by ccp. Soft Computing, 8(9):641–648, 2004. Camilo Rueda and Frank Valencia. A temporal concurrent constraint calculus as an audio processing framework. In SMC ’05, 2005. S. Safra. On the complexity of omega-automata. In Proceedings of the 29th Annual Symposium on Foundations of Computer Science (SFCS) ’88, pages Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra 319–327, Washington, DC, USA, 1988. IEEE Computer Society. Vijay A. Saraswat. Concurrent Constraint Programming. MIT Press, Cambridge, MA, 1992. Gerardo Sarria. Formal Models of Timed Musical Processes. PhD thesis, Universidad del Valle, Colombia, 2008. Vijay A. Saraswat, Radha Jagadeesan, and Vineet Gupta. Foundations of timed concurrent constraint programming. In Proceedings of the Ninth Annual IEEE Symposium on Logic in Computer Science, pages 71–80. IEEE Computer Press, 1994. Alexander Schimpf, Stephan Merz, and Jan-Georg Smaus. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Construction of b¨uchi automata for ltl model checking verified in isabelle/hol. In Proceedings of the 22nd International Conference on Theorem Proving in Higher Order Logics, TPHOLs ’09, pages 424–439, Berlin, Heidelberg, 2009. Springer-Verlag. Patrick S´enac, Pierre de Saqui-Sannes, and Roberto Willrich. Hierarchical time stream petri net: A model for hypermedia systems. In Proc. of the 16th International Conference on Application and Theory of Petri Nets, pages 451–470, London, UK, 1995. Springer-Verlag. Mauricio Toro, Carlos Ag´on, G´erard Assayag, and Camilo Rueda. Ntccrt: A concurrent constraint framework for real-time interaction. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of ICMC ’09, Montreal, Canada, 2009. Mauricio Toro and Myriam Desainte-Catherine. Concurrent constraint conditional branching interactive scores. In Proc. of SMC ’10, Barcelona, Spain, 2010. Mauricio Toro, Myriam Desainte-Catherine, and P. Baltazar. A model for interactive scores with temporal constraints and conditional branching. In Proc. of Journ´ees d’Informatique Musical (JIM) ’10, May 2010. Mauricio Toro, Myriam Desainte-Catherine, and Julien Castet. An extension of interactive scores for multimedia scenarios with temporal relations for micro and macro controls. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of Sound and Music Computing (SMC) ’12, Copenhagen, Denmark, July 2012. Mauricio Toro, Myriam Desainte-Catherine, and Camilo Rueda. Formal semantics for interactive music scores: A framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music, To be published., 2012. Paul Teehan, Mark R. Greenstreet, and Guy G. Lemieux. A survey and taxonomy of gals design styles. IEEE Design Test of Computers, 24(5):418–428, 2007. Mauricio Toro. Structured musical interactive scores (short). In Proc. of the doctoral consortium in ICLP ’10, Edinburgh, Scotland, UK, 2010. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Mauricio Toro. Structured Interactive Scores: From a simple structural definition to a real-time capable implementation. PhD thesis, Universit´e de Bordeaux, Bordeaux, September 2012. Frank D. Valencia. Temporal Concurrent Constraint Programming. PhD thesis, University of Aarhus, 2002. Frank D. Valencia. Decidability of infinite-state timed ccp processes and first-order ltl. Journal of Theoretical Computer Science - Expressiveness in concurrency, 330(3):557–607, 2005. Lindsay Vickery. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-linear structures for real-time interactive musical works. In Proceedings of the Australasian Computer Music Conference (ACMC) ’03, 2003. Lindsay Vickery. Interactive control of higher order musical structures. In Proc. of ACMC ’04, Victoria University, New Zealand, July 2004. ACMA. Lindsay Vickery. The possibilities of novel format structures through computer controlled live performance. In Proc. of ACMC ’11, 2011. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra More Future Work Score with conditions Event Structures semantics Block Diagram semantics Abstract semantics Structural Definition Ntcc model Ntcc and Faust interconnected Model checker Temporal Property Operational semantics Pure Data Program Score with audio proc. Score with interval durations Score with arbitrary durations Event Structures semantics Event Structures semantics Stand Alone Programs Programs Music Properties Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Limitation of time conditional branching b e Choose either d or e after aa c d ∆c = {3} ∆e = {2} ∆d = {1} ∆b = [1, 2] = = = = = = = Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 128
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail Red (r) ∆r ∆r sr er ∆r s r e r Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 129
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail φ ∆ ∆ ∆ ∆ ∆ ∆a ∆b ∆a ∆b ∆a Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 130
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail Green (g) Red (r) Lights (l) a ∆l ∆r ∆g ∆a = {0} ∆r ∆r ∆g ∆g sr sr er er sg sg eg eg a sl el ∆l ∆l Video (e) ee e e e l s l φ ≥ 10 ∆ = {0} Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 131
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Equivalence between interactive score constraints and its event traces Theorem The temporal constraints of a score are equivalent to the temporal constraints of the event structures semantics. Proof. This holds by replacing points in the temporal constraints of the score by its corresponding events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 132
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics Theorem The possible time units in which a point process can be launched in the operational semantics are included in the possible time units in which an event can appear in an event trace in the tes semantics [TDCR12]. Proof. By induction on the structure of the timed event structures (tes) semantics of a score. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 133
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics Theorem Let s be a score, ε∗ = des(normal(es(s))) = E, l∗ , R its event structures semantics, tc(ε∗ ) its temporal constraints, P = ScoreS,I,Pr,R the ntcc process that represents the score, [[P]] the denotation of the ntcc process, and Ti is the set of indexes j such that [[P]]j+1 entails launchi . It holds for all sequences in [[P]], n = card(E), that T1 × T2... × Tn ⊆ Solutions(tc(ε∗ )). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 134
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics ε∗ Values of Ti for process Scoreε∗ tc(ε∗ ) ((i)) Ti ⊆ [1, n∞] ti ∈ N (i) ∆ → (j) Ti = {1} ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti + ((i)) ∆ → (j) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti + ((i)) ∆ → ((j)) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti + (i) ∆ → ((j)) Ti = {1} ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti + Table : Temporal constraints to launch the events in the ntcc model Vs. temporal constraints of the event structures semantics. We denote an interactive object as ((i)) and a static one as (i). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 135
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 136
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra The novelty of our extension: Faust and ntcc interconnection 1. A model based on the Non-deterministic Timed Concurrent Constraint (ntcc) calculus for concurrency, user interactions and temporal relations, and 2. a model based on the Faust programming language for sound processing and micro controls. buton Label mouse down mouse up 0 1 user ntcc Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 137
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 138
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 139
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings a b [2s, 4s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Figure : The constraint graph Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 140
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3) @100 threeCords(f) output sk1 ek1 sk1 ek2 sk2 Figure : Faust’s Block diagram Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 141
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Figure : Implementation Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 142
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: Three user controlled arpeggios ∆ = 10 ∆ = 10 ∆ = 10 Figure : The double-headed arrow represents an inequality (≤) and a white-headed arrow represents an equality relation (=). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 143
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: An arpeggio without “clicks” a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1) Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s] Karplus (k2) Karplus' AC [0.5s, 0.5s] ∆k2 = [5s, 10s] [4.5s, 9.5s] Karplus (k3) Karplus' AC [0.5s, 0.5s] ∆k3 = [4s, 4s] [3.5s, 3.5s] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 144
Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: Sound source perception Karplus (k1) L Output (o1) [0s, 0s] ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] Karplus (k1) L Output (o1) ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] 500µs Time Time Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 145

More Related Content

PDF
Chances and Challenges in Comparing Cross-Language Retrieval Tools
Giovanna Roda
 
PDF
FORGE Keynote @ IT Mexicali, October 2015
FORGE project
 
PDF
cv_10
Craig Cooper
 
PDF
CETIS_John Robertson
The Open Education Consortium
 
PPTX
Approaches to supporting Open Educational Resource projects
R. John Robertson
 
PDF
Fossa 2010-ralf
fOSSa 2010 Main Presentations
 
PPTX
Interactive Video Search - Tutorial at ACM Multimedia 2015
klschoef
 
KEY
Gli standard per l’interoperabilità dei sistemi studenti in ambito Europeo
Simone Ravaioli
 
Chances and Challenges in Comparing Cross-Language Retrieval Tools
Giovanna Roda
 
FORGE Keynote @ IT Mexicali, October 2015
FORGE project
 
cv_10
Craig Cooper
 
CETIS_John Robertson
The Open Education Consortium
 
Approaches to supporting Open Educational Resource projects
R. John Robertson
 
Fossa 2010-ralf
fOSSa 2010 Main Presentations
 
Interactive Video Search - Tutorial at ACM Multimedia 2015
klschoef
 
Gli standard per l’interoperabilità dei sistemi studenti in ambito Europeo
Simone Ravaioli
 

Similar to Structured Interactive Scores (20)

PPTX
BuildingSMART Standards Summit 2015 - Technical Room - Linked Data for Constr...
Pieter Pauwels
 
DOC
Final CHO Internet of Things cho cho cho
kingdomofgodinfo
 
PDF
The 2nd Intl. Workshop on NL-based Software Engineering
Sebastiano Panichella
 
PPT
Iscram Multi Stakeholder Track And Labs Proposal 20090512
Paul Burghart
 
PDF
these_15-9
Bahjat Safadi
 
PPTX
Création de la banque de corpus CoMeRe : un partenariat Corpus-écrits – ORTO...
Thierry Chanier
 
PDF
Languages Methodologies And Development Tools For Multiagent Systems Second I...
ebedibukai
 
PDF
Coordination Models And Languages 6th International Conference Coordination 2...
tocanalises62
 
PDF
PATHS Final state of art monitoring report v0_4
pathsproject
 
PDF
CHARMe's presentation on WP300 - RV1 - EU Brussels Dec 2013
Raquel Alegre
 
DOC
Statistical and Empirical Approaches to Spoken Dialog Systems
butest
 
PPTX
1.0 Course Handout_ Introduction_IOT.pptx
byfexowwfeciurbwnw
 
PDF
Cbmn4104 multimedia networking dec10 edit-mac15
roslee amirdurahman
 
PPTX
requirements engineering - technologies
Katrien Verbert
 
PPTX
VIRTUAL CLASS ROOM
Bharat Rathore
 
DOCX
Documento Sena En Ingles (Ientilocos)[1]
IE Simona Duque
 
PPTX
Application of the Multiple Perspectives Model in an Undergraduate Course
Célio Gonçalo Marques
 
PPTX
Application of the Multiple Perspectives Model in an Undergraduate Course
Célio Gonçalo Marques
 
PDF
OpenDocument Traps
Marco Fioretti
 
PDF
Test
Anand PA
 
BuildingSMART Standards Summit 2015 - Technical Room - Linked Data for Constr...
Pieter Pauwels
 
Final CHO Internet of Things cho cho cho
kingdomofgodinfo
 
The 2nd Intl. Workshop on NL-based Software Engineering
Sebastiano Panichella
 
Iscram Multi Stakeholder Track And Labs Proposal 20090512
Paul Burghart
 
these_15-9
Bahjat Safadi
 
Création de la banque de corpus CoMeRe : un partenariat Corpus-écrits – ORTO...
Thierry Chanier
 
Languages Methodologies And Development Tools For Multiagent Systems Second I...
ebedibukai
 
Coordination Models And Languages 6th International Conference Coordination 2...
tocanalises62
 
PATHS Final state of art monitoring report v0_4
pathsproject
 
CHARMe's presentation on WP300 - RV1 - EU Brussels Dec 2013
Raquel Alegre
 
Statistical and Empirical Approaches to Spoken Dialog Systems
butest
 
1.0 Course Handout_ Introduction_IOT.pptx
byfexowwfeciurbwnw
 
Cbmn4104 multimedia networking dec10 edit-mac15
roslee amirdurahman
 
requirements engineering - technologies
Katrien Verbert
 
VIRTUAL CLASS ROOM
Bharat Rathore
 
Documento Sena En Ingles (Ientilocos)[1]
IE Simona Duque
 
Application of the Multiple Perspectives Model in an Undergraduate Course
Célio Gonçalo Marques
 
Application of the Multiple Perspectives Model in an Undergraduate Course
Célio Gonçalo Marques
 
OpenDocument Traps
Marco Fioretti
 
Test
Anand PA
 
Ad

Recently uploaded (20)

PPTX
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
Rodel Gordzkie
 
PDF
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
Sérgio Sacani
 
PDF
SEHH2274 Organic Chemistry Notes 1 Structure and Bonding.pdf
ivan1108lau
 
PPTX
BIOMOLECULES PPT........................
vachieagrawal1221
 
PDF
Cosmic Outliers: Low-spin Halos Explain the Abundance, Compactness, and Redsh...
Sérgio Sacani
 
PDF
Sciences of Europe No 170 (2025)
Sciences of Europe
 
PPTX
Microbiology with diagram medical studies .pptx
mohammedabbusiddiq
 
PPTX
TOTAL hIP ARTHROPLASTY Presentation.pptx
hararghemssc
 
PPT
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
joybejerano
 
PPTX
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
OliveJolly1
 
PPTX
neck nodes and dissection types and lymph nodes levels
DipuSarma
 
PDF
Biophysics 2.pdffffffffffffffffffffffffff
MiraMusleh
 
PPTX
Classification Systems_TAXONOMY_SCIENCE8.pptx
glofernignacio
 
PDF
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
Sérgio Sacani
 
PDF
lecture 2026 of Sjogren's syndrome l .pdf
ssuserf13226
 
PPTX
2. Earth - The Living Planet Module 2ELS
markjustinebarolobau
 
PPTX
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
PDF
Placing the Near-Earth Object Impact Probability in Context
Sérgio Sacani
 
PDF
ELS_Q1_Module-11_Formation-of-Rock-Layers_v2.pdf
savannaesar07
 
PPTX
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PrabakaranMP1
 
G5Q1W8 PPT SCIENCE.pptx 2025-2026 GRADE 5
Rodel Gordzkie
 
Formation of Supersonic Turbulence in the Primordial Star-forming Cloud
Sérgio Sacani
 
SEHH2274 Organic Chemistry Notes 1 Structure and Bonding.pdf
ivan1108lau
 
BIOMOLECULES PPT........................
vachieagrawal1221
 
Cosmic Outliers: Low-spin Halos Explain the Abundance, Compactness, and Redsh...
Sérgio Sacani
 
Sciences of Europe No 170 (2025)
Sciences of Europe
 
Microbiology with diagram medical studies .pptx
mohammedabbusiddiq
 
TOTAL hIP ARTHROPLASTY Presentation.pptx
hararghemssc
 
The World of Physical Science, • Labs: Safety Simulation, Measurement Practice
joybejerano
 
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
OliveJolly1
 
neck nodes and dissection types and lymph nodes levels
DipuSarma
 
Biophysics 2.pdffffffffffffffffffffffffff
MiraMusleh
 
Classification Systems_TAXONOMY_SCIENCE8.pptx
glofernignacio
 
CAPERS-LRD-z9:AGas-enshroudedLittleRedDotHostingaBroad-lineActive GalacticNuc...
Sérgio Sacani
 
lecture 2026 of Sjogren's syndrome l .pdf
ssuserf13226
 
2. Earth - The Living Planet Module 2ELS
markjustinebarolobau
 
ANEMIA WITH LEUKOPENIA MDS 07_25.pptx htggtftgt fredrctvg
vedvyas5668
 
Placing the Near-Earth Object Impact Probability in Context
Sérgio Sacani
 
ELS_Q1_Module-11_Formation-of-Rock-Layers_v2.pdf
savannaesar07
 
EPIDURAL ANESTHESIA ANATOMY AND PHYSIOLOGY.pptx
PrabakaranMP1
 
Ad

Structured Interactive Scores

  • 1. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra “If I cannot draw it, I do not understand it” –Anonymous. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 1
  • 2. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structured interactive scores: From a structural description of a multimedia scenario to a real-time capable implementation with formal semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. Ph.D defense Supervised by Myriam Desainte-Catherine and Camilo Rueda September 25th 2012 Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 2
  • 3. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Some domains shaped by the arose of technology Tape Music Electroacoustic Music Installations Interactive Multimedia Experimental Music Non-linear Multimedia Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 3
  • 4. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 4
  • 5. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Contemporary Dance ./videos/theatre.mov Minchoul synchronizes human gestes, video and sound in real-time in cleaning (2009). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 5
  • 6. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive multimedia installations ./videos/installation.mov Torres uses the participation of the audience to co-author the meaning of the installation in pollen (2008). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 6
  • 7. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Meta-instruments for physically disabled people ./videos/baopao.mov The Bao-Pao1 is a meta-instrument. As an example, this meta-instrument can control the start and end of notes. 1 http://www.bao-pao.com/index.php Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 7
  • 8. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 8
  • 9. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 9
  • 10. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 10
  • 11. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Cue-Lists model As an example, consider theater cue-list manager Qlab 2 . 2 http://figure53.com/qlab/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 11
  • 12. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timeline model As an example, consider sequencer Pro Tools 3 . 3 http://www.avid.com/US/resources/digi-orientation Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 12
  • 13. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time models are unrelated: Cue Lists Vs. Timeline Vs. It is argued in [DCAA12] that having both time models temporally related will bring new possibilities for the creation of interactive multimedia. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 13
  • 14. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] 3. Schedulers are not appropriate for real-time [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 14
  • 15. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Scheduler problem: aggravated under high cpu load • Most tools do not allow the users to interact with them without letting them experience noticeable delays • Karplus-Strong is a well-known digital signal processing algorithm to simulate the sound of metallic strings • A Karplus-Strong arpeggio implementation in Pure Data4 . ./sounds/barry-loaded.mov • A Karplus-Strong arpeggio implementation using Faust ([OFL04]) and interactive scores [TDCC12]. ./sounds/faust-loaded.mov Results from [TDCC12]. 4 Colin Barry’s implementation found at www.loomer.co.uk Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 15
  • 16. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Problems with most existing tools 1. No formal semantics [TDCR12] 2. Time models are unrelated [DCAA12] 3. Schedulers are not appropriate for real-time [TDCC12] 4. No hierarchy (e.g., movements, parts, measures, motives and notes) [Vic04] 5. No unified model for conditional branching and temporal relations [Vic03, Ran09, All09, Vic11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 16
  • 17. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 17
  • 18. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solution: Interactive Scores Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time A formalism for interactive multimedia developed at LaBRI: [BDC01, DCB03, ADC05, AADCR06, AADC07, ADCLA08, AADC08, Ran09, All09, OR09, BAM+ 09, Sar08, ABM+ 10], [TDCB10, TDC10, Tor10], [MADC11], [ADCT11, TDCC12, TDCR12, Tor12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 18
  • 19. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 19
  • 20. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra History of interactive scores • Boxes: A software to model a hierarchy and temporal constraints [BDC01] • A model of interactive scores [ADC05] • Hierarchical Time Stream Petri nets (hts petri nets) semantics for interactive scores [AADC07] • An extension of interactive scores with temporal reductions [ADCLA08] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 20
  • 21. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra History of interactive scores • First non-deterministic timed concurrent constraint (ntcc) model of interactive scores [AADCR06] • A model change the hierarchy of the temporal objects during execution [OR09] • A model to represent conditional branching and temporal relations separately [Ran09, All09] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 21
  • 22. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 22
  • 23. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Existing tools for interactive scores • Boxes [BDC01] • Iscore developed for OpenMusic [AADC08] • Acousmouscribe • Stand-alone i-score (currently maintained by LaBRI) • Prototypes using Ntccrt, a real-time interpreter for ntcc [TDCB10, TDC10, TDCC12] • Virage [BAM+ 09, ABM+ 10, MADC11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 23
  • 24. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Virage: An implementation of interactive scores ./videos/virage.mov Software Virage developed under Virage project [BAM+ 09]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 24
  • 25. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalisms to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 25
  • 26. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related formalisms to interactive scores • Globally Asynchronous, Locally Synchronous (gals) [TGL07] • Hierarchical time stream Petri nets (hts petri nets) [SSSW95] • Temporal constraints: qualitative [All83], quantitative [Gen98] and combination of both [Mei96] • Concurrent constraint programming (ccp) calculi [Sar92, SJG94, Sar08, OV08]. • Non-deterministic timed concurrent constraint programming (ntcc) [NPV02] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 26
  • 27. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-deterministic Timed Concurrent Constraint Programming (ntcc) • Process calculus to model reactive systems with discrete time, non-determinism, partial information and asynchrony [NPV02, Val02] • Includes a temporal logic to verify properties [NPV02] • A logic formula can be translated into a process and the strongest post-condition5 of a process into a B¨uchi automaton6 [Val05] 5 Sequences that the process can output under any environment. 6 A B¨uchi automaton recognizes sequences of infinite length. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 27
  • 28. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-deterministic Timed Concurrent Constraint Programming (ntcc) • Has been used for the verification and simulation of other interactive multimedia systems [RV02, RV05, RV04] • Has been used for interactive scores [AADCR06, Sar08, OR09] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 28
  • 29. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. Examples of multimedia interaction 2. Problems with most existing tools 3. Solution: Interactive scores 4. History of interactive scores 5. Tools for interactive scores 6. Related formalism to interactive scores 7. Related tools to interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 29
  • 30. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related tools to interactive scores • Score following (e.g., Antescofo [Con08, ECGJ11]) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 30
  • 31. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra An example of score following ./videos/antescofo.mov Example of the score following system Antescofo. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 31
  • 32. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Related tools to interactive scores • Score Following (e.g., Antescofo [Con08, ECGJ11]) • Asynchronous Dataflow Languages (e.g., Max/msp [PAZ98] and Pure Data[Puc96]) • Synchronous Dataflow Languages (e.g., CSound, Faust [OFL04], Esterel, Signal [GLGB87] and Lustre) • Machine Improvisation (e.g., OMax [ABC+ 06] and Continuator [Pac02]) • Computer Assisted Composition (e.g., OpenMusic’s Maquettes [BAA11]) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 32
  • 33. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Introduction 1. We saw some examples of multimedia interaction 2. There are problems with most existing tools 3. Our solution is interactive scores (is) 4. is is a mature formalism 5. There are several tools for is 6. Score following and Maquettes are related to is 7. gals, htspn and ntcc are related to is Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 33
  • 34. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 34
  • 35. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition [TDCR12] 2. Abstract semantics [TDCR12] 3. Operational semantics [TDCR12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 35
  • 36. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Temporal objects Green (g) Red (r) Sound (u) time Temporal objects are aligned on a timeline, thus they have nominal start time and nominal duration. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 36
  • 37. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Hierarchy of temporal objects Green (g) Red (r) Sound (u) Lights (l) Scenario (c) A temporal object can contain other objects to constraint the execution start and end times of its children. The hierarchical model has been previously discussed in [BDC01, All09]. We presented a new formalization in [TDCR12]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 37
  • 38. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Interactive objects Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d Interactive objects (gray circles) allow to control the start or end of other temporal objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 38
  • 39. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition: Temporal relations Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time Temporal relations constraint the set of possible execution times and durations of the temporal objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 39
  • 40. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Structural definition Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d ∆r ∆g ∆u ∆l ∆c {0} {0} {0} {0} {0} [0, ∞) [0, ∞) [0, ∞) Durations and temporal relations can be represented by point-to-point temporal relations labeled by integer intervals [Mei96]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 40
  • 41. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 41
  • 42. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) • Langerak’s timed event structures is a mathematical model to represent systems with non-determinism, real-time and concurrency [BKL98]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 42
  • 43. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e Figure : Circles are events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 43
  • 44. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e [0,7] [0,5] [4,4] Figure : Circles are events. Full-line arrows represent event delays. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 44
  • 45. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Timed Event Structures (tes) a b c d e [0,7] [0,5] [4,4] Figure : Circles are events. Full-line arrows represent event delays. Pointed-line arrows represent conflicts; when they have no direction, the conflict is symmetrical. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 45
  • 46. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal objects sc su sb sl sr sg sa sd ec eu er eg el STRUCTURAL DEFINITION tes semantics for interactive scores [TDCR12]. Colored circles represent events of static temporal objects. Gray circles represent events of interactive objects. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 46
  • 47. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Duration of temporal objects ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg sa sd ec eu er eg el STRUCTURAL DEFINITION Durations of the temporal objects are represent by event delays. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 47
  • 48. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal relations 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) sc su sb sl sr sg 0 sa sd ec eu er eg STRUCTURAL DEFINITION Red arrows are event delays of the form ei [0,∞) → ej , derived from the hierarchy. Black arrows are event delays derived from the explicit temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 48
  • 49. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Interactive scores 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg STRUCTURAL DEFINITION tes semantics is obtained by combining the events, the event delays from the duration of temporal objects, and the event delays from the temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 49
  • 50. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 50
  • 51. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Operational semantics: Intuition sc su sb sl sr sg sa sd ec eu er eg 0 0 0 0 (0 ,∞ )(0,∞) (0 , ∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg CONSTRAINT STORESTRUCTURAL DEFINITION x Agent that represents the behavior of point x • Ntcc semantics for interactive scores were introduced in [AADCR06] and later formalized in [TDCR12]. • Operational semantics respects the event structures semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 51
  • 52. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Hierarchical model of interactive scores 1. The structural definition is composed by temporal objects and point-to-point temporal relations 2. Abstract semantics are defined in timed event structures 3. Operational semantics are defined in ntcc Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 52
  • 53. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 53
  • 54. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition [TDCB10, TDC10, ADCT11] 2. Abstract semantics 3. Operational semantics [TDCB10, TDC10, ADCT11] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 54
  • 55. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model: Structural definition Verse 1 (v) Chorus (ch) Verse 3 (x) Verse 2 (w) Verse 4 (y) Chorus (c) ∆v ∆w∆ch ∆c ∆ = {0} ∆ = {0} ∆ = {0} ∆ = [2, 5] ∆ = [1, 4] ϕ > 0 ∨ ψ ≤ 3 φ > 0 ∧ ψ ≤ 5 d ∆x ∆y ∆a ∆b V = {φ, ψ, ϕ} main (m) ∆ = {0} ϕ > 0 ∨ ψ ≤ 3 V = {φ, ψ, ϕ} Timed conditional relation duration condition Variables Interactive object Choice The model was introduced in [TDCB10, TDC10, ADCT11]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 55
  • 56. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 56
  • 57. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: “Visible”-action events sa sb ea ecscsv ev d Structural Definition Colored circles are the “visible”-action events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 57
  • 58. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: “Invisible”-action events sa sb eb ea ecscsv ev d s a s b Structural Definition Black circles are the “invisible”-action events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 58
  • 59. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Temporal relations 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Temporal relations are translated to event delays. Event delays are represented as arrows. For simplicity, the arrows have no labels. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 59
  • 60. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: Conflicts 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Dashed lines represent symmetrical conflicts: It means that two events are mutually exclusive. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 60
  • 61. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra tes semantics: More conflicts 0 0 sa sb eb ea ecscsv ev d s a s b Structural Definition Dashed-line arrows represent asymmetrical conflicts: It means that one event must happen before another or it will not happen. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 61
  • 62. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Why visible and invisible events? • Because it is a simple way to represent conditional branching in a single bi-dimensional plane. A b s t r a c t i o n Tim e (Im plication) Values Paralelism (and) Alternative (or) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 62
  • 63. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra David Janin’s dimensions A b s t r a c t i o n Tim e (Im plication) Values Paralelism (and) Alternative (or) There are several dimensions in multimedia interaction7 . 7 http://www.labri.fr/perso/janin/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 63
  • 64. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition 2. Abstract semantics 3. Operational semantics Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 64
  • 65. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model: Operational semantics b a c when finish unless finish ∆b = 3 ∆c = 2 d ∆d = 1 ∆ = 0 ∆ = 1 ∆ = 1 ∆ = 1 ∆ = 1 ∆ = 0 V = {finish} Operational semantics for the conditional branching extension with loops were introduced in [TDCB10, TDC10, ADCT11]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 65
  • 66. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Time conditional model of interactive scores 1. Structural definition is given for scores with and without loops 2. Abstract semantics are defined in timed event structures for scores without loops 3. Operational semantics are defined in ntcc, support loops, but are not related to the abstract semantics, as it was the case in the hierarchical model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 66
  • 67. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 67
  • 68. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive scores with signal processing 1. Structural definition [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 68
  • 69. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores with signal processing Karplus (k1) Karplus (k2) Karplus (k3) Output An extension in which ntcc and the synchronous programming language with formal semantics Faust [OFL04] interact. Introduced in [TDCC12] and it is inspired from [All09]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 69
  • 70. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Temporal objects Karplus (k1) Karplus (k2) Karplus (k3) ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Output Temporal objects may contain a Faust audio process, before they used to control external audio processes. Input8 and Output streams are also temporal objects. 8 In this score there is no audio input. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 70
  • 71. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Temporal relations Karplus (k1) Karplus (k2) Karplus (k3) 100smp [2s, 4s] [0s, 0s] [0s, 0s] Output In addition to the usual temporal relations, dashed-line arrows represent high-precision temporal relations. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 71
  • 72. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Dataflow relations Karplus (k1) Karplus (k2) Karplus (k3) Output White-headed arrows represent dataflow relations: To describe how the sound is transferred from one object to another. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 72
  • 73. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Interactive Scores: Interactive objects Karplus (k1) Karplus (k2) Karplus (k3)a b Output Interactive objects are represented (and behave) as usual. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 73
  • 74. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 74
  • 75. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 75
  • 76. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation of Interactive Scores 1. Ntccrt [TAAR09] 2. Results on the conditional branching model [TDC10] 3. Results on the signal processing extension [TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 76
  • 77. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation: Interpreter for ntcc OpenMusic interface Ntccrt compiler Pure Data external Max/Msp external Common Lisp interface C++ interface Stand-alone program User Programmer Ntccrt9 is a simulation tool for ntcc introduced in [TAAR09]. Experience on the usage of Ntccrt is described in [ORS+ 11]. 9 http://sourceforge.net/projects/ntccrt/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 77
  • 78. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation: Example using Pure Data Structural Definition “Bang” objects are controlled by the Ntccrt plugin [TDC10]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 78
  • 79. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Ntccrt 2. Results on the conditional branching model 3. Results on the signal processing extension Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 79
  • 80. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the conditional branching extension 0
 5
 10
 15
 20
 25
 30
 35
 40
 45
 50
 0
 200
 400
 600
 800
 1000
 1200
 1400
 1600
 1800
 Average
dura*on
of
a
*me
unit
(ms)
 Number
of
points
in
the
Interac*ve
Score
 Stress test to compute the average duration of a discrete time unit of the simulation of a conditional branching score using Ntccrt [TDC10]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 80
  • 81. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Ntccrt 2. Results on the conditional branching model 3. Results on the signal processing extension Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 81
  • 82. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension Average relative jitter: The average time difference between the expected starting time of each string, with respect to the first string of an arpeggio, and the time obtained during execution. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 82
  • 83. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension Implementation 85% cpu load 3% cpu load Using object block˜ 1 in Pd 7991 ms 1 ms Using object z˜ 1 in Pd 9231 ms 2 ms Using Faust object in Pd 0.5 ms 0.5 ms Table : Average relative jitter of each string of an arpeggio with respect to the beep. All implementations use the Karplus-Strong algorithm to simulate the sound of the strings [TDCC12]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 83
  • 84. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Results on the signal processing extension In some cases –such as programs with feedback loops– our interactive scores implementation using Faust outperforms Pure Data under high-cpu load. Pure data’s performance is unacceptable, in such cases, for real-time interaction. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 84
  • 85. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Simulation 1. Our models are simulated using Ntccrt, a real-time capable interpreter for ntcc 2. Results on the conditional branching model show that we can execute up to 400 temporal objects 3. Results on the signal processing extension show that interactive scores outperforms Pure Data under high-cpu load Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 85
  • 86. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 86
  • 87. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) [NPV02] 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 87
  • 88. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification of interactive scores • Existing semantics for interactive scores were given in hts petri nets. • Existing verification tools for time Petri nets, such as Rom´eo [GLMR05] and Tapaal [BJS09], cannot be used to verify hts petri nets [SSSW95] because hts petri nets cannot be translated to time Petri nets [BD99]. • Existing verification tools for linear-time logic, such as : Uppaal [BDL+ 01], Kronos [BDM+ 98] and Spin [Hol97], cannot be used directly for ntcc. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 88
  • 89. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification of interactive scores • Results on bisimilarity for ccp [Ari10, ABPV12] are quite promising, but not yet applicable to ntcc. • Strongest post-condition of a process can be translated into a B¨uchi automaton [Val05], but using classic Schimpf’s algorithm [SMS09] is intractable because the encoding heavily relies on the complement of such automata [Saf88]. • We proposed a bounded-time finite-state-automata-based model checker for ntcc, ntccmc10 . 10 http://sourceforge.net/projects/ntccmc/ Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 89
  • 90. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 90
  • 91. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra cltl properties Examples of constraint linear-time logic (cltl) [NPV02] formulae are • “always the constraint pitch = 60 can be deduced from the output store”, namely (pitch = 60); and • “eventually, both object a and object b are launched at the same time”, namely ♦(launcha ˙∧launchb). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 91
  • 92. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Motivation 2. Constraint linear-time temporal logic (cltl) 3. Model checking algorithm Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 92
  • 93. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Is the language empty? Bounded-time automata-based model checking algorithm for ntcc A. intersection P → Pt F → Ft Ft Pt Pt → A Pt → A Ft → Pt Qt A A BF P t A B B Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Adaptation of classic model checking algorithm for linear-time logic [SMS09] to ntcc, based on the encoding of ntcc to B¨uchi automata given by Valencia et al. [Val05]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 93
  • 94. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc Bounded-time automata-based model checking algorithm for ntcc ntcc Process CLTL formulaF P t Number of discrete time units Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 94
  • 95. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Does a ntcc process P satisfy a CLTL formula F for a bounded time t ? Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → FtF P t Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 95
  • 96. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process F P t Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 96
  • 97. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A BF P t Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 97
  • 98. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 98
  • 99. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Bounded-time automata-based model checking algorithm for ntcc A. intersection P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t A B B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 99
  • 100. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Automatic verification of ntcc YES NO Is the language empty? Bounded-time automata-based model checking algorithm for ntcc A. intersection P F ntcc Process CLTL formula P → Pt P → Pt P to bounded-time P F to bounded-time FF → Ft F → Ft Ft Pt Bounded-time process to deterministic automaton (A) Pt → A Pt → A Pt → A Ft → Pt Ft → Pt Qt Bounded-time formula to bounded-time process A A A. complement A BF P t A B B A Automaton Does a a ntcc process P satisfy a CLTL formula F for a bounded time t ? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 100
  • 101. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Verification 1. Existing tools cannot be used directly to verify interactive scores and we want to prove properties of interactive scores automatically 2. In particular, constraint linear-time temporal logic (cltl) properties 3. We presented our bounded-time automata-based model checking algorithm for ntcc, ntccMC, for cltl properties Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 101
  • 102. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Conclusions Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 102
  • 103. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 103
  • 104. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Summary • A Formal model for multimedia interaction that combines • timeline model • cue-list model [TDCR12] • An extension for conditional branching [TDCB10, TDC10, ADCT11] • An extension for audio processing [Tor10, TDCC12] • A simulation tool [TAAR09, ORS+ 11] • A verification tool [Tor10] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 104
  • 105. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Contributions • 2 book chapters in “Constraint Programming in Music” [ADCT11, ORS+ 11] • 1 journal article in “Mathematics and Music” [TDCR12] • 2 papers in “Sound and Music Computing” conference [TDC10, TDCC12] • 1 extended abstract in “International Conference on Logic Programming” [Tor10] • 1 paper in “Journ´ees d’Informatique Musicale” [TDCB10] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 105
  • 106. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 106
  • 107. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • Time models are unrelated • In interactive scores, time models are temporally related Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends time Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 107
  • 108. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • Schedulers are not appropriate for real-time • Using interactive scores and Faust, it is possible to have real-time interaction even under high cpu load Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 108
  • 109. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No hierarchy • Interactive scores provide a hierarchy of temporal objects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 109
  • 110. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No formal semantics • We defined abstract and operational semantics for interactive scores Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 110
  • 111. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Solutions to the problem statements • No unified model for conditional branching and temporal relations • We provide a unified model for conditional branching and temporal relations in which all branches have the same set of possible durations Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 111
  • 112. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 112
  • 113. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Comparison w.r.t previous models of interactive scores • Model to change the hierarchy during performance [OR09] • Previous model developed at LaBRI [All09] • Our models [TDC10, TDCB10, TDCR12, TDCC12] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 113
  • 114. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Model to change the hierarchy during performance • Provides an elegant solution to change the structure of the hierarchy during performance • Has not been implemented Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 114
  • 115. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Previous model developed at LaBRI • Supports different mechanisms of temporal reductions • Average duration of a discrete time unit outperforms our simulations • Durations are intervals are limited to {0}, (0, ∞] and [0, ∞] • Conditional branching and temporal relations are presented separately • Processes associated are meant to be handled outside the formalism; in the implementation as external programs Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 115
  • 116. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Our models • Abstract semantics to describe the execution traces • Arbitrary integer intervals in the operational semantics • Unified model for conditional branching and temporal relations • Automatic verification • Does not allow temporal reductions • Simulation tool is a prototype; it is not user friendly • Average duration of time units is slower than in the implementation of previous model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 116
  • 117. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 117
  • 118. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) • Qlab and Ableton Live do not allow to model delays of 100 samples • Protools does not allow interaction In all the programs mentioned above, it is very hard to synchronize processes whose durations are integer intervals such as duration ∈ [2, 4]. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 118
  • 119. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 119
  • 120. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Future work directions • Event structures semantics for scores with loops • Model interactive theater performances • Model interactive museum exhibitions • Verify scores in which ntcc and Faust interact • Model music pieces combining the timeline and cue-list model Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 120
  • 121. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Music pieces combining both time models Baltazar et al. used i-score to compose miniature # 1 (2012). I-score controls the smoke, fan, sound and lights. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 121
  • 122. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Music pieces combining both time models ./videos/pascal.mp4 Baltazar et al. needed to have objects with flexible duration because the behavior of the smoke is unpredictable. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 122
  • 123. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conclusions 1. Summary 2. Solutions to the problem statements [TDCC12, TDCR12] 3. Comparison w.r.t previous models of interactive scores [TDCR12] 4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab and Protools [TDCC12] 5. Future work directions 6. Ongoing projects Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 123
  • 124. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
  • 125. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
  • 126. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Ongoing projects • Project Interactivity in the Writing of Time and Interactions (inedit) (2012-2015) founded by the french national research agency (anr). • Project Robust theories for Emerging Applications in Concurrency Theory: Processes and Logic Used in emergent Systems(react+) (2011-2013) founded by the colombian national research agency (Colciencias). • Project ossia (2012-2015) founded by the anr. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124
  • 127. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra M E R C I. Avez-vous des questions? Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 125
  • 128. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Outline of the presentation 1. Hierarchical Model 2. Time Conditional Model 3. Signal Processing Model 4. Simulation 5. Verification 6. Summary and Conclusions 7. Extra Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 129. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Antoine Allombert, G´erard Assayag, and Myriam Desainte-Catherine. A system of interactive scores based on petri nets. In Proc. of SMC ’07, Athens, Greece, 2007. Antoine Allombert, G´erard Assayag, and Myriam Desainte-Catherine. Iscore: a system for writing interaction. In Proc. of 3rd International Conference on Digital Interactive Media in Entertainment and Arts (DIMEA) ’08, pages 360–367, New York, NY, USA, 2008. ACM. Antoine Allombert, G´erard Assayag, M. Desainte-Catherine, and Camilo Rueda. Concurrent constraint models for interactive scores. In Proc. of Sound and Music Computing (SMC) ’06, Marseille, France, 2006. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 130. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra G´erard Assayag, Georges Bloch, Marc Chemillier, Arshia Cont, and Shlomo Dubnov. Omax brothers: a dynamic topology of agents for improvization learning. In Proc. of the 1st ACM workshop on Audio and music computing multimedia (AMCMM) ’06, pages 125–132, New York, NY, USA, 2006. ACM. Antoine Allombert, Pascal Baltazar, Rapha¨el Marczak, Myriam Desainte-Catherine, and Laurent Garnier. Designing an interactive intermedia sequencer from users requirements and theoretical background. In Proc. of International Computer Music Conference (ICMC) ’10, 2010. A. Aristizabal, F. Bonchi, L. Pino, and F. Valencia. Partition refinement for bisimilarity in ccp. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 131. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of the 27th ACM Symposium On Applied Computing (SAC 2012), 2012. Antoine Allombert and Myriam Desainte-Catherine. Interactive scores: A model for specifying temporal relations between interactive and static events. In Journal of New Music Research, 2005. Antoine Allombert, Myriam Desainte-Catherine, J. Larralde, and G´erard Assayag. A system of interactive scores based on qualitative and quantitative temporal constraints. In Proc. of 4th International Conference on Digital Arts (Artech) ’08, Porto, Portugal, 2008. The Artech International Association. Antoine Allombert, Myriam Desainte-Catherine, and Mauricio Toro. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 132. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Modeling temporal constrains for a system of interactive score. In G´erard Assayag and Charlotte Truchet, editors, Constraint Programming in Music, chapter 1, pages 1–23. Wiley, 2011. James F. Allen. Maintaining knowledge about temporal intervals. Communication of ACM, 26, 1983. Antoine Allombert. Aspects temporels d’un syst`eme de partitions num`eriques interactives pour la composition et l’interpr´etation. PhD thesis, Universit´e de Bordeaux, November 2009. Andr´es Aristiz´abal. Bisimilarity in concurrent constraint programming. In 26th International Conference on Logic Programming (ICLP) 2010, 2010. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 133. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Jean Bresson, Carlos Agon, and G´erard Assayag. Openmusic: visual programming environment for music composition, analysis and research. In Proceedings of the 19th ACM international conference on Multimedia, MM ’11, pages 743–746, New York, NY, USA, 2011. ACM. P. Baltazar, A. Allombert, R. Marczak, J.M. Couturier, M. Roy, A. S`edes, and M. Desainte-Catherine. Virage : Une reflexion pluridisciplinaire autour du temps dans la creation numerique. In in Proc. of JIM, 2009. Marc Boyer and Michel Diaz. Non equivalence between time petri nets and time stream petri nets. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 134. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of the The 8th International Workshop on Petri Nets and Performance Models, number 98 in 1, Washington, DC, USA, 1999. IEEE Computer Society. Anthony Beuriv´e and Myriam Desainte-Catherine. Representing musical hierarchies with constraints. In 7th International Conference on Principles and Practice of Constraint Programming, Musical Constraints Workshop, Paphos, 2001. Gerd Behrmann, Alexandre David, Kim G. Larsen, Oliver M¨oller, Paul Pettersson, and Wang Yi. Uppaal - present and future. In Proc. of 40th IEEE Conference on Decision and Control. IEEE Computer Society Press, 2001. Marius Bozga, Conrado Daws, Oded Maler, Alfredo Olivero, Stavros Tripakis, and Sergio Yovine. Kronos: A model-checking tool for real-time systems. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 135. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Anders Ravn and Hans Rischel, editors, Formal Techniques in Real-Time and Fault-Tolerant Systems, volume 1486 of Lecture Notes in Computer Science, pages 298–302. Springer Berlin / Heidelberg, 1998. Joakim Byg, Kenneth Yrke Jørgensen, and Jiˇr´ı Srba. Tapaal: Editor, simulator and verifier of timed-arc petri nets. In Proceedings of the 7th International Symposium on Automated Technology for Verification and Analysis (ATVA) ’09, pages 84–89, Berlin, Heidelberg, 2009. Springer-Verlag. Christel Baier, Joost-Pieter Katoen, and Diego Latella. Metric semantics for true concurrent real time. In Proc. of International Conference on Automata, Languages and Programming (ICALP) ’98, Berlin, Germany, 1998. Springer. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 136. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Arshia Cont. Antescofo: Anticipatory synchronization and control of interactive parameters in computer music. In Proc. of ICMC ’08, 2008. Myriam Desainte-Catherine, Antoine Allombert, and G´erard Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Computer Music Journal, To appear in fall, 2012. M. Desainte-Catherine and N. Brousse. Towards a specification of musical interactive pieces. In Proc. of the XIX Colloquium on Musical Informatics (CIM), Firenze, Italy, 2003. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 137. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Jos´e Echeveste, Arshia Cont, Jean-Louis Giavitto, and Florent Jacquemard. Formalisation des relations temporelles entre une partition et une performance musicale dans un contexte d’accompagnement automatique : Accompagnement musical automatique. In Journal Europ´een des Syst`emes Automatis´es, 2011. Rosella Gennari. Temporal resoning and constraint programming - a survey. CWI Quaterly, 11:3–163, 1998. Thierry Gautier, Paul Le Guernic, and L¨oic Besnard. Signal: A declarative language for synchronous programming of real-time systems. In Proc. of FPCA ’87, 1987. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 138. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Guillaume Gardey, Didier Lime, Morgan Magnin, and Olivier (H.) Roux. Rom´eo: A tool for analyzing time Petri nets. In 17th International Conference on Computer Aided Verification (CAV’05), volume 3576 of Lecture Notes in Computer Science, pages 418–423, Edinburgh, Scotland, UK, 2005. Springer. Gerard J. Holzmann. The model checker spin. Software Engineering, 23(5):279–295, 1997. Rapha¨el Marczak, Antoine Allombert, and Myriam Desainte-Catherine. Real-time temporal control of musical processes. In Proc. of the International Conferences on Advances in Multimedia (MMEDIA) ’11, 2011. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 139. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Itay Meiri. Combining qualitative and quantitative constraints in temporal reasoning. Artificial Inteligence, 87(1-2):343–385, 1996. M. Nielsen, C. Palamidessi, and F. Valencia. Temporal concurrent constraint programming: Denotation, logic and applications. Nordic Journal of Computing, 1(9):145–188, 2002. Yann Orlarey, Dominique Fober, and Stephane Letz. Syntactical and semantical aspects of faust. Soft Comput., 8(9):623–632, 2004. Carlos Olarte and Camilo Rueda. A Declarative Language for Dynamic Multimedia Interaction Systems. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 140. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of Mathematics and Computation in Music, volume 38, Berlin, Germany, july 2009. Springer. Carlos Olarte, Camilo Rueda, Gerardo Sarria, Mauricio Toro, and Frank Valencia. Concurrent constraints models of music interaction. In G´erard Assayag and Charlotte Truchet, editors, Constraint Programming in Music, chapter 6, pages 133–153. Wiley, Hoboken, NJ, USA., 2011. Carlos Olarte and Frank D. Valencia. The expressivity of universal timed ccp: undecidability of monoadic fltl and closure operators for security. In PPDP ’08: Proceedings of the 10th international ACM SIGPLAN conference on Principles and practice of declarative programming, pages 8–19. ACM, 2008. Fran¸cois Pachet. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 141. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Playing with virtual musicians: the continuator in practice. IEEE Multimedia, 9:77–82, 2002. Miller Puckette, Theodore Apel, and David Zicarelli. Real-time audio analysis tools for pd and max/msp. In Proc. of ICMC ’98, Ann Arbor, USA, 1998. M. Puckette. Pure data. In Proceedings of the International Computer Music Conference. San Francisco 1996, 1996. Nirina Ranaivoson. R´eflexion sur la mise en place des structures logiques dans un logiciel de partitions musicales interactives. Master’s thesis, Universit´e de Bordeaux, 2009. Camilo Rueda and Frank Valencia. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 142. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Proving musical properties using a temporal concurrent constraint calculus. In Proc. of the 28th International Computer Music Conference (ICMC) ’02, 2002. C. Rueda and F. Valencia. On validity in modelization of musical problems by ccp. Soft Computing, 8(9):641–648, 2004. Camilo Rueda and Frank Valencia. A temporal concurrent constraint calculus as an audio processing framework. In SMC ’05, 2005. S. Safra. On the complexity of omega-automata. In Proceedings of the 29th Annual Symposium on Foundations of Computer Science (SFCS) ’88, pages Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 143. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra 319–327, Washington, DC, USA, 1988. IEEE Computer Society. Vijay A. Saraswat. Concurrent Constraint Programming. MIT Press, Cambridge, MA, 1992. Gerardo Sarria. Formal Models of Timed Musical Processes. PhD thesis, Universidad del Valle, Colombia, 2008. Vijay A. Saraswat, Radha Jagadeesan, and Vineet Gupta. Foundations of timed concurrent constraint programming. In Proceedings of the Ninth Annual IEEE Symposium on Logic in Computer Science, pages 71–80. IEEE Computer Press, 1994. Alexander Schimpf, Stephan Merz, and Jan-Georg Smaus. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 144. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Construction of b¨uchi automata for ltl model checking verified in isabelle/hol. In Proceedings of the 22nd International Conference on Theorem Proving in Higher Order Logics, TPHOLs ’09, pages 424–439, Berlin, Heidelberg, 2009. Springer-Verlag. Patrick S´enac, Pierre de Saqui-Sannes, and Roberto Willrich. Hierarchical time stream petri net: A model for hypermedia systems. In Proc. of the 16th International Conference on Application and Theory of Petri Nets, pages 451–470, London, UK, 1995. Springer-Verlag. Mauricio Toro, Carlos Ag´on, G´erard Assayag, and Camilo Rueda. Ntccrt: A concurrent constraint framework for real-time interaction. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 145. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of ICMC ’09, Montreal, Canada, 2009. Mauricio Toro and Myriam Desainte-Catherine. Concurrent constraint conditional branching interactive scores. In Proc. of SMC ’10, Barcelona, Spain, 2010. Mauricio Toro, Myriam Desainte-Catherine, and P. Baltazar. A model for interactive scores with temporal constraints and conditional branching. In Proc. of Journ´ees d’Informatique Musical (JIM) ’10, May 2010. Mauricio Toro, Myriam Desainte-Catherine, and Julien Castet. An extension of interactive scores for multimedia scenarios with temporal relations for micro and macro controls. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 146. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra In Proc. of Sound and Music Computing (SMC) ’12, Copenhagen, Denmark, July 2012. Mauricio Toro, Myriam Desainte-Catherine, and Camilo Rueda. Formal semantics for interactive music scores: A framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music, To be published., 2012. Paul Teehan, Mark R. Greenstreet, and Guy G. Lemieux. A survey and taxonomy of gals design styles. IEEE Design Test of Computers, 24(5):418–428, 2007. Mauricio Toro. Structured musical interactive scores (short). In Proc. of the doctoral consortium in ICLP ’10, Edinburgh, Scotland, UK, 2010. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 147. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Mauricio Toro. Structured Interactive Scores: From a simple structural definition to a real-time capable implementation. PhD thesis, Universit´e de Bordeaux, Bordeaux, September 2012. Frank D. Valencia. Temporal Concurrent Constraint Programming. PhD thesis, University of Aarhus, 2002. Frank D. Valencia. Decidability of infinite-state timed ccp processes and first-order ltl. Journal of Theoretical Computer Science - Expressiveness in concurrency, 330(3):557–607, 2005. Lindsay Vickery. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126
  • 148. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Non-linear structures for real-time interactive musical works. In Proceedings of the Australasian Computer Music Conference (ACMC) ’03, 2003. Lindsay Vickery. Interactive control of higher order musical structures. In Proc. of ACMC ’04, Victoria University, New Zealand, July 2004. ACMA. Lindsay Vickery. The possibilities of novel format structures through computer controlled live performance. In Proc. of ACMC ’11, 2011. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127
  • 149. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra More Future Work Score with conditions Event Structures semantics Block Diagram semantics Abstract semantics Structural Definition Ntcc model Ntcc and Faust interconnected Model checker Temporal Property Operational semantics Pure Data Program Score with audio proc. Score with interval durations Score with arbitrary durations Event Structures semantics Event Structures semantics Stand Alone Programs Programs Music Properties Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127
  • 150. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Limitation of time conditional branching b e Choose either d or e after aa c d ∆c = {3} ∆e = {2} ∆d = {1} ∆b = [1, 2] = = = = = = = Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 128
  • 151. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail Red (r) ∆r ∆r sr er ∆r s r e r Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 129
  • 152. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail φ ∆ ∆ ∆ ∆ ∆ ∆a ∆b ∆a ∆b ∆a Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 130
  • 153. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Conditional branching semantics in detail Green (g) Red (r) Lights (l) a ∆l ∆r ∆g ∆a = {0} ∆r ∆r ∆g ∆g sr sr er er sg sg eg eg a sl el ∆l ∆l Video (e) ee e e e l s l φ ≥ 10 ∆ = {0} Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 131
  • 154. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Equivalence between interactive score constraints and its event traces Theorem The temporal constraints of a score are equivalent to the temporal constraints of the event structures semantics. Proof. This holds by replacing points in the temporal constraints of the score by its corresponding events. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 132
  • 155. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics Theorem The possible time units in which a point process can be launched in the operational semantics are included in the possible time units in which an event can appear in an event trace in the tes semantics [TDCR12]. Proof. By induction on the structure of the timed event structures (tes) semantics of a score. Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 133
  • 156. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics Theorem Let s be a score, ε∗ = des(normal(es(s))) = E, l∗ , R its event structures semantics, tc(ε∗ ) its temporal constraints, P = ScoreS,I,Pr,R the ntcc process that represents the score, [[P]] the denotation of the ntcc process, and Ti is the set of indexes j such that [[P]]j+1 entails launchi . It holds for all sequences in [[P]], n = card(E), that T1 × T2... × Tn ⊆ Solutions(tc(ε∗ )). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 134
  • 157. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Correctness of the operational semantics with respect to the event structures semantics ε∗ Values of Ti for process Scoreε∗ tc(ε∗ ) ((i)) Ti ⊆ [1, n∞] ti ∈ N (i) ∆ → (j) Ti = {1} ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti + ((i)) ∆ → (j) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti + ((i)) ∆ → ((j)) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti + (i) ∆ → ((j)) Ti = {1} ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti + Table : Temporal constraints to launch the events in the ntcc model Vs. temporal constraints of the event structures semantics. We denote an interactive object as ((i)) and a static one as (i). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 135
  • 158. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 136
  • 159. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra The novelty of our extension: Faust and ntcc interconnection 1. A model based on the Non-deterministic Timed Concurrent Constraint (ntcc) calculus for concurrency, user interactions and temporal relations, and 2. a model based on the Faust programming language for sound processing and micro controls. buton Label mouse down mouse up 0 1 user ntcc Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 137
  • 160. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Temporal object Temporal relation High-precision temporal relation Dataflow relation Interactive object Interactive score Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 138
  • 161. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 139
  • 162. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings a b [2s, 4s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Figure : The constraint graph Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 140
  • 163. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3) @100 threeCords(f) output sk1 ek1 sk1 ek2 sk2 Figure : Faust’s Block diagram Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 141
  • 164. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Example: An arpeggio with three strings Figure : Implementation Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 142
  • 165. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: Three user controlled arpeggios ∆ = 10 ∆ = 10 ∆ = 10 Figure : The double-headed arrow represents an inequality (≤) and a white-headed arrow represents an equality relation (=). Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 143
  • 166. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: An arpeggio without “clicks” a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1) Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s] Karplus (k2) Karplus' AC [0.5s, 0.5s] ∆k2 = [5s, 10s] [4.5s, 9.5s] Karplus (k3) Karplus' AC [0.5s, 0.5s] ∆k3 = [4s, 4s] [3.5s, 3.5s] Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 144
  • 167. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra Application: Sound source perception Karplus (k1) L Output (o1) [0s, 0s] ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] Karplus (k1) L Output (o1) ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] 500µs Time Time Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 145