QOrchestra and FlowML. High-level audio synthesis tools on top of SAOL
- 1. Bert Schiettecatte, VUB QOrchestra & FlowML High-level audio synthesis tools on top of SAOL
- 2. Introduction (1) This presentation: • Introduce an audio synthesis diagram format (FlowML) • Present a compiler from FlowML to SAOL (FlowML2SAOL) • Present a high-level authoring tool for SAOL (QOrchestra)
- 3. Introduction (2) The SA (Structured Audio) standard of Mpeg-4 audio consists of 5 elements: • SAOL (SA Orchestra Language): a signal- processing language for describing musical instruments • SASL (SA Score Language): a language for describing a performance using the instruments of a SAOL program
- 4. Introduction (3) • SASBF (SA Sample Bank Format): allows transmission of sample banks • Scheduler description: used to translate MIDI or SASL to events internally, in a decoder • Reference to MIDI: supported MIDI messages and their meaning
- 5. Available technology (1) Command-line encoders, decoders, profilers: • saolc: reference implementation of encoder/decoder • sfront: SAOL-to-C compiler with realtime I/O and some profiling support • sarun: compiles SAOL to a program for a virtual machine with a special instruction set, for efficient decoding
- 6. Available technology (2) Authoring tools: • QOrchestra: high-level synthesis diagram tool with SAOL export function, “non- realtime”, supports re-use of diagrams • CPS: low-level software synthesis tool with SAOL save function, “realtime” • VisOrc: Similar to CPS, with integrated sequencer
- 7. Available technology (3) IDEs: • SFX: SAOL editing environment with syntax highlighting, auditioning, some profiling support
- 8. QOrchestra (1) • Goal: experimental visual authoring tool for SAOL (GPL license), drawing space with primitive building blocks, connection tool, library for new blocks, … • Main idea: tool for people who don’t know SAOL, or don’t want to write SAOL code • Also a goal: case study of a complex diagram editor and object-oriented programming technology
- 9. QOrchestra (2) QOrchestra is NOT: • A low-level SAOL editor (e.g. CPS) • A complete SA authoring environment • A real-time software synthesis application • An IDE • Finished
- 10. QOrchestra (3)
- 11. FlowML (1) • QOrchestra project requirement: save, load and re-use synthesis diagrams • Available storage technology: XML (eXtensible Markup Language), a way to add meta-information to information (e.g. <author>Bert Schiettecatte</author>)
- 12. FlowML (2) • Tags for a specific document class are described in an XML DTD (document type definition) • XML technology is freely available, tested, reliable, very easy to use, … • FlowML = XML DTDs describing synthesis diagram formats on a high level + collection of synthesis constructs + synthesis environment-specific extension mechanism (~ C++ pragma’s)
- 13. FlowML (3) • Solved storage problems in QOrchestra: store diagrams and libraries as FlowML • Generating SAOL code from FlowML is relatively easy • Very easy to understand (stored as text file) • Could be a bridge between various software synthesis systems • Could be the right high-level tool for describing big synthesis systems (<> SAOL)
- 14. FlowML (4) • “Web-friendly”: easy to create browser plugins • Is not a programming language, but a high- level modelling language, in a way similar to UML (methodology for structuring OO designs) • FlowML is open and free • Extensions for future versions can be proposed and will be examined prior to becoming official (same license as HTML)
- 15. FlowML (5) FlowML diagram rules (informal): • Multiple outputs per block • Multiple connections per output • Multiple inputs per block • Only one connection per input • All signals are mono
- 16. FlowML (6) FlowML diagram rules (informal): • Every component instance has a unique ID used for referring • Every definition contains a description and a unique name used for instantiating • 2 types of signals: audio and control • Diagram/component I/O is done through special primitives defined in the standard
- 17. FlowML (7) Audio synthesis diagram example: Reverb mixer lvl1 lvl2 in1 in2 out delay audio_in time_in out xfade in1 in2 x out audio_in out out audio_out in in time_in out fback_in out level out drywet out
- 18. FlowML code for the reverb diagram FlowML (8) <?xml version='1.0'?> <GraphPrototype name="revbfilt"> <Description> Reverb (filtered delay). </Description> <Implementation> <Instance id="audio_in" type="audio_in" x="0" y="0"> <Connection to="in1" from="out" targetid="mixer"/> <Connection to="in1" from="out" targetid="xfade"/> </Instance> <Instance id="fback_in" type="control_in" x="0" y="0"> <Connection to="lvl2" from="out" targetid="mixer"/> </Instance> <Instance id="level" type="control_in" x="0" y="0"> <Connection to="lvl1" from="out" targetid="mixer"/> </Instance> <Instance id="mixer" type="mixer" x="0" y="0"> <Connection to="in" from="out" targetid="dly"/> </Instance> <Instance id="time_in" type="control_in" x="0" y="0"> <Connection to="time" from="out" targetid="dly"/> </Instance>
- 19. FlowML (9) FlowML code for the reverb diagram (cont’d) <Instance id="dly" type="delay" x="0" y="0"> <Connection to="in2" from="out" targetid="xfade"/> <Connection to="in2" from="out" targetid="mixer"/> </Instance> <Instance id="xfade" type="xfade" x="0" y="0"> <Connection to="in" from="out" targetid="audio_out"/> </Instance> <Instance id="drywet" type="control_in" x="0" y="0"> <Connection to="x" from="out" targetid="xfade"/> </Instance> <Instance id="audio_out" type="audio_out" x="0" y="0"/> </Implementation> </GraphPrototype>
- 20. SAOL Generator (1) • Originally intended to be a part of QOrchestra, but now a separate program • Reads FlowML synthesis diagrams and generates SAOL code with the behaviour of the diagram • Uses libraries of SAOL opcodes to simplify generated code: every block in the diagram translates to an opcode call
- 21. SAOL Generator (2) • Will support libraries containing FlowML diagrams and SAOL code (currently only FlowML) • Generated code will require preprocessing (e.g. #include) • Problem: separation authoring tool (including synthesizer) & code generator can cause differences in resulting sound
- 22. SAOL Generator (3) • Problem: Integrating software synthesizer/code generator might cause a problem: what about intellectual property?
- 23. SAOL Generator (4) generateOpcDecl(Prototype, Program, DeclsDone) VarDecls = new List Code = new List InstsDone = new Set Params = getInputNames(P) Outputs = getOutputNames(P) foreach O in Outputs add(Results, generateOpcResult(Prototype, O, Null, Program, VarDecls, Code, DeclsDone, InstsDone)) add(Code, makeReturn(Results)) return makeOpcDecl(getRate(getElement(Outputs)), getName(P), Params, VarDecls, Code) Code generation: opcode declaration
- 24. SAOL Generator (5) generateOpcResult(Prototype, N, C, Program, VarDecls, Code, DeclsDone, InstsDone) if isOutput(Prototype, N) X = getConnectionTo(N) return generateOpcResult(Prototype, getFrom(X), getFromConnector(X), Program, VarDecls, Code, DeclsDone, InstsDone) else if isInput(Prototype, N) return makeVarRef(N) else if contains(InstsDone, N) if hasMultipleOutputs(N) return makeArrayLookup(last(N), getNumber(N, C)) else return makeVarRef(last(N)) Code generation: opcode result (1/3) Output instance Input instance Instance already calculated
- 25. SAOL Generator (6) else add(InstsDone, N) if isGraphPrototype(N) and not contains(DeclsDone, N) add(DeclsDone, N) add(Program, generateOpcDecl(N, Program, DeclsDone)) Params = new List ConnectionsTo = getConnectionsTo(N) foreach X in ConnectionsTo add(Params, generateOpcResult(Prototype, getFrom(X), getFromConnector(X), Program, VarDecls, Code, DeclsDone, InstsDone)) A = makeOpcCall(N, Params) R = getRate(N, C) S = getNrOutputs(N) Code generation: opcode result (2/3) Instance in middle of diagram Satisfy instance inputs
- 26. SAOL Generator (7) if hasMultipleOutputs(N) or hasMultipleConnectionsPerOutput(N) add(Code, makeAssignment(last(N), next(N))) add(Code, makeAssignment(next(N), A) if hasMultipleOutputs(N) add(Vars, makeArrayDecl(last(N), R, S)) add(Vars, makeArrayDecl(next(N), R, S)) return makeArrayLookup(last(N), getNumber(N, C)) else if hasMultipleConnectionsPerOutput(N) add(Vars, makeVarDecl(last(N), R)) add(Vars, makeVarDecl(next(N), R)) return makeVarRef(last(N)) else return A Code generation: opcode result (3/3)
- 27. SAOL Generator (8) aopcode revbfilt(asig audio_in, ksig drywet, ksig fback_in, ksig level, ksig time_in) { asig dly_next, dly_last; dly_last = dly_next; dly_next = delay(mixer(audio_in, level, dly_last, fback_in), time_in); return xfade(drywet, audio_in, dly_last); } Generated SAOL code (reverb diagram) mixer lvl1 lvl2 in1 in2 out delay audio_in time_in out xfade in1 in2 x out audio_in out out audio_out in in time_in out fback_in out level out drywet out
- 28. The future • FlowML needs input to mature and become really interesting • Free software (e.g. FlowML parser, SAOL and CSound code generators, …) needs to be developed • Serious authoring tools are needed to make Mpeg-4 popular • Artists need to make a difficult transition
- 29. Conclusion (1) • FlowML might be interesting as a bridge between synthesis systems • FlowML hides much of the engineering side of audio synthesis and provides a high-level description format for musicians • FlowML allows to prototype a synthesizer quickly before optimizing it (e.g. compile to SAOL and hand-optimize)
- 30. Conclusion (2) • The popularity of SAOL will depend on its support in commercially available software synthesis applications
- 31. References and contact info • FlowML and QOrchestra http://www.bertschiettecatte.com/ • Mpeg-4 SA at the MLG, MIT Media Lab http://sound.media.mit.edu/mpeg4/ • Mpeg-4 SA book at Berkeley http://www.cs.berkeley.edu/~lazzaro/