The Green Lab - [05 B] Experiment design (advanced)
1 like1,844 views
This document discusses experiment design. It describes different types of experiment designs including those with one or multiple factors and treatments. More complex designs involve interactions between factors. Factorial designs can test all combinations of factors but become infeasible with many factors. Latin square designs provide an alternative for multiple factors. Proper instrumentation is also needed to monitor experiments without influencing the results. Overall, the key lessons are to choose a simple design when possible, maximize the use of available objects, and refer to checklists to design good experiments.
The Green Lab - [05 B] Experiment design (advanced)
- 1. 1 Het begint met een idee Experiment design (advanced) Ivano Malavolta
- 2. Vrije Universiteit Amsterdam Advanced design types Instrumentation 2 Roadmap Ivano Malavolta / S2 group / Experiment design
- 3. Vrije Universiteit Amsterdam 3 Line of reasoning Ivano Malavolta / S2 group / Experiment design
- 4. Vrije Universiteit Amsterdam 4 Advanced design types We can have the following cases: ● 1 factor and 2 treatments (1F-2T) ● 1 factor and >2 treatments (1F-MT) ● 2 factors and 2 treatments (2F-2T) ● >2 factors, each one with >=2 treatments (MF-MT) Ivano Malavolta / S2 group / Experiment design
- 5. Vrije Universiteit Amsterdam 5 2 factors ● Things become more complex ● Interaction must be modeled: ○ i : effect of treatment i (level of factor A) ○ j : effect of treatment j (level of factor B) ○ ( )ij : effect of the interaction between i and j ● Two factors may interact with each other Ivano Malavolta / S2 group / Experiment design
- 6. Vrije Universiteit Amsterdam 6 Definition of effect Ivano Malavolta / S2 group / Experiment design ● Null hypothesis: no difference in means ● Typical model of an outcome (dependent variable): ○ Yij = + Ti + error Outcome for subject j under treatment i Average of all values Effect of treatment i (offset)
- 7. Vrije Universiteit Amsterdam 7 Definition of effect Ivano Malavolta / S2 group / Experiment design ● = avg(Y) ● Ti = avg(Yi ) - avg(Y) + Ti (Y) Y
- 8. Vrije Universiteit Amsterdam 8 Interaction Interaction: non-additive effects between factors Additive model Yij = + i + j + error Non-Additive model Yij = + i + j + ( )ij + error https://goo.gl/XUq8tp
- 9. Vrije Universiteit Amsterdam 9 2 factors, 2 treatments (2F-2T) Ivano Malavolta / S2 group / Experiment design ● Example: ○ Factor 1: Connection Protocol ■ Treatment 1: HTTP ■ Treatment 2: HTTPS ○ Factor 2: Sorting Algorithm ■ Treatment 1: BubbleSort ■ Treatment 2: QuickSort
- 10. Vrije Universiteit Amsterdam 10 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design ● Consider all possible combinations of treatments ● Each treatment is randomly assigned to experimental objects ● Balanced design: each combination is assigned to an equal number of objects Factor 1: Connection Protocol Treatment 1: HTTP Treatment 2: HTTPS Factor 2: Sorting Treatment 1: BubbleSort Application 4,6 Application 1,7 Treatment 2: QuickSort Application 2,3 Application 5,8
- 11. Vrije Universiteit Amsterdam 11 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design ● i : mean of the dependent variable for treatment i ● i =avg(P) ● i : effect of treatment i (HTTP/S) of factor A (Conn. Protocol) ● j : effect of treatment j (Bubble/Quick) of factor B (Sorting) ● ( )ij : effect of the interaction between i and j
- 12. Vrije Universiteit Amsterdam 12 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design ● Null hypothesis: H0A : 1 = 2 = 0 ● Null hypothesis: H0B : 1 = 2 = 0 ● Null hypothesis: H0AB : ( )ij = 0 ∀ i,j
- 13. Vrije Universiteit Amsterdam 13 2F-2T: factorial design Ivano Malavolta / S2 group / Experiment design ● Alternative hypothesis: H1A : ∃ i | i ≠ 0 ● Alternative hypothesis: H1AB : ∃ (i,j) | ( )ij ≠ 0 ● Alternative hypothesis: H1B : ∃ j | j ≠ 0
- 14. Vrije Universiteit Amsterdam 14 2F-2T: 2-stage nested design Ivano Malavolta / S2 group / Experiment design ● Example: ○ Factor 1: Interface ■ Treatment 1: Web-based ■ Treatment 2: Client-based ○ Factor 2: Programming Language ■ Treatment 1,1: PHP ■ Treatment 1,2: ASP ■ Treatment 2,1: Java ■ Treatment 2,2: C++
- 15. Vrije Universiteit Amsterdam 15 2F-2T: 2-stage nested design Ivano Malavolta / S2 group / Experiment design ● One of the two factors has different treatments with respect to the other factor ● Balanced design, randomized application Factor 1: Interface Treatment 1: Web-based Treatment 2: Client-based Factor 2: Programming Language Factor 2: Programming Language Treatment 1,1: PHP Treatment 1,2: ASP Treatment 2,1: Java Treatment 1,2: C++ Application 1,3 Application 6,2 Application 7,8 Application 5,4
- 16. Vrije Universiteit Amsterdam 16 More than 2 factors Ivano Malavolta / S2 group / Experiment design ● Number of experimental groups explodes ● Total number of trials is at least n*n ● More trials = more subjects (larger sample size)
- 17. Vrije Universiteit Amsterdam 17 Factorial designs FULL-COVERAGE ○ every possible combination of all the alternatives of all the factors N = #trials k = #factors ni = #levels for i-th factor
- 18. Vrije Universiteit Amsterdam 18 Factorial designs Discovers the effects of each factor and its interactions with the other factors BEWARE: combinatorial curse N = #trials k = #factors ni = #levels for i-th factor SOLUTIONS?
- 19. Vrije Universiteit Amsterdam 19 Latin square designs Ivano Malavolta / S2 group / Experiment design ● Latin Square: an n x n array with n different symbols ● Divide factors in main factor and co-factors (or blocking factor) ○ Levels of the main factors are the "letters" ○ Levels of the co-factors are rows and columns ● All levels of the main factor occurs per each blocking factor
- 20. Vrije Universiteit Amsterdam 20 Latin square designs Ivano Malavolta / S2 group / Experiment design ● 3 factors ○ Code Size: Small, Medium, Large (Main Factor) ○ Programming Language: Java, C++, C (Blocking Factor) ○ Operating System: Windows, Linux, OS X (Blocking Factor) ● Total number of groups: ○ Full 33 factorial design: 27 ○ Latin Square: 9
- 21. Vrije Universiteit Amsterdam 21 Latin square designs Ivano Malavolta / S2 group / Experiment design Factor 1: Programming Language Treatment 1: Java Treatment 2: C++ Treatment 3: C Factor 2: Operating System Treatment 1: Windows Small Medium Large Treatment 2: Linux Medium Large Small Treatment 2: OS X Large Small Medium
- 22. Vrije Universiteit Amsterdam 22 Pitfalls of Latin squares Ivano Malavolta / S2 group / Experiment design ● Incomplete or partial design ○ Key: balancing + randomization ● Assumption: factors do not interact ● Randomization is limited by design
- 23. Vrije Universiteit Amsterdam 23 Latin squares applicabile to 1-factor studies ● 1 factor - 1 block
- 24. Vrije Universiteit Amsterdam 24 Latin squares applicabile to 1-factor studies ● 1 factor - >2 blocks It is equivalent to superimposing two different Latin squares:
- 25. Vrije Universiteit Amsterdam 25 Instrumentation Ivano Malavolta / S2 group / Experiment design
- 26. Vrije Universiteit Amsterdam 26 Instrumentation ● Instrumentation must not affect our control of the experiment Goal: Provide a way to conduct and monitor the experiment ● Types of instrumentation: ○ Objects (e.g. servers, apps) ○ Guidelines (checklists, documentation) ○ Measurement tools (power meters, profiling software, etc.) Ivano Malavolta / S2 group / Experiment design
- 27. Vrije Universiteit Amsterdam ● You know how to design an experiment ● Experiment design is an essential choice when doing an experiment ● Constraints on statistical methods ● If possible, use a simple design ● Maximize the usage of the available objects ○ automation may be your friend here 27 What this lecture means to you? Ivano Malavolta / S2 group / Experiment design
- 28. Vrije Universiteit Amsterdam 28 Ivano Malavolta / S2 group / Experiment design Readings Chapter 8 Chapter 5 IMPORTANT - Checklist for making a good experiment design (section 5.9)