Parallel R
6 likes3,906 views
This document discusses parallel MCMC and random number generators for parallel MCMC in R. It introduces parallel computing packages in R like foreach and parallel that allow running MCMC chains concurrently. It emphasizes the importance of using different random number generator seeds and initial values for each chain to ensure independence. The document also summarizes some random number generators available in R like the Mersenne Twister default RNG and others in the base package.
Parallel R
- 1. Parallel MCMC Random Number Generators Summary Parallel Bayesian computation in R ≥ 2.14 using the packages foreach and parallel Matt Moores Cathy Hargrave Bayesian Research & Applications Group Queensland University of Technology, Brisbane, Australia CRICOS provider no. 00213J Thursday September 27, 2012 BRAG Sept. 27 Parallel MCMC in R
- 2. Parallel MCMC Random Number Generators Summary Outline 1 Parallel MCMC Introduction R packages 2 Random Number Generators RNG and parallel MCMC RNGs available in R BRAG Sept. 27 Parallel MCMC in R
- 3. Parallel MCMC Introduction Random Number Generators R packages Summary Motivation Why parallel? large datasets many MCMC iterations multiple CPU cores now commonplace eg. Intel Core i5 and i7 even mobile phones have multicore CPUs BRAG Sept. 27 Parallel MCMC in R
- 4. Parallel MCMC Introduction Random Number Generators R packages Summary Parallel MCMC 2 kinds of parallelism: concurrent MCMC chains always applicable straightforward to implement concurrent updates within an iteration only useful for a very large parameter space ideally in a compiled language (eg. Rcpp with OpenMP) also implicit parallelism, eg. with Intel Math Kernel Library BRAG Sept. 27 Parallel MCMC in R
- 5. Parallel MCMC Introduction Random Number Generators R packages Summary Concurrent Chains BRAG Sept. 27 Parallel MCMC in R
- 6. Parallel MCMC Introduction Random Number Generators R packages Summary Simple Network Of Workstations R package snow by Luke Tierney, et al. spawns multiple copies of R provides several options for inter-process communication TCP sockets available on any platform, including Microsoft Windows Message Passing Interface (via the package Rmpi) Parallel Virtual Machine (via the package rpvm) NetWorkSpaces (via the package nws) can either run on a local machine or a cluster (eg. Lyra) BRAG Sept. 27 Parallel MCMC in R
- 7. Parallel MCMC Introduction Random Number Generators R packages Summary multicore R package by Simon Urbanek implemented using the POSIX fork system call available on Linux and Mac OS X clones the R instance (functions + data) takes advantage of copy-on-write will fork as many processes as there are available CPU cores, unless told otherwise BRAG Sept. 27 Parallel MCMC in R
- 8. Parallel MCMC Introduction Random Number Generators R packages Summary parallel R package parallel included in the core R distribution available in versions ≥ 2.14.0 incorporates subsets of snow, multicore, and rlecuyer sensible default behaviour BRAG Sept. 27 Parallel MCMC in R
- 9. Parallel MCMC Introduction Random Number Generators R packages Summary foreach "syntactic sugar" § l i b r a r y ( foreach ) library ( parallel ) library ( doParallel ) # w i l l a u t o m a t i c a l l y use a SOCK c l u s t e r on Windows # ( o t h e r w i s e uses m u l t i c o r e ) r e g i s t e r D o P a r a l l e l ( cores = d e t e c t C o r e s ( ) ) f o r e a c h ( i =1: getDoParWorkers ( ) ) %dopar% { # t h i s code w i l l be executed c o n c u r r e n t l y ... } BRAG Sept. 27 Parallel MCMC in R
- 10. Parallel MCMC Introduction Random Number Generators R packages Summary foreach with SNOW § l i b r a r y ( foreach ) library ( parallel ) library ( doParallel ) # setup l o c a l SOCK c l u s t e r f o r 4 CPU cores c l ← makePSOCKcluster ( 4 ) registerDoParallel ( cl ) f o r e a c h ( i =1: getDoParWorkers ( ) ) %dopar% { # t h i s code w i l l be executed c o n c u r r e n t l y ... } stopCluster ( cl ) BRAG Sept. 27 Parallel MCMC in R
- 11. Parallel MCMC Introduction Random Number Generators R packages Summary foreach with multicore § l i b r a r y ( foreach ) library ( parallel ) library ( doParallel ) # f o r k one c h i l d process f o r each CPU core c l ← makeForkCluster ( d e t e c t C o r e s ( ) ) registerDoParallel ( cl ) f o r e a c h ( i =1: getDoParWorkers ( ) ) %dopar% { # t h i s code w i l l be executed c o n c u r r e n t l y ... } BRAG Sept. 27 Parallel MCMC in R
- 12. Parallel MCMC Introduction Random Number Generators R packages Summary foreach with CODA If your Gibbs sampler returns an mcmc object, these can be conbined into an mcmc.list: § l i b r a r y ( coda ) samples . l i s t ← f o r e a c h ( i =1: getDoParWorkers ( ) , . combine=mcmc . l i s t , . m u l t i c o m b i n e =T ) %dopar% { # t h i s code w i l l be executed c o n c u r r e n t l y ... } BRAG Sept. 27 Parallel MCMC in R
- 13. Parallel MCMC Introduction Random Number Generators R packages Summary foreach with other libraries You need to declare any libraries that are used inside the child process. For example: § l i b r a r y ( mvtnorm ) l i b r a r y ( coda ) f o r e a c h ( i =1: getDoParWorkers ( ) , . packages=c ( "mvtnorm" , "coda" ) ) %dopar% { # t h i s code uses mcmc ( . . . ) and rmvnorm ( . . . ) ... } BRAG Sept. 27 Parallel MCMC in R
- 14. Parallel MCMC RNG and parallel MCMC Random Number Generators RNGs available in R Summary Random Number Generators for parallel MCMC The chains of our Gibbs sampler run independently, but: if the same RNG is seeded with the same value, all of the chains will generate the same random numbers in the same sequence - they will be identical! we either need to use: different seeds, or different random number generators for each chain (preferably both) it is also advisable to choose (or generate) different initial values in each chain of our Gibbs sampler BRAG Sept. 27 Parallel MCMC in R
- 15. Parallel MCMC RNG and parallel MCMC Random Number Generators RNGs available in R Summary Mersenne Twister The default RNG in R pseudo-random sequence with 32bit precision periodicity of 219937 − 1 takes 0.4 seconds to generate 107 random numbers on an Intel Core i5 running R 2.15.1 and Windows 7 open-source implementation available at: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html Matsumoto & Nishimura (1998) TOMACS 8: 3–30. BRAG Sept. 27 Parallel MCMC in R
- 16. Parallel MCMC RNG and parallel MCMC Random Number Generators RNGs available in R Summary Other RNGs in the base package Wichmann-Hill (1982) Applied Statistics 31, 188–190. Marsaglia-Multicarry (Usenet newsgroup sci.stat.math, 1997) Super-Duper (Reeds, J., Hubert, S. and Abrahams, M., 1982–4) For JAGS with up to 4 concurrent chains: § r n g I n i t s ← p a r a l l e l . seeds ( "base::BaseRNG" , 4 ) BRAG Sept. 27 Parallel MCMC in R
- 17. Parallel MCMC RNG and parallel MCMC Random Number Generators RNGs available in R Summary L’Ecuyer Available via R libraries rlecuyer or parallel Multiple independent streams of random numbers Periodicity ≈ 2191 (each stream is a subsequence of length 2127 ) 0.6 seconds to generate 107 random numbers via runif To initialize each child process in a SNOW cluster with an independent stream: § c l ← makeCluster ( 4 ) clusterSetRNGStream ( c l ) registerDoParallel ( cl ) L’Ecuyer, et al. (2002) Operations Research, 50(6): 1073–1075. BRAG Sept. 27 Parallel MCMC in R
- 18. Parallel MCMC Random Number Generators Summary Summary Most MCMC algorithms are "embarrasingly parallel" chains run independently (as long as the RNG is set up correctly) The R packages foreach and doParallel make parallelism easy, on any computing platform Related topics (not covered in this presentation): Running R on a supercomputer (eg. lyra.qut.edu.au) Cloud computing with Apache Hadoop GPU programming in R (nVidia CUDA) BRAG Sept. 27 Parallel MCMC in R
- 19. Appendix For Further Reading For Further Reading Norman Matloff The Art of R Programming. No Starch Press, 2011. M. Schmidberger, M. Morgan, D. Eddelbuettel, H. Yu, L. Tierney & U. Mansmann State of the Art in Parallel Computing with R. Journal of Statistical Software, 31(1), 2009. P. L’Ecuyer, R. Simard, E.J. Chen & W.D. Kelton An Object-Oriented Random-Number Package with Many Long Streams and Substreams. Operations Research, 50(6): 1073–1075, 2002. M. Matsumoto & T. Nishimura Mersenne Twister: A 623-Dimensionally Equidistributed Uniform Pseudo-Random Number Generator. ACM Transactions on Modeling and Computer Simulation, 8: 3–30, 1998. BRAG Sept. 27 Parallel MCMC in R