Managing large (and small) R based solutions with R Suite
0 likes614 views
The document discusses the use of R, an open-source programming language primarily for data analysis, highlighting its rapid advancements in analytics powered by GPU engines and a strong community support. It emphasizes the importance of structured software development for large-scale R projects, suggesting a comprehensive workflow using the R Suite, which includes package development, deployment, and continuous integration. The document also notes some limitations of R in production environments and offers guidelines for efficient R software development.
![Copyright (c) WLOG Solutions 49
RSuiteVersion: 0.10.212
RVersion: 3.3
Project: myproject
Repositories: MRAN[2017-09-28],
Dir[]
Artifacts: config_templ.txt
Example
PARAMETERS
● RVersion - R version for this
project
● Project - project name
● Repositories
○ MRAN[YYYY-MM-DD],
○ S3[URL]
○ URL[URL]
○ Dir[]
● Artifacts - what else should be
added to deployment
package.](https://image.slidesharecdn.com/witjakuczundatamassgdansksummit-171007152554/85/Managing-large-and-small-R-based-solutions-with-R-Suite-49-320.jpg)
Managing large (and small) R based solutions with R Suite
- 1. Copyright (c) WLOG Solutions Managing large (and small) R based solutions with R Suite Wit Jakuczun, WLOG Solutions 2017-09-29
- 2. Copyright (c) WLOG Solutions 2 Open-source is a leader the race in advanced analytics using GPU engines and power of its community.
- 3. Copyright (c) WLOG Solutions 3
- 4. Copyright (c) WLOG Solutions 4 Example?
- 5. Copyright (c) WLOG Solutions 5 4000x4 elastic-net models (CV-5) for 45Kx10K dataset in 1,5 minute!
- 6. Copyright (c) WLOG Solutions 6 Join 21st centuRy today!
- 7. Copyright (c) WLOG Solutions What is R? 7
- 8. Copyright (c) WLOG Solutions 8 Dynamically interpreted general programming language
- 9. Copyright (c) WLOG Solutions 9 Stable open-source product developed by R Foundation since ~1995 year.
- 10. Copyright (c) WLOG Solutions 10 Created for data analysis.
- 11. Copyright (c) WLOG Solutions 11 flights %>% group_by(year, month, day) %>% select(arr_delay, dep_delay) %>% summarise( arr = mean(arr_delay, na.rm = TRUE), dep = mean(dep_delay, na.rm = TRUE) ) %>% filter(arr > 30 | dep > 30) z <- scaled_input %>% layer_convolution2D(c(5,5), 32, pad = TRUE) %>% layer_max_pooling(c(3,3), c(2,2)) %>% layer_convolution2D(c(3,3), 48) %>% layer_max_pooling(c(3,3), c(2,2)) %>% layer_convolution2D(c(3,3), 64) %>% layer_dense(96) %>% layer_dropout(0.5) %>% layer_dense(num_output_classes, activation = activation_softmax()) Data preparation Predictive model building
- 12. Copyright (c) WLOG Solutions 12 R is a community.
- 13. Copyright (c) WLOG Solutions 13 CRAN 10K+ packages GitHub more and more popular
- 14. Copyright (c) WLOG Solutions 14 However, out-of-the-box R does not provide many essential features required In a large-scale production deployment!
- 15. Copyright (c) WLOG Solutions 15 R Software Development What is a large scale?
- 16. Copyright (c) WLOG Solutions 16 R software development vs R scripting
- 17. Copyright (c) WLOG Solutions 17 Large scale ~ 10K+ LOC Small scale ~ 1K LOC
- 18. Copyright (c) WLOG Solutions 18 Examples of large scale projects
- 19. Copyright (c) WLOG Solutions 19 Historical data Traffic forecasting Workforce optimiser Busines rules Human intervention Results Efficiency curves Workforce optimisation
- 20. Copyright (c) WLOG Solutions 20 Historical data Cash flow “forecasting” Optimisation phase I Optimisation phase II Business rules Human intervention Results FONG Cash optimization
- 21. Copyright (c) WLOG Solutions 21 Production Dev Continuous integration Version control R Studio Server Data-science team
- 22. Copyright (c) WLOG Solutions 22 R Software Development What does R give us “out-of-the-box”?
- 23. Copyright (c) WLOG Solutions 23 Dev Version control Continuous Integration & Deployment Prod Software development process
- 24. Copyright (c) WLOG Solutions 24 What is nice about R?
- 25. Copyright (c) WLOG Solutions 25 Package help system Package dependency system External data in packages Vignettes Tests
- 26. Copyright (c) WLOG Solutions 26 Where does R have its rough edges?
- 27. Copyright (c) WLOG Solutions 27 CRAN (MRAN) Github Other Dev environment Installed packages Local CRAN Source code repo
- 28. Copyright (c) WLOG Solutions 28 install.packages(“ggplot2”)
- 29. Copyright (c) WLOG Solutions 29 CRAN (MRAN) Github Other Dev environment Installed packages Local CRAN Source code repo
- 30. Copyright (c) WLOG Solutions 30 Teaser Alcohol concentration in blood
- 31. Copyright (c) WLOG SolutionsCopyright (c) WLOG Solutions 31 Instruction (Windows only!) 1. Install vanilla R 3.4.2 2. Download package (https://goo.gl/RnZAQg) 3. Unzip 4. Open CMD 5. Rscript R/master.R --port=7137 6. In browser open http://localhost:7137 http://www.sumsar.net/blog/2014/07/estimate-your-bac-using-drinkr/
- 32. Copyright (c) WLOG Solutions 32 R Software Development R Suite walk-through
- 33. Copyright (c) WLOG Solutions 33 Preliminaries
- 34. Copyright (c) WLOG Solutions 34 Install R https://cran.r-project.org/bin/windows/base/R-3.4.2-win.exe
- 35. Copyright (c) WLOG Solutions Install R Suite 35 http://rsuite.io/RSuite_Download.php
- 36. Copyright (c) WLOG Solutions Development cycle with R Suite 36 1. Start a project 2. Add a package to the project a. Develop the package (using devtools) 3. Add dependencies to the project 4. Build the package 5. Build a deployment package
- 37. Copyright (c) WLOG Solutions 37 Open tutorial http://rsuite.io/RSuite_Tutorial.php?article=basic_workflow.md
- 38. Copyright (c) WLOG Solutions 38
- 39. Copyright (c) WLOG Solutions 39 Project structure
- 40. Copyright (c) WLOG Solutions 40 deployment logs packages R tests import export work Where all the packages are installed.
- 41. Copyright (c) WLOG Solutions 41 deployment logs packages R tests import export work Where all the logs are stored.
- 42. Copyright (c) WLOG Solutions 42 deployment logs packages R tests import export work Where all the project packages are stored.
- 43. Copyright (c) WLOG Solutions 43 deployment logs packages R tests import export work Where master scripts (~ main) are stored.
- 44. Copyright (c) WLOG Solutions 44 deployment logs packages R tests import export work Where project tests are stored.
- 45. Copyright (c) WLOG Solutions 45 deployment logs packages R tests import export work (Optional) Raw data to be imported.
- 46. Copyright (c) WLOG Solutions 46 deployment logs packages R tests import export work (Optional) Results generated by our R program.
- 47. Copyright (c) WLOG Solutions 47 deployment logs packages R tests import export work (Optional) Temporary results.
- 48. Copyright (c) WLOG Solutions 48 Configuration files.
- 49. Copyright (c) WLOG Solutions 49 RSuiteVersion: 0.10.212 RVersion: 3.3 Project: myproject Repositories: MRAN[2017-09-28], Dir[] Artifacts: config_templ.txt Example PARAMETERS ● RVersion - R version for this project ● Project - project name ● Repositories ○ MRAN[YYYY-MM-DD], ○ S3[URL] ○ URL[URL] ○ Dir[] ● Artifacts - what else should be added to deployment package.
- 50. Copyright (c) WLOG Solutions 50 LogLevel: INFO N_days: 365 solver_max_iterations: 10 solver_opt_horizon: 8 Example config_templ.txt (config.txt) ● LogLevel - level for loggers ● _templ.txt - template for config ● config.txt - deployment version
- 51. Copyright (c) WLOG Solutions 51
- 52. Copyright (c) WLOG Solutions 52 master.R Here goes your code... Autodected path of master.R There can be many master scripts!
- 53. Copyright (c) WLOG Solutions 53 Put all logic into packages
- 54. Copyright (c) WLOG Solutions 54
- 55. Copyright (c) WLOG Solutions 55 Select external packages carefully. And control their versions!
- 56. Copyright (c) WLOG Solutions 56 data.table
- 57. Copyright (c) WLOG Solutions 57
- 58. Copyright (c) WLOG Solutions 58 print is not for logging. Forbidden
- 59. Copyright (c) WLOG Solutions 59 loginfo("Phase 1 passed") logdebug("Iter %d done", i) logwarning("Are you sure?") logerror("I failed :(")
- 60. Copyright (c) WLOG Solutions 60 pkg_loginfo("Phase 1 passed") pkg_logdebug("Iter %d done", i) pkg_logwarning("Are you sure?") pkg_logerror("I failed :(")
- 61. Copyright (c) WLOG Solutions 61 Automate building, deploying, testing, etc.
- 62. Copyright (c) WLOG Solutions Wit Jakuczun, PhD wit.jakuczun@wlogsolutions.com 62 Field tested R ecosystem for Enterprise http://rsuite.io