Apache Spark from Notebook to Cloud Native Application with Rebecca Simmonds
1 like483 views
The document discusses transitioning from notebook prototypes to cloud-native applications using Apache Spark, emphasizing the importance of tools, techniques, and robust architectures. It highlights the benefits of scalable, modular applications in cloud environments, along with visualizations and proof-of-concept demonstrations in Jupyter Notebook. The conclusion encourages developers to utilize resources like radanalytics.io for deploying their own applications.
Apache Spark from Notebook to Cloud Native Application with Rebecca Simmonds
- 1. #Py2SAIS Spark from Notebook to Cloud Native Application Rebecca Simmonds Senior Software Engineer rsimmond@redhat.com @becky_simmonds
- 2. #Py2SAIS To empower others with the tools and tips to go from prototype to production using Apache Spark Aim
- 4. #Py2SAIS 1. Use case 2. Problem domain 3. Data set 4. Tools and techniques Requirements
- 6. #Py2SAIS Variety Country Points Region Tinta de Toro Spain 98 Toro Cabernet Sauvignon US 70 Napa Valley Macauley US 50 Knights Valley
- 7. #Py2SAIS ● Open-source web application ● Create and share live code examples ● Python code ● It empowers users with visualisation tools Jupyter Notebook
- 8. #Py2SAIS Spark Driver Process Executors
- 10. #Py2SAIS ● Easy to setup and get going ● Lots of visualisations to practise with ● Great method for proof of concept Conclusions
- 12. #Py2SAIS 1. Cloud based for scale and portability 2. Tooling and techniques 3. Database/more robust store 4. Testing Next Steps
- 13. #Py2SAIS Applications that are: 1. designed to run in the cloud 2. scalable 3. modular 4. and resilient Cloud Native Applications
- 14. #Py2SAIS ● Allow you to package and isolate a runtime environment ● Easily portable to different environments ● Scalable ● Quick and easy to deploy Containers
- 15. #Py2SAIS Monolithic Architecture Microservices Architecture User Interface Business Logic Data Access Layer Database User Interface MicroserviceMicroserviceMicroservice Database Database Database
- 17. #Py2SAIS An open source community working to empower intelligent applications on kubernetes Projects and tutorials to empower developers with machine learning techniques Radanaytics.io
- 18. #Py2SAIS Oshinko
- 19. #Py2SAIS Oshinko Deployment Oshinko source to image
- 20. #Py2SAIS Architecture Postgresql Spark Spark Spark Wine Map Application Load and Calculate Response Response Request Response Request Job Web Browser
- 21. #Py2SAIS Demo
- 22. #Py2SAIS # test command os::cmd::try_until_text # what to test 'oc new-app --template=oshinko-python-spark-build-dc -p APPLICATION_NAME=winemap -p GIT_URI=https://github.com/radanalyticsio/winemap.git # expected result 'Success'
- 23. #Py2SAIS ● Jupyter notebook for prototyping ● VISIT radanalytics.io ● Deploy your own cloud native applications @becky_simmonds rsimmond@redhat.com https://radanalytics.io/applications/wine-map Conclusion