Docker data science pipeline
Download as PPTX, PDF3 likes1,317 views
The document discusses the implementation of a data science pipeline using Docker at ING, highlighting its role in creating, deploying, and running applications in a containerized environment. It details the integration of various tools such as GitLab, Hadoop, and OpenShift, focusing on aspects like continuous integration, model versioning, and orchestration. Additionally, it touches on agile methodologies and considerations for managing data science projects effectively.
Docker data science pipeline
- 1. Docker Datascience Pipeline Running datascience models on Docker at ING DataWorks Summit Berlin 18 -04-2018
- 2. • Big Data Engineer at ING • DB2, Oracle, AIX, Linux, Hadoop, Hive, Sqoop, Ansible • Let it all work on the Exploration Environment • @chiefware Lets introduce myself 2
- 3. Asking yourself how to get to production Depending on different tools Why I think you are here 3 You are running Datascience models Interested in Docker Asking yourself how to get to production
- 4. What is Docker 4 Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. By doing so, thanks to the container, the developer can rest assured that the application will run on any other Linux machine regardless of any customized settings that machine might have that could differ from the machine used for writing and testing the code
- 5. Agile way of working in Squads,Chapters and Tribes 5
- 6. big data exploration 6 Gitlab runner Citrix Server Dev node Hadoop Artifactory Gitlab Automation Server Access to cluster rdp, browser, putty Node for datascientists with tools and xrdp Datanode cluster Datanode cluster Pip repo Datasciences projects Details about processServer
- 8. Docker on big data exploration 8 Gitlab runner Citrix Server Dev node Hadoop Openshift POD/JOB Artifactory Gitlab Automation Server Access to cluster rdp, browser, putty Node for datascientists with tools and xrdp Datanode cluster Orchestration Containers docker containers Docker base Images Docker files and Datasciences projects
- 9. Netezza Hive Rapid experimentation Live model Deploymen t Code versioning Build Docker image(s) through Gitlab CI/Jenkins Put serialized model in shared directory Save Docker image(s) to Docker Registry Move serialized model to HDFS Data versioning On each commit Train and validate model Setup environment Docker Registry Tivoli scheduler triggers Monithor HDFS Add link to file to Monithor Add model metrics to Monithor Git repository with Hive queries Feature engineering Data exploration Cookiecutter Workspace management Data service Retraining Revalidatio n Monitoring Fallback DS code library Structured way of keeping track of dev models Continuous Integration (testing) Early CD facilitation Construct pipeline (Pachyderm or APApeline) Create virtual environment per project Checkpointin g Development Continuous Deployement Live
- 10. Dockerfile 10
- 11. Gitlab runner 11 To register run: gitlab-runner register gitlab-runner list
- 13. Spark 13 • Only submit and forget works in Docker • spark-submit deploy-mode cluster master yarn • kinit your keytab file for Kerberos • pip in virtual env
- 15. Demo Time 15
- 16. considerations 16 Jenkins instead of gitlab-runner Add gpu nodes to openshift How to use scheduler tool How to handle Kerberos files
- 17. 17 Questions?