PyconUK-2015
0 likes464 views
This document discusses using Docker and Docker Compose to build resilient data pipelines. It provides an example use case of processing news articles from different data sources and categories. The system is designed with microservices running in Docker containers that can interact and scale easily. Docker Compose is used to define and run the multi-container application. The keys to the design are separating each component into its own container to enable independent development, testing, and deployment across cloud infrastructure using Docker images.
PyconUK-2015
- 1. Resilient Data Pipelines with Docker and Docker- Compose Heeren Sharma Software Engineer @Cliqz heeren@cliqz.com @heerensharma
- 2. 80+ - Team size! ! 500,000 - DAU! ! 3 Million+ - Downloads (Germany only)! ! 1 billion+ - Indexed pages (We do not believe in indexing the web.)! ! 5 TB - In-Memory indexed (Based on open source and in-house build NoSQL stores.)
- 3. Storyfile FROM Introduction RUN Data-pipelines RUN whats-docker ADD use-case ./wrap-up CMD ./demo
- 4. “Data and data everywhere” “Design and development of Data Pipelines” “Cloud deployment has its own needs”
- 5. One’s processed Data is another system’s input data
- 6. What’s that whale named Docker ? "Docker allows you to package an application with all of its dependencies into a standardised unit for software development."
- 7. A little bit more Docker • Dockerfile ! FROM ubuntu:14.04! RUN pip install requests! ADD . /awesome-code! WORKDIR /awesome-code! CMD python revolutionary_app.py $ docker build -t=“your-awesome-image:v1” . $ docker run -d -p 80:5000 —name container-name your-awesome-image:v1 $ docker push your-awesome-image:v1 $ docker pull myregistry.com:8080/your-awesome-image:v1 $ docker ps! ! $ docker inspect <container-name/ID>! ! $ docker log <container-name/ID>! ! $ docker stop <container-name/ID>! ! $ docker exec -it <container-name/ID> your_command!
- 8. Simplified Use Case • Streaming data from different sources - Twitter, FB, feeds and customised scraping engine. • Different processing engines • Fast iterations over new requirements • Resilient system with focus over easy deployment
- 9. Use case - News Articles • Trending news articles from different domains • News Categorisation • Relevant news over search query • Traffic of news content fluctuates - pretty dynamic. • There is no universal right answer
- 10. System Design (1) Data Stream Processing Engine Processed Data (Redis) $ docker run -d —-name processed_data redismaster:v1 redis-server $ docker build -t=“redismaster:v1” . $ docker build -t=“datastream:twitter” . $ docker run -d —-name twitter_queue datastream:twitter python /code/format_data.py $ docker build -t —-name processing_engine .! $ docker run -d —-name magic_powerhouse —-link processed_data:db processing_engine python /code/ magic_script.py
- 11. New State of design
- 15. docker-compose to rescue queue:! build: .! command: python /news-swimlane/twitter-queue/read_queue.py! volumes:! - /ebs/data:/data! environment:! DATA_DIR: /data! PYTHONPATH: /news-swimlane! restart: always! update:! build: .! command: python /news-swimlane/server-redis/build_redis_exc.py! volumes:! - /ebs/data:/data! links:! - redismaster! - fetcher! environment:! REDIS_HOST: redismaster_1! DATA_DIR: /data! PYTHONPATH: /news-swimlane! restart: always! redismaster:! build: redis3-container! ports:! - "6379:6379"! volumes:! - /ebs/backup:/data! restart: always! command: /usr/local/bin/redis-server /master.conf! fetcher:! build: .! command: python /news-swimlane/fetcher/server_fetcher.py! ports:! - "80:5000"!
- 16. Key points: Design & Development • Micro - Services oriented design and henceforth development • Old/new components can be realised in form of docker containers • Different containers can readily interact among each other • Ease to test (Local environment) and no worries if it bursts out in production.
- 17. Deployment • Just install docker on remote (to be done with care) • Docker images can be pushed to remote repository (better say registry). • Make your instances autoscale in any cloud IaaS. • If instance go down, then new instance just pull docker images, and start docker containers.
- 18. Resources • Docker - https://www.docker.com/whatisdocker • Boot2docker - https://docs.docker.com/installation/mac/ • Docker-compose - https://docs.docker.com/compose/ • Very short video about Docker (American Style) - https:// www.youtube.com/watch?v=aLipr7tTuA4 - American style • https://www.youtube.com/watch?v=FdkNAjjO5yQ - Good resource to have a little deep insight about Containers