연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
66 likes4,308 views
This document discusses various Python and web development topics. It covers Python programming best practices and libraries like asyncio and NumPy. It also discusses front-end frameworks like Polymer and Electron. Additional sections provide overviews of containerization with Docker, message queues like ZeroMQ, log aggregation with Logstash, and cloud hosting options.
![Python 3.4 – asyncio loop
▪ loop.close()
..
import asyncio
import asyncio_redis
loop = asyncio.get_event_loop()
conn = loop.run_until_complete(
asyncio_redis.Connection.create('localhost', 6379))
conn.close()
loop.close()
Task was destroyed but it is pending!
task: <Task pending coro=<_reader_coroutine() running at /...
/lib/python3.4/site-packages/asyncio_redis/protocol.py:919>
wait_for=<Future pending cb=[Task._wakeup()]>>](https://image.slidesharecdn.com/pyconkr2015-lablup-final-150829145030-lva1-app6892/85/PyCon-KR-2015-63-320.jpg)
![Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my_timer():
i = 0
while True:
yield from asyncio.sleep(1)
print(i)
i += 1
try:
asyncio.async(my_timer(), loop=loop)
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
loop.close()
^CTask was destroyed but it is pending!
task: <Task pending coro=<my_timer() done, defined at test.py:4>
wait_for=<Future pending cb=[Task._wakeup()]>>](https://image.slidesharecdn.com/pyconkr2015-lablup-final-150829145030-lva1-app6892/85/PyCon-KR-2015-65-320.jpg)
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
- 4. ▪ ▪ ▪ / ▪ ▪ / ▪ TNF/ Needlworks ▪ … ▪ ▪ ▪ ▪ NBA ▪ 80 Gbps ▪ ▪ … http://www.lablup.com .
- 5. ▪ 21 , 20 , 19 ▪ ▪ ,
- 6. 21 ?
- 7. ▪ . ▪ .
- 8. . ..
- 9. ▪ ▪ ▪ / / / ▪ ▪ /
- 10. ▪ Three keyboardists but…
- 13. ! Python! … …
- 14. Python! ▪ ▪ : ▪ PEP-8 ▪ : ▪ ▪
- 15. Python T_T ▪ ▪ ▪ PyPy / Pyston ▪ : ▪ C ▪ Cython ▪
- 16. Python / ▪ ▪ CS101 ▪ “ ” ▪ ▪ indent … ▪ ▪ MATLAB numpy
- 17. : ▪ ▪ MATLAB : matplotlib, ipython notebook ▪ Ultra-complex figures ▪ ▪ ▪ ▪ /
- 18. : ▪ ▪ NBA ▪ ▪ configuration Python ▪ I/O stdout asyncio
- 19. It's time to go Python 3 ▪ Python 3 ▪ Django, numpy/scipy, … ▪ Python 3.3 ▪ PEP-393: str unicode codepoint ▪ narrow build UTF-16, wide build UTF-32 ▪ UTF-8/16/32 ▪ ASCII/Latin-1 ▪ / ·
- 20. It's time to go Python 3 ▪ Python 3.4 ▪ PEP-3156: asyncio coroutine ▪ I/O ▪ GIL (global interpreter lock) ▪ ▪ pip virtualenv ▪
- 21. ▪ RabbitMQ, Kafka, Celery, ... ▪ ▪ (broker) , ▪ redis …
- 22. – ZeroMQ ▪ ▪ Request-Reply ▪ 1 ( : HTTP-like server/client) ▪ Push-Pull ▪ N-to-1 ( : logging system, task partitioning) ▪ Publish-Subscribe ▪ 1-to-N / N-to-M ( : broadcast)
- 23. – ZeroMQ ▪ : ▪ zero broker, zero latency, zero administration, ... ▪ : connection ▪ TCP / UDP / UNIX domain / local "in-process" ▪ zeromq transport header (ZMTP) ▪ payload (64+ ) ▪ : persistent queue ▪ TCP reliability
- 24. – LogStash ▪ LogStash ▪ input / filter / output ▪ ZeroMQ, S3 ▪ ( !) Django User Kernel Logstash Server Database ZeroMQ (push-pull) + JSON AWS S3 msgpack
- 25. On-premise vs. Hosting vs. Cloud ▪ ▪ " " ... ▪ , , ▪ AWS ▪ ▪ ▪ … ▪ 2012 R2 / / .
- 26. Docker containers ▪ docker container ▪ ▪ VM deploy/destroy ▪ AWS EC2 Instance: ( ) ▪ pooling ▪ Amazon ECS? ▪ container (run task) latency ▪ …
- 27. Docker containers ▪ ▪ : instance ▪ Microsoft docker ▪ Azure Windows
- 28. Polymer library ▪ HTML5 ▪ HTML imports / Custom elements / Shadow DOM ▪ Web components ▪ “Polyfill” ▪ ▪ DOMelements
- 29. Polymer library ▪ ... ▪ ! ▪ Bootstrap + theme? ▪ ▪ HTML5 !
- 32. HTML imports! Reusable web components! Shadow DOM! Polyfill!
- 33. Polymer ▪ ▪ 7 ? ?
- 34. Electron: Cross-platform Polymer ▪ : cross-platform webapp container ▪ Electron ▪ Github Node.js ▪ Node Chromium wrapping ▪ / / ▪ ATOM ▪ ? ▪ Apache Cordova
- 35. ▪ ▪ ▪ ▪ Django UnitTest + Selenium ▪ Selenium webdriver: ▪
- 36. ▪ ▪ ▪ sphinx ▪ reST ▪ ▪ ▪ ! ▪
- 37. ▪ ▪ ▪ sphinx ▪ reST ▪ ▪ ▪ ! ▪
- 38. . .
- 39. , , + ,
- 40. ▪ + . ▪ : ▪ – ▪ , , scaling, ... ➞ ++ ▪ : ▪ – ▪
- 41. ▪ . ▪ ! ▪ ▪
- 47. …
- 48. , …
- 50. “ .”
- 51. ▪ React / Flux / AngularJS / Polymer / ... ▪ +UI ▪ Polymer 0.9 API .... ▪ (?) Bootstrap … ▪ I/O 2015 Polymer 1.0 ...
- 52. Polymer: it is too google to be true ▪ ▪ 0.5 - 0.8rc2 - 0.9 - … ▪ ▪ ( ) ▪ Google I/O - 1.0 :
- 53. Polymer: vulcanize ▪ Vulcanize ▪ : ▪ IE – + ▪ CSS – + ▪ Crisper javascript map shadow DOM root - +
- 54. Polymer: vulcanize ▪ Vulcanize ▪ : ▪ IE – ▪ CSS – ▪ Crisper javascript map shadow DOM root -
- 55. Polymer: polyfill ▪ ▪ ▪ (html import shadow DOM) polyfill ▪ ▪
- 56. Polymer + Django ▪ ▪ {{ … }} ▪ Polymer – ▪ Django – ▪ ▪ ▪ …
- 57. Polymer + Django + Security ▪ CORS (cross-origin resource scripting) ▪ Django – CORS header ( ) ▪ Polymer – vulcanize Javascript CORS ▪ Crisper CORS Javascript ▪ - ▪ XSS ▪ Django - CSRF token ▪ Polymer – header iron-ajax ▪
- 58. Polymer ▪ Production ready?! ▪ ▪ “Do not swim in sandocean. Swim in the ocean.” ▪ HTML5 ▪ ▪ X-tag, react, polymer… ▪ ?
- 59. Polymer ▪ Production ready?! ▪ ▪ “Do not swim in sandocean. Swim in the ocean.” ▪ HTML5 ▪ ▪ X-tag, react, polymer… ▪ ?
- 60. Electron + web ▪ ... ! ▪ ▪ (Noto ) ▪ ▪ CSS: . ▪ ?
- 61. : Django + RDS ▪ UTF-8 Python ? ▪ (cf., str / bytes / io / codecs) ▪ MySQL SQLite ▪ MySQL ▪ ForeignKey Django ▪ MySQL + Django + Korean = EPIC FAIL ▪ ( ) Postgres ... ▪ postgres ( / ) .
- 62. Python 3.4 – asyncio / coroutine ▪ . ▪ yield from ... ▪ AST static analysis coroutine yield from ▪ , 100% static analysis ▪ duck typing + dynamic method generation + method proxy pattern ... ▪ Python 3.5 await/async !
- 63. Python 3.4 – asyncio loop ▪ loop.close() .. import asyncio import asyncio_redis loop = asyncio.get_event_loop() conn = loop.run_until_complete( asyncio_redis.Connection.create('localhost', 6379)) conn.close() loop.close() Task was destroyed but it is pending! task: <Task pending coro=<_reader_coroutine() running at /... /lib/python3.4/site-packages/asyncio_redis/protocol.py:919> wait_for=<Future pending cb=[Task._wakeup()]>>
- 64. Python 3.4 – asyncio loop ▪ asyncio graceful shutdown , 1. loop._run_once() 2. loop.run_until_complete(asyncio.sleep(0)) 3. loop.run_until_complete(server.wait_closed()) ▪ , wait_closed() coroutine . ▪ event loop .
- 65. Python 3.4 – asyncio loop #! /usr/bin/env python3 import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def my_timer(): i = 0 while True: yield from asyncio.sleep(1) print(i) i += 1 try: asyncio.async(my_timer(), loop=loop) loop.run_forever() except KeyboardInterrupt: pass finally: loop.close() ^CTask was destroyed but it is pending! task: <Task pending coro=<my_timer() done, defined at test.py:4> wait_for=<Future pending cb=[Task._wakeup()]>>
- 66. Python 3.4 – asyncio loop #! /usr/bin/env python3 import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def my_timer(): i = 0 while True: yield from asyncio.sleep(1) print(i) i += 1 try: asyncio.async(my_timer(), loop=loop) loop.run_forever() except KeyboardInterrupt: for t in asyncio.Task.all_tasks(): t.cancel() try: loop._run_once() except asyncio.CancelledError: pass finally: loop.close()
- 67. asyncio_redis ▪ Redis asyncio ▪ Heisenbug ! 🚨 ▪ SCAN key ... ▪ while if ➞ ▪ https://github.com/jonathanslenders/asyncio- redis/issues/65 ▪ Connection pool API ▪ API call connection stateful API ( : SELECT) ▪ (aioredis)
- 68. : – ZeroMQ ▪ Asynchronous ? block ?? ▪ socket connect . ( ...) ▪ send/recv timeout ▪ aiozmq , blocking call asyncio.wait_for timeout . ▪ pyzmq , socket.poll(msec) timeout .
- 69. : – LogStash ▪ zmq input plugin s3 output plugin zmq recv_string ▪ 1: s3 output plugin “write” , PutObject DeleteObject . ▪ 2: config , output plugin log level · . ▪ 3: output plugin input plugin receive timeout .
- 70. : Docker ▪ ▪ docker registry "latest" latest ▪ 2.0 2.0.1 tag ▪ ▪ ▪ docker 1.8 , CoreOS (rkt) ▪ /API
- 71. : ▪ Jupyter / ipython ▪ scalable ▪ Jupyter ▪ Unix ▪ ▪ : ▪ ▪ – Sorna ▪ .
- 72. CodeON
- 73. ▪ Container Resource Consolidation ▪ Kubernetes ▪ paxos raft agreement key-value store . ▪ Ingen REPL (read-evaluate-print loop) ▪ stdout/stderr streaming ▪ interactive plot ▪ image, sound multimedia
- 74. ▪ ▪ ▪ User profiler + backtracer + ML with back propagation ▪ ▪ d3.js fluid nonlinear navigation UI ▪ ▪ ▪ ▪ ▪
- 75. ▪ . ▪ ... + + … ▪ . ▪ ?