How to deploy a new model in production without overhead
- 2. How to deploy a new model in production without overhead Laura Calem - AI in Production meetup - 12.10.18
- 3. Train Prod
- 6. Train Prod
- 7. Train Prod
- 8. Train Prod
- 10. Use Docker 1
- 15. New models 2
- 16. 2.1 Have an external module with a Batcher class
- 17. 2.1 Have an external module with a Batcher class Mother class
- 18. 2.1 Have an external module with a Batcher class Mother class ZMQ server
- 19. 2.1 Have an external module with a Batcher class Mother class ZMQ server Will handle receiving data
- 20. 2.1 Have an external module with a Batcher class Mother class ZMQ server Will handle receiving data Call a process_batch class
- 21. 2.1 Have an external module with a Batcher class Mother class ZMQ server Will handle receiving data Call a process_batch class Handle sending response
- 22. 2.1 Batcher mother class
- 23. 2.1 Batcher mother class class Batcher(object): def __init__(self): …
- 24. 2.1 Batcher mother class class Batcher(object): def __init__(self): … def get_parser(self): …
- 25. 2.1 Batcher mother class class Batcher(object): def __init__(self): … def get_parser(self): … def run_server(self): …
- 26. 2.1 Batcher mother class class Batcher(object): def __init__(self):
- 27. 2.1 Batcher mother class class Batcher(object): def __init__(self): parser = self.get_parser() self.args = parser.parse_args() # augment args if needed
- 28. 2.1 Batcher mother class class Batcher(object): def __init__(self): parser = self.get_parser() self.args = parser.parse_args() # augment args if needed self.batcher_init() # will be redefined self.host = self.args.host self.port = self.args.port
- 29. 2.1 Batcher mother class class Batcher(object): def run_server(self):
- 30. 2.1 Batcher mother class class Batcher(object): def run_server(self): self.context = zmq.Context() self.server = self.context.socket(zmq.REP)
- 31. 2.1 Batcher mother class class Batcher(object): def run_server(self): self.context = zmq.Context() self.server = self.context.socket(zmq.REP) zmq_bind_addr = "%s:%s" % (self.host, self.port) self.server.bind(zmq_bind_addr)
- 32. 2.1 Batcher mother class class Batcher(object): def run_server(self): self.context = zmq.Context() self.server = self.context.socket(zmq.REP) zmq_bind_addr = "%s:%s" % (self.host, self.port) self.server.bind(zmq_bind_addr) while True:
- 33. 2.1 Batcher mother class class Batcher(object): def run_server(self): self.context = zmq.Context() self.server = self.context.socket(zmq.REP) zmq_bind_addr = "%s:%s" % (self.host, self.port) self.server.bind(zmq_bind_addr) while True: message = self.server.recv()
- 34. 2.1 Batcher mother class class Batcher(object): def run_server(self): self.context = zmq.Context() self.server = self.context.socket(zmq.REP) zmq_bind_addr = "%s:%s" % (self.host, self.port) self.server.bind(zmq_bind_addr) while True: message = self.server.recv() response = self.handle_request(message)
- 35. 2.1 Batcher mother class class Batcher(object): def run_server(self): self.context = zmq.Context() self.server = self.context.socket(zmq.REP) zmq_bind_addr = "%s:%s" % (self.host, self.port) self.server.bind(zmq_bind_addr) while True: message = self.server.recv() response = self.handle_request(message) self.server.send(response)
- 36. 2.1 Batcher mother class class Batcher(object): def handle_request(self, message):
- 37. 2.1 Batcher mother class class Batcher(object): def handle_request(self, message): # handle message as images / filter / etc batch = self.create_batch(message)
- 38. 2.1 Batcher mother class class Batcher(object): def handle_request(self, message): # handle message as images / filter / etc batch = self.create_batch(message) result = self.process_batch(batch) # will be redefined
- 39. 2.1 Batcher mother class class Batcher(object): def handle_request(self, message): # handle message as images / filter / etc batch = self.create_batch(message) result = self.process_batch(batch) # will be redefined # format result into zmq response return self.format_result(result)
- 40. 2.1 Batcher mother class class Batcher(object): def __init__(self): … def get_parser(self): … def run_server(self): …
- 41. 2.2 Derive Batcher class in your new module
- 42. 2.2 Derive Batcher class in your new module batcher_init
- 43. 2.2 Derive Batcher class in your new module batcher_init process_batch
- 44. 2.2 Derive Batcher class in your new module batcher_init process_batch entrypoint
- 45. 2.2 Batcher child class
- 46. 2.2 Batcher child class class MyBatcher(Batcher): def batcher_init(self):
- 47. 2.2 Batcher child class class MyBatcher(Batcher): def batcher_init(self): # extract model config from args config = self.make_config(self.args)
- 48. 2.2 Batcher child class class MyBatcher(Batcher): def batcher_init(self): # extract model config from args config = self.make_config(self.args) self.model = MyModel(config)
- 49. 2.2 Batcher child class class MyBatcher(Batcher): def process_batch(self, batch):
- 50. 2.2 Batcher child class class MyBatcher(Batcher): def process_batch(self, batch): pred = self.model.predict(batch)
- 51. 2.2 Batcher child class class MyBatcher(Batcher): def process_batch(self, batch): tic = time.time() pred = self.model.predict(batch) toc = time.time() pred_time = toc - tic
- 52. 2.2 Batcher child class class MyBatcher(Batcher): def process_batch(self, batch): tic = time.time() pred = self.model.predict(batch) toc = time.time() pred_time = toc - tic return self.format_result(pred, pred_time)
- 53. 2.2 Batcher child class - Entrypoint if __name__ == "__main__":
- 54. 2.2 Batcher child class - Entrypoint if __name__ == "__main__": batcher = MyBatcher() batcher.run_server()
- 60. One api to rule them all
- 61. One api to rule them all docker compose
- 62. One api to rule them all docker compose api waits for every container
- 63. One api to rule them all docker compose api waits for every container api gets inference requests
- 64. One api to rule them all docker compose api waits for every container api gets inference requests api talks to containers
- 66. Final words
- 68. lvl 1
- 70. lvl 1 Manual training Script for inference
- 71. lvl 1 Manual training Script for inference No real prod
- 72. lvl 1 Manual training Script for inference No real prod lvl 2
- 73. lvl 1 Manual training Script for inference No real prod lvl 2
- 74. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training
- 75. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline
- 76. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline Fixed
- 77. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline Fixed lvl 3
- 78. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline Fixed lvl 3 Training pipeline
- 79. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline Fixed lvl 3 Training pipeline Prod pipeline
- 80. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline Fixed lvl 3 Training pipeline Prod pipeline Distributed
- 81. lvl 1 Manual training Script for inference No real prod lvl 2 Manual training Prod pipeline Fixed lvl 3 Training pipeline Prod pipeline Distributed Celery MRQ Python RQ
- 82. Thank you!