Towards a Serverless Platform for Edge AI
1 like494 views
The document discusses a proposed serverless platform for edge AI that incorporates model operationalization and cloud-based lifecycle management. It highlights various components such as model selectors, policies, and data locality awareness, while also addressing the design of a custom Kubernetes scheduler for effective data handling. Additionally, several challenges and considerations regarding the architecture and functionality of the system are mentioned.
![5
Serverless Model
{} Event (Request)
Trigger
Node
λ
λ
λ
λ
λ
λ
λ
λ
λ
Resource
λ
def handle(req):
s3 = boto3.client('s3')
with open(tmpfile, 'wb') as f:
s3.download_fileobj('bucket', req['obj'], f)
data = numpy.load(f)
m = train_model(data, req['train_params'])
s3.upload_fileobj(serialize(m), 'bucket', 'model'])
# ...
λλ
Function
Scheduler
Cloud Platform](https://image.slidesharecdn.com/presentation-hotedge19-trausch-190709232808/85/Towards-a-Serverless-Platform-for-Edge-AI-5-320.jpg)
![6
Deviceless Model
{} Event (Request)
Trigger
λλ
Function
Scheduler
??
Edge Cloud
Edge
Edge Cloud
Platform
λ
def handle(req):
s3 = boto3.client('s3')
with open(tmpfile, 'wb') as f:
s3.download_fileobj('bucket', req['obj'], f)
data = numpy.load(f)
m = train_model(data, req['train_params'])
s3.upload_fileobj(serialize(m), 'bucket', 'model'])
# ...](https://image.slidesharecdn.com/presentation-hotedge19-trausch-190709232808/85/Towards-a-Serverless-Platform-for-Edge-AI-6-320.jpg)
![8
@consumes.model(selector={
'type': 'image_classifier',
'data_tags': ['machine_x'],
'accuracy': '>=0.88'
})
def inference(model: Model, request):
data = request['input']
# data prep tasks
prediction = model.estimate(data)
@policy.deadline('2s')
@policy.fn(node = 'user_device',
capability = 'gpu')
@policy.data(network=['company_network'],
strict=True)
@consumes.data(
selector={'urn': 'mnist:data'},
holdout=0.2)
@produce.model(
type='classifier',
urn='mnist:model')
def train(data: Data, request) -> Model:
arr = data.to_ndarray()
return Model(train_model(arr))
@gate.bias(attribute = 'age',
predicate = '<0.8')
@gate.drift(metric = 'confidence',
predicate = '<0.2')
λ](https://image.slidesharecdn.com/presentation-hotedge19-trausch-190709232808/85/Towards-a-Serverless-Platform-for-Edge-AI-8-320.jpg)
![13
Dipl.-Ing. (MSc), BSc
Thomas Rausch
Research Assistant
TU Wien
Institute of Information Systems Engineering
Argentinierstrasse 8-194-02, Vienna, Austria
T: +43 1 58801-184838
E: trausch@dsg.tuwien.ac.at
https://dsg.tuwien.ac.at/staff/trausch
@consumes.model(selector={'urn': 'model:base'})
@consumes.data(batch = 100, selector=...)
@produces.model(type='regressor', urn='model:user:{usr}')
@policy.fn(node = 'local')
@policy.data(network = 'local', strict=True)
def refine(model: Model, data: Data):
ndarr = data.to_ndarray() # data artifact API
# transfer learning code
return refined_model
Network (edge, private)
node:{user}
container
Network (cloud)
f(x)
model u
data
data locality node
model b
λ
Function preprocessor
Scheduler
{} Event (Request)
Trigger
λλ
Function
Scheduler
Edge Cloud
Edge
Cloud
Platform
λ
def handle(req):
s3 = boto3.client('s3')
with open(tmpfile, 'wb') as f:
s3.download_fileobj('bucket', req['obj'], f)
data = numpy.load(f)
m = train_model(data, req['train_params'])
s3.upload_fileobj(serialize(m), 'bucket', 'model'])
# ...
λ](https://image.slidesharecdn.com/presentation-hotedge19-trausch-190709232808/85/Towards-a-Serverless-Platform-for-Edge-AI-13-320.jpg)
Towards a Serverless Platform for Edge AI
- 1. TU Wien, Vienna Austria Distributed Systems Group https://dsg.tuwien.ac.at Thomas Rausch @thrauat Waldemar Hummer Vinod Muthusamy Alexander Rashed Schahram Dustdar Towards a Serverless Platform for Edge AI IBM Research AI HotEdge’19, Renton, WA
- 2. 2 Drone With Accelerator Microsoft Build 2018 // Vision Keynote: https://www.youtube.com/watch?v=rd0Rd8w3FZ0
- 3. 3 Edge AI Accelerators Google Edge TPU NVIDIA Jetson Intel Neural Compute Stick Baidu Kunlun Microsoft Project BrainWave Huawei Atlas
- 4. 4 AI Operationalization Hummer et al., ModelOps: Cloud-based Lifecycle Management for Reliable and Trusted AI. IC2E’19. Process Train Validate Servee Model Runtime Monitoring Data Perf. Process Train Validate Serve Object Store Compute Cluster Learning Cluster Read Data Train Model Write Model Data Asset Trained Model ModelOps Platform
- 5. 5 Serverless Model {} Event (Request) Trigger Node λ λ λ λ λ λ λ λ λ Resource λ def handle(req): s3 = boto3.client('s3') with open(tmpfile, 'wb') as f: s3.download_fileobj('bucket', req['obj'], f) data = numpy.load(f) m = train_model(data, req['train_params']) s3.upload_fileobj(serialize(m), 'bucket', 'model']) # ... λλ Function Scheduler Cloud Platform
- 6. 6 Deviceless Model {} Event (Request) Trigger λλ Function Scheduler ?? Edge Cloud Edge Edge Cloud Platform λ def handle(req): s3 = boto3.client('s3') with open(tmpfile, 'wb') as f: s3.download_fileobj('bucket', req['obj'], f) data = numpy.load(f) m = train_model(data, req['train_params']) s3.upload_fileobj(serialize(m), 'bucket', 'model']) # ...
- 7. 7 Data and Models as first-class citizens Model Selectors Policies Gates AI Workflow Programming Model Deviceless function scheduling Policy enactment Context awareness Data locality awareness Execution Platform A Serverless Platform for Edge AI λ λ
- 8. 8 @consumes.model(selector={ 'type': 'image_classifier', 'data_tags': ['machine_x'], 'accuracy': '>=0.88' }) def inference(model: Model, request): data = request['input'] # data prep tasks prediction = model.estimate(data) @policy.deadline('2s') @policy.fn(node = 'user_device', capability = 'gpu') @policy.data(network=['company_network'], strict=True) @consumes.data( selector={'urn': 'mnist:data'}, holdout=0.2) @produce.model( type='classifier', urn='mnist:model') def train(data: Data, request) -> Model: arr = data.to_ndarray() return Model(train_model(arr)) @gate.bias(attribute = 'age', predicate = '<0.8') @gate.drift(metric = 'confidence', predicate = '<0.2') λ
- 9. 9 @consumes.model(selector={'urn': 'model:base'}) @consumes.data(batch = 100, selector=...) @produces.model(type='regressor', urn='model:user:{usr}') @policy.fn(node = 'local') @policy.data(network = 'local', strict=True) def refine(model: Model, data: Data): ndarr = data.to_ndarray() # data artifact API # transfer learning code return refined_model @consumes.model(selector={'urn': 'model:base'}) @consumes.data(batch = 100, selector=...) @produces.model(type='regressor', urn='model:user:{usr}') @policy.fn(node = 'local') @policy.data(network = 'local', strict=True) def refine(model: Model, data: Data): ndarr = data.to_ndarray() # data artifact API # transfer learning code return refined_model Network (edge, private) node:{user} container Network (cloud) f(x) model u data data locality node model b λ Function preprocessor Scheduler
- 10. 10 Data Locality Tradeoffs Cluster Middleware Cluster Middleware Cluster Middleware Cluster Middleware h Data proximity Container Image Deploy the container image to the edge? OR Send the data to the cloud? Edge
- 11. 11 Skippy Built on and Kubernetes Kubernetes daemon to discover node capabilities Custom Python-based Kubernetes scheduler ● Adds inter-node proximity and data locality as constraints ● Non-monolithic architecture Coming to GitHub soon™ λ
- 12. 12 Preprocess Train Inferenceλ λλ Scheduler + Simulator: https://git.dsg.tuwien.ac.at/serverless-edge-ai/sched-sim λ
- 13. 13 Dipl.-Ing. (MSc), BSc Thomas Rausch Research Assistant TU Wien Institute of Information Systems Engineering Argentinierstrasse 8-194-02, Vienna, Austria T: +43 1 58801-184838 E: trausch@dsg.tuwien.ac.at https://dsg.tuwien.ac.at/staff/trausch @consumes.model(selector={'urn': 'model:base'}) @consumes.data(batch = 100, selector=...) @produces.model(type='regressor', urn='model:user:{usr}') @policy.fn(node = 'local') @policy.data(network = 'local', strict=True) def refine(model: Model, data: Data): ndarr = data.to_ndarray() # data artifact API # transfer learning code return refined_model Network (edge, private) node:{user} container Network (cloud) f(x) model u data data locality node model b λ Function preprocessor Scheduler {} Event (Request) Trigger λλ Function Scheduler Edge Cloud Edge Cloud Platform λ def handle(req): s3 = boto3.client('s3') with open(tmpfile, 'wb') as f: s3.download_fileobj('bucket', req['obj'], f) data = numpy.load(f) m = train_model(data, req['train_params']) s3.upload_fileobj(serialize(m), 'bucket', 'model']) # ... λ
- 14. 14 Discussion ● Correct level of abstraction? ● API/SDK features? ● Validation criteria? ● Deviceless model (does it work?) ● Transparent data management ● Scheduler architecture ● Request routing architecture ● Proximity and bandwidth monitoring ● Learning optimal placements ● Model too high-level for scheduler ● “Bring-your-own-device” will fail Feedbacki Controversial pointsii Open issuesiii Failure risksiv