The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015
1 like684 views
The Mapbox Directions Pipeline aims to always have the freshest map data available for routing. It involves getting the latest OpenStreetMap data, pre-processing it for directions, loading the new data into API servers, and then repeating the process. Each step uses its own CloudFormation stack. The pipeline downloads planet files from OpenStreetMap, pre-processes them for different transport profiles, uploads the results to S3, and updates the API CloudFormation stacks to fetch the new data.
The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015
- 5. 1. Get&OpenStreetMap&data 2. Pre1process&for&Direc8ons 3. Load&new&data&into&API&servers 4. Repeat • Every'step'is'an'own'CloudForma6on'stack.
- 6. Overview 1. Get$OpenStreetMap$data 2. Pre'process,for,Direc0ons 3. Load,new,data,into,API,servers 4. Repeat • Every'step'is'own'CloudForma6on'stack.
- 7. OpenStreetMap • Planet.osm,is,all,OpenStreetMap,data,in,one,file • New,version,release,every,week • Big,file:,576,GB,uncompressed,,28,GB,as,compressed,protobuf • Incremental,diff,update,available,every,minute
- 8. OpenStreetMap • 1#EC2#c3.2xlarge#instance • Replays#latest#changesets#to#generate#a#new#planet#file • New#planet#every#2#1/2#hours,#uploaded#to#S3 • Updates#a#file#/latest#with#a#reference#to#most#recent#planet • Old#planets#are#purged#via#S3#Lifecycle#Object#ExpiraHon
- 9. Overview 1. Get$OpenStreetMap$data 2. Pre'process,for,Direc0ons 3. Load,new,data,into,API,servers 4. Repeat
- 10. Overview 1. Get&OpenStreetMap&data 2. Pre$process)for)Direc-ons 3. Load&new&data&into&API&servers 4. Repeat
- 11. Direc&ons*pre,processing • 1#EC2#r3.4xlarge#on#Spot#Pricing#per#profile • Fetch#latest#OSM#planet#from#S3#(discovery#via#/latest) • Run#preDprocessing#for#profile • Car:#6#Hours • Bicycle:#15#Hours • Walk:#23#Hours
- 13. Overview 1. Get&OpenStreetMap&data 2. Pre$process)for)Direc-ons 3. Load&new&data&into&API&servers 4. Repeat
- 14. Overview 1. Get&OpenStreetMap&data 2. Pre1process&for&Direc8ons 3. Load%new%data%into%API%servers 4. Repeat
- 15. Direc&ons*API • One%CloudForma/on%stack%per%Profile • Several%EC2s%r3.2xlarge • ELB/AutoScaling • On%start,%downloads%latest%Direc/ons%data%from%S3 • EC2%instances%are%cycled%on%CloudForma/on%parameter%update
- 16. Direc&ons*pre,processing aws cloudformation update-stack --stack-name "$ApiDirectionsStack" --use-previous-template --capabilities "CAPABILITY_IAM" --parameters $params ParameterKey=LatestTimstamp,ParameterValue="$TimeStamp"
- 17. Direc&ons*API*CloudForma&on*Template { "Parameters": { "LatestTimstamp": { "Type": "String" } }, "LaunchConfiguration": { "Type": "AWS::AutoScaling::LaunchConfiguration", "UserData": { { "Ref": "LatestTimstamp" } } }, "AutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "UpdatePolicy": { "AutoScalingRollingUpdate": { } } } }
- 19. The$Good • Few%moving%parts%and%easy%concept • Upda5ng%stack%parameter%mirrors%our%exis5ng%deployment%flow • Data<driven%approach%decouples%stacks • no%queues • Easy%1:n%distribu5on • Star5ng%new%stacks%is%easy,%they%just%pick%up%latest%state
- 20. The$Bad • Upda&ng)CloudForma&on)stack)parameters)needs)a)lot)of)IAM) permissions • Poten&al)security)problem)if)instance)is)hacked • AutoScaling)is)not)scopeable)by)resource • Poten&al)for)UPDATE_ROLLBACK_FAILED • Problem)with)UpdatePolicy,)AutoScaling)and)nonBboo&ng) EC2s
- 21. Open%Source • h#ps://wiki.openstreetmap.org/wiki/Osmosis • Process7and7forward7OpenStreetMap7data • h#p://project=osrm.org/ • Open%Source%Rou,ng%Machine7for7doing7Direc?ons7queries • Load7OpenStreetMap7data • Pre=processing7the7data7for7a7profile • Do7queries7against7the7pre=processed7data
- 23. 1. Get&OpenStreetMap&data 2. Pre1process&for&Direc8ons 3. Load&new&data&into&API&servers 4. Repeat @freenerd johan@mapbox.com