ソーシャルゲームログ解析基盤のHadoop活用事例
25 likes5,309 views
The document outlines a technical architecture involving Hadoop, Amazon EMR, and MongoDB for data processing and storage, emphasizing components such as Fluentd for data ingestion and sharding for MongoDB. It includes references to various AWS resources, data actions, and configurations relevant to performance metrics. Additionally, it mentions the use of EMR for data processing tasks with specified instance types and their resource allocations.
![8.5GB 1.4GB /
ID
Nov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345
[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45
Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345
[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5
8](https://image.slidesharecdn.com/20111108hadoop-111109033237-phpapp01/85/Hadoop-8-320.jpg)
![MongoDB
MongoDB ( )
"app" : "hogehoge",
"userid" : "12345",
ID
"dateint" : 20111101,
"hourint" : 23,
"actions" : [
"CONSUME_POWER", MongoDB Sharding
"ADD_MONEY"
],
"records" : [
"action" : "ADD_MONEY",
"timeint" : 235959,
]
10](https://image.slidesharecdn.com/20111108hadoop-111109033237-phpapp01/85/Hadoop-10-320.jpg)
ソーシャルゲームログ解析基盤のHadoop活用事例
- 1. Hadoop gumi 1
- 2. : Twitter:@CkReal gumi 2 AWS S3,EMR 2
- 3. 1. 2. Elastic MapReduce EMR 3. 4. EMR 3
- 4. 4
- 5. 2 ( 1 ) fluentd 5
- 6. 1 1,000 / AP APAP AP DB d fluentd fluentd mongos mongod(PRIMARY) DB config mongod(SECONDARY) DB fluentd mongos mongod(SECONDARY) config ReplicaSets & Sharding NFS 6
- 7. EMR 7
- 8. 8.5GB 1.4GB / ID Nov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345 [BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45 Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345 [BeforePower] 25 [AfterPower] 20 [ConsumePower] 5 8
- 9. NFS NFS 9
- 10. MongoDB MongoDB ( ) "app" : "hogehoge", "userid" : "12345", ID "dateint" : 20111101, "hourint" : 23, "actions" : [ "CONSUME_POWER", MongoDB Sharding "ADD_MONEY" ], "records" : [ "action" : "ADD_MONEY", "timeint" : 235959, ] 10
- 11. EMR Hive Pig Hadoop Streaming Hadoop Streaming (Python) 11
- 12. m2.4xlarge × 1 4.9GB 85 EMR(m2.xlarge) × 5 4.9GB 44 m2.4xlarge × 1 7.2GB 138 EMR(m2.xlarge) × 5 7.2GB 69 (Macbook Air) 3.6GB 30 … 12
- 13. EMR CPU 13
- 14. 14
- 15. ( ) NFS Amazon S3 EMR S3 EMR S3 EMR config MongoDB mongos 15
- 16. S3 boto S3 c3cmd S3 EMR Mapper,Reducer,Python2.7 MongoDB pymongo MongoDB EMR Client Tool(Ruby) EMR 16
- 17. EMR 17
- 18. EMR S3 EC2⇔S3 20MB/sec Hadoop HadoopStreaming EMR 18
- 19. GB/ 19
- 20. 20
- 21. 21