Big Data and Machine Learning Workshop - Day 3 @ UTACM

1
۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬
‫ﺻﺪﯾﻘﯽ‬ ‫ﺍاﻣﯿﺮ‬
‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
@amirsedighi :‫ﺗﻮ4ﺘﺮ‬
:‫ﺍاﯾﻤﯿﻞ‬sedighi@gmail.com
‫‌ﻫﺎ‬‫ﺭر‬‫ﺍاﺑﺰﺍا‬ ‫ﻭو‬ ‫‌ﻫﺎ‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ،‫ﺩدﺍاﺩدﻩه‬ ‫‌ﻫﺎﯼی‬‫ﻥن‬‫ﺟﺮﯾﺎ‬ ،(‫‌ﺩدﺍاﺩدﻩه‬‫ﻪ‬ ِ‫)ﻣ‬ ‫‌ﺑﺰﺭرﮒگ‬‫ﯼی‬‫‌ﻫﺎ‬‫ﻩه‬‫ﺩدﺍاﺩد‬ - ‫ﺳﻮﻡم‬ ‫ﺭرﻭوﺯز‬

‫ﺻﺪﯾﻘﯽ‬ ‫ﺍاﻣﯿﺮ‬
:‫ﻣﻮﺳﺲ‬
2
‫ﻣﻌﺮﻓﯽ‬
http://recommender.ir
http://helio.ir
http://commentum.ir
@amirsedighi :‫ﺗﻮ4ﺘﺮ‬
:‫ﺍاﯾﻤﯿﻞ‬sedighi@gmail.com

3
‫‌ﺁآﻣﻮﺯزﯾﻢ‬‫ﯽ‬‫ﻣ‬ ‫ﺍاﻣﺮﻭوﺯز‬ ‫ﺁآﻧﭽﻪ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
(BigData Processing) (‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫‌ﺩدﺍاﺩدﻩه/ﺩدﺍاﺩد‬‫ﻥن‬‫‌ﺩدﯾﺘﺎ/ﮐﻼ‬‫ﮓ‬‫)ﺑﯿ‬ ‫‌ﺩدﺍاﺩدﻩه‬‫ﻪ‬ ِ‫ﻣ‬ ‫ﭘﺮﺩدﺍاﺯزﺵش‬ ‫ﻣﻔﺎﻫﯿﻢ‬ ‫ﺑﺎ‬ ‫ﺁآﺷﻨﺎﯾﯽ‬
(Data Streaming) ‫ﺩدﺍاﺩدﻩه‬ ‫‌ﻫﺎﯼی‬‫ﻥن‬‫ﺟﺮﯾﺎ‬ ‫ﭘﺮﺩدﺍاﺯزﺵش‬ ‫ﻣﻔﺎﻫﯿﻢ‬ ‫ﺑﺎ‬ ‫ﺁآﺷﻨﺎﯾﯽ‬
‫ﺩدﺍاﺩدﻩه‬ ‫‌ﻫﺎﯼی‬‫ﻥن‬‫ﺟﺮﯾﺎ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫ﺣﻮﺯزﻩه‬ ‫ﺩدﺭر‬ ‫ﺻﻨﻌﺖ‬ ‫ﻭو‬ ‫‌ﮐﺎﺭر‬‫ﻭو‬‌‫ﺐ‬‫ﮐﺴ‬ ‫‌ﻫﺎﯼی‬‫ﺯز‬‫ﻧﯿﺎ‬ ‫ﺑﺎ‬ ‫ﺁآﺷﻨﺎﯾﯽ‬
‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‌‫ﺵش‬‫ﭘﺮﺩدﺍاﺯز‬ ‫ﺯزﯾﺮﺳﺎﺧﺖ‬ ‫‌ﺗﺮﯾﻦ‬‫ﺭر‬‫ﻣﺸﻬﻮ‬ ‫ﺑﮑﺎﺭرﮔﯿﺮﯼی‬ ‫ﻧﻤﻮﻧﻪ‬ ‫ﯾﮏ‬
‫‌ﺷﯿﻢ‬‫ﯽ‬‫ﻣ‬ ‫ﺁآﺷﻨﺎ‬ ‫ﺣﻮﺯزﻩه‬ ‫ﺍاﯾﻦ‬ ‫ﭘﺮﮐﺎﺭرﺑﺮﺩد‬ ‫‌ﺑﺎﺯز‬‫ﻦ‬‫ﻣﺘ‬ ‫ﺍاﺑﺰﺍاﺭرﻫﺎﯼی‬ ‫ﺑﺎ‬

4
‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
‫ﭘﺎﯾﻪ‬ ‫ﻣﻔﺎﻫﯿﻢ‬ ‫ﻭو‬ ‫ﺗﺎﺭرﯾﺨﭽﻪ‬
‫ﺳﺮﻓﺼﻞ‬
‫‌ﻫﺎ‬‫ﺭر‬‫ﺍاﺑﺰﺍا‬ ‫ﻭو‬ ‫‌ﻫﺎ‬‫ﺵش‬‫ﺭرﻭو‬

5
‫ﻭوﺍاﻗﻌﯿﺖ‬ ‫ﺗﺎ‬ ‫ﺗﺼﻮﺭرﺍاﺕت‬ ‫ﺍاﺯز‬

6
‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬
‫‌ﺩدﺍاﺩدﻩه‬‫ﻪ‬ ِ‫ﻣ‬
‫‌ﺩدﺍاﺩدﻩه‬‫ﻥن‬‫ﮐﻼ‬
‫‌ﺩدﯾﺘﺎ‬‫ﮓ‬‫ﺑﯿ‬
Big-Data
‫ﻣﻘﺪﻣﻪ‬

7
:‫ﭘﯿﺶ‬ ‫ﺳﺎﻝل‬ ‫ﭼﻨﺪ‬ ‫ﻫﻤﯿﻦ‬
‫ﺩدﯾﺮﻭوﺯز‬

8

9

10

11
640KB
1.2MB

12
‫‌ﻫﺎ‬‫ﺭر‬‫ﻫﻨﺠﺎ‬ ‫ﺮ‬K‫ﺗﻐ‬

13
‫ﺷﺪ‬ ‫ﺷﺮﻭوﻉع‬ ‫ﮐﺠﺎ‬ ‫ﺍاﺯز‬ ‫ﺩدﺍاﺳﺘﺎﻥن‬

14

15

16

17

18

19

20
‫ﺩدﺍاﺩدﻩه‬ ‫ﭘﺮﺩدﺍاﺯزﺵش‬ ‫ﻭو‬ ‫ﺗﮑﺜﯿﺮ‬ ،‫ﺗﻮﻟﯿﺪ‬ ‫‌ﻫﺎﯼی‬‫ﺵش‬‫ﺭرﻭو‬ ‫ﺗﮑﺎﻣﻞ‬

21
‫ﮐﺎﺭرﺑﺮﺍاﻥن‬ ‫ﺗﻨﻮﻉع‬ ‫ﻭو‬ ‫ﺗﻌﺪﺍاﺩد‬ ‫ﺭرﺷﺪ‬ ‫ﺍاﺯز‬ ‫ﻧﺎﺷﯽ‬ ‫ﺍاﺛﺮﺍاﺕت‬

22
Welcome to the Machine‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

23
(‫)ﺧﻮﺩدﻣﺎﻧﯽ‬ ‫ﺗﻌﺮﯾﻒ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
‫‌ﺷﻮﻧﺪ‬‫ﯽ‬‫ﻣ‬ ‫ﻣﺴﺌﻠﻪ‬ ‫ﺍاﺯز‬ ‫ﺑﺨﺸﯽ‬ ‫‌ﻫﺎ‬‫ﻩه‬‫ﺩدﺍاﺩد‬
(… SQL-Server ،MySQL ،‫)ﺍاﮐﺴﻞ‬ ‫ﻣﺘﺪﺍاﻭوﻝل‬ ‫‌ﻫﺎﯼی‬‫ﺵش‬‫ﺭرﻭو‬ ‫ﻧﺎﺗﻮﺍاﻧﯽ‬

24
‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫ﺍاﺯز‬ ‫ﮔﺎﺭرﻧﺘﺮ‬ ‫ﺗﻌﺮﯾﻒ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

25
‫ﺍاﻧﺪﺍاﺯزﻩه‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

26
‫ﺳﺮﻋﺖ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

27
‫ﺗﻨﻮﻉع‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

28
‫ﺻﺤﺖ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

29
‫ﺛﺎﻧﯿﻪ‬ ‫ﯾﮏ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

30
‫ﻣﻘﺪﻡم‬ ‫ﺧﻂ‬

31
Data Scientist
Data Engineer

32

33
‫‌ﻫﺎ؟‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫ﮐﺪﺍاﻡم‬
‫ﻭوﺏب‬ ‫ﺻﻔﺤﺎﺕت‬

34
!‫ﭘﺴﻨﺪﯾﺪﻡم‬

35
(‫ﮐﺎﺭرﺑﺮﺍاﻥن‬ ‫)ﻧﻈﺮ‬ ‫‌ﻫﺎ‬‫ﺖ‬‫ﮐﺎﻣﻨ‬

36
‌‫ﻮ‬‫ﻭوﯾﺪﺋ‬

37
‫ﻣﺎﻟﯽ‬ ‫‌ﻫﺎﯼی‬‫ﺶ‬‫ﺗﺮﺍاﮐﻨ‬

38
‫‌ﻫﺎ‬‫ﺖ‬4‫ﺗﻮ‬

39
‫ﺗﺼﻮﯾﺮ‬

40
‫ﻻﮒگ‬

41
‫ﻻﮒگ‬
‫ﮐﺎﺭرﺑﺮﺍاﻥن‬ ‫‌ﻫﺎﯼی‬‫ﮏ‬‫ﮐﻠﯿ‬

42
‫ﻻﮒگ‬
‫‌ﺳﺮﻭوﺭر‬‫ﺏب‬‫ﻭو‬ ‫ﻻﮒگ‬

43
‫ﻻﮒگ‬
‫ﺍاﭘﻠﯿﮑﯿﺸﻦ‬ ‫ﻻﮒگ‬

44
‫ﻻﮒگ‬
‫ﺳﯿﺴﺘﻢ‬ ‫ﻻﮒگ‬

45
‫ﻻﮒگ‬
‫ﺳﯿﺴﺘﻢ‬ ‫ﻻﮒگ‬
(‫‌ﻫﺎ‬‫ﺰ‬‫‌ﭼﯿ‬‫ﺖ‬‫)ﺍاﯾﻨﺘﺮﻧ‬ ‫ﺯزﻧﺪﮔﯽ‬ ‫ﻻﮒگ‬

46
‫ﺍاﺻﻠﯽ‬ ‫ﻣﻨﺒﻊ‬

47
‫ﺍاﺻﻠﯽ‬ ‫ﻣﻨﺒﻊ‬
http://hadoopilluminated.com/hadoop_illuminated

48
‫‌ﻫﺎ‬‫ﺖ‬‫ﻣﺤﺪﻭوﺩدﯾ‬
‫ﭘﺮﺩدﺍاﺯزﺵش‬ ‫ﺑﺮﺍاﯼی‬ (Platform) ‫‌ﺳﺎﺯزﻩه‬‫ﻦ‬‫ﺑ‬ ‫ﯾﮏ‬ ‫ﺩدﺳﺘﺮﺱس‬ ‫ﺩدﺭر‬ ‫ﻭو‬ ‫ﺍاﺭرﺯزﺍاﻥن‬ ‫ﺗﺠﻬﯿﺰﺍاﺕت‬ ‫ﺗﻮﺳﻂ‬ ‫ﭼﮕﻮﻧﻪ‬
‫ﺑﺒﯿﻨﯿﻢ؟‬ ‫ﺗﻬﯿﻪ‬ ‫‌ﺑﺰﺭرﮒگ‬‫ﯼی‬‫‌ﻫﺎ‬‫ﻩه‬‫ﺩدﺍاﺩد‬

49
(Scalability) ‫‌ﭘﺬﯾﺮﯼی‬‫ﺱس‬‫ﻣﻘﯿﺎ‬

50

51
Scale-Up

52
Scale-Out
Scale-Up

53

54

55
‫ﺳﺮﺁآﻏﺎﺯز‬

56
1999

57
1999
2002

58
1999
2002

59
‫ﭼﮕﻮﻧﻪ؟‬
Q!‫‌ﻭوﺭرﯾﻢ؟‬‫ﺎ‬‫ﺑﯿ‬ ‫ﺑﺪﺳﺖ‬ ‫ﺧﻄﯽ‬ ‫ﭘﺬﯾﺮﯼی‬ ‫ﻣﻘﯿﺎﺱس‬ ‫ﻗﯿﻤﺖ‬ ‫ﺍاﺭرﺯزﺍاﻥن‬ ‫ﻣﺎﺷﯿﻦ‬ ‫ﺯزﯾﺎﺩدﯼی‬ ‫ﺗﻌﺪﺍاﺩد‬ ‫ﮐﻤﮏ‬ ‫ﺑﻪ‬ ‫ﭼﮕﻮﻧﻪ‬

60
(Map-Reduce) ‫ﮐﺎﻫﺶ‬ ‫ﻧﮕﺎﺷﺖ‬
Q!‫‌ﻭوﺭرﯾﻢ؟‬‫ﺎ‬‫ﺑﯿ‬ ‫ﺑﺪﺳﺖ‬ ‫ﺧﻄﯽ‬ ‫ﭘﺬﯾﺮﯼی‬ ‫ﻣﻘﯿﺎﺱس‬ ‫ﻗﯿﻤﺖ‬ ‫ﺍاﺭرﺯزﺍاﻥن‬ ‫ﻣﺎﺷﯿﻦ‬ ‫ﺯزﯾﺎﺩدﯼی‬ ‫ﺗﻌﺪﺍاﺩد‬ ‫ﮐﻤﮏ‬ ‫ﺑﻪ‬ ‫ﭼﮕﻮﻧﻪ‬
A(Map-Reduce) ‫ﮐﺎﻫﺶ‬ ‫ﻧﮕﺎﺷﺖ‬

61
1999
2002
2003 ..

62
1999
2002
2006200

63
‫ﭘﺎﯾﻪ‬ ‫ﻣﻔﺎﻫﯿﻢ‬ ‫ﻭو‬ ‫ﺗﺎﺭرﯾﺨﭽﻪ‬
‫‌ﻫﺎ‬‫ﺭر‬‫ﺍاﺑﺰﺍا‬ ‫ﻭو‬ ‫‌ﻫﺎ‬‫ﺵش‬‫ﺭرﻭو‬

64
‫ﭘﺮﺩدﺍاﺯزﺵش‬ ‫ﺗﻮﺍاﻥن‬ ‫ﺍاﻓﺰﺍاﯾﺶ‬ - ‫ﭘﺮﺳﺶ‬
Q‫ﺍاﯾﺸﺎﻥن‬ ‫ﺍاﺯز‬ .‫ﺍاﺳﺖ‬ ‫ﮔﺮﻓﺘﻪ‬ ‫ﻗﺮﺍاﺭر‬ ‫ﻋﻠﯽ‬ ‫ﻭو‬ ‫ﺳﺎﺳﺎﻥن‬ ،‫ﭘﯿﺎﻡم‬ ‫ﺍاﺧﺘﯿﺎﺭر‬ ‫ﺩدﺭر‬ ‫ﻓﺮﺩدﻭوﺳﯽ‬ ‫ﺷﺎﻫﻨﺎﻣﻪ‬ ‫ﺟﻠﺪ‬ ‫ﯾﮏ‬
.‫ﺑﺸﻤﺎﺭرﻧﺪ‬ ‫ﺭرﺍا‬ ‫ﺳﻬﺮﺍاﺏب‬ ‫ﻭو‬ ‫ﺭرﺳﺘﻢ‬ ،‫ﺭرﻭوﺩدﺍاﺑﻪ‬ ‫ﺍاﺳﺎﻣﯽ‬ ‫ﻣﻤﮑﻦ‬ ‫ﺭرﻭوﺵش‬ ‫‌ﺗﺮﯾﻦ‬‫ﻊ‬‫ﺳﺮﯾ‬ ‫ﺑﻪ‬ ‫‌ﺍاﻧﺪ‬‫ﻪ‬‫ﺧﻮﺍاﺳﺘ‬
‫‌ﮐﻨﯿﺪ؟‬‫ﯽ‬‫ﻣ‬ ‫ﭘﯿﺸﻨﻬﺎﺩد‬ ‫ﺭرﻭوﺷﯽ‬ ‫ﭼﻪ‬

65
‫ﭘﯿﺎﻡم‬
‫ﺳﺎﺳﺎﻥن‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﺳﻬﺮﺍاﺏب‬ ‫ﻭو‬ ‫ﺭرﺳﺘﻢ‬ ،‫ﺭرﻭوﺩدﺍاﺑﻪ‬

66
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

67
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

68
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

69
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

70
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

71
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﺭرﻭوﺩدﺍاﺑﻪ‬
‫ﺭرﺳﺘﻢ‬
‫ﺳﻬﺮﺍاﺏب‬

72
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

73
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

74
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬

75
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
map() shuffle() reduce()splitting()

76
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
‫ﻋﻠﯽ‬
map() shuffle() reduce()
[k1,v1] by k1 [k1,[v1, v2, v3 …]]

77

78
‫ﺍاﻃﻤﯿﻨﺎﻥن‬ ‫ﺿﺮﯾﺐ‬ ‫ﺍاﻓﺰﺍاﯾﺶ‬ - ‫ﭘﺮﺳﺶ‬
Q‫ﮐﺎﻣﭙﯿﻮﺗﺮ‬ ‫ﻫﺮ‬ ‫ﺍاﯾﻨﮑﻪ‬ ‫ﺑﻔﺮﺽض‬ .‫‌ﺍاﯾﻢ‬‫ﻩه‬‫ﮐﺮﺩد‬ ‫ﺗﻬﯿﻪ‬ ‫ﻗﯿﻤﺖ‬ ‫ﺍاﺭرﺯزﺍاﻥن‬ ‫ﮐﺎﻣﭙﯿﻮﺗﺮ‬ ۳۶۵ ‫ﺑﺎ‬ ‫ﺧﻮﺷﻪ‬ ‫ﯾﮏ‬
‫ﯾﮏ‬ ‫ﺭرﻭوﺯزﺍاﻧﻪ‬ ‫ﻣﺘﻮﺳﻂ‬ ‫ﻃﻮﺭر‬ ‫ﺑﻪ‬ (‫ﺧﺮﺍاﺑﯽ‬ ‫ﻧﻮﻉع‬ ‫)ﻫﺮ‬ ‫ﺷﻮﺩد‬ ‫ﺧﺮﺍاﺏب‬ ‫ﯾﮑﺒﺎﺭر‬ ‫ﺳﺎﻟﯽ‬ ‫ﺗﻨﻬﺎ‬ ،‫‌ﻗﯿﻤﺖ‬‫ﻥن‬‫ﺍاﺭرﺯزﺍا‬
‫ﭘﺮﺩدﺍاﺯزﺵش‬ ‫ﺳﻼﻣﺖ‬ ‫ﻭو‬ ‫‌ﻫﺎ‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫ﺳﻼﻣﺖ‬ ‫ﭼﮕﻮﻧﻪ‬ ‫ﺷﺮﺍاﯾﻄﯽ‬ ‫ﭼﻨﯿﻦ‬ ‫ﺩدﺭر‬ .‫ﺩدﺍاﺷﺖ‬ ‫ﺧﻮﺍاﻫﯿﻢ‬ ‫ﺧﺮﺍاﺑﯽ‬
‫ﮐﻨﯿﻢ؟‬ ‫ﺗﻀﻤﯿﻦ‬ ‫ﺭرﺍا‬ (‫ﻣﺤﺴﺒﺎﺗﯽ‬ ‫ﻋﻤﻠﯿﺎﺕت‬ ‫)ﺗﮑﺮﺍاﺭرﭘﺬﯾﺮﯼی‬

79
(DFS) ‫ﺷﺪﻩه‬ ‫ﺗﻮﺯزﯾﻊ‬ ‫ﻓﺎﯾﻞ‬ ‫ﺳﺎﻣﺎﻧﻪ‬

80
(DFS) ‫ﺷﺪﻩه‬ ‫ﺗﻮﺯزﯾﻊ‬ ‫ﻓﺎﯾﻞ‬ ‫ﺳﺎﻣﺎﻧﻪ‬

81
‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﺰ‬‫ﺳﺎﯾ‬ ‫ﺑﻼﮎک‬

82
‫‌ﺷﺪﻩه‬‫ﻊ‬‫‌ﺗﻮﺯزﯾ‬‫ﻞ‬‫‌ﻓﺎﯾ‬+ ‫‌ﺷﺪﻩه‬‫ﻊ‬‫ﺗﻮﺯزﯾ‬ ‫ﭘﺮﺩدﺍاﺯزﺵش‬ = ‫ﻫﺪﻭوﭖپ‬
Hadoop = HDFS + MapReduce

83
‫ﺭرﻋﯿﺘﯽ‬ /‫ﺍاﺭرﺑﺎﺏب‬

84
‫ﺭرﻋﯿﺘﯽ‬ /‫ﺍاﺭرﺑﺎﺏب‬

85
‫ﻫﺪﻭوﭖپ‬

86
‫ﺩدﯾﮕﺮ‬ ‫‌ﻫﺎﯼی‬‫ﺭر‬‫ﺍاﺑﺰﺍا‬ ‫ﺑﺎ‬ ‫ﺁآﺷﻨﺎﯾﯽ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
‫‌ﺑﺎﺯز‬‫ﻦ‬‫ﻣﺘ‬ ‫ﻗﺪﺭرﺗﻤﻨﺪ‬ ‫ﺍاﺑﺰﺍاﺭر‬ ‫ﭼﻨﺪ‬ ‫ﺑﺎ‬ ‫ﺁآﺷﻨﺎﯾﯽ‬

(Transfer) ‫ﺟﺎﺑﺠﺎﯾﯽ‬
‫‌ﺳﺎﺯزﯼی‬‫ﻩه‬‫ﺫذﺧﯿﺮ‬
(Join) ‫ﺍاﺭرﺗﺒﺎﻁط‬ ‫ﺑﺮﻗﺮﺍاﺭرﯼی‬
(Index) ‫‌ﮔﺬﺍاﺭرﯼی‬‫ﺲ‬‫ﺍاﻧﺪﯾ‬
(Analytics) ‫ﺗﺤﻠﯿﻞ‬
(Aggregate) ‫ﺗﺠﻤﯿﻊ‬
(Visualize) ‫‌ﺳﺎﺯزﯼی‬‫ﻪ‬‫ﻧﻤﺎﯾ‬
87
‫ﻣﺎ‬ ‫‌ﻫﺎﯼی‬‫ﯼی‬‫ﻧﯿﺎﺯزﻣﻨﺪ‬
Image ref: http://supplychaininsightsglobalsummit.com/steps-4-5-to-regain-manufacturing-prowess/

88
‫‌ﺩدﺍاﺩدﻩه‬‫ﻩه‬‫ﭘﺎﯾﮕﺎ‬ ‫ﺍاﻧﻮﺍاﻉع‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

89
SQL ‫ﻭوﺍاﺳﻂ‬ ‫ﺍاﻧﻮﺍاﻉع‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
• Hive
• Drill
• Impala

90
‫ﺩدﺍاﺩدﻩه‬ ‫ﺍاﻧﺘﻘﺎﻝل‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

91
‫ﮔﺬﺍاﺭرﯼی‬ ‫ﺍاﻧﺪﯾﺲ‬ ‫ﻭو‬ ‫ﺟﺴﺘﺠﻮ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

92
‫ﺻﻒ‬ ‫ﻣﺪﯾﺮﯾﺖ‬ ‫ﻭو‬ ‫ﺭرﺳﺎﻧﯽ‬ ‫ﭘﯿﺎﻡم‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

93
‫ﻓﺎﯾﻞ‬ ‫ﻻﮒگ‬ ‫ﻣﺪﯾﺮﯾﺖ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

94
‫ﺩدﺍاﺩدﻩه‬ ‫ﺟﺮﯾﺎﻥن‬ ‫ﭘﺮﺩدﺍاﺯزﺵش‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
Apache Flink

95
‫ﻣﺎﺷﯿﻦ‬ ‫ﯾﺎﺩدﮔﯿﺮﯼی‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
Apache Hama
TensorFlow
SystemML

96
‫ﺳﺎﺯزﯼی‬ ‫ﭘﯿﺎﺩدﻩه‬
‫ﺑﻨﻮﯾﺴﯿﻢ‬ ‫ﮐﺪ‬

97
.‫ﮐﻨﯿﺪ‬ ‫ﻣﺮﺗﺐ‬ ‫ﻓﺮﺍاﻭوﺍاﻧﯽ‬ ‫ﺍاﺳﺎﺱس‬ ‫ﺑﺮ‬ ‫ﻭو‬ ‫ﺑﺸﻤﺎﺭرﯾﺪ‬ ‫ﺭرﺍا‬ ‫ﺷﺎﻫﻨﺎﻣﻪ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﻭوﺍاﮊژ‬ ‫ﻫﺪﻭوﭖپ‬ ‫ﮐﻤﮏ‬ ‫ﺑﻪ‬

98
‫ﻣﻠﺰﻭوﻣﺎﺕت‬ - ‫ﻧﮕﺎﺷﺖ‬ ‫ﮐﺎﻫﺶ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
Prerequisites:
• Windows, Linux or Mac
• Java 1.7
• Apache Maven 3

99
‫ﻫﺪﻭوﭖپ‬ ‫ﺩدﺭرﯾﺎﻓﺖ‬
‫ﻫﺎﯼی‬ ‫ﻧﺴﺨﻪ‬ ‫ﯾﺎ‬ ‫ﮐﻨﯿﺪ‬ ‫ﮐﻤﭙﺎﯾﻞ‬ ‫ﻭو‬ ‫ﺑﮕﯿﺮﯾﺪ‬ ‫ﺭرﺍا‬ ‫ﮐﺪ‬ ‫ﺳﻮﺭرﺱس‬ ‫ﻣﯿﺘﻮﺍاﻧﯿﺪ‬ .‫ﮐﻨﯿﺪ‬ ‫ﻧﺼﺐ‬ ‫ﺭرﺍا‬ ‫ﻫﺪﻭوﭖپ‬ ‫ﺑﺎﯾﺪ‬ ‫ﺍاﺑﺘﺪﺍا‬
.‫ﮐﻨﯿﺪ‬ ‫ﻧﺼﺐ‬ ‫ﻭو‬ ‫ﺩدﺍاﻧﻠﻮﺩد‬ ‫ﺭرﺍا‬ ‫ﺑﺎﯾﻨﺮﯼی‬

100
‫ﻫﺪﻭوﭖپ‬ ‫ﺩدﺭرﯾﺎﻓﺖ‬
‫ﻫﺎﯼی‬ ‫ﻧﺴﺨﻪ‬ ‫ﯾﺎ‬ ‫ﮐﻨﯿﺪ‬ ‫ﮐﻤﭙﺎﯾﻞ‬ ‫ﻭو‬ ‫ﺑﮕﯿﺮﯾﺪ‬ ‫ﺭرﺍا‬ ‫ﮐﺪ‬ ‫ﺳﻮﺭرﺱس‬ ‫ﻣﯿﺘﻮﺍاﻧﯿﺪ‬ .‫ﮐﻨﯿﺪ‬ ‫ﻧﺼﺐ‬ ‫ﺭرﺍا‬ ‫ﻫﺪﻭوﭖپ‬ ‫ﺑﺎﯾﺪ‬ ‫ﺍاﺑﺘﺪﺍا‬
.‫ﮐﻨﯿﺪ‬ ‫ﻧﺼﺐ‬ ‫ﻭو‬ ‫ﺩدﺍاﻧﻠﻮﺩد‬ ‫ﺭرﺍا‬ ‫ﺑﺎﯾﻨﺮﯼی‬

101
‫ﻫﺪﻭوﭖپ‬ ‫ﮐﺪ‬ ‫ﮐﻤﭙﺎﯾﻞ‬
‫ﮐﺪ‬ ‫ﮐﻤﭙﺎﯾﻞ‬ ‫ﻭو‬ ‫ﺩدﺭرﯾﺎﻓﺖ‬

102
‫ﺑﺎﯾﻨﺮﯼی‬ ‫ﻧﺼﺐ‬
(OS X) ‫ﺑﺎﯾﻨﺮﯼی‬ ‫‌ﻫﺎﯼی‬‫ﻪ‬‫ﻧﺴﺨ‬ ‫ﻧﺼﺐ‬
brew install hadoop

103
:‫ﺷﻮﺩد‬ ‫ﻓﺮﻣﺖ‬ ‫ﺍاﺳﺘﻔﺎﺩدﻩه‬ ‫ﺑﺮﺍاﯼی‬ ‫ﺑﺎﯾﺪ‬ ‫ﺩدﯾﮕﺮ‬ ‫ﺩدﯾﺴﮏ‬ ‫ﻫﺮ‬ ‫ﻣﺜﻞ‬ HDFS
bin/hdfs namenode -format
:‫ﮐﻨﯿﺪ‬ ‫ﺍاﺳﺘﻔﺎﺩدﻩه‬ ‫ﺯزﯾﺮ‬ ‫ﺩدﺳﺘﻮﺭر‬ ‫ﺍاﺯز‬ HDFS ‫ﮐﺮﺩدﻥن‬ ‫ﺭرﻭوﺷﻦ‬ ‫ﺑﺮﺍاﯼی‬
sbin/start-dfs.sh
:‫ﺑﺴﺎﺯزﯾﺪ‬ ‫ﻓﻮﻟﺪﺭر‬ POSIX ‫ﻓﺎﯾﻞ‬ ‫ﺳﺎﻣﺎﻧﻪ‬ ‫ﻫﺮ‬ ‫ﻣﺜﻞ‬ ‫ﻣﯿﺘﻮﺍاﻧﯿﺪ‬
bin/hdfs dfs -mkdir /acm
:‫ﮐﻨﯿﺪ‬ ‫ﺍاﺳﺘﻔﺎﺩدﻩه‬ ‫ﺩدﺳﺘﻮﺭر‬ ‫ﺍاﯾﻦ‬ ‫ﺍاﺯز‬ YARN ‫ﮐﺮﺩدﻥن‬ ‫ﺭرﻭوﺷﻦ‬ ‫ﺑﺮﺍاﯼی‬
sbin/start-yarn.sh
:‫ﮐﻨﯿﺪ‬ ‫ﺍاﺳﺘﻔﺎﺩدﻩه‬ ‫ﺯزﯾﺮ‬ ‫ﺩدﺳﺘﻮﺭر‬ ‫ﺍاﺯز‬ HDFS ‫ﺩدﺭر‬ ‫ﻓﺎﯾﻞ‬ ‫ﺩدﺍاﺩدﻥن‬ ‫ﻗﺮﺍاﺭر‬ ‫ﺑﺮﺍاﯼی‬
bin/hdfs dfs -put yourFile.dat /acm

104
‫ﭘﺮﻭوﮊژﻩه‬ ‫ﺍاﻭوﻟﯿﻪ‬ ‫ﺳﺎﺧﺘﺎﺭر‬ ‫ﺳﺎﺧﺖ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
Create the Maven project:
mvn archetype:generate
-DarchetypeGroupId=org.apache.maven.archetypes
-DgroupId=ir.ac.ut.acm.wordcount
-DartifactId=wordcount
-DinteractiveMode=false

105
‫ﭘﺮﻭوﮊژﻩه‬ ‫ﺍاﻭوﻟﯿﻪ‬ ‫ﺳﺎﺧﺘﺎﺭر‬ ‫ﺳﺎﺧﺖ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
Create the project:
mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=ir.ac.ut.acm.wordcount -DartifactId=wordcount -DinteractiveMode=false

106
…‫‌ﻫﺎ‬‫ﯽ‬‫ﻭوﺍاﺑﺴﺘﮕ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
<dependency>
<groupid>org.apache.hadoop</groupid>
<artifactid>hadoop-client</artifactid>
<version>2.7.2</version>
</dependency>
Add the Hadoop Client the project POM:

107
Add the following plugins to the POM:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<conﬁguration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>ir.ac.ut.acm.wordcount.WordCount</mainClass>
</manifest>
</archive>
</conﬁguration>
</plugin>

108
Add the following plugins to the POM:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<conﬁguration>
<source>1.7</source>
<target>1.7</target>
</conﬁguration>
</plugin>

109
‫ﺟﺎﻭوﺍا‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
Create WordCount.java
package ir.ac.ut.acm.wordcount;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class WordCount {
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private ﬁnal static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}

110
‫ﺟﺎﻭوﺍا‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Conﬁguration conf = new Conﬁguration();
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
Complete WordCount.java

111
…‫ﺍاﺟﺮﺍا‬ ‫ﻭو‬ ‫ﺍاﺟﺮﺍاﯾﯽ‬ ‫ﺑﺴﺘﻪ‬ ‫ﺳﺎﺧﺖ‬ ‫ﻭو‬ ‫ﮐﻤﭙﺎﯾﻞ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
mvn clean install

112
…‫ﺍاﺟﺮﺍا‬ ‫ﻭو‬ ‫ﺍاﺟﺮﺍاﯾﯽ‬ ‫ﺑﺴﺘﻪ‬ ‫ﺳﺎﺧﺖ‬ ‫ﻭو‬ ‫ﮐﻤﭙﺎﯾﻞ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
hadoop jar target/wordcount-1.0-SNAPSHOT.jar input.txt wcount

113
!‫ﺍاﺳﺖ‬ ‫ﺷﺪﻩه‬ ‫ﻣﺮﺗﺐ‬ ‫ﻧﺘﯿﺠﻪ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬

114
‫ﻣﺮﺟﻊ‬‫ﺍاﻥن‬‫ﺮ‬‫ﺗﻬ‬ ‫ﺩدﺍاﻧﺸﮕﺎﻩه‬ ACM - ۱۳۹۵ ‫ﺗﺎﺑﺴﺘﺎﻥن‬ - ‫‌ﻣﺎﺷﯿﻦ‬‫ﯼی‬‫ﯾﺎﺩدﮔﯿﺮ‬ ‫ﻭو‬ ‫ﺑﺰﺭرﮒگ‬ ‫‌ﻫﺎﯼی‬‫ﻩه‬‫ﺩدﺍاﺩد‬ ‫‌ﻫﺎﯼی‬‫ﺩد‬‫ﮐﺎﺭرﺑﺮ‬ ‫ﺑﺮ‬ ‫ﮔﺬﺭرﯼی‬
http://www.infogovsolutions.com/defensible-disposition-of-structured-data-2/
http://pamneely.com/6-ways-search-will-change/
http://www.forbes.com/sites/gilpress/2013/05/28/a-very-short-history-of-data-science
http://www.slideshare.net/accavdar/big-data-tech-stack
http://wallup.net/preview/?wallpaper=glaciers-landscape-snow-water-drops-water-nature-iceberg-underwater
http://wikipediea
http://hadoopilluminated.com/hadoop_illuminated
http://wiki.apache.org/hadoop/
http://tuttlem.github.io/2014/01/30/create-a-mapreduce-job-using-java-and-maven.html
https://getblueshift.com/setting-up-hadoop-2-4-and-pig-0-12-on-osx-locally/
‫ﺳﺎﺯزﯼی‬ ‫ﭘﯿﺎﺩدﻩه‬ ‫ﮐﺪ‬

Big Data and Machine Learning Workshop - Day 3 @ UTACM

More Related Content

Viewers also liked (20)

More from Amir Sedighi (16)

Big Data and Machine Learning Workshop - Day 3 @ UTACM