Jumpstarting big data projects / Architectural Considerations of HDInsight Applications
Download as PPTX, PDF4 likes1,953 views
The document discusses the technical specifications and workload configurations for implementing big data projects using HDInsight, particularly in the automotive sector, involving numerous vehicle diagnostics and data processing capabilities. It details the challenges presented by small file management in Hadoop while elaborating on data preparation, customer segmentation, and machine learning techniques like Mahout. It also includes a sample job configuration for running Oozie workflows and the monitoring of processing logs.
![$coreConfig = @{
"io.compression.codec"="org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.ha
doop.io.compress.BZip2Codec";
"io.sort.mb" = "1024";
}
$mapredConfig = new-object 'Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.DataObjects.AzureHDInsightMapReduceConfiguration'
$mapredConfig.Configuration = @{
"mapred.tasktracker.map.tasks.maximum"="2";
}
$clusterConfig = New-AzureHDInsightClusterConfig -ClusterSizeInNodes $numberNodes `
| Set-AzureHDInsightDefaultStorage -StorageAccountName $fqStorageAccountName -StorageAccountKey $storageAccountKey `
-StorageContainerName ($storageContainer.Name)
$continueCheck = Read-Host "Attach additional storage accounts? (yes to continue)"
if ($continueCheck -eq "yes")
{
foreach($asa in 1..5) {
$newStorageAccountName = ($clusterPrefix + [DateTime]::Now.ToString("yyyyMMddHHmmss") + "a" + $asa)
New-AzureStorageAccount -StorageAccountName $newStorageAccountName -Location "North Europe"
$clusterConfig = $clusterConfig | Add-AzureHDInsightStorage `
-StorageAccountName ($newStorageAccountName + ".blob.core.windows.net") `
-StorageAccountKey (Get-AzureStorageKey $newStorageAccountName).Primary
}
}
$clusterConfig = $clusterConfig | Add-AzureHDInsightConfigValues -Core $coreConfig -MapReduce $mapredConfig
# "At this point we are able to create a hdinsight cluster with a customised configuration"
http://aka.ms/HDIconfiguration](https://image.slidesharecdn.com/jumpstartingbigdataprojects-150203040114-conversion-gate01/85/Jumpstarting-big-data-projects-Architectural-Considerations-of-HDInsight-Applications-76-320.jpg)
![0
200
400
600
800
1000
1200
Jobs StorageTotal
2014-03-26 22:28:37,321 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-140326181153083-oozie-hdp-W] ACTION[0000000-
140326181153083-oozie-hdp-W@pig-node-01] callback for action [0000000-140326181153083-oozie-hdp-W@pig-node-01]
2014-03-26 22:28:37,472 INFO PigActionExecutor:539 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie-hdp-W]
ACTION[0000000-140326181153083-oozie-hdp-W@pig-node-01] action completed, external ID [job_201403261811_0001]
2014-03-26 22:28:37,562 WARN PigActionExecutor:542 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie-hdp-
W] ACTION[0000000-140326181153083-oozie-hdp-W@pig-node-01]
Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
2014-03-26 22:28:38,101 INFO ActionEndXCommand:539 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie-
hdp-W] ACTION[0000000-140326181153083-oozie-hdp-W@pig-node-01] ERROR is considered as FAILED for SLA
2014-03-26 22:28:38,228 WARN JPAService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor [WorkflowActionGetJPAExecutor]
ended with an active transaction, rolling back
2014-03-26 22:28:38,343 INFO ActionStartXCommand:539 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie-
hdp-W] ACTION[0
High Latency Timeout](https://image.slidesharecdn.com/jumpstartingbigdataprojects-150203040114-conversion-gate01/85/Jumpstarting-big-data-projects-Architectural-Considerations-of-HDInsight-Applications-93-320.jpg)
Jumpstarting big data projects / Architectural Considerations of HDInsight Applications
- 11. Up to 75 control units in 1 vehicle About 1,000 individual possible extra equipments 1 GB car software, 15 GB data on board (incl. navi) 2,000 user functions implemented 12,000 types of error stored onboard for diagnosis Daily up to 60,000 car diagnosis worldwide
- 12. </meldungText><antwort>False</antwort><wert>na</wert></meldung><steuergeraet sgbdVariante="SMG_60"><steuergeraeteFunktion zeitstempel="2013-04-30T09:00:37.9926171-04:00" endDate="2013-04-30T09:00:38.1158609-04:00" jobName="STATUS_FAHRZEUGTESTER"><datensatz satzNr="1"><result name="JOB_STATUS">OKAY</result><result name="_TEL_ANTWORT">80 F1 18 70 70 02 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 82 6B 00 6D 6B 39 CD 14 00 14 00 00 0E 00 15 00 0A 00 19 00 0C 00 12 00 15 85 57 71 88 81 C0 7D 73 C2 08 01 05 02 F7 00 FF FF 01 73 00 00 02 A8 00 C2 00 01 E0 00 00 00 00 00 00 3D 01 00 00 00 01 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 FD 01 E1 02 05 01 F8 03 4F FF AD 04</result><result name="_TEL_AUFTRAG">83 18 F1 30 02 01</result><result name="STAT_KL15_ROH">0</result><result name="STAT_KLR_EIN_ROH">0</result><result name="STAT_WAKE_UP_ROH">1</result><result name="STAT_ISTGANG_TEXT">Neutral</result> <sgFunktion zeitstempel=“2013-04-30T10:33:37.0834084+02:00" endDate="2013-04-30T10:33:37.9310504+02:00" jobName="_FLM_LESEN_BOSCH"><datensatz satzNr="1"><result name="FLM_DATEN_1">00 00 00 03 02 08 C6 56 46 4C 4D 39 00 16 4B B2 00 00 00 32 00 00 06 99 00 00 00 65 00 00 18 6E 00 00 00 73 00 00 00 20 00 00 00 73 00 00 00 00 00 00 10 69 00 00 0F 53 00 00 00 2C 00 00 00 0A 00 00 79 6D 00 00 B7 34 00 00 D3 9E 4A 4C 41 52 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2C 00 00 00 00 00 00 1A 5C 00 15 4B CA 00 00 44 08 00 00 2D 39 00 00 1E 45 00 00 26 89 00 00 1E EB 00 00 0C 65 00 00 04 47 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 27 00 00 01 1E 00 00 02 AB 00 00 07 71 00 00 13 D7 00 00 36 48 00 15 91 AD 00 00 3F 97 00 00 19 C1 00 00 07 F9 00 00 02 D4 00 00 00 BD 00 00 00 20 00 16 1C 42 00 00 18 B1 00 00 09 40 00 00 08 9F 00 00 04 3A 00 00 01 3E 00 01 8C D7 00 00 61 A3 00 00 37 9D 00 00 1E 78 00 00 14 96 00 00 0A 71 00 00 05 49 00 00 02 B1 00 00 00 A7 00 00 00 1D 00 00 00 09 00 00 00 05 00 00 00 00 00 00 00 00 00 00 23 BB 00 00 2F 84 00 00 14 EF 00 00 09 40 00 00 04 71 00 00 03 34 00 00 02 12 00 00 01 AC 00 00 01 59 00 00 0B C4 00 00 00 06 00 00 00 38 00 00 00 19 00 00 00 01 00 00 00 00 00 00 00 04 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 52 4F 54 48 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 56 30 00 00 00 03 00 11 00 01 01 06 00 00 00 00 00 00 00 00 00 01 00 00 00 0E 00 05 00 1A 00 12 00 00 00 26 00 00 00 00 00 0B 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 44 44 00 43 00 16 00 08 00 0D 00 04 00 02 00 00 00 02 00 11 00 20 00 1A 00 0A 00 15 00 0F 00 1B 00 13 00 08 00 08 00 00 00 00 00 07 00 0E 00 08 00 04 00 02 00 01 00 00 00 6D 00 03 00 02 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0A 00 21 00 15 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 05 1F 00 00 00 00 00 00 00 00 00 1F 00 03 00 02 00 00 00 00 00 00 00 20 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 62 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2E 00 00 1B 00 19 00 18 00 0D 00 00 00 00 00 00 00 01 00 01 00 02 00 00 06 00 01 E6 00 00 12 00 03 00 02 00 07 00 00 00 00 00 00 00 00 00 00 00 00 00 04 00 02 01 BA 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 24 00</result><result name="FLM_DATEN_2">08 00 00 00 00 00 00 00 00 00 00 0C 00 80 1B 00 45 10 00 A6 0D 00 51 16 00 59 44 00 00 EB 00 00 CA 00 00 49 00 00 17 00 10 00 0C 00 05 00 04 00 06 00 02 00 01 00 00 00 00 12 00 00 3A… Here!
- 32. Hadoop does not do well with lots of small files http://aka.ms/HDI_smallfiles
- 35. Clean & prepare data Customer polarisation & segmentation
- 38. - SQL-like abstraction of MapReduce - Compatible with other RDBMs - Simple aggregations - DBAs - Complex Joins - Unstructured data - UDF is complex
- 39. - Slice & dicing (or complex query) - Streaming objects - UDF - Software developers - Integration with analytical tools/RDBMs
- 40. Feature vector per customer Scoring and prioritisation
- 44. Mahout Azure Machine Learning
- 47. Distributed similarity measure Version number of Mahout
- 48. Input Output
- 49. Limit of #similar items per item Max #preferences per user/item
- 52. 1. 2. 3. 4.
- 55. hadoop jar C:appsdistmahout-0.9mahout-core-0.9-job.jar org.apache.mahout.classifier.df.tools.Describe -p wasb:///user/hdp/testdata/KDDTrain+.arff -f wasb:///user/hdp/testdata/KDDTrain+.info -d N 3 C 2 N C 4 N C 8 N 2 C 19 N L Describing the columns
- 57. Leaf size
- 58. Data Data Set
- 61. Test Data Data Set / Descriptor File Forest Model
- 62. Analysing the classification results Use MapReduce Jobs Output
- 64. accuracy = #correctly classified instances #classified instances
- 67. Scoring and prioritization Read & process data for the day Extract into outputs
- 76. $coreConfig = @{ "io.compression.codec"="org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.ha doop.io.compress.BZip2Codec"; "io.sort.mb" = "1024"; } $mapredConfig = new-object 'Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.DataObjects.AzureHDInsightMapReduceConfiguration' $mapredConfig.Configuration = @{ "mapred.tasktracker.map.tasks.maximum"="2"; } $clusterConfig = New-AzureHDInsightClusterConfig -ClusterSizeInNodes $numberNodes ` | Set-AzureHDInsightDefaultStorage -StorageAccountName $fqStorageAccountName -StorageAccountKey $storageAccountKey ` -StorageContainerName ($storageContainer.Name) $continueCheck = Read-Host "Attach additional storage accounts? (yes to continue)" if ($continueCheck -eq "yes") { foreach($asa in 1..5) { $newStorageAccountName = ($clusterPrefix + [DateTime]::Now.ToString("yyyyMMddHHmmss") + "a" + $asa) New-AzureStorageAccount -StorageAccountName $newStorageAccountName -Location "North Europe" $clusterConfig = $clusterConfig | Add-AzureHDInsightStorage ` -StorageAccountName ($newStorageAccountName + ".blob.core.windows.net") ` -StorageAccountKey (Get-AzureStorageKey $newStorageAccountName).Primary } } $clusterConfig = $clusterConfig | Add-AzureHDInsightConfigValues -Core $coreConfig -MapReduce $mapredConfig # "At this point we are able to create a hdinsight cluster with a customised configuration" http://aka.ms/HDIconfiguration
- 79. Example: Oozie job configuration nameNode=wasb://container_name@storage_name.blob.core.windows.net jobTracker=jobtrackerhost:9010 queueName=default oozie.wf.application.path=wasb:///user/admin/examples/apps/ooziejobs outputDir=ooziejobs-out oozie.use.system.libpath=true
- 82. 12/29 6/29 Customer ID URL Item Time Customer ID Item
- 85. Raw Data Storage Processing Processed Data Storage Validation & Troubleshooting Raw Data Storage Processing Processed Data Storage
- 92. AuthorizationError, Availability, AverageE2ELatency, AverageServerLatency, ClientTimeoutError, NetworkError, PercentAuthorizationError, PercentNetworkError, PercentSuccess, ServerTimeoutError, Success, ThrottlingError Timestamp, TotalBillableRequests, TotalEgress, TotalIngress, TotalRequests Application Storage throttling When? Data exchange 0 2E+09 4E+09 6E+09 8E+09 1E+10 1.2E+10 1.4E+10 Jobs Storage Sum of TotalIngress Sum of TotalEgress
- 93. 0 200 400 600 800 1000 1200 Jobs StorageTotal 2014-03-26 22:28:37,321 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-140326181153083-oozie-hdp-W] ACTION[0000000- 140326181153083-oozie-hdp-W@pig-node-01] callback for action [0000000-140326181153083-oozie-hdp-W@pig-node-01] 2014-03-26 22:28:37,472 INFO PigActionExecutor:539 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie-hdp-W] ACTION[0000000-140326181153083-oozie-hdp-W@pig-node-01] action completed, external ID [job_201403261811_0001] 2014-03-26 22:28:37,562 WARN PigActionExecutor:542 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie-hdp- W] ACTION[0000000-140326181153083-oozie-hdp-W@pig-node-01] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2] 2014-03-26 22:28:38,101 INFO ActionEndXCommand:539 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie- hdp-W] ACTION[0000000-140326181153083-oozie-hdp-W@pig-node-01] ERROR is considered as FAILED for SLA 2014-03-26 22:28:38,228 WARN JPAService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor [WorkflowActionGetJPAExecutor] ended with an active transaction, rolling back 2014-03-26 22:28:38,343 INFO ActionStartXCommand:539 - USER[Admin] GROUP[-] TOKEN[] APP[receipts-products-mahout] JOB[0000000-140326181153083-oozie- hdp-W] ACTION[0 High Latency Timeout