Big Data & Hadoop Tutorial

Editor's Notes

• #13: Web and e-tailing- eBay is using Hadoop technology and the Hbase database, which supports real-time analysis of Hadoop data, to build a new search engine for its auction site. eBay has more than 97 million active buyers and sellers and over 200 million items for sale in 50,000 categories. The site handles close to 2 billion page views, 250 million search queries and tens of billions of database calls daily. The company has 9 petabytes of data stored on Hadoop and Teradata clusters, and the amount is growing quickly.TelecommunicationsChina Mobile; Data Mining platform for Telecom Industry, 5-8 TB/day CDR , Network Signaling DataCurrent Solutions such as Oracle DB, SAS (Data Mining), Unix Servers and SAN aren’t sufficient to store and process such a vast amount of dataNeed faster data processing to Precision marketing, Network Optimization, Service Optimization and Log Processing
• #14: Government-AADHAR by Govt. Of India; 5 MB Data per resident, maps to about 10-15 PB of raw data. The Hadoop stack: HDFS (Hadoop distributed file system) is used to provide high data read/write throughput in the order of many terabytes per day. Distributed architecture enables scale out as needed. Hive is used for building the UIDAI data warehouse, HBase for indexed lookup of records across millions of rows, Zookeeper as a distributed coordination service for server instances, and Pig as an ETL (extract, transform and load) solution for loading data into Hive.Healthcare and Life Sciences- Life sciences research firm NextBio uses Hadoop and HBase to help big pharmaceutical companies conduct genomic research. The company embraced Hadoop in 2009 to make the sheer scale of genomic data-analysis more affordable. The company's core 100-node Hadoop cluster, which has processed as much as 100 terabytes of data, is used to compare data from drug studies to publically available genomics data. Given that there are tens of thousands of such studies and 3.2 billion base pairs behind each of the hundreds of genomes that NextBio studies, it's clearly a big-data challenge.
• #15: Banks and Financial services:3 of the top 5 Banks run Cloudera HadoopJPMorgan Chase uses Hadoop technology for a growing number of purposes, including fraud detection, IT risk management and self service. With over 150 petabytes of data stored online, 30,000 databases and 3.5 billion log-ins to user accounts, data is the lifeblood of JPMorgan Chase.Retail:Sears is an American multinational mid-range department store chain headquartered in Hoffman Estates, Illinois). It Moved to Hadoop from Hadoop from Teradata and SAS to avoid archiving and deleting its meaningful sales and other customer activity data. 300-Node Hadoop cluster helps Sears to keep its 100% data (~2PB) available to BI rather than a meager 10% as was the case with Non-Hadoop solutions. Walmart; migrated data from its existing Oracle, Neteeza, Oracle and Greenplum gear to its 250-Node Hadoop Cluster.
• #16: Why Oracle, HP, IBM and other Enterprise Technology giants are in Red on growth. Sears:Sears wanted to personalize marketing campaigns, coupons, and offers down to the individual customer, but the existing legacy systems were incapable of supporting that.Sears' process for analyzing marketing campaigns for loyalty club members used to take six weeks on mainframe, Teradata, and SAS servers. The new process running on Hadoop can be completed weekly. For certain online and mobile commerce scenarios, Sears can now perform daily analyses. What's more, targeting is more granular, in some cases down to the individual customer. Moving up the stack, Sears is consolidating its databases to MySQL, InfoBright, and Teradata--EMC Greenplum, Microsoft SQL Server, and Oracle (including four Exadata boxes) are on their way out.Sears routinely replaces legacy Unix systems with Linux rather than upgrade them, and it has retired most of its Sun and HP-UX servers. Microsoft server and development technologies are also on the way out.
• #17: - 2 PB of data--mostly structured and unstructured data such as customer transaction, point of sale, and supply chain.- Because of Archiving Need 90% of the ~2PB of Data is not available for BI
• #18: 300-Node Hadoop cluster helps Sears to keep its 100% data (~2PB) available to BI rather than a meager 10% as was the case with Non-Hadoop solutions. Sears now keeps all of its data down to individual transactions (rather than aggregates) and years of history (rather than imposing quarterly windows on certain data, as it did previously). That's raw data, which Sears can refactor and combine as needed quickly and efficiently within Hadoop.To give a sense of how early Sears was to Hadoop development, Wal-Mart divulged early this year that it was scaling out an experimental 10-node Hadoop cluster for e-commerce analysis. Sears passed that size in 2010.Has its own Hadoop solutions subsidiary MetaScale, to provide hadoop services to other retail companies on the line of AWS.
• #19: Accessible: Hadoop runs on large clusters of commodity machines or cloud computing services such as Amazon EC2Robust: Since Hadoop can run on commodity cluster, its designed with the assumption of frequent hardware failure, it can gracefully handle such failure and computation don’t stop because of few failed devices / systemsScalable:Hadoop scales linearly to handle large data by adding more slave nodes to the clusterSimple : Its easy to write efficient parallel programming with Hadoop
• #29: We will cover other Hadoop Components in detail in future sessions of this course.
• #45: Data transferred from DataNode to MapTask process. DBlk is the file data block; CBlk is the file checksum block. File data are transferred to the client through Java niotransferTo (aka UNIX sendfilesyscall). Checksum data are first fetched to DataNode JVM buffer, and then pushed to the client (details are not shown). Both file data and checksum data are bundled in an HDFS packet (typically 64KB) in the format of: {packet header | checksum bytes | data bytes}.2. Data received from the socket are buffered in a BufferedInputStream, presumably for the purpose of reducing the number of syscalls to the kernel. This actually involves two buffer-copies: first, data are copied from kernel buffers into a temporary direct buffer in JDK code; second, data are copied from the temporary direct buffer to the byte[] buffer owned by the BufferedInputStream. The size of the byte[] in BufferedInputStream is controlled by configuration property "io.file.buffer.size", and is default to 4K. In our production environment, this parameter is customized to 128K.3. Through the BufferedInputStream, the checksum bytes are saved into an internal ByteBuffer (whose size is roughly (PacketSize / 512 * 4) or 512B), and file bytes (compressed data) are deposited into the byte[] buffer supplied by the decompression layer. Since the checksum calculation requires a full 512 byte chunk while a user's request may not be aligned with a chunk boundary, a 512B byte[] buffer is used to align the input before copying partial chunks into user-supplied byte[] buffer. Also note that data are copied to the buffer in 512-byte pieces (as required by FSInputChecker API). Finally, all checksum bytes are copied to a 4-byte array for FSInputChecker to perform checksum verification. Overall, this step involves an extra buffer-copy.4. The decompression layer uses a byte[] buffer to receive data from the DFSClient layer. The DecompressorStream copies the data from the byte[] buffer to a 64K direct buffer, calls the native library code to decompress the data and stores the uncompressed bytes in another 64K direct buffer. This step involves two buffer-copies.5.LineReader maintains an internal buffer to absorb data from the downstream. From the buffer, line separators are discovered and line bytes are copied to form Text objects. This step requires two buffer-copies.The client creates the file by calling create() on Distributed FileSystem (step 1). Distributed FileSystem makes an RPC call to the namenode to create a new file in the filesystem’s namespace, with no blocks associated with it (step 2). The namenode performs various checks to make sure the file doesn’t already exist, and that the client has the right permissions to create the file.
• #46: The client opens the file it wishes to read by calling open() on the FileSystemobject,which for HDFS is an instance of DFS(step 1).DistributedFileSystem calls the namenode, using RPC, to determine the locations of the blocks for the first few blocks in the File (step 2). For each block, the namenode returns the addresses of the datanodes that have a copy of that block