Apache PIG Relational Operations
0 likes645 views
The document explains various Pig relational operators, including order, limit, distinct, and group. The order operator is used for sorting data, the limit operator restricts the number of records displayed, the distinct operator removes duplicate data, and the group operator organizes data into groups based on specified keys. Examples are provided for each operator, demonstrating their usage in practical scenarios with datasets.
Apache PIG Relational Operations
- 1. Pig Relational Operators - I: Order, Distinct, Limit, GroupBy
- 2. Pig Relational Operator: ORDER This helps to sort the data based on Ascending or Descending manner. 1. Sorting for numerical fields are based on numerically. 2. Sorting for chararray fields are based on lexically. 3. Sorting for bytearray fields are based on lexically. 4. Nulls are considered to be smaller than other values. Therefore it will always come first or last during ascending or descending the results. Let’s perform this with the help of an example; grunt> dataTransaction = Load ‘/home/hduser/datasets/store.csv’ using PigStorage(‘,’) AS (Product_Name:chararray,CustomerName:chararray,Transaction_I D:byearray,TransAmt1:bytearray,TransAmt2:bytearray, TransAmt3:bytearray, Place:chararray, Department:chararray); Rupak Roy
- 3. Pig Relational Operator: ORDER grunt> orderbyName = ORDER dataTransaction by Name; Example 2: grunt> orderbyNameNsymbol= order datatransaction by date, symbol; Example 3: grunt> desc= order datatransaction by close desc, open; Here close column will have descending order and since we didn’t mentioned any order for open column it will take ascending order by default. Rupak Roy
- 4. Pig Relational Operator: LIMIT Limit simply limits the number of records to display. For example, if we have 1 million rows and columns and if we dump the results for testing or any purpose it will take a lot of time to finish displaying the results one by one which is very time consuming process, so to make sure our required script is working it is better to view some results rather than displaying the whole results. However Pig will still read all the records even we limit the display of results by (assume) 20 records, but it will also display 20 different records each time even we use the same limit query. We can overcome this issue by using ORDER operator immediately after the limit operator and will guarantee the same 20 records each time when we use the same limit query. Rupak Roy
- 5. Pig Relational Operator: LIMIT grunt> Trecords= LIMIT dataTransaction 20; grunt> dump Trecords; grunt> dump Trecords; #it will display 20 different records. #to overcome the limit issue grunt> order= order dataTransaction by $0; grunt> Trecords= Limit order 20; grunt> dump Trecords; Again we will test for the same results; grunt> Trecrods = Limit order 20; Rupak Roy
- 6. Pig Relational Operator: DISTINCT This operator simply removes the duplicate data. grunt> RD = DISTINCT dataTransaction; Note: Distinct operator makes use of a combiner or we can say semi-reducer between the map phase and reducer phase to remove the duplicates. Rupak Roy
- 7. Pig Relational Operator: GROUP Group operator is one of the important functions for grouping the data from a large pool of datasets. grunt> grouping = GROUP dataTransaction by Place; grunt> describe grouping; grunt>cnt = foreach grouping GENERATE group, COUNT( $1); grunt> dump cnt; By applying this to our dataset store.csv we can find from how many places each customer purchase similar products. Note: to verify this result, load the dataset in excel (since this is a subset of a dataset and will be suitable to view it in excel). Use the filter function then select only Christy Britain, we will see she have purchased the similar products from 6 different places. Rupak Roy
- 8. Group on multiple keys: grunt> grouped = GROUP dataTransaction by (Place, Department); Pig Relational Operator: GROUP Rupak Roy
- 9. Next More into advanced relational operators like foreach, filter, join and more. Rupak Roy