Big Data - Load CSV File & Query the EZ way - HPCC Systems
0 likes1,391 views
The document outlines the process of loading CSV data and querying it using HPCC Systems' Enterprise Control Language (ECL), a C++ based query language designed for big data platforms. It details steps for uploading files to the Thor cluster, querying capabilities, and provides resources for learning and documentation. Additionally, it compares ECL with Hadoop's Pig, emphasizing ECL's expressiveness and richness.
![Query w/ ECL
Com := DATASET(‘~test::complaints’,ComS,
CSV(HEADING(1), SEPARATOR([',','t'])));
ComS :=RECORD
UNSIGNED3 ComplaintID;
STRING23 Product;
STRING38 State;
………………………….
………………………….
STRING31Consumer_disputed;
END;
Ma := Com(State = ‘MA’);
Ma; //output
WHERE `State` = ‘MA’
File Type
File Location,!
“FROM Table”
“USE DATABASE;”
“SELECT * ….”
Schema](https://image.slidesharecdn.com/meetupbasicscsvdemo-140718031142-phpapp01/85/Big-Data-Load-CSV-File-Query-the-EZ-way-HPCC-Systems-10-320.jpg)
Big Data - Load CSV File & Query the EZ way - HPCC Systems
- 1. HPCC Systems Loading csv Data & Querying By Fujio Turner @FujioTurner
- 2. Non-Indexed Full Data Set 1 20 Customers Development Business http://hpccsystems.com/why-hpcc/benchmarks
- 3. ECL (Enterprise Control Language) C++ based query language SQL w/ JOINS Map/Reduce GraphDB Machine Learning Simple to Complex Queries
- 4. “I’m sub-second fast.” “I can query all or part of your data.” Architecture Thor Roxie Hard Disk Index(optional) Hard Disk Index(optional) In-memory Index SSD Either/Both
- 5. Example File Load File into HPCC Query CSV data sample source http://catalog.data.gov/dataset/consumer-complaint-database
- 6. Administrator Web GUI! on IP / Url of HPCC install Port 8010
- 7. 4. add ,t 5. 1. Upload file*! 2. Distribute to cluster! 3. Name of file in cluster! 4. Most CSV have t! 5. Push to cluster *2GB file size limit through web No limit if uploaded via SOAP Load !! ! ! Data
- 8. *optional file rename Loaded In Thor Cluster
- 9. How do I Query HPCC Systems ? What Is ECL? ECL (Enterprise Control Language) is a C++ based query language for use with HPCC Systems Big Data platform. ECLs syntax and format is very simple and easy to learn.! ! Note - ECL is very similar to Hadoop’s pig ,but! more expressive and feature rich.
- 10. Query w/ ECL Com := DATASET(‘~test::complaints’,ComS, CSV(HEADING(1), SEPARATOR([',','t']))); ComS :=RECORD UNSIGNED3 ComplaintID; STRING23 Product; STRING38 State; …………………………. …………………………. STRING31Consumer_disputed; END; Ma := Com(State = ‘MA’); Ma; //output WHERE `State` = ‘MA’ File Type File Location,! “FROM Table” “USE DATABASE;” “SELECT * ….” Schema
- 11. 1. Go to playground! 2. Edit ECL! 3. Pick “thor” Cluster! 4. Submit Practice http://www.meetup.com/HPCC-SV/pages/ECL_EXAMPLE__- _CSV_LOAD_and_QUERY
- 12. Schema Made EZ http://hpccsystems.com/demos/data-profiling-demo CSV IN Schema Click OUT Storing a new file and want to make a quick schema? ! Take a small part of your CSV data and go to the link below to make an ECL Schema
- 13. ECL Guide http://hpccsystems.com/download/docs/ecl-language-reference JOIN! MERGE! LENGTH! REGEX! ROUND! SUM! COUNT! TRIM! WHEN! AVE! ABS! CASE! DEDUP! NORMALIZE! DENORMALIZE! IF! SORT! GROUP! more ….
- 14. For More HPCC! “How To’s”! Go to SlideShare http://www.slideshare.net/FujioTurner/
- 15. Watch how to install HPCC Systems in 5 Minutes Download HPCC Systems Open Source Community Edition http://hpccsystems.com/download/ http://www.youtube.com/watch?v=8SV43DCUqJg or Source Code https://github.com/hpcc-systems