Exploring the Enron Email Dataset with Kiji and Hive
2 likes1,981 views
The document outlines the creation of an external table for emails with various structured fields including timestamp and content. It provides SQL queries to analyze the email data, including grouping by sender and extracting sentiment metrics. Additionally, it includes methods for calculating average sentiments and outputs related to email communication from the 'enron.com' domain.
![CREATE EXTERNAL TABLE emails (
mid STRUCT<ts: TIMESTAMP, value: STRING>,
dateLong STRUCT<ts: TIMESTAMP, value: BIGINT>,
fromStr STRUCT<ts: TIMESTAMP, value: STRING>,
toStr STRUCT<ts: TIMESTAMP, value: STRING>,
subject STRUCT<ts: TIMESTAMP, value: STRING>,
body STRUCT<ts: TIMESTAMP, value: STRING>,
) STORED BY 'org.kiji.hive.KijiTableStorageHandler'
WITH SERDEPROPERTIES (
'kiji.columns' = ‘info:mid[0],info:date[0],info:from[0],info:to[0],’
+ ‘info:subject[0],info:body[0]’
) TBLPROPERTIES (
'kiji.table.uri' = ' kiji://.env/enron_email/emails '
);](https://image.slidesharecdn.com/exploringtheenronemaildatasetwithkijiandhive-130913140714-phpapp01/85/Exploring-the-Enron-Email-Dataset-with-Kiji-and-Hive-9-320.jpg)
![SELECT
lword AS word,
sum(sentiment) AS totalsentiment
FROM (
SELECT
mid.value AS mid,
lower(word) AS lword,
sentiment.value AS sentiment
FROM emails
LATERAL VIEW explode(sentences(body.value)[0]) wds AS word
WHERE regexp_replace(fromStr.value,".*@","")=="enron.com"
) subquery
GROUP BY lword
ORDER BY totalsentiment ASC;](https://image.slidesharecdn.com/exploringtheenronemaildatasetwithkijiandhive-130913140714-phpapp01/85/Exploring-the-Enron-Email-Dataset-with-Kiji-and-Hive-22-320.jpg)
Exploring the Enron Email Dataset with Kiji and Hive
- 2. ● ● ● ●
- 4. ● ● ● ● ●
- 7. …
- 8. ● ● ● ● ●
- 9. CREATE EXTERNAL TABLE emails ( mid STRUCT<ts: TIMESTAMP, value: STRING>, dateLong STRUCT<ts: TIMESTAMP, value: BIGINT>, fromStr STRUCT<ts: TIMESTAMP, value: STRING>, toStr STRUCT<ts: TIMESTAMP, value: STRING>, subject STRUCT<ts: TIMESTAMP, value: STRING>, body STRUCT<ts: TIMESTAMP, value: STRING>, ) STORED BY 'org.kiji.hive.KijiTableStorageHandler' WITH SERDEPROPERTIES ( 'kiji.columns' = ‘info:mid[0],info:date[0],info:from[0],info:to[0],’ + ‘info:subject[0],info:body[0]’ ) TBLPROPERTIES ( 'kiji.table.uri' = ' kiji://.env/enron_email/emails ' );
- 10. SELECT fromStr.value AS fromStr, count(1) AS count FROM emails GROUP BY fromStr.value ORDER BY count DESC LIMIT 10;
- 12. SELECT fromStr.value AS fromStr, trim(splitToStr) AS toStr, count(1) AS count FROM emails LATERAL VIEW explode(split(toStr.value,',')) tos AS splitToStr GROUP BY fromStr.value,trim(splitToStr) ORDER BY count DESC LIMIT 10;
- 14. ● ● ● ○ ○
- 17. SELECT ((year(datelong.ts)-1999)*52+weekofyear(datelong.ts)) AS weeknum, avg(sentiment.value) AS avgsentiment, stddev(sentiment.value) AS stddevsentiment, count(1) AS nummessages FROM emails WHERE regexp_replace(fromStr.value,".*@","")=="enron.com" GROUP BY ((year(datelong.ts)-1999)*52+weekofyear(datelong. ts));
- 22. SELECT lword AS word, sum(sentiment) AS totalsentiment FROM ( SELECT mid.value AS mid, lower(word) AS lword, sentiment.value AS sentiment FROM emails LATERAL VIEW explode(sentences(body.value)[0]) wds AS word WHERE regexp_replace(fromStr.value,".*@","")=="enron.com" ) subquery GROUP BY lword ORDER BY totalsentiment ASC;
- 26. ● ● ● ●
- 27. ● ● ●