Social Stream Analysis Use Cases
0 likes996 views
The document discusses using WSO2's big data analytics platform to analyze social media streams like Twitter data. It describes how the WSO2 Business Activity Monitor can collect data from various sources, and the Complex Event Processor can process multiple event streams and identify patterns. An example use case presented is analyzing Twitter tags over time to identify trends and public interest around certain topics, and notifying users via email of interesting tweets. The proposed solution architecture shows how a Twitter agent can collect data and publish it to BAM for storage and processing by CEP queries.
![Twiaer
Agent
//Configuring
Agent
AgentConfiguraXon
agentConfiguraXon
=
new
AgentConfiguraXon();
agentConfiguraXon.setTrustStorePassword("/Users/suho/wso2/dev/twiaer-‐sample/wso2cep-‐3.0.0/repository/"
+
"resources/security/client-‐truststore.jks");
agentConfiguraXon.setTrustStore("wso2carbon”);
Agent
agent
=
new
Agent(agentConfiguraXon);
//New
Data
publisher
to
BAM
final
AsyncDataPublisher
asyncDataPublisher
=
new
AsyncDataPublisher("tcp://localhost:7611",
"admin",
"admin",
agent);
//Defining
the
Stream
String
STREAM_NAME
=
"twiaer_data_stream";
String
VERSION
=
"1.0.0";
String
streamDefiniXon
=
"{"
+
"
'name':'"
+
STREAM_NAME
+
"',"
+
"
'version':'"
+
VERSION
+
"',"
+
"
'nickName':
'Twiaer_Data_Stream',"
+
"
'descripXon':
'Stream
of
twiaer
data',"
+
"
'payloadData':["
+
"
{'name':'createdAt','type':'LONG'},"
+
"
{'name':'user','type':'STRING'},"
+
"
{'name':'tweet','type':'STRING'}"
+
"
]"
+
"}";
asyncDataPublisher.addStreamDefiniXon(streamDefiniXon,
STREAM_NAME,
VERSION);
23](https://image.slidesharecdn.com/nsaforenterprises-socialstreamanalysisusecases-feb2014-140227013703-phpapp02/85/Social-Stream-Analysis-Use-Cases-23-320.jpg)
![CEP:
Retweet
NoXficaXon
from
twiaerStream[tweet
contains
'RT
']
insert
into
reTweetStream;
33](https://image.slidesharecdn.com/nsaforenterprises-socialstreamanalysisusecases-feb2014-140227013703-phpapp02/85/Social-Stream-Analysis-Use-Cases-33-320.jpg)
Social Stream Analysis Use Cases
- 1. Big Brother for Enterprises: Social Media Stream Use Cases Samisa Abeysinghe VP Developer Evangelism 26 Feb 2014
- 2. About the Presenter ๏ ๏ 2 Samisa Abeysinghe VP Developer Evangelism samisa@wso2.com Samisa Abeysinghe, Vice President of Developer Evangelism joined the company in September 2005. Prior to the current role, Samisa used to be VP of Engineering and managed the development of WSO2 Carbon based product plaOorm.
- 3. About WSO2 ๏ ๏ ๏ Global enterprise, founded in 2005 by acknowledged leaders in XML, web services technologies, standards and open source Provides only open source plaOorm-‐as-‐a-‐service for private, public and hybrid cloud deployments ๏ ๏ 3 All WSO2 products are 100% open source and released under the Apache License Version 2.0. Is an AcXve Member of OASIS, Cloud Security Alliance, OSGi Alliance, AMQP Working Group, OpenID FoundaXon and W3C. Driven by InnovaXon ๏ Launched first open source API Management soluXon in 2012 ๏ Launched App Factory in 2Q 2013 ๏ Launched Enterprise Store and first open source Mobile soluXon in 4Q 2013
- 4. What WSO2 Deliver 4
- 5. NSA like Monitoring for Your Enterprise ๏ Analyze volumes of data ๏ Address correlation complexities in analytics ๏ Off line vs Real time operations implications ๏ Some operations got to be in real time, else the value is lost ๏ Summarized data over time (and other) dimensions for analytics A “Big Brother” that keeps an eye on the whole enterprise 5
- 6. Why should I bother? ๏ Deal with high volume (terra bites) of information ๏ In order to make decisions ๏ Real time & Offline ๏ Take action 6
- 7. WSO2 :ONLY COMPLETE & INTEGRATED Platform ๏ Complete and integrated for ๏ Data capture ๏ Analysis: both real-time and batch ๏ Visualization ๏ Action taking business process execution Making data driven intelligence for your enterprise easy 7
- 10. WSO2 Big Data Analytics Platform for Your Enterprise WSO2 Big Data Analy0cs Pla4orm for Your Enterprise 10
- 11. Key Elements Data CollecXon Taking AcXon Data Analysis Data VisualizaXon 11
- 12. WSO2 Big Data Analytics Platform for Your Enterprise Data Collection 12 Data Analysis Data Visualization & Taking Action
- 13. Use Case Scenario: Analyzing Twiaer Streams 13
- 14. Social media … ! o Test o Test
- 15. Processing Social Media !!!
- 16. Why Monitor Social Media ? ● To get useful insight about the market ○ CompeXtor Analysis ○ Customer Trend and Interest ● IdenXfy and Compare Trends ○ Monitoring the brand ○ Comparing the trends of the public interest 16
- 17. WSO2 Business AcXvity Monitor (BAM) ● Can capture data from any applicaXon/service ● APIs, Services or source including file-‐based data ○ WSO2 has lightweight agents to provide ■ raw data from numerous sources ■ as well as support for ● gateways (lightweight or heavyweight) ● Service and API proxies ● Data imports from various sources (including file based data) ● Any other source (extended agents) 17
- 18. WSO2 Business AcXvity Monitor (BAM) 18
- 19. WSO2 Complex Event Processor (CEP) ● CEP process mulXple event streams generated within the enterprise to construct data abstracXon and idenXfy meaningful paaerns among those streams ○ Powered by WSO2 Siddhi ■ ■ Processes more than 2.5M events/sec on single server (commodity hardware) Has a powerful and extensible SQL Like Siddhi query language ○ Integrates with any enterprise system for event capture ■ Support mulXple transports & event formats: HTTP, JMS & SMTP on XML, JSON, Text & Map ○ Efficient large volume event publishing to enterprise systems and custom noXficaXons ■ ■ ■ 19 XML, JSON, Text over HTTP, JMS & SMTP SMS Support MySQL DB and Cassandra Support
- 20. WSO2 Complex Event Processor (CEP) 20
- 21. E.g. Usecases ● Finding out the trend of #Cloud Twiaer tags over Xme ● IdenXfy the public interest on cloud market leaders based on tweets on #Cloud Twiaer tags ● Comparing interest trends on cloud market ● NoXfying via Email when an interesXng tweet was posted. 21
- 23. Twiaer Agent //Configuring Agent AgentConfiguraXon agentConfiguraXon = new AgentConfiguraXon(); agentConfiguraXon.setTrustStorePassword("/Users/suho/wso2/dev/twiaer-‐sample/wso2cep-‐3.0.0/repository/" + "resources/security/client-‐truststore.jks"); agentConfiguraXon.setTrustStore("wso2carbon”); Agent agent = new Agent(agentConfiguraXon); //New Data publisher to BAM final AsyncDataPublisher asyncDataPublisher = new AsyncDataPublisher("tcp://localhost:7611", "admin", "admin", agent); //Defining the Stream String STREAM_NAME = "twiaer_data_stream"; String VERSION = "1.0.0"; String streamDefiniXon = "{" + " 'name':'" + STREAM_NAME + "'," + " 'version':'" + VERSION + "'," + " 'nickName': 'Twiaer_Data_Stream'," + " 'descripXon': 'Stream of twiaer data'," + " 'payloadData':[" + " {'name':'createdAt','type':'LONG'}," + " {'name':'user','type':'STRING'}," + " {'name':'tweet','type':'STRING'}" + " ]" + "}"; asyncDataPublisher.addStreamDefiniXon(streamDefiniXon, STREAM_NAME, VERSION); 23
- 24. Visualizing #Cloud Twiaer Social Stream 24
- 25. 25
- 26. Finding out the trend of #Cloud Twiaer tags over Xme ๏ ๏ 26 insert overwrite table Twi@erDataStreamStatsPerMinute select count(1) as total_tweet_count, year(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )) as year, month(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐ MM-‐dd HH:mm:ss.SSS' )) as month,day(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )) as day,hour(from_unixXme(cast(created_at/ 1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )) as hour, minute(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )) as minute,concat(substring(from_unixXme(cast(created_at/1000 as BIGINT), 'yyyy-‐MM-‐ dd HH:mm:ss'),0,16),':00') as Xme from TwiaerDataStream group by year(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )), month(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )),day(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )),hour(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )),minute(from_unixXme(cast(created_at/1000 as BIGINT),'yyyy-‐MM-‐dd HH:mm:ss.SSS' )),substring(from_unixXme(cast(created_at/1000 as BIGINT), 'yyyy-‐MM-‐ dd HH:mm:ss'),0,16);
- 27. 27
- 28. IdenXfy the public interest on cloud market leaders insert overwrite table CountOnCompanyTwi@erDataStream select 'IBM' as company, count(1) as cnt from TwiaerDataStream where instr(lower(tweet), 'ibm') >0 ; insert overwrite table CountOnCompanyTwi@erDataStream select 'Google' as company, count(1) as cnt from TwiaerDataStream where instr(lower(tweet), 'google') >0 ; insert overwrite table CountOnCompanyTwi@erDataStream select 'VMware' as company, count(1) as cnt from TwiaerDataStream where instr(lower(tweet), 'vmware') >0 ; insert overwrite table CountOnCompanyTwi@erDataStream select 'Azure' as company, count(1) as cnt from TwiaerDataStream where instr(lower(tweet), 'azure') >0 ; 28
- 29. 29
- 30. Comparing interest trends on cloud market 30
- 31. 31
- 32. NoXfying via Email when an interesXng tweet was posted! & Take AcXon 32
- 33. CEP: Retweet NoXficaXon from twiaerStream[tweet contains 'RT '] insert into reTweetStream; 33
- 35. Log Analysis Demos ….. http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/ http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/ 35
- 37. Call to acXon page ๏ ๏ Big Data Webinar Series: hap://wso2.com/landing/wso2-‐bigbrother-‐webinar-‐series/ ๏ WSO2 Business AcXvity Monitor: hap://wso2.com/products/business-‐acXvity-‐monitor/ ๏ WSO2 BAM Docs: hap://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor +DocumentaXon ๏ WSO2 Complex Event Processor: hap://wso2.com/products/complex-‐event-‐processor/ ๏ 37 Be your own NSA: hap://wso2.com/landing/nsa/ WSO2 CEP Docs: hap://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor +DocumentaXon
- 38. Contact us !