Networks All Around Us: Extracting networks from your problem domain
1 like562 views
This document summarizes a presentation on analyzing networks in problem domains. It provides examples of different types of networks that can be analyzed, including founder networks, website behavior networks, and online social networks. It also describes various tools and techniques for social network analysis, such as calculating centrality, clustering, and dispersion. The presentation emphasizes how to identify relevant entities and relationships to model a problem domain as a property graph and analyze it using graph databases and network analysis libraries.
![THEN USE
SNA
LIBRARIES
#
# Example - calculate friendship dispersion
#
di_graph = nx.DiGraph()
all_edges = util.json_cr_file_2_array('jsondump/links.json')
for edge in all_edges:
if 'type' in edge and edge['type'] == 'partnership':
di_graph.add_edge(edge['domain1'], edge[‘domain2'])
dispersion = nx.dispersion(di_graph)](https://image.slidesharecdn.com/metisgraphtalk-160304060644/85/Networks-All-Around-Us-Extracting-networks-from-your-problem-domain-20-320.jpg)
Networks All Around Us: Extracting networks from your problem domain
- 1. M E T I S M E E T U P Networks All Around Us: Analyzing Networks in your Problem Domain | 3/3/2016 Russell Jurney http://bit.ly/socialnetworkanalysis2
- 3. BACKGROUND Serial Entrepreneur Contributed code to Apache Druid, Apache Pig, Apache DataFu, Apache Whirr, Azkaban, MongoDB Apache Commi?er Three-Bme O'Reilly Author Started & Shipped Product at E8 Security Ning, LinkedIn, Hortonworks veteran
- 4. 2009 2010 2011 2012 2014 EXAMPLES OF NETWORKS
- 5. FOUNDER NETWORKS node = company edge = employment transition as in people who… …worked at one startup, founded another
- 6. WEBSITE BEHAVIOR node = web page edge = user browses one page, then another
- 7. ONLINE SOCIAL NETWORKS node = linkedin profile, edge = linked connection
- 8. EMAIL INBOX node = email address, edge = sent email
- 9. MARKETS node = company, edge = partnership
- 10. MARKET REPORTS
- 12. TINKERPOP “Marko Rodriguez is the Doug Cutting of graph analytics.” —Mark Twain
- 13. PROPERTY GRAPHS
- 15. PROPERTY GRAPHS IN YOUR DOMAIN identify entities identify relationships specify schema (or not) populate graph database learn to think in graph walks (hard) query in batch query in realtime
- 16. POPULATING A PROPERTY GRAPH // Add nodes while((json = company_reader.readLine()) != null) { document = jsonSlurper.parseText(json) v = graph.addVertex('company') v.property("_id", document._id) v.property("domain", document.domain) v.property("name", document.name) }
- 17. POPULATING A PROPERTY GRAPH // Get a graph traverser g = graph.traversal() while((json = links_reader.readLine()) != null) { document = jsonSlurper.parseText(json) // Add edges to graph v1 = g.V().has('domain', document.home_domain).next() v2 = g.V().has('domain', document.link_domain).next() v1.addEdge(document.type, v2) }
- 19. final Graph g = TinkerFactory.createClassic(); try (final OutputStream os = new FileOutputStream(“jsondump/links.json")) { GraphSONWriter.build().create().writeGraph(os, g); } EXPORT LINKS AS JSON
- 20. THEN USE SNA LIBRARIES # # Example - calculate friendship dispersion # di_graph = nx.DiGraph() all_edges = util.json_cr_file_2_array('jsondump/links.json') for edge in all_edges: if 'type' in edge and edge['type'] == 'partnership': di_graph.add_edge(edge['domain1'], edge[‘domain2']) dispersion = nx.dispersion(di_graph)
- 21. TOOLS OF SNA SNA = Social Network Analysis centrality clustering block models cores dispersion center-pieces
- 22. CENTRALITY Centrality is a way of measuring how central or important a particular node is in a social network. OR What nodes should I care about?
- 23. SINGLE-RELATIONAL CENTRALITY(S) # all-links-the-same-type-centrality g.V().out().groupCount() # things-humans-walk-centrality g.V().hasLabel(‘human’).out(‘walks’).groupCount() # things-dogs-eat-centrality g.V().hasLabel(‘dog’).out(‘eats’).groupCount()
- 24. MULTI-RELATIONAL CENTRALITY(S) # things-eaten-by-things-humans-walk-centrality g.V().hasLabel(‘human’).out(‘walks’).out(‘eats’).groupCount() # things-hated-by-things-humans-pet-centrality g.V().hasLabel(‘human’).out(‘pets’).out(‘hates’).groupCount() # things-that-pet-things-that-eat-mice-centrality g.V().in(‘eats’).in(‘pets’).groupCount()
- 25. CENTRALITIES degree centrality closeness centrality betweenness centrality eigenvector centrality
- 26. DEGREE CENTRALITY in-degree centrality is nice… it works even if you’re missing a node’s outbound links
- 27. DEGREE CENTRALITY # computation count connections …its that simple in-degree centrality = popularity out-degree centrality = gregariousness # meaning risk of catching cold
- 28. DEGREE CENTRALITY IN GREMLIN # all-links-the-same-type-centrality g.V().out().groupCount()
- 29. CLOSENESS CENTRALITY # computation count hops of all shortest paths distance from all other nodes reciprocal of farness # meaning communication efficiency spread of information
- 30. CLOSENESS CENTRALITY IN GREMLIN closenessCentrality = g.V().as(“a”).repeat(both(‘relationship_type').simplePath()).emit().as("b") .dedup().by(select(“a","b")).path() .group().by(limit(local, 1)).by(count(local) .map {1/it.get()}.sum())
- 31. BETWEENNESS CENTRALITY # computation count of times node appears in shortest paths… …between all pairs of nodes # meaning control of communication between other nodes
- 32. EIGENVECTOR CENTRALITY # computation counts connections of connected nodes more connected neighbors matter more # meaning influence of one node on others pagerank is an eigenvector centrality
- 33. EIGENVECTOR CENTRALITY IN GREMLIN g.V() .repeat(out(‘relationship_type’).groupCount(‘m').by('unique_key')) .times(n).cap('m')
- 34. CLUSTERING
- 35. CLUSTERING property based clustering: k-means graph based clustering: modularity property graph based clustering: CESNA
- 36. BLOCK MODELS how much do clusters connect? are links reciprocal? circos are helpful
- 37. CORES
- 38. DISPERSION Romantic Partnerships and the Dispersion of Social Ties: A Network Analysis of Relationship Status on Facebook
- 39. CENTER-PIECE SUBGRAPHS *Slide stolen from Tong, Faloutsos, Pan