Network analysis lecture

Editor's Notes

• #3: Let’s talk about network analysis. Starting with “what is a network”. I can start talking about nodes and edges and maths and stuff, but it’s easier to start by showing you.
• #4: A network is a set of things that are linked together. Networks are usually visualized as a set of points (“nodes”) connected by lines (“edges”). “relationships”: This can be as simple as “a relationship exists”, or as complex as “this probability matrix describes the complex relationship between the states in these nodes”
• #5: Classic networks include things like communications and power grids; in my world, they also explain things like the movement of water supplies between streams, rivers, farms and processing plants. As you look at these examples, I want you to think about the types of questions you could start asking with this network data. For example, in infrastructure, you often only have access to the junctions and the fact that there *is* a connection between two points. With transport, the dataset gets richer. You not only have the nodes and links between them, you also have timetables that list the average time between stations (and the current state of the network) and the switching costs of changing lines at a station. Aside: The London Underground map is one of my favorite network visualizations: a wonderful simplification of a complex system.
• #6: And here the dataset gets richer still: this is just my Facebook network; I have many other networks that I connect to people with, and overlapping uses for those networks. I can also start investigating the information that’s carried across those networks, and their effect on my state (e.g. my political opinions).
• #7: Many datasets can be framed as networks. Here, the Spotify API gives me relationships between its artists; I can also create some of my own relationship data from this API by looking at which songs and artists are on the same playlists.
• #8: Much of text analysis can also be framed as networks. Here’s a matrix showing words that occur together in sentences and how many times they’ve co-occurred in the dataset. If we see every words as a node, and every nonzero co-occurrance score as a link, we’ve got ourselves a network. This can also be applied at the document level, e.g. Jonathan Stray’s Overview project analysed networks of documents to find civilian deaths in the Iraq War.
• #9: So why do we bother representing things as networks? After all, we could list the songs that are played together most, or the stations with the most travellers. The bottom line is that, when you look at something as a network, you can start to see which things have the most important relationships in your network, and where to concentrate effort if you want to affect it all (e.g. who do want to retweet your ideas?). We’re going to look at network analysis at 3 levels: node, group and network.
• #10: But first, some nomenclature I’m using computer science language for this. Other groups that study networks and their words for nodes and edges are: Here: network, node, edge maths: graph, vertex, arc/edge Physics: network, site, bond Sociology: network, actor, relation Biology: network, node, edge
• #11: This is all valid python code (you can use it to generate a network diagram with NetworkX - see next slide). Different representations are useful for different things (if you’re coding up your own algorithms): Diagram: good for explaining a network (especially if interactive) Adjacency matrix: good for dense graphs (can also use scipy.sparse to use this for sparse graphs) Adjacency list: good for sparse graphs (e.g. social networks tend to be sparse); used by NetworkX Edge list: good general representation Maths: good for describing algorithms. V = vertices (nodes); E=edges; e=map from edges to nodes. n is the number of nodes; m is the number of edges
• #12: I’ve listed several python libraries for network management at the end of these slides. The one we’re using here is NetworkX. It produces ugly graphs, but has a good set of network analysis tools. NB Use nx.DiGraph() if you want a directed graph
• #13: NetworkX centrality functions: http://networkx.lanl.gov/reference/algorithms.centrality.html
• #14: Simplest form of centrality “degree” = how many direct links connect to this node Note that degree centrality is normalized (divided) by the largest possible number of connections per node: in this case, 9. Degree centrality is not a great measure of power: what’s important is the number of nodes that the node can easily reach, and the highest-ranked node might be part of a clique (e.g. not well connected to the outside world).
• #15: Between = how many nodes are there between two nodes? Nodes with high betweenness have influence over the flow of information or goods through a network: they bridge separate communities (good) but also often are a single point of failure in communications between those communities (bad).
• #16: Closeness = has the shortest average path to all other nodes in the network. Nodes with high closeness have great influence over the rest of the network, especially if influence diminishes with path length; these points are also good places to observe all information flows from.
• #17: Eigenvector centrality measures how much influence a node has in the whole network, taking account of their connections to other highly-connected nodes. These are the “kings” of your network - they might not have great closeness or betweenness, but they do wield a lot of influence. PageRank is based on eigenvector centrality. NB You’ll need to look at the eigenvectors of the adjacency matrix to build this one, and like neural networks, eigenvector centrality algorithms won’t always converge to a solution.
• #18: All available in networkx Social networks = short path lengths, high clustering, skewed degree distributions. Small worlds = lots of highly-connected small groups with fewer connections to other groups: Saw this effect in the Ebola response contact-tracking.
• #19: Let’s look at communities: groupings within your network. These are useful for questions like “how is a network likely to split into groups” and “how do I efficiently influence this network”. Note that when we have a community, we can study it as a network in its own right, including finding the most important nodes in it. “Small world theory” = there are roughly 6 steps on the shortest path between each pair of nodes in the world (see also “6 degrees of Kevin Bacon” http://en.wikipedia.org/wiki/Six_degrees_of_separation). The maths works out at roughly s = ln(n)/ln(k) where n is the population size and k is the average number of connections per node. For k=30, s is usually roughly 6. NetworkX community functions: http://networkx.lanl.gov/reference/algorithms.community.html
• #20: First, let’s cover networks where there isn’t a path from every node to every other node in the network. These networks are called “disconnected” networks and can be interesting because of the lack of connections between groups (e.g. you’re trying to most efficiently connect up different transport systems).
• #21: These are group measures based on the numbers of links K-core: Every node in the clique is connected to K or more other nodes in the clique. Clique-level analysis and node-level analysis interact with each other, e.g. if you find a set of cliques in a network, you can then look for and use the central nodes in those cliques.
• #22: K-cores and cliques don’t always find the natural cliques in a graph (especially one containing human relationship). N-cliques: “friend of friend” cliques; use Bron and Kerbosch algorithm. Issues include nodes that contribute to the clique aren’t included in it. P-clique addresses some of this. Other approaches: n-clans, k-plexes etc.: see http://faculty.ucr.edu/~hanneman/nettext/C11_Cliques.html#nclique
• #24: Achoo! Diffusion model used when it’s important that you find *everybody* in contact, e.g. for Ebola, you have to assume that everyone an infectious person is in contact with is a potential carrier. Here, we assume that node 9 changes state first; in the next step of the algorithm, the nodes directly connected to it (0,1,7) change state; in the next step, the nodes connected to (0,1,7) change state, etc. etc. Thought experiment: infections are time-sensitive, e.g. you get infected, then either get better or die. How would you represent this in a network? What would you expect to happen in a small-world network? 
• #25: Only if… Diffusion models for more complex choices, e.g. whether to go see a movie, based on your friends’ opinions plus reading movie reviews. In complex contagion, a node changes state based on the state of *all* its neighbors, and often also on outside information; just because 9 is in one state, 1 doesn’t have to change to that state too (but it might change state with probability p).
• #26: Network diagrams are still the best way to describe networks Edge bundling is useful for small world networks Metanodes are useful for large networks of communities An adjacency matrix can help if it’s nicely grouped, but sometimes it’s just more confusing. Explaining graphs to the C-suite? Use visual cues they’re used to. Carefully. Some examples are in the Visualisation Periodic Table at http://www.visual-literacy.org/periodic_table/periodic_table.html