Analysing Github events with Neo4j
1 like595 views
This document describes how to import and analyze GitHub event data stored in JSON files using the Neo4j graph database. It provides examples of Cypher queries that can be used to analyze relationships between users, repositories, pull requests, forks and other GitHub elements from the events data. These include queries to find the most active user, most forked repository, number of comments on pull requests before being merged, and users who have worked on the same repositories through merged pull requests.
![MERGE (u:User {name:'pixelfreak2005'})
CREATE (ev:PullRequestEvent {time:toInt(1401606356) })
MERGE (u)-[:DO]->(ev)
MERGE (pr:PullRequest {html_url:'https://github.com/pixelfreak2005/liqiud_android_packages_apps_Settings/pull/2'})
SET pr += { id:toInt(16573622), number:toInt(2), state:'open'}
MERGE (ev)-[:PR_OPEN]->(pr)
MERGE (ow:User {name:'pixelfreak2005'})
MERGE (or:Repository {id:toInt(20338536), name:'liqiud_android_packages_apps_Settings'})
MERGE (or)-[:OWNED_BY]->(ow)
MERGE (pr)-[:PR_ON_REPO]->(or)](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-8-320.jpg)
![#ForkEvent
MERGE (u:User {name:'rudymalhi'})
CREATE (ev:ForkEvent {time:toInt(1401606379) }) MERGE (u)-[:DO]->(ev)
CREATE (fork:Fork:Repository {name:'Full-Stack-JS-Nodember'})
MERGE (ev)-[:FORK]->(fork)-[:OWNED_BY]->(u)
MERGE (bro:User {name:'mgenev'})
MERGE (br:Repository {id:toInt(15503488), name:'Full-Stack-JS-Nodember'})-[:OWNED_BY]->(bro)
MERGE (fork)-[:FORK_OF]->(br)](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-9-320.jpg)
![#IssueCommentEvent
You can check if the issue is related to a P.R. and build the complete P.R. schema
MERGE (u:User {name:'johanneswilm'})
CREATE (ev:IssueCommentEvent {time:toInt(1401606384) })
MERGE (u)-[:DO]->(ev)
MERGE (comment:IssueComment {id:toInt(44769338)})
MERGE (ev)-[:ISSUE_COMMENT]->(comment)
MERGE (issue:Issue {id:toInt(34722578)})
MERGE (repo:Repository {id:toInt(14487686)})
MERGE (comment)-[:COMMENT_ON]->(issue)-[:ISSUE_ON]->(repo)
SET repo.name = 'diffDOM'
MERGE (owner:User {name:'fiduswriter'})
MERGE (comment)-[:COMMENT_ON]->(issue)-[:ISSUE_ON]->(repo)-[:OWNED_BY]->(owner)](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-10-320.jpg)
![who did the most events ?
!
MATCH (u:User)-[r:DO]->()
RETURN u.name, count(r) as events
ORDER BY events DESC
LIMIT 1](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-12-320.jpg)
![which repo has been the most touched ?
!
MATCH (repo:Repository)<-[r]-()
RETURN repo.name, count(r) as touchs
ORDER BY touchs DESC
LIMIT 1](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-13-320.jpg)
![which repo has been the most forked ?
!
MATCH (repo:Repository)<-[:FORK_OF]-(fork:Fork)<-[:FORK]-
(event:ForkEvent)
RETURN repo.name, count(event) as forks
ORDER BY forks DESC
LIMIT 1](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-14-320.jpg)
![which repo has the most merged PRs ?
!
MATCH (repo:Repository)<-[:PR_ON_REPO]-
(pr:PullRequest)<-[merge:PR_MERGE]-()
RETURN repo.name, count(merge) as merges
ORDER BY merges DESC
LIMIT 1](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-15-320.jpg)
![how much forks are resulting in an open PR ?
!
MATCH p=(u:User)-[:DO]->(fe:ForkEvent)-[:FORK]->(fork:Fork)
-[:FORK_OF]->(repo:Repository)<-[:PR_ON_REPO]-(pr:PullRequest)
-[:PR_OPEN]-(pre:PullRequestEvent)<-[:DO]-(u2:User)<-[:OWNED_BY]-
(f2:Fork)<-[:BRANCH_OF]-(br:Branch)<-[:FROM_BRANCH]-(pr2:PullRequest)
WHERE u = u2 AND fork = f2 AND pr = pr2
RETURN count(p)](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-16-320.jpg)
![Number of comments on a PR before the PR is merged ?
!
MATCH p=(ice:IssueCommentEvent)-[:ISSUE_COMMENT]->(comment:IssueComment)
-[:COMMENT_ON]->(issue:Issue)-[:BOUND_TO_PR]->(pr:PullRequest)
<-[:PR_MERGE]-(pre:PullRequestEvent)
WHERE ice.time <= pre.time
WITH pr, count(comment) as comments
RETURN avg(comments)](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-18-320.jpg)
![Top contributor ?
Which user has the most merged PRâs on repositories
not owned by him
!
MATCH (u:User)-[r:DO]->(fe:PullRequestEvent)-[:PR_OPEN]->(pr:PullRequest {state:'merged'})
-[:PR_ON_REPO]-(repo:Repository)-[:OWNED_BY]->(u2:User)
WHERE NOT u = u2
RETURN u.name, count(r) as prs
ORDER BY prs DESC
LIMIT 1](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-19-320.jpg)
![Relate together Users having Merged PR's on same
repositories, could serve as Follow Recommendations Engine!
!
MATCH p=(u:User)-[:DO]-(e:PullRequestEvent)-->(pr:PullRequest {state:'merged'})-
[:PR_ON_REPO]->(r:Repository)<-[:PR_ON_REPO]-(pr2:PullRequest
{state:'merged'})--(e2:PullRequestEvent)<-[:DO]-(u2:User)
WHERE NOT u = u2
WITH nodes(p) as coll
WITH head(coll) as st, last(coll) as end
MERGE (st)-[r:HAVE_WORKED_ON_SAME_REPO]-(end)
ON MATCH SET r.w = (r.w) + 1
ON CREATE SET r.w = 1](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-20-320.jpg)
![avg time between a repo is forked and this fork result in
an opened PR ?
!
MATCH p=(u:User)-[:DO]->(fe:ForkEvent)-[:FORK]->(fork:Fork)-[:FORK_OF]
->(repo:Repository)<-[:PR_ON_REPO]-(pr:PullRequest)-[:PR_OPEN]-
(pre:PullRequestEvent)
<-[:DO]-(u2:User)<-[:OWNED_BY]-(f2:Fork)<-[:BRANCH_OF]-(br:Branch)<-
[:FROM_BRANCH]-(pr2:PullRequest)
WHERE u = u2 AND fork = f2 AND pr = pr2
RETURN count(p), avg(pre.time - fe.time) as offsetTime](https://image.slidesharecdn.com/github-events-neo4j-141113044131-conversion-gate02/85/Analysing-Github-events-with-Neo4j-24-320.jpg)
Analysing Github events with Neo4j
- 2. #whoami âą Christophe Willemsen âą Bruges (Belgium) âą Created Graphgen (graphgen.neoxygen.io) ! ! ! ! âą @ Graph Aware
- 4. #github-data-archive âą Github Events âą archive available @ http://www.githubarchive.org/ âą events json files per hour âą approx. 10k events per hour âą ! the file in itself is not valid json, all file rows are valid json
- 5. #event-types âą CommitCommentEvent âą CreateEvent âą DeleteEvent âą DeploymentEvent âą DeploymentStatusEvent âą DownloadEvent âą FollowEvent âą ForkEvent âą ForkApplyEvent âą GistEvent âą GollumEvent âą IssueCommentEvent âą IssuesEvent âą MemberEvent âą PageBuildEvent âą PublicEvent âą PullRequestEvent âą PullRequestReviewCommentEvent âą PushEvent âą ReleaseEvent âą StatusEvent âą TeamAddEvent âą WatchEvent
- 6. #gh4j Github Events importer for Neo4j Parse file + build customized Cypher Statements for each event + load in Neo4j
- 7. #PullRequestEvent Payload and informations from the past âą You get information of the PR âą You can also build informations about the repo, who is owning it for e.g. âą On which branch âą Depending of the P.R. Action (open/close/merge), you can determine for a close/merge who opened first the PR and from which fork it is coming
- 8. MERGE (u:User {name:'pixelfreak2005'}) CREATE (ev:PullRequestEvent {time:toInt(1401606356) }) MERGE (u)-[:DO]->(ev) MERGE (pr:PullRequest {html_url:'https://github.com/pixelfreak2005/liqiud_android_packages_apps_Settings/pull/2'}) SET pr += { id:toInt(16573622), number:toInt(2), state:'open'} MERGE (ev)-[:PR_OPEN]->(pr) MERGE (ow:User {name:'pixelfreak2005'}) MERGE (or:Repository {id:toInt(20338536), name:'liqiud_android_packages_apps_Settings'}) MERGE (or)-[:OWNED_BY]->(ow) MERGE (pr)-[:PR_ON_REPO]->(or)
- 9. #ForkEvent MERGE (u:User {name:'rudymalhi'}) CREATE (ev:ForkEvent {time:toInt(1401606379) }) MERGE (u)-[:DO]->(ev) CREATE (fork:Fork:Repository {name:'Full-Stack-JS-Nodember'}) MERGE (ev)-[:FORK]->(fork)-[:OWNED_BY]->(u) MERGE (bro:User {name:'mgenev'}) MERGE (br:Repository {id:toInt(15503488), name:'Full-Stack-JS-Nodember'})-[:OWNED_BY]->(bro) MERGE (fork)-[:FORK_OF]->(br)
- 10. #IssueCommentEvent You can check if the issue is related to a P.R. and build the complete P.R. schema MERGE (u:User {name:'johanneswilm'}) CREATE (ev:IssueCommentEvent {time:toInt(1401606384) }) MERGE (u)-[:DO]->(ev) MERGE (comment:IssueComment {id:toInt(44769338)}) MERGE (ev)-[:ISSUE_COMMENT]->(comment) MERGE (issue:Issue {id:toInt(34722578)}) MERGE (repo:Repository {id:toInt(14487686)}) MERGE (comment)-[:COMMENT_ON]->(issue)-[:ISSUE_ON]->(repo) SET repo.name = 'diffDOM' MERGE (owner:User {name:'fiduswriter'}) MERGE (comment)-[:COMMENT_ON]->(issue)-[:ISSUE_ON]->(repo)-[:OWNED_BY]->(owner)
- 11. Letâs have some fun and try some queries ! demo
- 12. who did the most events ? ! MATCH (u:User)-[r:DO]->() RETURN u.name, count(r) as events ORDER BY events DESC LIMIT 1
- 13. which repo has been the most touched ? ! MATCH (repo:Repository)<-[r]-() RETURN repo.name, count(r) as touchs ORDER BY touchs DESC LIMIT 1
- 14. which repo has been the most forked ? ! MATCH (repo:Repository)<-[:FORK_OF]-(fork:Fork)<-[:FORK]- (event:ForkEvent) RETURN repo.name, count(event) as forks ORDER BY forks DESC LIMIT 1
- 15. which repo has the most merged PRs ? ! MATCH (repo:Repository)<-[:PR_ON_REPO]- (pr:PullRequest)<-[merge:PR_MERGE]-() RETURN repo.name, count(merge) as merges ORDER BY merges DESC LIMIT 1
- 16. how much forks are resulting in an open PR ? ! MATCH p=(u:User)-[:DO]->(fe:ForkEvent)-[:FORK]->(fork:Fork) -[:FORK_OF]->(repo:Repository)<-[:PR_ON_REPO]-(pr:PullRequest) -[:PR_OPEN]-(pre:PullRequestEvent)<-[:DO]-(u2:User)<-[:OWNED_BY]- (f2:Fork)<-[:BRANCH_OF]-(br:Branch)<-[:FROM_BRANCH]-(pr2:PullRequest) WHERE u = u2 AND fork = f2 AND pr = pr2 RETURN count(p)
- 18. Number of comments on a PR before the PR is merged ? ! MATCH p=(ice:IssueCommentEvent)-[:ISSUE_COMMENT]->(comment:IssueComment) -[:COMMENT_ON]->(issue:Issue)-[:BOUND_TO_PR]->(pr:PullRequest) <-[:PR_MERGE]-(pre:PullRequestEvent) WHERE ice.time <= pre.time WITH pr, count(comment) as comments RETURN avg(comments)
- 19. Top contributor ? Which user has the most merged PRâs on repositories not owned by him ! MATCH (u:User)-[r:DO]->(fe:PullRequestEvent)-[:PR_OPEN]->(pr:PullRequest {state:'merged'}) -[:PR_ON_REPO]-(repo:Repository)-[:OWNED_BY]->(u2:User) WHERE NOT u = u2 RETURN u.name, count(r) as prs ORDER BY prs DESC LIMIT 1
- 20. Relate together Users having Merged PR's on same repositories, could serve as Follow Recommendations Engine! ! MATCH p=(u:User)-[:DO]-(e:PullRequestEvent)-->(pr:PullRequest {state:'merged'})- [:PR_ON_REPO]->(r:Repository)<-[:PR_ON_REPO]-(pr2:PullRequest {state:'merged'})--(e2:PullRequestEvent)<-[:DO]-(u2:User) WHERE NOT u = u2 WITH nodes(p) as coll WITH head(coll) as st, last(coll) as end MERGE (st)-[r:HAVE_WORKED_ON_SAME_REPO]-(end) ON MATCH SET r.w = (r.w) + 1 ON CREATE SET r.w = 1
- 21. QUESTIONS ?
- 22. âą More queries in the gist file : https://gist.github.com/ikwattro/ 071d36f135131e8e4442 âą Not valid with Github Live API (different payload) âą zipped db file http://bit.ly/1BaMCy9
- 24. avg time between a repo is forked and this fork result in an opened PR ? ! MATCH p=(u:User)-[:DO]->(fe:ForkEvent)-[:FORK]->(fork:Fork)-[:FORK_OF] ->(repo:Repository)<-[:PR_ON_REPO]-(pr:PullRequest)-[:PR_OPEN]- (pre:PullRequestEvent) <-[:DO]-(u2:User)<-[:OWNED_BY]-(f2:Fork)<-[:BRANCH_OF]-(br:Branch)<- [:FROM_BRANCH]-(pr2:PullRequest) WHERE u = u2 AND fork = f2 AND pr = pr2 RETURN count(p), avg(pre.time - fe.time) as offsetTime