Gitbase, SQL interface to Git repositories

Editor's Notes

• #2: Hi, I am Javi Fontán and I am an engineer at source{d}. I'm going to show you what's gitbase and some of its implementation details.
• #4: At source{d} we make tools to do large scale code analysis and machine learning on source code. I am not going to talk about the Machine Learning stuff today. All the tools I'm going to talk about are Open Source.
• #5: All the tools we make that deal with access to repositories, parse source code, detect language and do searches are under an umbrella called the source{d} Engine. gitbase is one of these tools.
• #7: gitbase is a database server that uses mysql wire protocol and SQL dialect to query git repositories. This means that you can connect to it with MySQL clients and libraries and use a common SQL dialect to make queries. The storage for this database are plain git repositories, no preprocessing of these repos are needed.
• #8: This is a typical use of gitbase. We provide it with a directory full of git repositories and it starts listening in a port. You should probably find the port 3306 familiar.
• #9: Now we can connect to it using a MySQL compatible client. In this case we are using the standard MySQL client tool and get a list of the tables. You'll also find these tables familiar if you've been using Git.
• #10: In this example we are using count and group by to get the number of commits per repository. As you can see it looks like a normal SQL query that you can do to your databases.
• #11: Here's the result.
• #12: In this other example we use a filter and a join to count only blobs that belong to the HEAD commit of each repository. You my find NATURAL JOIN weird but it’s an INNER JOIN that automatically does the matching by all columns with the same name. In this case it matches all the commits pointed by references named “HEAD” and blobs contained in those commits.
• #13: This is what you get in mysql client.
• #14: This other example is a more complex one. We are not only getting information from git itself but also use other components to extract more data: * LANGUAGE function detects the programming language from a file, this is used to filter only the files written in Go language at the end of the query * UAST parses the file and generates what we call a Universal Abstract Syntax Tree. That is, a standardized AST that let us query it in the same way for any languages supported. This AST can be queried with XPATH and that's the part with roleFunction and roleDeclaration in the query.
• #15: Here's an excerpt of the output in one repository.
• #17: gitbase uses several libraries to accomplish this: * go-mysql-server is a library to create databases. Uses parts of vitess project for the wire protocol and parse sql. It also has the database engine. * bblfsh is responsible to generate and query the Universal AST mentioned previously. * enry is a port of linguist library to Go to detect programming languages. * go-git is a library to access git repositories. It's like libgit2 but implemented in Go. * to implement indexes we use pilosa, that is a library to deal with bitmap indexes, and bolt db as key value store for bitmap to value mapping
• #18: * gitbase uses Volcano or iterator model. This is the architecture used by most relational databases. A tree of iterators is generated from the query and each node gets information from the lower one, transforms or filters the data that is consumed then by another iterator on top. * gitbase has only one database with tables for each type of object. They contain the data for all the repositories. * it has some extra functions to get info not stored in the repositories like languages and Universal ASTs
• #19: We map every git object type to its own table. As such you can query table commits or refs.
• #20: And we also have some helper tables that allow us to do relations. One special one is files that have the information of a file from a commit. This allows easier queries than using relations to match commits with tree entries and blobs.
• #22: We ran into lots of performance issues accessing git objects. Even if we are actively working on making our libraries faster accessing objects we are also trying to read as few objects as possible.
• #23: Some of the techniques we use are: * cache as much data as we can so we don't have to access it in disk or uncompress again * squashed tables, that let us access the objects directly when possible instead of iterating all and filtering the ones we want * create indexes for values that we access more frequently
• #24: In this example we want to get all the repositories in which HEAD commit has author name Javi Fontan. Without any optimization we iterate all the references and all the commits. The where nodes filter the objects we are interested in and the join matches the data from the iterators in both sides. Here we do a full scan of references table and we do N commit full scans, one for each reference that reaches the join. As you can see this is not super optimal.
• #25: Squashed tables technique uses some peculiarities from git. We know that each ref tip is only one commit and we can do pushdown of some filters to the tables, that is, pass the filtering to the storage layer where we can get the object directly instead of iterating and filtering. In this example we substitute the general join into a chain of iterators. The references iterator is able to get directly the references from all repositories with an specific name, in this case HEAD. Then we use the commit hash to feed the next iterator so we only get HEAD commits and filter by author. Now we are only getting the commits that are HEAD of some repo, a much cheaper operation.
• #26: Another technique we use is creating indexes for some columns. These indexes are not git indexes but user defined ones. Depending on the usage we may be interested in different columns so we let the user pick which ones they want. There may be people that want to pinpoint commits from specific users or do some analysis on files written in some language. Generating indexes take quite some time and space and that’s another reason to let the user pick their indexes.
• #27: We use bitmaps as they are easily composable. We also store the packfile and offset for the rows so we bypass git indexing and read directly from the files. In this example we create an index for the author name of commits. After it is completed we can get exactly the commits that match a filter by author name.
• #28: And that’s what I had prepared for today. If you want to know more about gitbase or any related library you can ask me and my colleagues or go to our project pages shown here. Thank you!