Machine learning on source code
1 like564 views
The document discusses the application of machine learning on source code, detailing various tools and techniques for analyzing and processing code, such as code classification, comment sentiment analysis, and type inference. It highlights datasets, including GitHub data, and introduces projects like Source{d} for mining code and the Modelforge tool for model management. Additionally, it explores practical problems solved by applying machine learning in language processing and code analysis.
![>>> engine.repositories.references.head_ref
.commits.tree_entries.blobs
.classify_languages()
.filter('lang = "Python"')
.extract_uasts()
.query_uast('//*[@roleIdentifier]')
.extract_tokens("result", "tokens")
.select("blob_id", "path", "tokens")](https://image.slidesharecdn.com/machinelearningonsourcecode-180627180730/85/Machine-learning-on-source-code-32-320.jpg)
![Splitting and normalization
_tcp_socket_connect -> [tcp, socket, connect]
AuthenticationError -> [authentication, error]
authentication, authenticate -> authenticate
01.
02.
03.](https://image.slidesharecdn.com/machinelearningonsourcecode-180627180730/85/Machine-learning-on-source-code-45-320.jpg)
Machine learning on source code
- 3. GitHub
- 5. MLonCode
- 6. Examples
- 8. class ??? : def connect(self, dbname, user, password, host, port): # ... def query(self, sql): # ... def close(self): # ... 01. 02. 03. 04. 05. 06. 07.
- 9. class Database : def connect(self, dbname, user, password, host, port): # ... def query(self, sql): # ... def close(self): # ... 01. 02. 03. 04. 05. 06. 07.
- 10. class Foo: def bar(self, qux): # ... def baz(self, waldo): # ... def do(self, really): # ... 01. 02. 03. 04. 05. 06. 07.
- 11. twitter/mysql Tokutek/mysql-5.5 facebook/mysql-5.6 percona/percona-xtrabackup Tokutek/mariadb-5.5 percona/percona-server atcurtis/mariadb webscalesql/webscalesql-5.6 alibaba/AliSQL mysql/mysql-server Details Projects similar to MariaDB/server?
- 13. Similar code detection • By style • By structure • By identifiers provides us Global graph Licenses Refactoring
- 15. Code style class foobar: def connecttoserver(self): myserverhost = globalconfig.server.host
- 16. class FooBar: def connect_to_server(self): myServerHost = globalConfig.server.host
- 17. Many other applications • Prediction of class, function, variable names • Type inference • Which comments do not make sense? • Which comments are funny? • Which APIs are bad or misused?
- 18. Your code as a crime scene • Idea: mine the development history • Book by Adam Tornhill • codescene.io
- 20. Data
- 21. Datasets • GHTorrent - everything except Git repositories, 70GB • Public Git Archive - only Git repositories, 3TB • GitHub Data in Google BigQuery • rovers & borges - DIY
- 22. PGA • 270k of siva files • CSV index Details in the paper.
- 25. Plumbing
- 26. source{d} engine • siva or bare git repository loader for Apache Spark • Classification, parsing (thanks to bblfsh), filtering, checkouting, scaling • Apache license GitHub
- 27. source{d} engine >>> from sourced.engine import Engine >>> engine = Engine(spark, "/path/to/siva/files", "siva") >>> engine.repositories.references.head_ref .commits.tree_entries.blobs .classify_languages() .select("blob_id", "path", "lang") .show()
- 28. function definition identifier value: nearest_neighbors arguments body identifier value: self identifier value: origin keyword argument comment value: origin can be... conditional identifier value: k literal value: 10 condition true branch false branch builtin value: isinstance identifier value: origin builtin value: tuple builtin value: tuple builtin value: list AST
- 29. How to parse • Regular expressions - Pygments, highlight.js • Abstract syntax tree (AST) - ANTLR • Compilation
- 31. Universal AST • ± uniform structure • ± standard node types (roles) • XPath queries • 4 traversal orders dashboard.bblf.sh
- 32. >>> engine.repositories.references.head_ref .commits.tree_entries.blobs .classify_languages() .filter('lang = "Python"') .extract_uasts() .query_uast('//*[@roleIdentifier]') .extract_tokens("result", "tokens") .select("blob_id", "path", "tokens")
- 34. Powerful
- 35. modelforge Automatically fetch from the internet "Model Store" Flexible, modern binary format - ASDF Versioning Reproducability Support various programming languages GitHub
- 36. Machine Learning
- 37. hercules • Go • Mining Git history • Relies on go-git, Babelfish and Tensorflow • Apache license GitHub
- 38. MLonCode in hercules • Structural hotness • Comment sentiment
- 39. source{d} ml • Python • Targets large-scale MLonCode • Runs on top of source{d} engine, modelforge and Tensorflow • Apache license GitHub
- 40. Solved practical problems • Identifier splitting • O(1) code similarity search • Topic modeling • Embeddings
- 42. V1 ⇔ "foo" V2 ⇔ "bar" V3 ⇔ "integrate" distance(V1, V2) < distance(V1, V3) distance(Vi, Vj) = arccos Vi⋅Vj ∥Vi∥∥∥Vj ∥∥ Scalar product Norm
- 43. How to estimate ?Vi ⋅ Vj
- 44. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None 01. 02. 03. 04. 05. 06. 07. 08.
- 45. Splitting and normalization _tcp_socket_connect -> [tcp, socket, connect] AuthenticationError -> [authentication, error] authentication, authenticate -> authenticate 01. 02. 03.
- 46. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None database, connect , user , password , host , port , tcp, socket , authenticate , error, close 01. 02. 03. 04. 05. 06. 07. 08. >>> 2 2 2 2 2 2 2
- 48. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None connect , user , password , host , port , tcp, socket , authenticate , error, close 01. 02. 03. 04. 05. 06. 07. 08. >>> 2 2 2 2 2 2 2
- 50. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None connect, user, password, host, port 01. 02. 03. 04. 05. 06. 07. 08. >>>
- 52. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None tcp, socket, connect, host, port 01. 02. 03. 04. 05. 06. 07. 08. >>>
- 54. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None authenticate , user, password, error, socket, close 01. 02. 03. 04. 05. 06. 07. 08. >>> 2
- 56. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None authenticate, user, password 01. 02. 03. 04. 05. 06. 07. 08. >>>
- 58. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None authenticate, error, socket, close 01. 02. 03. 04. 05. 06. 07. 08. >>>
- 60. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None authenticate, error 01. 02. 03. 04. 05. 06. 07. 08. >>>
- 62. class Database: def connect(self, user, password, host, port): self._tcp_socket_connect(host, port) try: self._authenticate(user, password) except AuthenticationError as e: self.socket.close() raise e from None socket, close 01. 02. 03. 04. 05. 06. 07. 08. >>>
- 65. Incidence matrix • number of times and were together • Also known as the co-occurrence matrix Cij Cij = i j
- 66. Pointwise Mutual Information (PMI) Vi ⋅ Vj = P M Iij = log Cij ∑ C ∑ N k=1 Cik ∑ N k=1 Cjk
- 71. Swivel Multi-GPU Multi-node Quality tricks Swivel: Improving Embeddings by Noticing What's Missing by Shazeer et.al. Tensorflow implementation
- 72. • afoo • qux • myfoo • baz • mfoo • wibble • dofoo • quux • dfoo • testing • ifoo Nearest to“foo”
- 73. Analogies “bug” - “test” + “expect” = “suppress” “database” - “query” + “tune” = “settings” “send” - “receive” + “pop” = “push”
- 74. Typos • recieve = receive • grey = gray • calback = callbak = callback
- 75. Summary
- 76. Summary MLonCode is fun There is data There are tools Community is forming Idea ? Doubt ? Write us !
- 78. Thank you egor@sourced.tech egorbu egor_bu sourcedtech blog.sourced.tech Awesome #MLonCode bit.ly/2MAU7qG