HailDB: A NoSQL API Direct to InnoDB

10 likes2,215 views

HailDB is an open source project that provides a NoSQL API directly to InnoDB. It exposes the internal InnoDB API through cursor-based functions for transactions, key lookups, index scans, updates, inserts, and deletes. HailDB can be used to build embedded, fast, concurrent relational databases or as the storage engine for SQL databases by using the API instead of SQL. The API provides functions for initialization, configuration, DDL statements, transactions, cursors, tuples, full table scans, inserts, and updates.

Technology

HailDB A NoSQL API direct to InnoDB Stewart Smith

Embedded, fast, concurrent, relational database

High-Performance Storage Services Using HailDB and Java Ballroom E, 2:50pm Thursday

ib_savepoint_take() ib_savepoint_release() ib_savepoint_rollback()

Position a cursor, perform row operations

Represents columns from table record or secondary index

ib_tpl_t tpl = ib_clust_read_tuple_create(cursor)

$ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);$

ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

HailDB: A NoSQL API Direct to InnoDB

1. HailDB A NoSQL API direct to InnoDB Stewart Smith

2. What is HailDB?

3. What is InnoDB?

4. ACID database enigne

5. Commonly found in MySQL

6. Originally was separate

7. Been in MySQL “forever”

8. HailDB and InnoDB have...

9. Features you'd expect

10. Exposed through SQL

11. Really an interface to an InnoDB API

12. InnoDB internal API

13. Complex and sometimes MySQL specific

14. HailDB history

15. Embedded InnoDB

16. A couple of releases...

17. Abandoned?

18. Enter HailDB

19. Completely free and open source project

20. http://www.haildb.com

21. Launchpad

22. $ bzr branch lp:haildb

23. Tarball releases

24. Packaged in Ubuntu Natty

25. Packaged in drizzle-developers PPA

26. Also RPMs

27. (Currently) no Win32 release

28. Source compatible with embedded_innodb

29. libhaildb.so

30. haildb.h

31. Pandora-build provides autoconf foo

32. What HailDB Provides

33. Cursor based API

34. Transactions

35. Key lookups

36. Index scans

37. DIY Joins

38. Updates

39. Inserts

40. Deletes

41. Use cases

42. Embedded, fast, concurrent, relational database

43. Not arbitrary queries

44. Not SQL

45. High-Performance Storage Services Using HailDB and Java Ballroom E, 2:50pm Thursday

46. Engine for SQL DB

47. Using the API

48. ib_init()

49. ib_cfg_set_int()

50. ib_cfg_set_bool_on|off()

51. ib_cfg_set_text()

52. Set callbacks

53. ib_startup(“barracuda”)

54. ib_shutdown(flag)

55. DDL

56. ib_database_create() ib_database_drop()

57. ib_table_create()

58. ib_table_drop()

59. ib_table_truncate()

60. Transactions

61. ib_trx_begin()

62. ib_trx_commit() ib_trx_rollback()

63. ib_savepoint_take() ib_savepoint_release() ib_savepoint_rollback()

64. Cursors

65. ib_cursor_open_table()

66. ib_cursor_close()

67. ib_cursor_reset()

68. Position a cursor, perform row operations

69. Tuples

70. Represents columns from table record or secondary index

71. ib_tpl_t tpl = ib_clust_read_tuple_create(cursor)

72. ib_col_set_value()

73. ib_col_get_value()

74. Examples

75. Full Table Scan

76. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

77. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

78. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

79. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

80. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

81. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

82. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

83. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

84. ib_tpl_t tpl= ib_clust_read_tuple_create(crsr); err= ib_cursor_first(crsr); while (err == DB_SUCCESS) { err= ib_cursor_read_row(crsr, tpl); /* Handle locking and timeout errors */ ib_tuple_read...(); err= ib_cursor_next(crsr); /* Handle locking and timeout errors */ tpl= ib_tuple_clear(tpl); } ib_tuple_delete(tpl);

85. Insert a row

86. ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

87. ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

88. ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

89. ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

90. ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

91. ib_err_t err; ib_tpl_t tpl; tpl= ib_clust_read_tuple_create(crsr); /* INSERT … VALUES (“a”, “b”, 100); */ ib_col_set_value(tpl, 0, “a”, strlen(“a”)); ib_col_set_value(tpl, 1, “b”, strlen(“b”)); ib_tuple_write_i32(tpl, 2, 100); err= ib_cursor_insert_row(crsr, tpl); /* Handle errors. e.g. duplicate key */ ib_tuple_delete(tpl);

92. Updating a row

93. Delete

94. ib_tuple_delete()

95. WHERE clause

96. Table statistics

97. HailDB

98. Questions?

Editor's Notes

#2: Talk to me!
#10: ACID compliance Transactions, savepoints, crash recovery, concurrency etc
#14: Row buffer format etc
#15: Row buffer format etc
#16: Clean, easy to use API, shared library. Based on early InnoDB Plugin
#17: Clean, easy to use API, shared library. Based on early InnoDB Plugin
#18: Oracle certainly doesn't like to announce things.
#19: So I created HailDB. Take existing Embedded InnoDB code. Fix bugs, add features. Update InnoDB based on.
#20: So I created HailDB. Take existing Embedded InnoDB code. Fix bugs, add features. Update InnoDB based on.
#26: Karmic, Lucid and Maverick
#27: I don't personally use RPM based distros... but there are people making RPMs
#28: Karmic, Lucid and Maverick
#29: Since we've cleaned up what symbols are being exported, we've bumped the so version Added APIs Existing apps, change what you link against, header name, and recompile.
#30: Since we've cleaned up what symbols are being exported, we've bumped the so version Added APIs Existing apps, change what you link against, header name, and recompile.
#31: Since we've cleaned up what symbols are being exported, we've bumped the so version Added APIs Existing apps, change what you link against, header name, and recompile.
#32: Detect which HailDB version is present etc.
#43: Even where you may have connected to an external RDBMS. Need concurrency that sqlite doesn't provide Need less overhead than InnoDB in MySQL
#44: That's what SQL is for
#45: Use a SQL database. HailDB can be used to build a SQL database though.
#47: Like the HailDB engine in Drizzle. Our work-in-progress next gen way to interface between innobase (HailDB) and the DB kernel
#49: Call first. Sets up a few internal things.
#52: Used to set configuration options. HailDB does not provide config file parsing or command line option parsing.
#53: Log messages (defaults to printing to stderr) trx_is_interrupted (can interrupt threads stuck on locks) Panic handler (elegantly shut down the rest of your app on corruption)
#54: Starts InnoDB. Reads tablespaces, does recovery etc.
#55: The various innodb shutdown options NORMAL, NO_IBUF_MERGE, NO_BUFPOOL_FLUSH
#56: Schema lock. Shared for read, exclusive for write.
#57: Creates the directories for a database. Where file-per-table ibd files go. Otherwise, not really that useful :)
#58: Create a table. Done inside a transaction. Get DDL lock. Schema specified by API calls, not SQL. No foreign keys (yet)
#59: You can do several DDL operations in the same DDL transaction. You can even (within limits) do transactions on other tables. DFE in DDL txn
#60: Same as TRUNCATE TABLE in SQL
#61: Simple calls for begin/commit/rollback.
#62: Supports isolation levels
#63: Rather obvious. You can also query the state of the transaction, or in deadlock, ib_trx_release() if txn has been rolled back by HailDB
#64: Named savepoints. Maps exactly to what you'd expect from SQL usage. In future we'll have a lightweight “statement” savepoint
#65: Can have a cursor not yet attached to a transaction.
#66: There's also using_id() methods.
#67: When you're done
#68: Lets you reuse the data sturctures, increased perf
#69: Don't really need for INSERT. Do need for UPDATE and DELETE of course.
#71: So can lookup the primary key from a secondary index without having to read the clustered index
#72: To read/write row in clustered index. Other types for searching secondary index, redaing row from secondary index
#73: Used to set values. e.g. doing an UPDATE or INSERT row operation.
#74: A suite of calls both for specific data types (such as 64bit int) and pointer to it in memory (useful for BLOBs).
#75: A suite of calls both for specific data types (such as 64bit int) and pointer to it in memory (useful for BLOBs).
#93: A little more involved. Read a row, copy it to a new tuple, update that tuple, call ib_cursor_update_row()
#95: On a positioned cursor
#96: Search an index (or do a full table scan), examining rows until you find the ones you're looking for.
#97: Same ones used by MySQL and Drizzle optimisers. Likely not too useful in most HailDB using apps.
#98: And that's HailDB. A rather simple interface to an advanced transactional engine.

HailDB: A NoSQL API Direct to InnoDB

More Related Content

What's hot (20)

Similar to HailDB: A NoSQL API Direct to InnoDB (20)

Recently uploaded (20)

HailDB: A NoSQL API Direct to InnoDB

Editor's Notes