HBase RowKey design for Akka Persistence
8 likes2,265 views
The document discusses the design of rowkeys in HBase within the context of the Akka persistence plugin, focusing on the principles of persistent actors and event sourcing. It outlines the use of commands and events to manage state and persistence, along with optimizations for reading and writing data in HBase. Furthermore, it addresses potential issues like hot-spotting and emphasizes efficient recovery strategies through snapshots and event replays.
![state until [E8]
Snapshots
S8
!
!
Journal
E1 E2 E3 E4
E5 E6 E7 E8
!
!
Snapshot Store
snapshot!](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-29-320.jpg)
![state until [E8]
Snapshots
S8
!
!
Journal
E1 E2 E3 E4
E5 E6 E7 E8
crash!
!
!
Snapshot Store
snapshot!
S8](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-30-320.jpg)
![state until [E8]
Snapshots
restart!
replay!
S8
!
!
Journal
E1 E2 E3 E4
E5 E6 E7 E8
!
!
Snapshot Store
S8](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-34-320.jpg)
![state until [E8]
Snapshots
S8
!
!
Journal
E1 E2 E3 E4
E5 E6 E7 E8
We could delete these!
!
!
Snapshot Store
S8](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-35-320.jpg)
![trait MySummer extends Processor {!
var nums: List[Int]!
var total: Int!
!
def receive = {!
case "snap" => saveSnapshot(total)!
case SaveSnapshotSuccess(metadata) => // ...!
case SaveSnapshotFailure(metadata, reason) => // ...!
}!
}!
Snapshots, save
Async!](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-36-320.jpg)
![Journal Plugin API
!
def asyncWriteMessages(!
messages: immutable.Seq[PersistentRepr]!
): Future[Unit]!
!
def asyncDeleteMessagesTo(!
persistenceId: String, !
toSequenceNr: Long, !
permanent: Boolean!
): Future[Unit]!
!
@deprecated("writeConfirmations will be removed.", since = "2.3.4")!
def asyncWriteConfirmations(!
confirmations: immutable.Seq[PersistentConfirmation]!
): Future[Unit]!
!
@deprecated("asyncDeleteMessages will be removed.", since = "2.3.4")!
def asyncDeleteMessages(!
messageIds: immutable.Seq[PersistentId], !
permanent: Boolean!
): Future[Unit]!](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-40-320.jpg)
![Optimising writes
001-a-0001,
001-b-0001,
001-a-0051
id = “a”
002-a-0002,
002-b-0002,
002-a-00052
003-a-0003,
003-b-0003,
003-a-0053
049-a-0049,
049-b-0049,
049-a-0099
…
“partition” seeds
[HMaster]](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-52-320.jpg)
![Optimising writes
001-a-0001,
001-b-0001,
001-a-0051
id = “a”
002-a-0002,
002-b-0002,
002-a-00052
003-a-0003,
003-b-0003,
003-a-0053
049-a-0049,
049-b-0049,
049-a-0099
…
“partition” seeds
[HMaster]](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-53-320.jpg)
![Optimising writes
001-a-0001,
001-a-0051
id = “a”
002-a-0002,
002-a-00052
003-a-0003,
003-a-0053
049-a-0049,
049-a-0099…
[HMaster]
Write load spread to cluster!](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-54-320.jpg)
![Optimising writes
001-a-0001,
001-a-0051
id = “a”
002-a-0002,
002-a-00052
003-a-0003,
003-a-0053
049-a-0049,
049-a-0099…
Writes to different regions! [HMaster]](https://image.slidesharecdn.com/akka-persistence-hbase-plugin-2014-140716084322-phpapp01/85/HBase-RowKey-design-for-Akka-Persistence-55-320.jpg)
HBase RowKey design for Akka Persistence
- 1. RowKey design in HBase Akka Persistence Plugin Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL Konrad `@ktosopl` Malawski
- 3. Konrad `@ktosopl` Malawski typesafe.com geecon.org Java.pl / KrakowScala.pl sckrk.com / meetup.com/Paper-Cup @ London GDGKrakow.pl meetup.com/Lambda-Lounge-Krakow hAkker @
- 6. Compared to “Good ol’ CRUD Model” state “Mutable Record” state = apply(es) “Series of Events”
- 7. super quick domain modelling! sealed trait Command! case class GiveMe(geeCoins: Int) extends Command! case class TakeMy(geeCoins: Int) extends Command Commands - what others “tell” us; not persisted case class Wallet(geeCoins: Int) {! def updated(diff: Int) = State(geeCoins + diff)! } State - reflection of a series of events sealed trait Event! case class BalanceChangedBy(geeCoins: Int) extends Event! Events - reflect effects, past tense; persisted
- 8. var state = S0 ! persistenceId = “a” ! PersistentActor Command ! ! Journal
- 9. PersistentActor var state = S0 ! persistenceId = “a” ! ! ! Journal Generate Events
- 10. PersistentActor var state = S0 ! persistenceId = “a” ! ! ! Journal Generate Events E1
- 11. PersistentActor ACK “persisted” ! ! Journal E1 var state = S0 ! persistenceId = “a” !
- 12. PersistentActor “Apply” event ! ! Journal E1 var state = S0 ! persistenceId = “a” ! E1
- 13. PersistentActor ! ! Journal E1 var state = S0 ! persistenceId = “a” ! E1 Okey!
- 14. PersistentActor ! ! Journal E1 var state = S0 ! persistenceId = “a” ! E1 Okey!
- 15. PersistentActor ! ! Journal E1 var state = S0 ! persistenceId = “a” ! E1 Ok, he got my $.
- 16. PersistentActor class BitCoinWallet extends PersistentActor {! ! var state = Wallet(coins = 0)! ! def updateState(e: Event): State = {! case BalanceChangedBy(coins) => state.updatedWith(coins)! }! ! // API:! ! def receiveCommand = ??? // TODO! ! def receiveRecover = ??? // TODO! ! }!
- 17. persist(e) { e => }
- 18. PersistentActor def receiveCommand = {! ! case TakeMy(coins) =>! persist(BalanceChangedBy(coins)) { changed =>! state = updateState(changed) ! }! ! ! ! ! ! ! } async callback
- 19. persist(){} - Ordering guarantees ! ! Journal E1 var state = S0 ! persistenceId = “a” ! C1 C2 C3
- 20. ! ! Journal E1 var state = S0 ! persistenceId = “a” ! C1 C2 C3 Commands get “stashed” until processing C1’s events are acted upon. persist(){} - Ordering guarantees
- 21. ! ! Journal var state = S0 ! persistenceId = “a” ! C1 C2 C3 E1 E2 E2E1 events get applied in-order persist(){} - Ordering guarantees
- 22. C2 ! ! Journal var state = S0 ! persistenceId = “a” ! C3 E1 E2 E2E1 and the cycle repeats persist(){} - Ordering guarantees
- 24. Eventsourced, recovery /** MUST NOT SIDE-EFFECT! */! def receiveRecover = {! case replayedEvent: Event => ! state = updateState(replayedEvent)! } re-using updateState, as seen in receiveCommand Akka Persistence ScalaDays
- 25. Snapshots
- 27. Eventsourced, snapshots def receiveCommand = {! case command: Command =>! saveSnapshot(state) // async!! } /** MUST NOT SIDE-EFFECT! */! def receiveRecover = {! case SnapshotOffer(meta, snapshot: State) => ! this.state = state! ! case replayedEvent: Event => ! updateState(replayedEvent)! } snapshot!? how?
- 28. …sum of states… Snapshots ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8
- 29. state until [E8] Snapshots S8 ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 ! ! Snapshot Store snapshot!
- 30. state until [E8] Snapshots S8 ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 crash! ! ! Snapshot Store snapshot! S8
- 31. Snapshots ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 crash! ! ! Snapshot Store S8
- 32. “bring me up-to-date!” Snapshots ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 restart! replay! ! ! Snapshot Store S8
- 33. “bring me up-to-date!” Snapshots restart! replay! S8 ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 ! ! Snapshot Store S8
- 34. state until [E8] Snapshots restart! replay! S8 ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 ! ! Snapshot Store S8
- 35. state until [E8] Snapshots S8 ! ! Journal E1 E2 E3 E4 E5 E6 E7 E8 We could delete these! ! ! Snapshot Store S8
- 36. trait MySummer extends Processor {! var nums: List[Int]! var total: Int! ! def receive = {! case "snap" => saveSnapshot(total)! case SaveSnapshotSuccess(metadata) => // ...! case SaveSnapshotFailure(metadata, reason) => // ...! }! }! Snapshots, save Async!
- 37. Snapshot Recovery class Counter extends Processor {! var total = 0! ! def receive = {! case SnapshotOffer(metadata, snap: Int) => ! total = snap! ! case Persistent(payload, sequenceNr) => // ...! }! }
- 39. Akka Persistence TCK class JournalTCKSpec extends JournalSpec {! lazy val config = ConfigFactory.parseString(“").! withFallback(ConfigFactory.load())! }! ! ! ! ! class SnapshotStoreTCKSpec extends SnapshotStoreSpec {! lazy val config = ConfigFactory.parseString(“").! ! ! ! ! ! ! ! ! ! ! withFallback(ConfigFactory.load())! }!
- 40. Journal Plugin API ! def asyncWriteMessages(! messages: immutable.Seq[PersistentRepr]! ): Future[Unit]! ! def asyncDeleteMessagesTo(! persistenceId: String, ! toSequenceNr: Long, ! permanent: Boolean! ): Future[Unit]! ! @deprecated("writeConfirmations will be removed.", since = "2.3.4")! def asyncWriteConfirmations(! confirmations: immutable.Seq[PersistentConfirmation]! ): Future[Unit]! ! @deprecated("asyncDeleteMessages will be removed.", since = "2.3.4")! def asyncDeleteMessages(! messageIds: immutable.Seq[PersistentId], ! permanent: Boolean! ): Future[Unit]!
- 42. Hbase table data layout a-0001,a-0002,a-0003 b-0001,b-0002,b-0003
- 43. Hbase table data layout a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 HBase regions
- 44. Hbase table data layout a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 HBase regions (lexicographical order)
- 45. Optimising writes a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” id = “b”
- 46. Optimising writes a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” id = “b”
- 47. Optimising writes a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” id = “b” HOT region
- 48. Optimising writes a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” id = “b” HOT region
- 49. Optimising writes a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” id = “b” impossible to spread load!
- 52. Optimising writes 001-a-0001, 001-b-0001, 001-a-0051 id = “a” 002-a-0002, 002-b-0002, 002-a-00052 003-a-0003, 003-b-0003, 003-a-0053 049-a-0049, 049-b-0049, 049-a-0099 … “partition” seeds [HMaster]
- 53. Optimising writes 001-a-0001, 001-b-0001, 001-a-0051 id = “a” 002-a-0002, 002-b-0002, 002-a-00052 003-a-0003, 003-b-0003, 003-a-0053 049-a-0049, 049-b-0049, 049-a-0099 … “partition” seeds [HMaster]
- 54. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… [HMaster] Write load spread to cluster!
- 55. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… Writes to different regions! [HMaster]
- 57. Optimising reads a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” replay!
- 58. Optimising reads a-0001,a-0002,a-0003 b-0001,b-0002,b-0003 id = “a” replay! Ordered, batch read, super efficient!!!
- 59. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 60. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 61. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 62. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 63. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 64. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 65. Optimising writes 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… this is madness! replay!
- 66. Optimising reads
- 67. Optimising recovery 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay!
- 68. Optimising recovery 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… async! replay! + re-sequence
- 69. Optimising recovery 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… async! small batches replay! + re-sequence
- 70. Optimising recovery 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay! (to seqNr = 2)
- 71. Optimising recovery 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… replay! (to seqNr = 2)
- 72. Optimising recovery 001-a-0001, 001-a-0051 id = “a” 002-a-0002, 002-a-00052 003-a-0003, 003-a-0053 049-a-0049, 049-a-0099… for short recovery = no need to check all servers! replay! (to seqNr = 2)
- 73. Akka Persistence ScalaDays 2014 Akka Persistence Plugins • Journals / Snapshot Stores (http://akka.io/community/) • Cassandra • HBase • Kafka • DynamoDB • MongoDB • shared LevelDB journal for testing
- 74. Akka Persistence ScalaDays 2014 Links • http://akka.io • https://groups.google.com/forum/#!forum/akka-user ! • https://github.com/ktoso/akka-persistence-hbase ! • http://www.slideshare.net/alexbaranau/intro-to-hbase- internals-schema-design-for-hbase-users • http://blog.sematext.com/2012/04/09/hbasewd-avoid- regionserver-hotspotting-despite-writing-records-with- sequential-keys/ • https://github.com/OpenTSDB/asynchbase
- 75. ©Typesafe 2014 – All Rights Reserved