Getting to Know the Cassandra Codebase
- 1. Cozy with CassandraGetting to know the Cassandra CodebaseGary Dusbabek • Rackspace@gdusbabekCassandra Summit • Mission Bay Conference Center • San Francisco • 10 August 2010
- 2. OutlineCode ThemesStartup SequenceKey ClassesRead PathWrite PathStages & ThreadingBootstrap & StreamingTests & IDE considerationsAdding API methodsQuestions
- 5. Themes and PatternsSingletons and statics
- 6. Themes and PatternsStages &Thread pools
- 8. CassandraDaemonLoads configurationTransport initializationStorage (Keyspace initialization)CommitLogrecoveryStorageService.initServer()Initializes CassandraServerPasses it off to transport
- 9. CassandraServerImplements IDL interface methods (cassandra.thrift, cassandra.genavro)Good place to start diving when adding or troubleshooting API methods
- 10. ConfigurationDatabaseDescriptorVia CassandraDaemon.setup()Looks for config path, loads yamlDoesn’t spin anything upDefines system tablesKS and CF described by CFMetaData and KSMetaData
- 11. CodeSomeClasses
- 12. Main ControllersEnd with *Service or *ManagerStorageService, MessagingServiceCompactionManager, HintedHandoffManager, StageManager, StreamInManager
- 13. StorageProxyPut & Get methodsCollection of static methodsMerges local and distributed operationsTracks latencyExposed via StorageProxyMBean
- 14. StorageServiceinitServer()—Starts servicesRegisters verb handlers (in MessagingService)Main event respondersRepository of replication strategies and TokenMetadataRing topology & token information
- 15. MessagingServiceVerb handlers reside hereSets up socket listenersGateway for outbound messagesMS.sendRR()MS.sendOneWay()Inbound tooMS.receive()
- 16. Table & ColumnFamilyStoreAlso RowMutationLow-level storage operationso.a.c.db.*SSTableLocal operations
- 17. Read+Write Paths
- 19. Reading StorageProxyRangesCollectorsLocal & remote branches
- 20. Reading StorageProxy localTable, ColumnFamilyStoreCFSMake QueryFilterQuery MemtablesQuery SSTablesCoalesce in iteratorso.a.c.db packageo.a.c.db.filter
- 21. Reading StorageProxy remoteread commandResponse handlerSend to remote nodes
- 22. WritingSocket->CassandraServerValidationConvert to Mutation (IDL object)Penalties!
- 23. WritingStorageProxyblocking/non-blocking mutate local/remote branchRowMutationone ColumnFamily perColumnFamilyCollection of column modifications
- 24. WritingRM.apply->Table.applyWrite to CLIterate over RM CFsCFS.apply()Overwrites results on pre-existing column families
- 25. WritingRM is serialized into a Message and sent to other nodesWaits for ACKs depending on CL
- 27. StagesSEDAhttp://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdfo.a.c.concurrent.StageManagerRead, mutation, stream, gossip, response, anti entropy, load balance, migrationThread pools that consume tasks
- 28. ThreadsMessagingService.listen() spawns thread.Each incoming connection spawns a new short-lived thread (IncomingTcpConnection)Non-stream ops go to MS.messageDeserializerExecutor_Stream ops handled there.Anti-entropy repair
- 30. <=0.6 BootstrapingA wants data, B has data.StreamingRequestMessage A->BHandled on B by StreamRequestVerbHandlerFor each range StreamOut.transferRanges()Flush, anticompactionStreamInitiateMessage B->A for each range transferMeanwhile, back on A…StreamInitiateVerbHandler gets the SIM from B, does some nesting.StreamInitiateDone A->BBack to B…StreamInitiateDoneHandler gets the SID from ACalls StreamOutManager.startNext() which sends a single file to AMessagingService on A picks this up and the file is streamed.Sstable is createdSTREAM_FINISHED A->BB gets rid of the file, calls SOM.startNext()
- 31. 0.7 BootstrappingA wants data, B has dataStreamRequestMessage A->BOn B, StreamRequestVerbHandlerIf single file, sends it.If range, StreamOut.transferRangesForRequest()Send next file (first will contain meta data about all files)On A, IncomingStreamReader.read() Data is received, sstable createdAck, request next file
- 32. TestsTestable & UntestableUnit testsant clean build testSystem testsant gen-thrift-pynosetests test/system/test_thrift_server.py
- 33. IDEConfiguration file must be in the classpathTreat as sourcelib vs build/libLog at debug
- 34. IDE-ea -Xms128M –Xmx2G -Dcom.sun.management.jmxremote.port=8081 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false-Dcassandra-foreground=yes -Dlog4j.configuration=log4j-server.properties-Dmx4jport=9081
- 35. Adding API methodsSame goes for modifyingDefine method and structures in IDLinterface/cassandra.thriftRegenerate filesant gen-thrift-java gen-thrift-pyImplement methods in o.a.c.thrift.CassandraServerCreate a system test (tests/system/test_thrift_server.py)
Editor's Notes
- #3: Talk about Cassandra processes and the classes that relate to them.
- #4: Data operated on,then passed off to another layer.Evident on R/W pathOnion analogy
- #5: DisjointCoupled when neccessary
- #6: Model of class/object designSuck it upNot a class project
- #15: ServicesGossiperMessagingServiceMigrationManagerBootstrapPreloaded cacheHandle to server-related tasks.
- #28: layers