CUBRID Inside - Architecture, Source & Management Components
Download as PPTX, PDF1 like3,409 views
CUBRID is an open-source relational database management system designed for internet portal services, offering features like a multi-threaded architecture, various APIs, and a graphical interface for management. The document details CUBRID's system architecture, including its server, client, and broker subsystems, as well as its processes for query handling and transaction management. It also emphasizes the importance of following Creative Commons attribution and licensing conditions when reusing or modifying the work.
CUBRID Inside - Architecture, Source & Management Components
- 1. CUBRID InsideFirst Look Inside CUBRIDJan 18, 2009
- 2. 저작권 Copyright NoticeCREATIVE COMMONS 저작자표시-동일조건변경허락 2.0 대한민국이용자는 아래의 조건을 따르는 경우에 한하여 자유롭게이 저작물을 복제, 배포, 전송, 전시, 공연 및 방송할 수 있습니다.이차적 저작물을 작성할 수 있습니다.다음과 같은 조건을 따라야 합니다:저작자표시. 귀하는 원저작자를 표시하여야 합니다. What does "Attribute this work" mean? The page you came from contained embedded licensing metadata, including how the creator wishes to be attributed for re-use. You can use the HTML here to cite the work. Doing so will also include metadata on your page so that others can find the original work as well. 동일조건변경허락. 귀하가 이 저작물을 개작, 변형 또는 가공했을 경우에는, 이 저작물과 동일한 이용허락조건하에서만 배포할 수 있습니다. 귀하는, 이 저작물의 재이용이나 배포의 경우, 이 저작물에 적용된 이용허락조건을 명확하게 나타내어야 합니다.저작권자로부터 별도의 허가를 받으면 이러한 조건들은 적용되지 않습니다.Nothing in this license impairs or restricts the author's moral rights.CREATIVE COMMONS Attribution-Share Alike 2.0 KoreaYou are free:to Share — to copy, distribute and transmit the workto Remix — to adapt the workUnder the following conditions:Attribution. You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). What does "Attribute this work" mean? The page you came from contained embedded licensing metadata, including how the creator wishes to be attributed for re-use. You can use the HTML here to cite the work. Doing so will also include metadata on your page so that others can find the original work as well. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to this web page.Any of the above conditions can be waived if you get permission from the copyright holder.Nothing in this license impairs or restricts the author's moral rights.http://creativecommons.org/licenses/by-sa/2.0/kr/Copyright 2009 Search Solution Corporation. All Rights Reserved.
- 3. CUBRID
- 4. ContentsSystem Architecture and ProcessesDesign & Implementation ConceptsModule and Source StructureIn The Next CUBRID Inside4/41 First Look Inside CUBRID
- 7. CUBRID Processes and IPC
- 8. Query Processing WorkflowSystem Architecture and Processes15/41 First Look Inside CUBRID
- 9. System Architecture and ProcessesWhat is CUBRID?CUBRID isOpen Source DBMS owned and developed by NHNHosted at nForge siteTargeted to a database platform for Internet portal serviceRelational database system with object extensions – Object-Relational Model1,000,000 line of source code scaled projectEngine – C/C++, CUBRID Manager (GUI) and JDBC driver – JavaCUBRID providesFull-featured database server engineServer and Broker architectureGUI management and query toolCUBRID ManagerVarious API (JDBC, ODBC(OLE DB), PHP, and CCI)Python and Ruby as contributed projects1.16/41 First Look Inside CUBRID
- 10. System Architecture and ProcessesSystem Architecture1.2Server SubsystemMulti-threaded storage engine part which runs as DB serverConsists of Transaction Manager, Object Manager, Lock Manager, Query Manager, Log Manager, and Storage ManagerClient SubsystemClient library part providing native C APIConsists of Query Parser & Optimizer, Object & Lock Cache, Transaction Manager and Object ManagerBroker SubsystemLinked with client library and implemented CCI protocolDoes job queuing, connection pooling, monitoring and loggingConnectorsJDBC driverCCI libraryMany drivers are implemented with CCI library – ODBC, OLEDB, PHP, Python, and Ruby7/41 First Look Inside CUBRID
- 11. System Architecture and ProcessesSystem Architecture1.2CUBRIDManagerGUIInterfacePHPOLE DBRubyCUBRIDManagerJDBCODBCCCIPythonCM ServerJobQueuingMonitoringConnectionPoolingLoggingAdminUtilityBrokerCreate, Delete, Copy, RenameQueryParser &OptimizerLockCacheObjectManagerTransactionManagerAdd VolumeLoad /UnloadBackup /RestoreQueryManagerLockManagerServerCompact /OptimizeStorageManagerLogManagerCheck /DiagActiveLogFile BasedObjectsDataVolumeIndexVolumeTempVolumeArchiveLog8/41 First Look Inside CUBRID
- 12. 1.2CUBRIDManagerGUIInterfacePHPOLE DBRubyCUBRIDManagerJDBCODBCCCIPythonCM ServerCreate, Delete, Copy, RenameJobQueuingMonitoringConnectionPoolingLoggingAdminUtilityBrokerClient LibraryAdd VolumeNative C APIParserObjectManagerSchemaManagerTransactionManagerLoad /UnloadQueryTransformWorkspace ManagerBackup /RestoreQueryOptimizerMemory ManagerCompact /OptimizePlanGenerationCheck /DiagCommunication ModuleStorage EngineServerCommunication ModuleTransactionManagerLogManagerLockManagerQueryManagerAccessMethodB+TreeModuleFile ManagerSystemCatalog ModuleSystem ArchitectureBuffer ManagerDisk ManagerActiveLogFile BasedObjectsDataVolumeIndexVolumeTempVolumeArchiveLog9/41 First Look Inside CUBRID
- 13. System Architecture and ProcessesCUBRID Processes and IPCMaster Process ‘cub_master’Daemon process listening on the TCP port for clients’ connectioncsql, broker, and admin utilitiesRegistrar of server processes using Unix Domain SocketServer Process ‘cub_server’Multi-threaded process serving clients’ requestsHas thread pool and job queueAccesses database volume and log filesOne server process per one databaseBroker Process ‘cub_broker’Daemon process listening on the TCP port for connectors’ connectionControls cub_cas processes (fork/kill) with connection queueCAS Process ‘cub_cas’Connects to the server process in behalf of connectorsAccording to the database and user nameCSQL Program ‘csql’Interactive SQL execution1.310/41 First Look Inside CUBRID
- 14. System Architecture and ProcessesCUBRID Processes and IPC1.3JDBC driverCCI libraryconnectquery &resultquery &resultport listeningcub_brokercubrid_broker.confforkparsedescriptor passAPWeb Servercub_cascub_casshared memorycubridcs.socubridcs.socsqlconnectcubridcs.sorequest &responserequest &responseparseBrokerTCPjob queuemulti-threadport listeningdescriptor passparsecub_mastercub_servercubrid.socubrid.confUDSActiveServerStandbyServermount(read/write)readregistervolume filelog filedatabases.txtvolume filelog fileReplicationcub_admincubridsa.so11/41 First Look Inside CUBRID
- 15. System Architecture and ProcessesQuery Processing Workflow1.412/41 First Look Inside CUBRID
- 16. System Architecture and ProcessesQuery Processing Workflow1.4XASL TypeUNIONExtended Access Spec Language(XASL)DIFFERENCEINTERSECTIONOBJFETCHQuery ManagerTransactionManagerLock ManagerSETFETCHBUILD_LISTBUILD_VALUEQuery Result CacheQuery Plan CacheQuery Evaluator(Interpreter)SCANMERGE_LISTScan ManagementREADUPDATEStorage ManagerDELETEINSERT13/41 First Look Inside CUBRID
- 17. Design & Implementation Concepts2Client-Server and Standalone Mode
- 18. Object, Class and OID
- 19. Representation, Record Descriptor, and Memory Object Pointer
- 20. Workspace – Object & Lock Caching
- 21. 2PL vs. MVCC
- 22. XASL (Extended Access Spec Language)
- 23. Heap, B+tree, Catalog, and Temp
- 24. Click Counter14/41 First Look Inside CUBRID
- 25. Design & Implementation ConceptsClient-Server and Standalone Mode2.115/41 First Look Inside CUBRID
- 26. Design & Implementation ConceptsObject, Class and OID2.216/41 First Look Inside CUBRID
- 27. Design & Implementation ConceptsRepresentation, Record Descriptor, and Memory Object Pointer2.317/41 First Look Inside CUBRID
- 28. Design & Implementation ConceptsWorkspace – Object & Lock Caching2.418/41 First Look Inside CUBRID
- 29. Design & Implementation Concepts2PL vs. MVCC2.519/41 First Look Inside CUBRID
- 30. Design & Implementation ConceptsXASL (Extended Access Spec Language)2.620/41 First Look Inside CUBRID
- 31. Design & Implementation ConceptsHeap, B+tree, Catalog, and Temp2.721/41 First Look Inside CUBRID
- 32. Design & Implementation ConceptsClick CounterIn Terms of UserWhenever the user clicks an article, the read counter of the article is increased.Counts the clicks on the articleIn Terms of Service Application DeveloperThe read counter is very important for bulletin board type service.Combine getting article data and increasing read counter togetherget_article() + increase_counter() = get_article_and_increase_counter()(SELECT article_info) + (UPDATE read_counter+1)In Terms of Database EngineerDo SELECT and UPDATE in one SQL statementSELECT trigger? SELECT FOR UDPATE?UPDATE the record with the short term lockExampleSELECT doc_id, title, INCR(read_counter), post_date FROM board WHERE doc_id=?2.822/41 First Look Inside CUBRID
- 33. Design & Implementation ConceptsClick Counter2.8Don’t walk the same passage twice!When you get there, do as many things as possible.Killing two birds with one stone.23/41 First Look Inside CUBRID
- 34. Module and Source Structure3Overall Structure
- 37. Server Storage Management Component
- 38. Client Storage Management Component
- 40. Job and Thread Management Component
- 42. Broker Component24/41 First Look Inside CUBRID
- 43. Module and Source StructureOverall Structure41 First Look Inside CUBRID25/3.1C APITransaction ManagementCommon & SupportBrokerJob & Thread ManagementCCI LibraryServer Storage ManagementCUBRID ManagerQuery ProcessingClient-Server CommPHP ConnectorQuery ProcessingNative C APIQuery ProcessingJob & Thread ManagementUtilitiesObject ManagementMemory ManagerJDBC DriverClient Storage ManagementJava Stored Procedure
- 44. 41 First Look Inside CUBRID26/Module and Source Structure3.1src/brokerCUBRIDManagerGUIPHPOLE DBRubyJDBCODBCCCIPythonCM Serversrc/querysrc/storagesrc/transactionsrc/compatCreate, Delete, Copy, RenameJobQueuingMonitoringConnectionPoolingLoggingClient LibraryAdd VolumeNative C APIParserObjectManagerSchemaManagerTransactionManagerLoad /UnloadQueryTransformWorkspace ManagerBackup /RestoreQueryOptimizerMemory ManagerCompact /OptimizePlanGenerationCheck /DiagCommunication Modulesrc/cmserversrc/communicationsrc/jdbcsrc/objectsrc/oledbsrc/phpsrc/ccisrc/odbcsrc/optimizersrc/basesrc/executablessrc/heaplayerssrc/parsersrc/threadsrc/connectionStorage EngineCommunication ModuleTransactionManagerLogManagerLockManagerQueryManagerAccessMethodB+TreeModuleFile ManagerSystemCatalog ModuleBuffer ManagerDisk Manager
- 45. Module and Source StructureQuery Processing ComponentScanner/ParserParses query string and builds parse treeUses ANTLR v1.x as parser generatorWill be replaced with ‘bison’ at R2.0Source files in src/parserData structure: PT_PARSER, PT_NODE, …Semantic CheckerType check and binding, name resolution, semantic checking, view translation, and so onSource files in src/parserParse tree traversing function: parser_walk_tree()OptimizerRewrite optimization by heuristic rulesSelect a query execution plan by CBO (Cost Based Optimization) methodSource files in src/parser and src/optimizerData structure: QO_NODE, QO_PLANNER, …XASL GeneratorGenerate XASL tree from the query execution planXASL contains scan information (heap/index/list file/set/method scan), value list, and predicatesSource files in src/parserData structure: XASL_NODE, REGU_VARIABLE, …Query Manager (Executor)Executes XASL tree (XASL interpreter)Includes query evaluator, query plan cache, query result cache, query result file, and scan managementSource files in src/queryMain function: qexec_execute_mainblock()Cursor ManagerExtracts tuple from the list file page shipped from the server Query ManagerSource files in src/query3.227/41 First Look Inside CUBRID
- 46. Module and Source StructureQuery Processing Component3.2Object Management ComponentBrokerSemantic CheckerScanner/ParserOptimizerXASL GeneratorCursor ManagerClient-Server Communication LayerClient-Server Communication LayerServerQuery Plan/Result CacheQuery ManagerTransaction Management ComponentQuery Evaluator(Interpreter)Storage Management Component(Server)Scan Management28/41 First Look Inside CUBRID
- 47. Module and Source StructureTransaction Management ComponentTransaction ManagerControls transaction – start, commit/abort, savepoint, and top operationWorks tightly with lock and log manager Supports 2PC protocol for global transactionInitialize other related modules such as lock, log, and recovery managerSource files in src/transactionData structure: LOG_TDES, TRANTABLE, …Object LocatorControls the flow of objects between the workspaces and the database pagesTranslates between memory representation and disk representationCaches objects from the server into the workspace with the granted locksCached object and lock provides fast access to the objects and reduces interacting with the lock manager on the serverSource files in src/transactionLock ManagerBased on strict 2PL protocolIncludes support for OR featureclass lock and instance lockDeadlock detector with WFGAs separate threadSource files in src/transactionData structure: LK_ENTRY, LK_TRAN_LOCK, …Log ManagerTransaction logging and recoveringREDO/UNDO recovery protocolWAL protocolPhysical logging and logical loggingSupports group commit and asynchronous commit featureSource files in src/transacitonData structure: LOG_GLOBAL, LOG_HDRPAGE, LOG_PAGE, 3.329/41 First Look Inside CUBRID
- 48. Module and Source StructureTransaction Management Component3.3Storage Management Component(Client)Object Management ComponentTransaction Manager(Client)BrokerObject Locator(Client)Client-Server Communication LayerClient-Server Communication LayerServerObject Locator(Server)Lock ManagerTransaction Manager(Server)Storage Management Component(Server)Log Manager30/41 First Look Inside CUBRID
- 49. Module and Source StructureServer Storage Management ComponentFile I/O ManagerProvides disk I/O access functions and implements file structureCoordinates the allocation and deallocation of unstructured files and pagesFile>Sector>PageFile manager requests sectors from the disk managerSource files in src/storageIdentifier: vdes, PAGEIDDisk ManagerVolume file management – implements volume structure in a OS fileA volume corresponds to a singe fileA set of contiguous pages of fixed sizeSource files in src/storageIdentifier: volidPage Buffer ManagerData buffer pool with LRU replacement algorithmUses buffer hash table and fix/unfix protocol (latch)Source files in src/storageIdentifier: VPIDData structure: PGBUF_BUFFER_POOLSlotted Page ManagerManages insertions, deletions, and modifications of records on pagesEach record has an associated slot identifierSource files in src/storageOID = (pageid, slotid, volid)Overflow File ManagerLarger object than the size of a page is placed on overflow pagesSource files in src/storageObject Heap ManagerInserts/deletes/updates objects within a filePermanent unchangeable OID is assigned to an object when it is stored on a heapProvides heap scan method (sequential scan) and direct fetch methodAn object heap holds the instances of one classSource files in src/storageIdentifier: HFIDB+-tree ManagerImplements prefix B+-tree index with KVLProvides index scan method – unique find and range search3.431/41 First Look Inside CUBRID
- 50. Module and Source StructureServer Storage Management Component3.4BrokerClient-Server Communication LayerClient-Server Communication LayerServerQuery Processing Component(Server)Transaction Management Component(Server)Object Heap ManagerB+-tree ManagerLarge Object ManagerExtendible Hash ManagerFile ManagerSlotted Page ManagerOverflow File ManagerPage Buffer ManagerCatalog ManagerDisk ManagerI/O Manager32/41 First Look Inside CUBRID
- 51. Module and Source StructureClient Storage Management ComponentWorkspace ManagerMaintains a cache of database objects within the virtual memory address space of client processMaintains hash table that maps a OID into MOPMOP has OID field and a pointer to memory objectSource files in src/objectQuick Fit AllocatorMemory manager for workspaceSource files in src/object, src/baseGarbage CollectorTo collect and free the unused(dangled) memory objectsNormally GC is disabledObject and Lock CachingWhen the contents of object(s) is(are) required, the object locator is called to fetch the object(s) from the database via the server-side object locatorThe object locator translate the disk representation of the objet into the memory representation and allocates MOP to points itWhile accessing the object in the database, the requested lock is held by the lock manager in the server and the memory object has the cached lock modeThe cached objects can be decached at any time when its state is not consistent with the disk objects, or will be decached at the time of transaction abortAt the time of transaction commit, only the cached locks are released; objects remains with null lock3.533/41 First Look Inside CUBRID
- 52. Module and Source StructureClient Storage Management Component3.5Object Management ComponentBrokerGarbage CollectorQuick File AllocatorWorkspace ManagerTransaction Management Component Object Locator (Client)Client-Server Communication LayerClient-Server Communication LayerServer34/41 First Look Inside CUBRID
- 53. Module and Source StructureObject Management ComponentObject AccessorInterface for object creation, deletion, inspection, and modificationSource files in src/objectObject RepresentationTransforms (or translates) between the disk representation and the memory representation of the objectResolves byte ordering alsoSource files in src/object, src/baseFunctions: or_xxx(), tf_mem_to_disk(), tf_disk_to_mem(), …Data Type and DomainInternal data structure for data type and domainSource files in src/objectData structure: TP_DOMAINSetsSource files in src/objectData structure: SETOBJ, COLSchema ManagerImplements OR model regarding database schemaDefinitions of the class object – attributes, method, class inheritance, and conflict resolutionMemory representation of class objectSource files in src/objetData structure: SM_CLASS, SM_ATTRIBUTE, …Authentication and AuthorizationUser authentication and database privilegeUser, GRANT, REVOKE, …Source files in src/objectTrigger ManagerImplements trigger featureSource files src/objectDynamic LoaderSupport dynamic loading of shared object as ‘method’Depreciated featureLarge ObjectImplements glo, elo, fbo classes and supporting methodsSource files in src/object3.635/41 First Look Inside CUBRID
- 54. Module and Source StructureObject Management Component3.6Object AccessorBrokerSchema ManagerAuthentication & AuthorizationTrigger ManagerSetDynamic LoaderData Type and DomainObject RepresentationStorage Management Component(Client)Client-Server Communication LayerClient-Server Communication LayerServer36/41 First Look Inside CUBRID
- 55. Module and Source StructureJob and Thread Management ComponentConnection ManagerConnection establishment and management and request handlerSource files in src/communication, src/connectionData structure: CSS_QUEUE_ENTRY, CSS_CONN_ENTRY, …Job QueueMaster thread monitors socket fds (select()), receives clients’ request packet, en-queues it, and then wake up a worker thread in the thread poolSource files in src/connectionData structure: CSS_JOB_ENTRY, …Critical SectionImplements semaphoreSource files in src/threadSystem ThreadsMaster threadDeadlock detection threadRun deadlock detection algorithm periodicallyResolves deadlock by aborting unilaterally victim transaction(s)Checkpoint threadExecutes periodic checkpoint actionOOB handling threadCatch OOB signalPage flush threadFlushes dirty data buffer pages to the volume files in the backgroundTo reduce burden of page replacementLog flush threadFlushes dirty log buffer pages to the log file in the backgroundImplements group commit and asynchronous commit featuresSource files in src/thread3.737/41 First Look Inside CUBRID
- 56. Module and Source StructureClient-Server Communication LayerSocket TransceiverSocket utility functionsTCP/IP, Unix domain socket, and named pipeSource files in src/connectionLinux and Windows portPacket HandlerSends requests and receives response between client and serverPacket types: request packet, data packet, close packet, out-of-band packet, error packet, abort packetPacket format: packet header + payloadQueuing request and data packetsSource files in src/connectionCommunication Interface LayerRPC stub style functionsImplements object serializationSource files in src/communication3.838/41 First Look Inside CUBRID
- 57. Module and Source StructureBroker ComponentBroker ProcessAccepts connection requests from the Connectors (JDBC, CCI, …)Allocates and/or assigns CAS process to the requestControls and monitors CAS processesHas four threadsmain: manages cas processreceiver_thread: does accept() and enqueues new jobdispatch_thread: dequeue job and assigns cas processcas_monitor_thread: monitors cas process to restartSource files in src/brokerCAS ProcessLinked with CUBRID client librarySo, run as a single threadReceives and processes requests from the application(Connectors)Connect to databasePrepare and execute queryFetch the resultCommit/rollback transactionGet/set parametersHas four statesIDLE: not connected with APBUSY: processing requestCLIENT_WAIT: connected and waiting for request; within a transactionCLOSE_WAIT: connected and waiting for request; out of a transactionSource files in src/broker3.939/41 First Look Inside CUBRID
- 58. In The Next CUBRID Inside440/41 First Look Inside CUBRID
- 59. In The Next CUBRID Inside4What topic do you want to see covered in the next CUBRID Inside?Suggesting Topics41/41 First Look Inside CUBRID
- 60. Question?