HBase System Tables / Metadata Info
- 1. HBase System Tables / Metadata Info Wellington Chevreuil
- 2. Overview ● General metadata files ● Table level metadata files ● Meta Table ● Namespace Table ● ACLs Table ● Snapshot metadata files ● Hands-on labs
- 3. General Metadata Files ● Simple files defining hbase unique identification and version, respectively ○ /hbase/hbase.id ■ Created once the cluster is being set up, or when Master is restarted and no id file is found ○ /hbase/hbase.version ■ Created only upon cluster install or upgrade ● Written in protobuff format ○ For implementation details: org.apache.hadoop.hbase.ClusterId, org.apache.hadoop.hbase.util.FSUtils ● Most issues will be related with corruption ● Duplicate hbase.id may prevent Replication from start (often occurs if the whole "/hbase" is copied between clusters when bootstrapping backup cluster)
- 4. Table Level Metadata Files ● /hbase/data/default/test-1/.tabledesc/.tableinfo.0000000001 ○ This is where several table structure information such as CFs, coprocessors, encoding, etc, are stored. ○ This is same info shown by hbase shell describe command. ● /hbase/data/default/test-1/40f1ab8c352a23f54b343de5e3125f4e/.regioninfo ○ Keeps info such as region name, start and end keys ● /hbase/data/default/test-1/40f1ab8c352a23f54b343de5e3125f4e/recovered.edits/1 27.seqid ○ Empty file, named with the region initialisation sequence id ● Most issues related to HDFS corruption ○ HBCK provides -fixHdfsOrphans and -fixTableOrphans to regenerate those files. More info on the HBCK session
- 5. HBase Meta ● Records regions assignments; ● First table to be onlined upon initialisation (HBase fails if it can't bring it online); ● Is queried by clients when these try to access specific rowkeys for the 1st time; ○ Once the specific region assignment is known, client keeps a cache ● Has same hdfs structure from user tables: It only has a single region, with a different naming convention, then user tables.
- 6. HBase Meta ● It's normally updated by Master's AssignmentManager class (more details on HBase RIT session) ● Reflects current regions assignments state: ● How many regions are online for this test-1 table?
- 7. Namespace Table ● Records available namespaces: ● Must be online, so that hbase is operational. ● Has same hdfs structure from user tables:
- 8. Namespace Table ● Most common issue involving namespace table is HBASE-14190: ○ Currently, namespace table does not have assignment priority over user tables ○ On cluster with too many regions, namespace table can get relegated for more than 5 minutes to get online ○ Causes Master startup to crash, due to timeout determined by hbase.master.namespace.init.timeout (default 5 minutes) ● Other problems would be caused by any sort of table hdfs file corruption (much rare)
- 9. ACLs Table ● Table where HBase internal ACLs are stored. ● Only created once hbase.security.authorization is set. ● Queried by hbase shell user_permission command, or scanning it directly: ● ACLs are also stored on ZK under /hbase/acl znode. When updating ACLs, HBase will also try to edit this znode.
- 10. Snapshot metadata files ● Created under /hbase/.hbase-snapshot/SNAPSHOT_NAME hdfs dir. Example: ○ /hbase/.hbase-snapshot/snap-1/.snapshotinfo ○ /hbase/.hbase-snapshot/snap-1/data.manifest ● Store snapshots details such as table schema, hfiles containing data for the time snapshot has been taken and general snapshot information. ● These files are also serialised within protobuff. Contents can be viewed with snapshotinfo command:
- 11. HBase Meta - Possible issues ● HDFS files corruption ○ Would require meta reconstruction, HBCK/HBCK2 would be useful ● Meta not consistent with hdfs table/regions structure ○ Usually due uncompleted or broken region splits; ● Region not assigned to any RS ○ Caused by some failed previous region assignment