Parquet overview
Download as PPT, PDF5 likes7,539 views
Parquet is a column-oriented storage format for Hadoop that supports efficient compression and encoding techniques. It uses a row group structure to store data in columns in a compressed and encoded column chunk format. The schema and metadata are stored in the file footer to allow for efficient reads and scans of selected columns. The format is designed to be extensible through pluggable components for schema conversion, record materialization, and encodings.
Parquet overview
- 1. Parquet overview Julien Le Dem Twitter http://parquet.github.com
- 2. Format Schema definition: for binary representation Layout: currently PAX, supports one file per column when Hadoop allows block placement policy. Not java centric: encodings, compression codecs, etc are ENUMs, not java class names. i.e.: formally defined. Impala reads Parquet files. Footer: contains column chunks offsets 2
- 3. Format ⢠Row group: A group of rows in columnar format. ⢠Max size buffered in memory while writing. ⢠One (or more) per split while reading. ⢠roughly: 10MB < row group < 1 GB ⢠Column chunk: The data for one column in a row group. ⢠Column chunks can be read independently for efficient scans. ⢠Page: Unit of compression in a column chunk ⢠Should be big enough for compression to be efficient. ⢠Minimum size to read to access a single record (when index pages are available). ⢠roughly: 8KB < page < 100KB 3
- 4. Dremelâs shredding/assembly Schema: message Document { required int64 DocId; Columns: optional group Links { DocId repeated int64 Backward; Links.Backward repeated int64 Forward; } Links.Forward repeated group Name { Name.Language.Code repeated group Language { Name.Language.Country required string Code; Name.Url optional string Country; } optional string Url; }} Reference: http://research.google.com/pubs/pub36632.html ⢠Each cell is encoded as a triplet: repetition level, definition level, value. ⢠This allows reconstructing the nested records. ⢠Level values are bound by the depth of the schema: They are stored in a compact form. Example: Max repetition level Max definition level DocId 0 0 Links.Backward 1 2 Links.Forward 1 2 Name.Language.Code 2 2 Name.Language.Country 2 3 Name.Url 1 2 4
- 5. Abstractions ⢠Column layer: ⢠Iteration on triplets: repetition level, definition level, value. ⢠Repetition level = 0 indicates a new record. â˘When dictionary encoding and other compact encodings are implemented, can iterate over encoded or un-encoded values. ⢠Record layer: ⢠Iteration on fully assembled records. â˘Provides assembled records for any subset of the columns, so that only columns actually accessed are loaded. 5
- 6. Extensibility ⢠Schema conversion: ⢠Hadoop does not have a notion of schema. ⢠However Pig, Hive, Thrift, Avro, ProtoBufs, etc do. ⢠Record materialization: ⢠Pluggable record materialization layer. ⢠No double conversion. ⢠Sax-style Event base API. ⢠Encodings: ⢠Extensible encoding definitions. ⢠Planned: dictionary encoding, zigzag, rle, ... 6
- 7. Extensibility ⢠Schema conversion: ⢠Hadoop does not have a notion of schema. ⢠However Pig, Hive, Thrift, Avro, ProtoBufs, etc do. ⢠Record materialization: ⢠Pluggable record materialization layer. ⢠No double conversion. ⢠Sax-style Event base API. ⢠Encodings: ⢠Extensible encoding definitions. ⢠Planned: dictionary encoding, zigzag, rle, ... 6