Avro and its awesomeness
0 likes105 views
The document discusses the relationship between Avro topics, database table schemas, and metadata models, highlighting aspects like data safety, automatic compression, and cross-language portability. It explores the complexities of schema evolution, compatibility checks, and the allowed changes across different versions for producers and consumers. The content also emphasizes the learning curve associated with using Kafka and the tools involved in managing schemas.
Avro and its awesomeness
- 1. Avro
- 4. Topic <=> DB Table Schema <=> DB Column metadata Model
- 5. • Data safety • Data compressed automatically • Documentation • Cross language portability • Evolution • Many tools and moving parts • Learning curve • Difficult to debug • Complexity (comes with Kafka) + - Avro Schema
- 6. Demo
- 9. Compatibility Type Changes allowed Check against which schemas Upgrade first BACKWARD •Delete fields •Add optional fields Last version Consumers BACKWARD_TRANSITIVE •Delete fields •Add optional fields All previous versions Consumers FORWARD •Add fields •Delete optional fields Last version Producers FORWARD_TRANSITIVE •Add fields •Delete optional fields All previous versions Producers FULL •Add optional fields •Delete optional fields Last version Any order FULL_TRANSITIVE •Add optional fields •Delete optional fields All previous versions Any order NONE •All changes are accepted Compatibility checking disabled Depends
- 13. C1 P1 Schema-Forward(Fixed by Backward) C2 C2 V1 V2
- 21. • Java-Avro codegen • Reflect • And why reflect?