apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apache Kafka by Hugo Guerrero, Red Hat
0 likes511 views
The document discusses event-driven APIs and their implementation using Apache Kafka, covering challenges such as integration issues and the importance of contract-first development. It highlights AsyncAPI as a tool for defining event-driven APIs and emphasizes the need for a schema registry for effective management of data schemas in a Kafka environment. Additionally, the document outlines registry capabilities and provides resources for implementation and management within cloud environments.
apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apache Kafka by Hugo Guerrero, Red Hat
- 1. August 26, 2021 Event-driven APIs & Apache Kafka Hugo Guerrero APIs & Messaging Developer Advocate Red Hat
- 2. Hugo Guerrero Mexican @ MA, USA APIs & Event-Driven Specialist | Open Source Advocate | History, Travel & Food Enthusiast @hguerreroo apidays Hong Kong Agenda Event-driven APIs Event-driven development Kafkaesque Challenges Contract-First The REST of the World AsyncAPI Spec Event Schemas Schemas as Contracts Registry for Schemas
- 3. Event-driven APIs & Modern Applications apidays Hong Kong
- 4. What is the situation? EVENT-DRIVEN APIS & APACHE KAFKA 4 @hguerreroo Distributed Decoupled Highly Connected apidays Hong Kong
- 5. What is the situation? EVENT-DRIVEN APIS & APACHE KAFKA 5 @hguerreroo Distributed Decoupled Highly Connected Microservices Request / Response Events apidays Hong Kong
- 6. What is the situation? EVENT-DRIVEN APIS & APACHE KAFKA 6 @hguerreroo Distributed Decoupled Highly Connected Microservices Request / Response Events apidays Hong Kong Events
- 7. The Traditional Workflow EVENT-DRIVEN APIS & APACHE KAFKA 7 Project Planning The business experts spend time writing up a project plan for a series of services and potentially a user interface Send to Developers The various development teams attempt to interpret the project plan and implement their code so that they are compatible Iterate and Integrate The developer teams iteratively try to resolve integration issues and spend loads of time ensuring compatibility apidays Hong Kong
- 8. Kafkaesque EVENT-DRIVEN APIS & APACHE KAFKA 8 Photo credit: dbeck03 on VisualHunt ENDPOINT DOCUMENTATION ? DATA FORMAT AND VALIDATION ? MIS-UNDERSTANDING
- 9. Kafkaesque EVENT-DRIVEN APIS & APACHE KAFKA 9 Photo credit: dbeck03 on VisualHunt INFRASTRUCTURE ACCESS ? PUBLISHERS & CONSUMERS COMPLIANCE ? EVENTS AVAILABILITY ? SLOW TIME TO MARKET, POOR QUALITY
- 10. Event-Driven APIs and Contract-First Workflows apidays Hong Kong
- 11. EVENT-DRIVEN APIS & APACHE KAFKA 11 Why Contract-First? @hguerreroo ▸ Because you want to allow people to work independently ▸ Because you want to ensure consistency ▸ Because you need strong guarantees about service contracts ▸ Because you, your team, your customers, and your partners can collaborate ▸ Because you can save time by using code generators and testing tooling apidays Hong Kong
- 12. How does it work on REST APIs? EVENT-DRIVEN APIS & APACHE KAFKA 12 Source: https://swagger.io/docs/specification/about/ https://www.apicur.io/ https://openapi-generator.tech/ https://microcks.io/ Build an API Specification Using a tool like OpenAPI, write an API specification FIRST Publish the API Specification Using a tool like Swagger or Apicur.io, publish the API specification where others have access and can collaborate Generate Code, Mocks and Testing Using a tool like OpenAPI Generator to create the API stubs for both client and server applications. You can also generate “mock” services with Microcks. apidays Hong Kong
- 13. Event-driven Secret Weapon EVENT-DRIVEN APIS & APACHE KAFKA 13 An open source initiative that seeks to improve the current state of Event-Driven Architectures. Created to describe Event-driven APIs using MQTT, AMQP, Apache Kafka, ... “Sister” specification to OpenAPI: same base syntax, use JSON Schema underhood. Allows definition of examples AsyncAPI https://www.asyncapi.com/ apidays Hong Kong
- 14. AsyncAPI EVENT-DRIVEN APIS & APACHE KAFKA 14 asyncapi.org apidays Hong Kong Protocol agnostic: ▸ AMQP ▸ MQTT ▸ NATS ▸ HTTP ▸ WebSockets ▸ Kafka ▸ STOMP ▸ JMS Open Source Tooling: ▸ Online playground (playground.asyncapi.org) ▸ Code generators ▸ Documentation generators ▸ Editors ▸ Mocking ▸ Testing
- 15. Unified API contracts EVENT-DRIVEN APIS & APACHE KAFKA 15 @hguerreroo POST /register { "fullName":"John Doe", "email":"john@example.io", "age":36 } TOPIC user-registration { "id":"38102178-dbe5-4e81-aa65-8f6df1be4d44", "fullName":"John Doe", "email":"john@example.io", "age":36, "sendAt":"1603786974956", } OpenAPI 3 Contract AsyncAPI 2 Contract apidays Hong Kong
- 16. AsyncAPI Anatomy EVENT-DRIVEN APIS & APACHE KAFKA 16 https://www.asyncapi.com/docs/getting-started/coming-from-openapi Info: metadata and documentation for this asynchronous API: version, title, description, contact and licensing terms Servers: list of brokers with binding details (URLs and security protocols) Channel Item: destination that will be used by the API to publish or consume events. Destination may be mapped to a topic, queue, address name depending on the protocol binding. Operation: whether the application using the AsyncAPI should subscribe or produce messages. Holds its own descriptive metadata Message: the content-type and structure of message headers and payload. May reference schemas of different types (AsyncAPI, JSON, Avro). May contain examples! Components: Common definitions (schemas, traits, messages) like in OpenAPI v3 apidays Hong Kong
- 17. AsyncAPI Anatomy EVENT-DRIVEN APIS & APACHE KAFKA 17 https://www.asyncapi.com/docs/getting-started/coming-from-openapi Info: metadata and documentation for this asynchronous API: version, title, description, contact and licensing terms Servers: list of brokers with binding details (URLs and security protocols) Channel Item: destination that will be used by the API to publish or consume events. Destination may be mapped to a topic, queue, address name depending on the protocol binding. Operation: whether the application using the AsyncAPI should subscribe or produce messages. Holds its own descriptive metadata Message: the content-type and structure of message headers and payload. May reference schemas of different types (AsyncAPI, JSON, Avro). May include examples! Components: Common definitions (schemas, traits, messages) like in OpenAPI v3 apidays Hong Kong
- 19. EVENT-DRIVEN APIS & APACHE KAFKA 19 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | | | | | | | | | | | | | | | | Apache Kafka Send Data Retrieve Data Consumer apidays Hong Kong @hguerreroo Contract-First API Development
- 20. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 20 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | | | | | | | | | | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer apidays Hong Kong @hguerreroo
- 21. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 21 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | | | | | | | | | | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer v2 v2 v2 v1 apidays Hong Kong @hguerreroo
- 22. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 22 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | | | | | | | | | | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer v2 v2 v2 v1 apidays Hong Kong @hguerreroo
- 23. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 23 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | | | | | | | | | | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer v2 v2 v2 v1 apidays Hong Kong @hguerreroo
- 24. What issues needs to be addressed in a Kafka world? ▸ Kafka is not aware of data types or schema formats. ▸ API schemas are subject to change. ▸ Central registry where the data schemas are stored and accessible. EVENT-DRIVEN APIS & APACHE KAFKA 24 Photo credit: Harald Groven on VisualHunt
- 25. A Registry for Event Schemas apidays Hong Kong
- 26. Using the Registry for Schema Management EVENT-DRIVEN APIS & APACHE KAFKA 26 Producer Consumer Deserializer Serializer Registry | | | | | | | | | | | | | | | | | | | | | Apache Kafka Get or Register Schema by Id Retrieve Schema by Id Topic B (JSON ) Topic C (Protobuf) Topic A (Avro) Send Serialized Data Retrieve Serialized Data apidays Hong Kong @hguerreroo
- 27. EVENT-DRIVEN APIS & APACHE KAFKA What are the required capabilities of the registry? 27 The registry needs to address the following 3 pillars for a successful management @hguerreroo You need to be able to manage all the different types of artifacts, be able to browse, download and document in single place. Artifact Management Support for different types of data structures, like Apache Avro, Google Protocol Buffers, or JSON Schemas. Data Formats Artifacts follow content evolution and are controlled by enabling content rules, including Validity and Compatibility. Versioning apidays Hong Kong
- 28. EVENT-DRIVEN APIS & APACHE KAFKA Registry 28 Use Cases Source: https://github.com/Apicurio/apicurio-registry API specification registry for API consumers Shared Data Types API Designs Shared data types (schemas) across API and event driven architectures Schema Registry Schema registry for Kafka serializers/deserializers apidays Hong Kong
- 29. ▸ Implementation preview of the CNCF Schema Registry API. ▸ Part of the CloudEvents Working Group efforts for standardization and interoperability. EVENT-DRIVEN APIS & APACHE KAFKA CNCF’s Schema Registry API 29 CNCF Schema Registry API support Source: https://github.com/cloudevents/spec/blob/master/schemaregistry/schemaregistry.md apidays Hong Kong
- 30. ▸ Getting Started ● https://developers.redhat.com/blog/2019/12/16/getting-started-with-red-hat-integration-service-registry ● https://developers.redhat.com/blog/2020/12/09/new-features-and-storage-options-in-red-hat-integration-service-registry-1 -1-ga ▸ Install and configure Service Registry for AMQ Streams ● Installing Service Registry from the OpenShift OperatorHub ● Configuring Service Registry with AMQ Streams storage on OpenShift ▸ Validating Schemas using Kafka Client Serializer/Deserializer ● Validating schemas using Kafka client serializers/deserializers ● Strategies to look up a schema ▸ Service Registry Java Client ● Managing Service Registry content using the Java client EVENT-DRIVEN APIS & APACHE KAFKA Useful Bookmarks 30 apidays Hong Kong @hguerreroo