Apache Kafka and Blockchain - Comparison and a Kafka-native Implementation

Building a Secure, Tamper-Proof & Scalable
Blockchain on Top of Apache Kafka
Kai Waehner
Technology Evangelist
Confluent
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
confluent.io
kai-waehner.de
Stephen Reed
CEO, Co-founder
AiB
stephen@ai-blockchain.com
LinkedIn
ai-blockchain.com

kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda

Blockchain for Supply Chain Management
(Manufacturing, Logistics, …)
3
https://coletiv.com/blog/blockchain-supply-chain-management-system/
https://www.disruptordaily.com/blockchain-use-cases-supply-chain-management/

Blockchain in Healthcare
4
• Use Cases
• Management of patient data, especially
access control
• RVU Agreements with Ai-Blockchain
• Management and use of academic research
data
• Logistics / Supply chain for drugs, medical
technology, and other products
• Often no need for Blockchain /
DLT
• Only for Identity
Management
• Processing and (big) data storage outside
blockchain
https://digiforest.io/en/blog/blockchain-examples-in-healthcare

Blockchain for
Financial Services
5
https://www.accenture.com/t20170120T074124Z__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/Consulting/Accenture-Banking-on-Blockchain.pdf%23zoom=50

Bitcoin – The Tip of the Iceberg!
7
‘Bitcoin’ compared to
‘blockchain’, ‘distributed
ledger’, etc.

Blockchain vs. Distributed Ledger Technology (DLT)
8
https://www.da.ventures/post/gaining-clarity-on-key-terminology-bitcoin-versus-blockchain-versus-distributed-ledger-technology

Blockchain Concepts
• Permissionless or permissioned (or consortium)
• Consensus algorithm (Proof of work, Proof of Stake, or Voting systems)
• Blockchain: Global Consensus across all nodes
• DLT: Consensus without having to validate across the entire Blockchain
• Blockchain
• Growing list of records, called blocks, linked using cryptography
• Each block contains a cryptographic hash of the previous block, a timestamp, and transaction data
11

Challenges and Concerns of Blockchains
• Technical and organizational
complexity
• Transaction speed and
scalability
• Energy consumption
• Security
• Data tenancy
• Lifecycle costs
• Ease of integration

Purpose-built solutions
• Ripple: Payment settling, currency exchange and remittance system
• Libra: Global payment system and financial infrastructure
• IOTA: IoT transactions
Non-Blockchain DLTs
to solve the challenges to provide better performance and scalability
13
Open and flexible solution for many use cases
• Apache Kafka: Open, scalable, real time event
streaming
Technology Partners
• Ai-Blockchain: “AiB”

Event Streaming in the Finance Industry
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/

Tamper-Proof Event Streaming for
(some of the) Use Cases in the Finance Industry
Real Time Processing Digital Transformation Strategic Goals
Short-Sale Risk Calculation
/ Trade Approval
Mainframe Offloading
and Replacement
Instant Payment
Next-Best
Offer
Robot Process Automation
(e.g. Know Your Customer, KYC)
Customer Service
(e.g. Chat Bots)
IT
ModernizationRegulatory
Reporting
Fraud Detection
Anomaly Detection
Across Assets and Locations
Derivatives
Pricing Compliance
Trading Post-
Processing
Strategic
Planning and
Simulations

STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

Event Streaming Platform –
The Immutable Commit Log
Time
P
C1 C2
C3

Event Streaming Platform –
A Distributed System for 24/7 and Zero Data Loss
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
Microservices
Mainframes
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Stream processing apps
Connectors
Connectors
Stream processing apps

Kafka Connect
with Ethereum Connector
Kafka Cluster
Blockchain
Integration
Domain-Driven Design and Decoupled Applications
Legacy
Integration
Custom
Application
ETL Tool
with Mainframe Connector
Java / C++ /
Go / Python / KSQL
Schema
Registry
Event Streaming Platform
Blockchain Domain Payment Domain Fraud Domain
Audit Logs,
RBAC, etc.

Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
High Throughput
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
24

Kafka Clusters
can stretch over
Regions
Zero Downtime + Zero Data loss
(RPO=0 and RTO=0)
e.g. Stretched over US East + Mid +
West
Automate Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without
Custom Code
Multi-Region Cluster
(Only available in Confluent Platform)
26

Example of a Multi-Region Cluster in a Bank
Large FinServ Customer
Payment
Log
Payment
Log
Location Location
synchronous
asynchronous
● ‘Payment’ transactions enter
from us-east and us-west with
fully synchronous replication
● ‘Log’ and ‘Location’ information
in the same cluster use async -
optimized for latency
● Automated disaster recovery
(zero downtime, zero data loss)
Result: Clearing time from ‘deposit’ to
‘available’ goes from 5 days to 5 seconds
(including security checks)

Real-time
High Throughput
Decentralized database
Distributed log of records
Immutable log
Replication
High availability
Decoupling of applications / clients
Role-based access control to data
Tamper-Proof
Encrypted payloads
Deployment over independent organizations
is not a Blockchain!
X

Kafka AND Blockchain
32
Kafka
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)
Blockchain
(Bitcoin)
Blockchain
(Ethereum)
Kafka-native
Blockchain
Kafka AS Blockchain
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)

Kafka AND Blockchain:
Financial services platform
combining Kafka and Blockchains
• Provides the speed and convenience of
traditional exchanges and the security of
non-custodial approaches.
• Invest in, make payments with and trade
Bitcoin, Ethereum, NEO, and other digital
assets.
• The exchange is the first of its kind, offering
non-custodial cross-chain trading with the
full power of a real order book
• Deterministic replayability in its exact order
at any time
33
https://community.nash.io/t/first-of-its-kind-non-custodial-exchange-using-confluent-cloud/8254

Kafka AS Blockchain:
Consensus algorithm
in Hyperledger Fabric
using Kafka to be fault tolerant,
highly scalable and performant
34
https://hyperledger-fabric.readthedocs.io/en/release-2.0/kafka.html
https://www.meetup.com/Dubai-Kafka/events/268001616/ (presented by Farrukh Munir)

PRODUCTS
Built FIRST DocuSign Blockchain
product.
Allows for the writing of contracts on
the blockchain with proof of trust
Software agents keep the blockchain
tamper proof and immutable
Current POC’s across eCommerce,
Healthcare and Supply Chain
SOLUTIONS
• Artificial Intelligence, Blockchain, Ecommerce,
Healthcare, Media, Payments and Supply Chain
• Securely store commercial and non-commercial
transactions
• Energy efficient and ESG
GROWTH OPPORTUNITIES
How do we expand globally with new customers
using AiB products?
AiB offers services for scalability to meet all of the
demands of global businesses offering architecture
assessment diagrams to clients.
TECHNOLOGY
We employ one of a kind Artificial Agents on
the Blockchain to give world class security
and to ensure all data is immutable.
PLATFORM
Its most visible success is in the unique
approach to financial transactions through its
implementation of its Artificial Intelligent
Blockchain Cloud Payment Platform.
Enterprise
Grade
Platform
COMPANY
Since its founding in 2014, Ai-
Blockchain has built a
combination of innovative IP
and blockchain products
that have been recognized in
the marketplace and USPTO.
Potential
Customer
Use Cases
and
Problems
Ai-Blockchain
Products
Ai-Blockchain
Solutions and
advantages
Industry
Expansion
Ai-Blockchain – Patented Private-Permissioned Blockchain
for Smart Contracts

AiB’s BRICK Messaging Platform
For security, privacy and scalability, the AiB’s BRICK
platform is a platform agnostic layer that is built for
enterprise and micro payments in the cloud.
Third-Party Audited
AiB’s technology stack has been
security audited and stress tested by
third-party global cloud providers.
Full-Stack Capability
Deploying nodes and writing smart contracts
up to front end development and deployment.
Thought Leaders
Consistently asked to
speak at international
events to share our
Blockchain expertise.
Blockchain Experts
Established in 2014
Ai Blockchain’s team of over
100 Blockchain developers &
experts have been building
solutions for over six years
and are ready today for any
and all market
opportunities.
Ai-Blockchain – Proprietary Blockchain Technology Stack

39
KafkaBlockchain Architecture
Producers reuse the KafkaBlockchain
library and Tamper-Evident Records.
Consumers reuse the KafkaBlockchain library and Tamper-Evident Records.

KafkaBlockchain – Producer, Consumer & Verifier
Open Source Apache
Licensed on GitHub

41

Demonstration:
run-kafka-blockchain.demo.sh
• Demonstrate putting messages into a Kafka blockchain with one partition, one
producer and one consumer.
• Describe the single blockchain partition
• Demonstrate verifying messages with one consumer
Demonstration:
run-kafka-blockchain-multiple-partition.demo.sh
• Demonstrate putting messages into a Kafka blockchain with five partitions, one
producer and one consumer.
• Describe the five blockchain partitions
• Demonstrate verifying messages with five cooperating consumers - one for each
partition.
43
KafkaBlockchain – Live Demo
https://github.com/ai-coin/KafkaBlockchain

Next steps…

When to use
Kafka vs. a “real blockchain framework”?
45
Use Apache Kafka
with AIBlockchain for
• Enterprise infrastructure
• Open, scalable, real-time
requirements
• Flexible architectures for many
use cases
Use Hyperledger,
Ethereum, et al. for
• Deployment over various
independent organizations
• Participants verify the
distributed ledger contents
themselves.
• Specific use cases
• Server-side managed and
controlled by multiple
organizations
• Scenarios where the
business value overturns the
added complexity and
project risk
Use Kafka and Blockchain
together to combine the
benefits of both
(where this makes sense!)

What about long-term storage in Kafka?
46

Today, Kafka works well for
recent events, short
horizon storage, and
manual data balancing
Kafka’s present-day design offers
extraordinarily low messaging latency by
storing topic data on fast disks that are
collocated with brokers. This is usually good.
But sometimes, you need to store a huge
amount of data for a long time.
Blockchain is such a use case!
Kafka
Processing
App
Storage
Transactions, auth,
quota
enforcement,
compaction, ...

Confluent Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
Store Forever – AiB’s Tamper-Proof Blockchain
with KafkaBlockchain
Older data is offloaded to inexpensive object storage,
permitting it to be consumed at any time. Using AiB, storage
can be made tamper-proof and immutable
Save $$$
Storage limitations, like capacity and duration, are effectively
uncapped.
Instantaneously scale up and down
Your Kafka clusters will be able to automatically self-balance
load and hence elastically scale

Tamper-proof, encrypted event streaming
Time
P
C1
C2
Payment
Real-time data processing and historical analytics with one scalable infrastructure
Compliance check on historical data
Real-time
fraud detection

Questions? Feedback?
Let’s connect!
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
confluent.io
kai-waehner.de
Stephen Reed
CEO, Co-founder
AiB
stephen@ai-blockchain.com
LinkedIn
ai-blockchain.com

Apache Kafka and Blockchain - Comparison and a Kafka-native Implementation

More Related Content

What's hot (20)

Similar to Apache Kafka and Blockchain - Comparison and a Kafka-native Implementation (20)

More from Kai Wähner (20)

Recently uploaded (20)

Apache Kafka and Blockchain - Comparison and a Kafka-native Implementation