Practical Tools for Enterprise Uses of Hyperledger Fabric (Audit and System Operations)

© Hitachi America, Ltd. 2020. All rights reserved.
Practical Tools for Enterprise Uses of Hyperledger Fabric
(Audit and System Operations)
Taku Shimosawa, Tatsuya Sato
Hitachi America, Ltd.
https://github.com/shimos/bcverifier
Hyperledger Global Forum 2020

Agenda
• Blockchain Verifier: A tool to check the blockchain ledger
• Operations SC: smart contract-based system operations
for blockchain-based system

1. Blockchain Verifier

Motivation: Why do we need to verify?
• Requirement by regulations
– Auditing is often required in enterprise systems by either rules or laws
– Evidences that the blockchain is properly working and the data stored is
correct are essential in the systems.
• e.g. See “Ability to provide interfaces to audit” in JIRA (FAB-105)
• Local forge of the blocks/states
– Data stored in the blockchain are not altered, and states are consistent
with the blockchain
• Double check (another implementation) on smart contracts
– Ensures that certain properties (such as invariants) hold in the smart
contracts especially across multiple versions

Blockchain Verifier : Goals
• Verify
– To verify the integrity of a ledger
• Check
– To perform additional checks in the transaction stored in a ledger
• Report
– To extract evidences from a ledger to help an auditing report for the
blockchain-based system

Blockchain Verifier
Blockchain Verifier
Block
Block
Block
Block
Blockchain
(ledger)
# Hash PrevHash
0 0xabcdef01 --
1 0xc9a78563 0xabcdef01
2 0x112358d5 0xc9a78563
… … …
Verification items
• Verify hash values (Block)
• Verify signatures (Block, Transaction)
• …
Tx SC Func Input Output
1-1 Transfer(X, Y, 10) X = 200
Y = 120
X = 190
Y = 130
… …
Check items
• Check input versions
• Check invariant
• …
Blocks:
Transactions:
Verify
Check
OK: Blocks #0 to #99
OK: TXs #0-0 to #99-30
Verification Result
(evidence)
Report

What checks/verifications?
• Properties
– Immutability
• Data are not modified from those
that are obtained in consensus
– Consensus
• All the participants agree the
blocks and the transactions
– Consistency (Isolation)
• Transactions are serializable
– Correctness
• Smart contracts are correctly
executed.
• Level
– Platform
• Independent of applications
• Common checks to certain
platform
– Application
• Depending on applications/smart
contracts
• Application developers are
responsible

Check Items
Platform
(Hyperledger Fabric)
Application
Immutability Hash values (Block, Transaction)
Signatures (Block, Endorsement)
Hash comparison (w/other peer)
-
Consensus Signers (Block, Endorsement)
Policy check
Hash comparison (w/other peer)
(Re-execution of smart contract)
(Endorsers)
Consistency Versions of read-write set -
Correctness (Re-execution of smart contract) Application-specific checks (State,
Transaction)

Application Level Check
• Check logics to be written by application developers
• Examples
– Run another logic equivalent to the smart contract
– Check an invariant in the contract
• Such as Sum(In) == Sum(Out)
– Perform additional checks against external systems
• Blockchain Verifier provides API to access keys/transactions for the
check logic, and invoke the logic after platform-level checks.

Interface for Application-Level Checker
• To traverse transactions,
– Implement AppTransactionChecker interface
• tx.getInput()
– Returns the read set
• tx.getOutput()
– Returns the write set
• tx.getState()
– Returns the state at the time the transaction is executed
• To traverse keys,
– Implement AppStateChecker interface
• state.getKeys()
– Returns the keys available at the transaction tx
• state.getValue(key)
– Returns the value for the key
• keyValue.getHistory()
– Returns the history for the key

Trivial example for FabCar sample
• Functions in FabCar
– createCar
• Adds a car into the ledger
– changeCarOwner
• Changes the car owner
• Checker example
– createCar does not change any other keys than that for the new car
– changeCarOwner does not change any other keys than that for the
target car
– createCar overwrites the new key
• Actually, the chaincode does not check if the key already exists or not
Bad Example of chaincode calls:
createCar(CAR3, { Make: Honda, Owner: John });
createCar(CAR3, { Make: Toyota, Owner: Ann });

Blockchain Verifier (Concept)
Blockchain Verifier
Input (Network plugin)
Hyperledger Fabric Ledger File
Hyperledger Fabric Query
Hyperledger Indy
…
Ledger
Peer
Node
Block, Tx Data
Check Logic (Check plugin)
Hyperledger Fabric Block
Hyperledger Fabric Transaction
Hyperledger Indy Block
…
Application-Specific
Check Logic
Hyperledger Fabric
network
B. Application Check API
(A) Platform-Level Checks
Evidences
(JSON)
Implemented Future Work
Hyperledger
Indy
network

Blockchain Platforms
• Plugin-based design
– Allows support for multiple blockchain platforms
– As a start point, only Hyperledger Fabric plugins are available
• We highly appreciate contribution for plugins for other platforms!

Implementation: Preliminary version
• Blockchain Verifier (bcverifier) Ver. 0.2.0
– https://github.com/shimos/bcverifier
– Approved as one of the Hyperledger Labs (Feb 2020)
– Basic supports for
• Verification of Hyperledger Fabric blocks
– RW set for private data is also checked (if private data store is available)
• Verification of Hyperledger Fabric transactions
• Application-specific checkers
– License: Apache-2.0

Input (Hyperledger Fabric)
• Block Files (fabric-block)
– /var/hyperledger/production/ledgersData/chains/(channel name)
– Optional: Private collections (LevelDB)
• /var/hyperledger/production/ledgersData/pvtdataStore
• Network Profile + Cryptographic Artifacts (fabric-query)
– Host name of the peer, etc. in network profile
– Service discovery also may be utilized
– Private key and certificate to connect to a peer
– No private collections supported (no query API)

Example Output
$ node ./dist/cli.js start -n fabric-block -c ./blockfile_000000
Checked by fabric-block
Config: ./blockfile_000000
Blocks:
Block Range: Block 0 to Block 7
Checks performed: 52 (8 blocks)
Checks passed: 52 (8 blocks)
Checks failed: 0 (0 blocks)
Transactions:
Checks performed: 38 (8 transactions)
Checks passed: 38 (8 transactions)
Checks failed: 0 (0 transactions)
All checks finished successfully.

Open Discussions
• Report format
– How to report?
• Evidence format
– Should some formal language be used to describe what are checked?
– Similar to test framework (assert), but needs to clearly what are
compared (or performed) in some formal way.
• API
– Is this sufficient?

Summary
• Blockchain Verifier
– Verifies and checks blocks and transactions, and generates a report
– Is aimed at helping audits on blockchain-based systems
– Is open-source and now under development at
• We welcome your suggestions & contributions!

2. Operations SC: smart contract based system operations for
blockchain-based systems
(*) [1] https://ieeexplore.ieee.org/abstract/document/8328745
[2] https://arxiv.org/abs/1901.11249
[1] Smart-Contract Based System Operations for Permissioned Blockchain, BSC 2018, p.6
[2] Design and Evaluation of Smart-Contract-based System Operations for Permissioned
Blockchain-based Systems, arXiv:1901.11249, p.11, 2019

Background
• Toward production uses, system operations become more important
– e.g., Upgrading a SC and the applications, taking snapshot of ledger data
• Target: Blockchain-based system built across multiple management domains
Network
BizSC
Organization A
BizSC
Organization C
BizSC
Organization B
Consensus
DC1 DC2 DC3
Consensus
Org. A’s
Node
Org. C’s
Node
Org. B’s
Node
(*) BC: Blockchain, SC: Smart Contract, DC: Data Center,
BizSC: SC defining logics for business transactions
: need to collaborate with other organizations
• Problem: Difficult to execute inter-organizational system operations
e.g., Deploy the same SC in the same period
Each domain
has separated
admins
A B C
Admin

Problems about the system operations for BC-based systems
Conventional operation management tools (e.g., Job mgmt. servers, IaC tools):
- Enable admins to do general (= single-organizational) operations efficiently
But do not cover with inter-organizational operations
BizSC
Org A
BizSC
Org C
BizSC
Org B
CA B
Admin
Gap of operations
(procedures, triggers,
config parameters)
SC
v2.2
SC
v2.1Way 1:
A single admin
operates all nodes
Way 2:
Each org’s admin operates
their own nodes
(*) IaC: Infrastructure as Code
Problem 2:
Different configs may prevent the system
from working
Problem 1:
- The admin is SPOT (Single Point of Trust)
- Cannot access to nodes owned by other
orgs because of lack of permissions
Node
NO
Permission
How to do
inter-org ops

Smart contract based system operations method
To define system operational flow as a SC (referred to Operation SC / OpsSC)
- Cross-domain operations w/o SPOT and sharing credentials by BC consensus
- Unified procedures with unified config parameters based on SC
Network
Org A
BizSC
Org C
OpsSC
Org B
OpsSCOpsSC
0. Invoke TX
of OpsSC
2. Share parameters,
Control workflows
over SC
1. Establish
consensus Node
1. Establish
consensus
3. Execute
operations
based on SC
3. Execute
operations
Unified operations
(e.g. SC based
unified backup)
3.
BizSCBizSC
Snapshot()
- Cmds: [“zip /ledger…”…
- Timing: “every 23:55”
…
(*) SC: Smart Contract, TX: Transaction,
SPOT: Single Point of Trust

Demo: OpsSC for New Chaincode Lifecycle in Hyperledger Fabric v2.0
• Defined End-to-End Chaincode mgmt. processes (from proposal to deployment) as a OpsSC
• OpsSC manages operational workflow and issues SC events on the operation instructions
• OpsAgent for each org executes operations based on the events to ALL peers for the org
OpsSC for CC mgmt.
Portal Portal
Ops
Agent1
Org1’s
Peer
Org1 Org2 Org3
Admins
1. Propose new CC 2. Vote the proposal
Ops
Agent2
Org2’s
Peer
Ops
Agent3
Org3’s
Peer
Portal
3. Deploy the CC based on the OpsSC
- Download -> Install -> Approve -> Commit
Overview Portal Screen for OpsSC
Drop in at the Hitachi booth to see the demo !!

Related work and Our future work
• Fabric Interop Working Group (*):
– Purpose: To promote the interoperability of Fabric network service
• Focusing on a scenario that new organization joins a running Fabric network
– Approach: Create artifacts for the join request with
“Consortium Management Chaincode (CMCC)”
– The concept is very similar with ours although the scope is slightly different
– Our OpsSC could be positioned as a form or application of the CMCC
(*) Cf. https://wiki.hyperledger.org/display/fabric/Fabric+Interop+Working+Group
Related work
• Expand the proposed method implementations to support various operations
• Upstream them to the Hyperledger Fabric community
Our Future Work

Summary of the presentation
• Verifies and checks blocks and transactions, and generates a report
• Is aimed at helping audits on blockchain-based systems
• Is now one of the Hyperledger Labs and now under development at
• We welcome your suggestions & contributions!
1. Blockchain verifier
• Execute inter-organizational operations on blockchain-based systems
• Primary idea is to define operational flows as smart contract
• Chaincode mgmt. on Hyperledger Fabric v2.0 as an example of OpsSC
• We plan to expand and upstream them to the Fabric community
2. OpsSC: Smart contract based system operations

Details
• Input
– Ledger file of Hyperledger Fabric (Stored in /var/hyperledger in a peer container)
– Information necessary for query (private key, certificate, peer address etc.)
• Checks performed
– Block
• Hash for the data
• Hash for the header of the previous block
• Metadata signature (an orderer’s signature for the block)
• Last config index signature (an orderer’s signature for the index)
– Normal Transaction
• Proposer’s certificate, endorser’s certificate and signature
• Hash for private RW set, Hash for hashed RW set (if private data is available)
• Output
– JSON
• Whether the checks are successful or not and what is compared

Example Output : Checking a block (#2)
{
"number": 2,
"results": [
{
"result": "OK", “predicate": "EQ",
"operands": ["Block(2).PreviousHash", "HashForPrev(Block(1))" ]
},
{
"result": "OK", "predicate": "EQ",
"operands": ["Block(2).Hash", "HashForSelf(Block(2))"]
},
{
"result": "OK", "predicate": "LE",
"operands": ["Block(2).Metadata[1].LastConfig.Value","Block(2).Number"]
},
{
"result": "OK", "predicate": "INVOKE",
"operands": [ "VerifyMetadataSignature", "Block(2)",
"Block(2).Metadata[1].LastConfig", "Block(2).Metadata[1].Signature[0]" ]
}, ...
Checked for equality
“Previous Hash” field in the block Calculated hash value for the block 1
Block number

Private Collections
• If a data collection is available,
– bcverifier checks the hash value for the private RW set of transactions
– bcverifier also checks the hash values for the keys and the values in the
private RW set
• If the collection is not available, bcverifier just ignores the private
part of the transaction
Transaction (RWSet)
Hashed Key Hashed Value
559aea… ad5736…
df7e70... 27badc…
Side DBKey Value
A 100
B 200
RWSet Hash: 17f3d8…
Private collection

Now working
• Code testing
• More support for Hyperledger Fabric
– Endorsement Policy check
• Integration of the results of the application checker to the output

Appendix. Assumed BC-based system configuration in production phase
Network
BizSC
Ledger
Organization A
BizSC
Ledger
Organization C
BizSC
Ledger
Organization B
BC-based System
Consensus
DC1 DC2 DC3
Consensus
• Each organization owns the nodes
Org. A’s
Node
Org. C’s
Node
Org. B’s
Node
• These nodes are located in different management
domains (e.g., DC of each organization)
(*) BC: Blockchain, SC: Smart-contract,
BizSC: SC defining logics for business transactions, DC: Data Center
Each domain has separated admins

• System operations are tasks executed by admins to maintain the system
• Two types of system operations for BC-based systems
- Single-organizational: Operations closed to one organization
- Inter-organizational: Operations which need to collaborate with other
organizations (specialized for BC-based systems)
Appendix. System operations for BC-based systems
BizSC
Organization A
BizSC
Organization C
BizSC
Organization B
e.g., Modify UIs for each
org’s portal
e.g., Deploy the same SC in the same period
(*) SC: Smart-Contract

Appendix. Architecture Design for Fabric-based system - Background
• From Fabric v1.0, “execute-order-validate architecture” was introduced
– Only specific peers (=nodes) execute SCs (including OpsSCs)
– All peers receive the result of SC execution delivered via special nodes
named ordering nodes
TX
Invoke TX
BizSC BizSC BizSC
Ordering nodes
Execute SC
Update ledger using delivered result
NOT every node executes SCs
Difficult to realize our proposed
method straightforwardly

Appendix. Architecture Design for Fabric-based system – improvement of proposal
• Introduce OpsSC-event-listening agents (OpsAgents)
Ops.
Agent
Ops.
AgentOpsSC
BizSC
Ops
Agent
4. Execute ops. based on the event
3. Issue ops. instructions
as a event
1. Establish consensus
2. Manage SC exec. status
Snapshot()
- Cmds: “tar cfvz …”
・・・
[“tar cfvz …”]
OpsEvent
Event
Listener
> tar cfvz …
OpsSC issues SC events on operation instructions according
to SC execution status
OpsAgent for each org receives the events and executes operations
based on the events to ALL nodes owned by this org
“Custom event” function provided by Fabric support implementing the above

Increase operations which are executed by each org and must be used same parameters
Appendix. Example of OpsSC (for New Chaincode Lifecycle in Fabric v2.0)
New Chaincode Lifecycle introduced from Fabric v2.0
1. Install
1. Install
1. Install
2. Instantiate
V1.0:
2 steps
Org1:
Org2:
Org3:
CC source code CC definition (e.g., policy)
1. Install
1. Install
1. Install
2. Approve
V2.0:
3 steps
Org1:
Org2:
Org3: 3. Commit
2. Approve
2. ApproveCentralized !!
Check number of approvalsDecentralized !!
Prototyped a OpsSC for the Chaincode management processes
– Define End-to-End Chaincode mgmt. processes (from proposal to deployment)
as a OpsSC (implemented as a user Chaincode)
Eliminated centralized process in installing / upgrading Chaincode
Remaining issue
To improve
(*) Chaincode (CC) is SC in Fabric

Appendix. Sequence diagram of OpsSC for New Chaincode Lifecycle
Org1 Org2
OpsCC
Org1 Org2
OpsPortals OpsAgents
Org1 Org2
Peers
Org1’s
Admin
ProposeNewCC (Source, Definition)
… … …
VoteForProposal()
Org2’s
Admin
ProposalEvent
Check the
num of votes
DeployForOrgEvent
Download, Install, Approve the CC
SendResult()
DeployEvent
Check the num
of peer approvals
CommitCC
SendResult()

Appendix. Sequence diagram of OpsSC for New Chaincode Lifecycle
Org1 Org2
OpsCC
Org1 Org2
OpsPortals OpsAgents
Org1 Org2
Peers
Org1’s
Admin
ProposeNewCC (Source, Definition)
… … …
VoteForProposal()
Org2’s
Admin
ProposalEvent
Check the
num of votes
DeployForOrgEvent
SendResult()
DeployEvent
Check the num
of peer approvals
CommitCC
SendResult()
- Control operational flows on CC mgmt.
- Share and record status and parameters
Automated and unified deployment
based on the OpsCC
Manual proposal and
voting inter-orgs.

Appendix. Implementation variations of OpsSCs
General-purpose OpsSCSpecific-purpose OpsSC
SC and agent are implemented
ONLY for an individual specific ops.
Effective for usecases that:
- want to represent complicated
procedures and flows
- common, typical, standard ops.
Show an example of this type
in this presentation
Implemented as a template applicable to
various ops.
- Should predefine concreate ops. procedures
(e.g., command sequence) according the
template and store them on StateDB
- Prepare agents (with limited privileges for Fabric)
just execute ops. according to the command
sequence embedded in OpsSC events
Effective for usecases that:
- execute simple commands
- want to change ops. Flexibly
((*): Refer to the previous papers)
Switch
according
to tasks
and/or
phase

Practical Tools for Enterprise Uses of Hyperledger Fabric (Audit and System Operations)

More Related Content

What's hot (20)

Similar to Practical Tools for Enterprise Uses of Hyperledger Fabric (Audit and System Operations) (20)

More from LFDT Tokyo Meetup (20)

Recently uploaded (20)

Practical Tools for Enterprise Uses of Hyperledger Fabric (Audit and System Operations)