"Image Sensor Formats and Interfaces for IoT Applications," a Presentation from Sony

Copyright © 2017 Sony Corporation 1
Tatsuya Sugioka
May 2017
Image Sensor Formats and Interfaces
for IoT Applications

1. Introduction
• Image sensor technology evolution
2. Image sensor output formats introduction
• Crucial formats for embedded vision systems
3. Image sensor output interface choices
• Interface technology landscape and comparison
Agenda

Sony’s Image Sensors Have Been Evolving for Mobile
2 0 0 8
2 0 1 2
2 0 1 6
2 0 2 0
Pixel
AD
Su p er Slo w
1 K fp s
BI
Stack
…
Du al Cam era
M u lti-fram e
p rocessin g
Clear N ig h t View
I m ag in atio n !
Rich Function
PDAF/HDR/EI S
H ig h sp eed
Low Pow er
High
Sensitivity
Zoom
I n n ova tive Tech n olog y ( H istory & Tren d )
An y Scen e!
Sem i G.S 2 4 0 fp s

A Different Dimension of Evolution: IoT Sensors
Sensor business explosion
…not physical explosion;)
sensor
controller
Artificial
Intelligence
IoT Host / Cloud
M ob ile / p h on e
Trillion
IoT
Sensors
~2035

History of Image Sensor Architecture Evolution
Fro n t-I llu m in ated
Back -illu m in ated Stack ed
Stru ctu re
Circu it
CCD
ch arg e tran sfer
CM OS
co lu m n QV
pixel array pixel array
pixel array
ADCCDSCDS
ADC
CM OS
co lu m n -p arallel ADCs
Integration
・Massively parallel ADCs
・Function extension
EIS/HDR/ROI
ADCs
CDS
CDS: Correlated Double Sampling

I m a g e sen so r o u tp u t fo rm a ts a n d in terfa ces fo r I o T a p p lica tio n s
CMOS
Image Sensor
DSP / ISP / ADAS…
Video Timing Clock domain Output Clock domain
Video
Timing
Divider
Output
Divider
Asynch
FIFO
Pixel
Array
Image
pipeline
Shutter/Read-out controller
Timing Handler
ADC
MIPI
Tx logic
MIPI
PHY
ADC
SCL
Asynchrono
used to ad
associ
Form atter I n terfaceexample of
Sensorblock diagram
Image
pipeline
FIFO
MIPI
CSI-2
MIPI
PHYs
Timing controller
Today, my talk will cover…
First topic

ROI : Region of Interest
EIS : Electronic Image Stabilization
Sensoroutput image
ROI area(2 spots)setting image
Subject
In-Focus
Out-Focus
PSF（ Point Spread Function）
Long exposure
Short exposure
HDR image
PD : Phase Detection
(Auto Focus)
HDR : High Dynamic Range
Processing
unit 1
Pre-process
Processing
Unit 2
Gy ro
I /F
Im ag e Sensor
EIS block
Shaking Stabilized
Sensor Output Formats Enable Various Features

Sensor
Gyro
IC
Sensor
Gyro
IC
ISP
System without Gyro data insertion function
ISP
Frame data
Gyro data:
System with Gyro data insertion function
Frame data
with Gyro data
ISP need to
connect each pin
Frame data
Output data image
Gyro data
Frame data
Gyro data
…
Gyro data
Gyro data has sampling timing information
(H count)
H count
N-1th Gyro data
H count
N …H count
N+1
Example Format for EIS: Gyro Data
Insertion Function
Makes your system simpler!

Sensor
Gyro
IC
ISP
Image data
with Additional data
Frame data
Additional data
Frame data
…
Lineblanking
PH
Pixel data
FS
PF
FE
PH Temperature/Gyro/Phase Difference data PF
Temperature
sensor
or
or
Phase
difference
calculation
Additional data
Different
Data Type
Features Beyond EIS Realized Via Data
Insertion Function
Many more features enabled!
Example : MIPI data insertion

Long exposure frame Short exposure frame HDR image
×over-exposure ×noise in dark
◯ Perfect
combined
Format for HDR: Example Image

Example Format for HDR with MIPI Virtual
Channel (1)
Blanking
Lineblanking
PH
Blanking
Long
Exposure
Frame
Short
Exposure
Frame
FS
PF
PH PF
FE
Frame blanking
FS
FE
Frame Start/End is generated for
each frame
Virtual Channel is used for
recognitionof Long and Short
VC DT WC ECC
0 2B XXXX XX
Example:RAW10
VC DT WC ECC
1 2B XXXX XX
Example:RAW10
VC DT WC ECC
0 00 XXXX XX
VC DT WC ECC
1 00 XXXX XX
VC DT WC ECC
0 01 XXXX XX
VC DT WC ECC
1 01 XXXX XX

Example Format for HDR with MIPI
Virtual Channel (2)
PF
PH
(Visible)
FE
PH
(Visible)
FE
PF
FS
PF
PH
(Visible)
FE
FS
PH
(Visible)
FE
PF
VSync
FS
Long
Exposure
Frame
(Nth)
FS
Long
Exposure
Frame
(N+1th)
Short
Exposure
Frame
(Nth)
Short
Exposure
Frame
(N+1th)
If Data of Long and Short are overlapped,
Receiver can recognize by using Virtual
Channel.
Readout timing gap
because of
Exposure time
Overlap
Flexibility of Exposure time control

World of ROI (Region of Interest)  IoT Applications
ROI
Fa ctory a u tom a tion
： p rod u cts recog n ition
ROI
- face
- n u m b er
( licen se p late)
W ea ra b le ： p olice-ca m
Su rveilla n ce ： tota l secu rity solu tionGa m e： U I sen sin g
ROI
V2 V
( lo w BW )
Au tom otive： V2 V, V2 X
Con cep t : To d eliver on ly th e d a ta th a t you n eed w h en it’s n eed ed .
Ben efits : red u ces p ow er con su m p tion / en h a n ces p rocessin g sp eed / solves b a n d w id th
lim ita tion / sa ve d a ta stora g e sp a ce

Total System Performance Improvement
The sensor ROI improves total system latency in case of small ROI window
under the same backend processing capability.
Sen sor I SP
M em o ry
Face
D etect
Pro cessor
V
I
N
V
O
T
Camera process
Po st
Down scale & Reformat
ROI extraction : Sensor vs Processor
ROI function
ON / OFF
0
0.2
0.4
0.6
0.8
1
1.2
320x240 640x480 1280x960
Latency[a.u.]
ROI w in d ow size [p ix . ]
System La ten cy Com p a rison
SensorROI OFF
SensorROI ON
Normalized
Sensor full frame:
5120x3840 pix.

USB3.0
Encode
ROI
Packet Decode
Input
image
Output
imageTx Rx
Low band transmission
channel
ROI
information
Image Sensor
face
detect
All
DataTx Rx
Full Pixel Image
ROI ImageROI Encode ROI Decode on / off
“Smart ROI” Proof of Concept Demonstration

Full Pixel Image vs ROI Image Demo Videos
Full Pixel Image Transmission ROI Image Transmission
Encode
ROI
Packet Decode
Input
image
Output
imageTx Rx
face
detect
All
DataTx Rx
Full Pixel Image
Data rate reduction
with frame rate increase!
Frame rate
bypass
Data rate
Frame rate
Data rate

System Performance Improvement Demo Videos
Full Pixel Image Transmission + Face Detection ROI Image Transmission + Face Detection
Encode
ROI
Packet Decode
Input
image
Output
imageTx Rx
face
detect
All
DataTx Rx
Full Pixel Image
enable
ROI helps face-detection
processing and increases
frame rate!
Frame rate
Data rate
Frame rate
Data rate

Automotive Use-case: Long Channel
Limit Breakthrough
 Required resolution estimates: 75mm/pixel case
- 50m distance recognition: FOV 120deg. 1280H  2Mpix~
- 200m distance recognition: FOV 100deg. 4000H  8Mpix~
 15m long reach camera: 1Gbps/Lane limited
- NO ROI : 2Mp/16bit/60fps  50m recognition range [limited]
- Smart ROI : 8Mp(effective)/16bit/60fps  200m range enabled
blinker
Transmissionchannellength
Achievable
Datarate
5Gbps @ 3m
1Gbps @ 10m
Front
sense
Side
sense
rear
sense
Source:
mipi alliance

ROI common platform accelerates more efficient systems
ROI parameters
# of windows
Window size
Window shape
Window overlap
Resolution etc…
Different types of image sensors
Multiple input ports
Smart ROI Smart ROI Smart ROI
Smart ROI ready ISP
Common framework is needed
ROI Format Standardization Initiated
in MIPI Alliance
Source:
MIPI Alliance

I m a g e sen so r o u tp u t fo rm a ts a n d in terfa ces fo r I o T a p p lica tio n s
CMOS
Image Sensor
DSP / ISP / ADAS…
Video Timing Clock domain Output Clock domain
Video
Timing
Divider
Output
Divider
Asynch
FIFO
Pixel
Array
Image
pipeline
Shutter/Read-out controller
Timing Handler
ADC
MIPI
Tx logic
MIPI
PHY
ADC
SCL
Asynchrono
used to adj
associa
Form atterexample of
Sensorblock diagram
Image
pipeline
FIFO
MIPI
CSI-2
MIPI
PHYs
Timing controller
I n terface
Today, my talk will cover…
Second topic

Image Sensor Output Interface Comparison
SONY
SLVS-EC
Sub-LVDS MIPI (CSI-2)
D-PHY v1.2
MIPI(CSI-3)
MPHY
MIPI (CSI-2)
C-PHY v1.0
Data rate
(Actual rate)
2.304Gbps
(1.843Gbps)
0.576Gbps
(0.576Gbps)
Up to 2.5Gbps 1.5G, 3G, 6Gbps
(1.2G, 2.4G,4.8Gbps)
Up to 2.5Gs/s
(5.75Gbps @3wire)
Number of Lane 8 10 or more 2, 4, 8 4 3
Error correction FEC (RS) None None Resend/ARQ(CRC) None
Line coding 8b10b None None 8b10b 16 to 7 Mapper +
differential ENDEC
Clocking scheme EmbeddedClock DDR Source
Synchronous Clock
DDR Source
Synchronous Clock
EmbeddedClock EmbeddedClock
Equal length
betweenLanes
Not necessary Necessary
(Data-Clock Skew)
Calibration required
(RX Skew calibration)
Not necessary Not necessary
Supposed RX ASIC,FPGA ASIC,FPGA ASIC,FPGA ASIC,FPGA FPGA?
 Researchedby Sony corporation
 As of Dec.27th
2016

Interface Output Driver Topology Comparison
1 0 0 Mbps 1 Gbps
1 mW
1 0 mW
1 0 0 mW
Power(perpin)
1 0 Gbps
DDR
LVDS
CMOS
DDR2
DDR3
Speed ( per pin)
CMOS
CMOS
Terminated
SLVS
High Speed & Low Power
for consumer product
High Speed & High Power
CML
SATA/PCIe/HDMI
CML/LVDS Style Trend : SLVS topology
1.8V
0V
0.15V
0.825V
0.825VVIN VINB
VOUT
VOUTB
0.4V
0V
0.1V
VIN VINB
VOUT
VOUTB
0.2V
0.1V
0.4V
3mA
2mA
3mA
3mA
3mA
2mA
=
=
電流
電圧
電圧
電流
の面積：送信電力
の面積：無効電力
①
①
②③
④
② ③
④
⑤
⑤
⑥
⑥
⑦
⑦
1.8V
Efficiency 4.17% Efficiency 50%
SLVS topologyLVDS topology
Source term.
Tr. SW
Tr. SW w/ 50ohm term.
1.8V
0.9V
0
VDD
0.3V
0
0.4V
Vo lt Tran sm itted Pow er
I n efficien t Pow er
cu r r en tcu r r en t
Vo lt
◆ Power consumption is crucial for Image sensor characteristics
 VML(Voltage Mode Logic) / SLVS(Scalable Low Voltage Signaling)

Interface Clocking Scheme Comparison
◆ Source-synchronous vs. embedded-clock (MIPI example)
0
1
2
3
4
5
6
7
8
9
D-PHY(IMX237) C-PHY(PANAMA)
消費電力(mW/Gbps)
D
PR
PO
SE
D
CL
RE
×0.66
D-PHY mode C-PHY mode
Power
Data Lane0
Data Lane1
Data Lane2
Data Lane3
Clock Lane
Camera Module Processor
D-PHY
Camera Module Processor
Trio 0
Trio 1
Trio 2
C-PHY
Embeddedclock by unique encodingDedicated clock lane required
for source-synchronous

What Is the Best Interface for IoT Sensors?
◆ MIPI alliance starts standardizing interface for IoT applications
1
10
100
1000
10000
0 10 20 30 40 50 601M 10M 100M 1G 10G
Total bandwidth [bps]
ChannelLength[cm]
Control/
Backbone
network
MIPI
CAN,FlexRay,etc.
Viewing Sensing
CMOS
Ethernet
SERDES
(LVDS) MI PI
Lon g -rea ch
Ta rg et?
Landscape of automotive (Camera) interface standards
Source : MIPI alliance

Which PHY Scheme Wins for IoT Sensor Interface
Standard? ... Stay Tuned!
MIPI case Assu m ed to p o lo g y D evelo p m en t item s Ch allen g es
D -PH Y
sch em e
・TX/Em phasis
・RX EQ/skew Cal for Lanes
・pow er line transm ission?
・?
・low er pow er/EMI issue?
・skew calibration for long
reach
・?
C-PH Y
sch em e
・TX/Em phasis
・RX EQ/skew Cal for w ires
・?
・skew calibration for long
reach
・new coding schem e?
・?
M -PH Y
sch em e
・TX/Em phasis
・RX EQ
・?
・new protocol for MI PI ?
・?
REG
Power
LPLP
Power
HS-TX
Emp
HS-TX
Emp
RT
HS-RX
EQ
RT
HS-RX
EQ
φ
LPLP
REG
Power
RT
HS-TX
Emp
HS-RX
EQ
Power
RT
HS-TX
Emp
HS-RX
EQ
REG
PowerPower
HS-TX
Emp
RT
HS-RX
EQ
HS-TX
Emp
RT
HS-RX
EQ
～5m?
～5m?
Power line
～5m?
Power line
Power line
CK
Data

1. More efficient embedded vision systems can be realized via evolving
image sensor output formats
2. Dramatic extension of the range of imaging applications in IoT will be
driven by the emerging format, “Smart ROI”
3. The right choice of image sensor output interface makes your system
much more competitive. Standardization for IoT has been initiated to
meet embedded vision requirements.
Key Take-aways from My Talk

• Open discussion would be appreciated
Thank You

"Image Sensor Formats and Interfaces for IoT Applications," a Presentation from Sony

More Related Content

What's hot (20)

Similar to "Image Sensor Formats and Interfaces for IoT Applications," a Presentation from Sony (20)

More from Edge AI and Vision Alliance (20)

Recently uploaded (20)

"Image Sensor Formats and Interfaces for IoT Applications," a Presentation from Sony