![2 Y. Liu et al.
sensing, storage, and computation onto focal planes with novel circuit designs or new
materials. This chapter aims to describe the proposed image processing algorithms
and neural networks of in-sensor computing, as well as their applications in machine
vision and robotics. The goal of this chapter is to help developers, researchers, and
users of unconventional visual sensors understand their functioning and applications,
especially in the context of autonomous driving.
1.1 Introduction
The importance of vision as a means of perception cannot be overstated, as it enables
efficient collection and interpretation of information [1]. To apply this capability to
fields like machine vision, robotics, Internet of Things (IoT), and artificial intelli-
gence (AI), there is a pressing need to develop visual information processing methods
and technologies that operate at ultra-high speeds while consuming minimal energy
[2, 3]. The conventional machine vision systems and their associated technologies
face major constraints in terms of system latency, power consumption, and privacy
issues [2, 3]. Unlike the mammalian retina mechanism that rapidly processes raw
signals through several layers of cells, the visual signal digitization, storage, and
transmission processes involved in conventional machine vision systems can intro-
duce significant time latency, which hinders quick responses to dynamic changes and
results in inefficiencies due to irrelevant data processing [2, 3]. Additionally, exter-
nal image processors like CPU/GPU/VPU/DSPs consume high amounts of power,
which is unfavorable for portable tasks [2, 3]. The overwhelming amount of data gen-
erated by ubiquitous sensors may obscure the useful information, thus necessitating
the extraction of critical information by terminal sensors to reduce data movement
from the sensing chip to processing units [4, 5]. Moreover, privacy-sensitive scenar-
ios may require the extraction of crucial information from raw analog signals rather
than collected images.
To address these challenges, a paradigm shift towards in-sensor computing is pro-
posed [6]. This approach is inspired by the mammalian retina (Fig.1.1a) and involves
the vision sensor not only acquiring visual information but also processing it to pro-
duce highly compressed information instead of video frames (Fig.1.1c). In-sensor
computing offers image-free visual signal processing, which ensures data confiden-
tiality. This interdisciplinary field encompasses various technologies, including sen-
sors, analogue signal processing, near-sensor computing, and in-memory computing
(Fig.1.3). In-sensor computing devices are sensors that integrate perception, tem-
porary storage, and data processing and analysis with raw analogue signals within
the sensing chip. While near-sensor computing can reduce the physical distance
between sensing and computing, data movement from sensors to processors is still
necessary. In-memory computing uses memristors for both memory and computing
[7], utilizing tunable resistance as the synaptic weights.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-17-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 3
Retina Cone and rod
Pre-processed
signals
Photoreceptors
Biopolar cells
Ganglion cells
CCD/CMOS
ADC
Memory units Processing units
Load
Photoreceptors
In-sensor computing chip
Photoreceptors
Multi-layer register arrays
Pre-processed/
processed signals
a
b
c
Fig. 1.1 The origin of in-sensor visual computing. a The concept of in-sensor computing is
bio-inspired by the retina mechanism where visual signals can be generated and pre-processed by
different types of cells [8]. b Conventional machine vision system: light density needs to be read
out first and converted to digital data which being loaded into memory and then processing units for
meaningful information extraction. c Visual data can be generated, stored, and processed in sensor
through the bio-inspired hardware design
This chapter firstly illustrates the common in-sensor visual computing hardware
architecture. Then various emerging in-sensor computing visual sensors are intro-
duced in terms of hardware, software, algorithms, and applications within the cate-
gory of in-sensor computing architecture. Finally, a summary and future prospective
of in-sensor visual computing technology are made in the conclusion.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-18-320.jpg)
![4 Y. Liu et al.
1.2 In-Sensor Computing Devices
1.2.1 Architecture
In recent years, progress has been made in the development of in-sensor computing
devices. By far, there are mainly two types of in-sensor computing architectures:
(1)In-Sensorarchitecturebyintegratingsensing,memory,andcomputingunits:
A Focal-Plane Sensor Processor (FPSP) [9] integrates visual sensing, storage and
computing units on the focal plane under the architecture of cellular neural networks
(Fig.1.2a). As for each Processing Element (PE), the generated analogue signals
from the pixel can be transferred to the temporal memory units through the bus and
processed using ALU units and registers. Each PE plays a role as a cell interacting
with its neighbours for signal exchange and processing. Hence, the in-sensor visual
inference is realised by the hardware cellular neural network and its synaptic weights
in memory. The representative devices under the FPFS architecture mainly include
the SCAMP Pixel Processor Array (PPA), Q-Eye [10] MIPA4k [9], Asynchronous-
Synchronous Focal-Plane Sensor-Processor Chip (ASAP) [9], KOVA1 [11], and Ais-
torm Mantis2 [12], where the SCAMP PPA is comparatively mature with continuous
research and application outputs.
...
...
a
b
Photoreceptors Processing elements
Registor
Memory ALU
...
...
Ni1
Multi-layer register network
y1
y2
y3
yn
...
...
N11 N1j
Ni1 Nij
N11
N1i
Wi Wj
Si
SiO2
Detect-and-memorize materials
Electrode
Light
P1
PN
Wij
W1
ij
Wn
ij
...
...
P1
PN
Pixel
Multi-layer neural network
y1
y2
y3
yn
...
Fig. 1.2 In-sensor perception and computing architectures and their associated artificial net-
works. a An in-sensor cellular network can be built with an array of PEs which integrates sensing,
memory, and computing units. b A neural network with detect-and-memorise materials](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-19-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 5
Table 1.1 In-sensor computing architecture comparison between FPFS and DAM scheme
Scheme Maturity
level
Speed Application General-
purpose
Programmable Efficiency
FPFS
[15, 16]
Mature High-speed Many Yes Yes High
DAM
[5, 6, 17–19]
Immature Ultra-fast Few No Partial Ultra-high
Fig. 1.3 The position of
in-sensor computing in the
existing knowledge
S
e
n
s
i
n
g
M
e
m
o
r
y
Computing
Near-sensor
computing
In-sensor
computing
I
n
-
m
e
m
o
r
y
c
o
m
p
u
t
i
n
g
M
o
r
p
h
o
l
o
g
i
c
a
l
c
o
m
p
u
t
i
n
g
(2) Detect-and-memorise materials for in-sensor computing architectures:
Material-based detect-and-memorise (DAM) devices (Fig.1.2b) have recently been
proposed to mimic the functional mechanism of the photonic synapses to implement
artificial neural networks [5, 13]. Among these emerging materials and devices, the
memristor is representative as it facilitates sensing, temporal memory, and comput-
ing capability when combined with other photo-sensitive devices [14]. Specifically,
visual signals generated from photoreceptors such as photodiodes can be further pro-
cessed within the artificial networks composed of memristors with tunable resistance
as the weights.
Table1.1 shows the difference between the two rising in-sensor computing archi-
tectures. As can be seen from Table1.1, the DAM-based in-sensor computing sensors
are new and immature compared to the scheme by sensor, memory, and computing
integration. Hence, this chapter mainly reviews devices and algorithms based on the
first architecture scheme (Fig.1.3).
1.2.2 Focal-Plane Sensor Processor (FPFS)
Conventional sensors mainly play the role of information collectors. In recent years,
with the development of techniques on integrated circuit design and the growing
need for low-power and lower-latency edge-computing, a sensor has gradually been](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-20-320.jpg)
![6 Y. Liu et al.
Table 1.2 List of in-sensor and in-memory processing vision systems
System names Resolution In-sensor storage Speed (FPS)
Aistorm Mantis2 96.×96 Gray-scale image 50K [12]
Eye-RIS 176.×144 7 gray + 4 binary 10K [22]
Photodiode array – – 20M bins [14]
SCAMP PPA 256.×256 7 gray + 13 binary 100K [23]
Kovilta’s KOVA1 96.×96 – .>1K [24]
MIPA4k 64.×64 – .>1K [25]
DVS In-senor
pre-processing
Binary events .>10K [26]
integrated with the ability of signal processing independent from general-purpose
computers. The goal of near-sensor processing is to use a dedicated machine learning
accelerator chip on the same printed circuit board [20], or perhaps 3D-stacked with
the CMOS image chip [21]. Despite the fact that this allows CMOS image chip data
to be processed closer to the sensor rather than in the cloud, data transport expenses
between the sensing chip and the processing chip still exist. In contrast, the in-sensor
computing paradigm aims to embed processing capability for each individual pixel.
This section introduces classic in-sensor visual computing devices. Table1.2 lists the
differences among these above-mentioned in-sensor computing devices.
1.3 SCAMP-5d Vision System and Pixel Processor Array
1.3.1 Introduction
SCAMP vision system is one of the emerging in-sensor visual computing devices.
Currently, the most up-to-date version of SCAMP series system is the SCAMP-5d
(Figs.1.4 and 1.5) which consists of 256.×256 processing elements (PEs) weighted
171g with a normal lens. SCAMP-5d vision system is a general-purpose pro-
grammable massively parallel vision system [23] that was invented, designed, and
developed by University of Manchester. By far, SCAMP-5d enjoys many applica-
tions in the field of robotics [27–30] and computer vision [31–33]. As for the PPA
shown in Figs.1.3 and 1.4, the photo-detector converts light into an analogue sig-
nal which can be directly parallelly processed on AREG. Different from the current
hardware design structure of computer vision systems, the PPA gets rid of the Ana-
logue/Digital Conversion (ADC) after sensing and directly operates on analogue
electric current using an arithmetic unit, hence accelerating the signal processing
speed and, in the meantime, avoiding the bottleneck of ADC and data transmission
process. However, errors can be introduced when performing arithmetic operations
or temporal information storage on AREG [6].](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-21-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 7
Fig. 1.4 SCAMP-5d vision system and the pixel processor array (PPA). SCAMP-5d consists of
PPA with 256.×256 Processing Elements (PE) and ARM Micro-controller where parallel image
processing is conducted on PPA by directly operating on analogue signal (electric current from
PIX, which is proportional to the light intensity) within AREG and bit operation within in DREG.
Hence, there is no need for time-consuming and energy-inefficient analogue-to-digital conversion.
Bio-inspired by the efficient information processing of neurons connected by synapses, PPA is
designed to have highly interconnected PE and registers where information can be shared and
accessed adjacently enabling efficient parallel machine vision computing. ARM micro-controller
is in charge of sending instructions to the PPA, receiving the processed information from the PPA,
and more fully-connected layers for deeper CNN
In terms of hardware techniques, the PPA integrates information storage on reg-
isters, image processing and analogue information operation (arithmetic operation,
shifting, etc.), digital/bit operation, and logical operations. As can be seen from
Fig.1.4, for each Processing element (PE), there are seven read/write AREG (A to
F) which can be used for signed value storage and computation with basic arith-
metic operations, such as addition, subtraction, division, etc. In addition, thirteen
1-bit DREG (R0 to R12) in each PE (256.×256 in total) can execute the Boolean
logical operations, such as AND, OR, XNOR, and NOT [15] with information after
binary thresholding on AREG. Each register in PE executes identical instructions
synchronously under SIMD instructions, hence enabling parallel image processing.
In addition, the FLAG register can activate different areas of registers given corre-
sponding patterns for more flexible operation. With the neighbour access function
where each pixel is able to communicate with its four neighbours (north, west, east,
south), an efficient parallel image shifting can be implemented easily. Instructions for
thePPAaredispatchedbytheARM-basedmicro-controllerwithaCortexM0running
at 204MHz. The analogue operations is executed at 5MHz and digital at 10MHz.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-22-320.jpg)
![8 Y. Liu et al.
Fig. 1.5 The development process of SCAMP series vision system from the University of Manch-
ester.ThisreviewmainlyfocusesontheSCAMP-3andSCAMP-5visionsystembecausetheirhigher
resolution and performance would enable more research and applications. Source Piotr Dudek’s
talk in Second International Workshop on Event-based Vision and Smart Cameras (CVPRW) [39]
Other I/O functions, such as USB2.0, GPIO, SPI, and UART, are performed on
Cortex M4 Core [15]. Notice that other names for a similar type of focal-plane sensor
processor can be seen from [9] with names, e.g. ASPA (Asynchronous-synchronous
Focal Plane Sensor Processor Array), FPSP (Focal-Plane Sensor-Processor).
The PPA is a hardware implementation of Cellular Neural Network (CeNN) with
the new optimisation on a mixture of both analogue and digital computing using
AREGs and DREGs, respectively. The studies based on the PPA reviewed in this
work utilise the parallel nature of the CeNN architecture for efficient and high-
performance computing, where each “cell” is intricately connected with its four
neighboursandinformationcanbesharedefficiently.Hence,thePPAcanbemodelled
as a CeNN architecture for visual information computing. The CeNN processing
circuit architecture was first proposed by Leon Chua and Lin Yang [34], followed
by the CeNN universal machine [35] as a prototype. After that, as an invention
of new circuit architecture and a parallel computing paradigm, it enjoys widespread
popularity in academia with a substantial number of research outputs and applications
in pattern recognition [36], image processing [37], and biological vision modelling
[38]. With above-mentioned hardware features, the SCAMP PPA mainly consists of
the following advantages over conventional machine systems.
Efficiency and Low Latency: It is obvious to see from Fig.1.1c that in-sensor
computing gets rid of signal digitisation, transmission, and storage processes onto
external devices, hence enabling high-speed image processing [23] and CNN infer-
ence [40] which can be integrated with agile mobile robot platforms [27–30]. In
addition, the PE distribution and simultaneous instruction execution on PEs allow](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-23-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 9
efficient parallel signal processing. Carey et al. [23] demonstrate an object detection
with a frame rate of 100,000 per second using the SCAMP vision system and Liu et
al. [40] propose binary shallow neural network on the PPA with binary classification
problem at up to 17,000 FPS. These works show the efficiency of image processing
in a sensor once the parallelism mechanism of the PPA is fully taken advantage of.
Low power consumption: According to Fig.1.1c, there are no external process-
ing units or data processing needed, hence the power consumption can be saved to a
large degree. The maximum power cost of the SCAMP-3 vision system for a complex
object tracking and counting system is 29 mW [41]. And the overall power consump-
tion on image processing and CNN inference tasks within a SCAMP-5 vision system
is lower than 2 W [42]. This feature makes the SCAMP vision system suitable for
mobile platforms, usually with short battery life. In addition, according to the power
consumption test from work [43], given 8 popular kernel filters, the SCAMP PPA
generates the same convolution results with much lower power consumption (.>20
times) at a higher speed (.>100 times) compared to common CPUs and GPUs.
Data Security and Privacy Protection: An unique but non-negligible feature of
in-sensor analogue computing with the PPA is its inherent feature of data security
and privacy protection. Data security is feasible because of the focal-plane analogue
information processing without ADC, extra data recording, storage, or intermediate
transmission procedures. Usually, the only output after analogue computing is the
extracted useful target information without redundant information, which is hardly
reversible to get the original data for sensitive information or user re-identification
[44]. Data security and privacy protection have become prominent challenges with
the emergence of the internet of things. Smart devices such as autonomous vehicles,
domestic robots, and smart household appliances are usually equipped with percep-
tual sensors and collect data pervasively in public and private spaces, threatening
users’ privacy and data security. Conventional sensors usually directly upload raw
data to the cloud for data processing [45], which can be a fault line of data security.
When data is processed manually or the network is attacked, crucially sensitive data
can be directly obtained. The acquired data can then be applied to determine individ-
uals’ habits (e.g., motion sensors) or to conduct surveillance (e.g., facial recognition
systems), which can cause significant violations of EU GENERAL DATA PROTEC-
TION REGULATION Article 25.1
In sensor computing first enables only valuable
information to be extracted as it’s output, without redundant information. Moreover,
the minimised raw data are further mocked by the analog signals, which leads to
re-identification almost impossible. Hence, users’ privacy can be strictly protected
with in-sensor processing mechanisms.
1 https://gdpr-info.eu/art-25-gdpr/.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-24-320.jpg)
![10 Y. Liu et al.
1.3.2 Algorithms and Applications
This section mainly illustrates algorithms and their potential applications (Fig.1.7)
based on the SCAMP-5 vision systems (Table1.3 and Fig.1.6).
1.3.2.1 Image Enhancement
Image enhancement comes along with the imaging process on the PPA compared
to the conventional image enhancement which only happens after the image data is
captured. Later, other methods are exploited on different image processing tasks. For
example, Wang et al. [67] proposed a simple coarse grain mapping method to process
bigger images than the PPA resolution itself by temporarily storing sub-images into
different registers.
Table 1.3 List of main studies with the SCAMP PPA
Image processing methods Applications References
Background extraction Segmentation [46, 47]
Contour extraction Object detection [48–50]
Skeleton extraction Shape simplification [51–53]
HDR Image enhancement [31, 48, 54, 55]
Feature corner/edge extraction Edge/feature-based VO [33]
Target detection/localisation High-speed object tracking [23, 56, 57]
Neural network High-level inference [40, 42, 58–61]
Depth estimation/visual
odometry
Robot navigation [29, 30, 32, 60, 62–65]
Automatic code generation Neural network inference, face
detection
[43, 66]
Fig. 1.6 Examples of two images with(left)/without(right) HDR algorithms towards the same scene
in an outdoor environment](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-25-320.jpg)
![12 Y. Liu et al.
HDR (Fig.1.6) is a basic low-level image pre-processing method to obtain rich
image information even facing extreme lighting conditions, such as the mixture of
dim and strong light intensity. However, conventional image sensors rely on either
a global or rolling shutter to form an image, which limits the efficiency of HDR
imaging [31, 68]. Back in 2006, Dudek [48] proposed sensor-level adaptive sensing
and image processing with SCAMP-3 [41, 69], where different exposure settings
are combined for an image with a high dynamic range. Martel et al. [55] make
significant contributions in this area using the PPA. The first HDR image generation
in-sensor is from [54] where pixel-wise exposure can be controlled to generate HDR
images, followed by automotive applications [70]. Furthermore, Bose et al. [32] take
advantageoftheHDRimagetoextractedgesastherobustinputinformationforvisual
odometry estimation. However, the usage of iterative exposure for different regions
of the image slows down the image pre-processing. To speed up the HDR imaging,
Martel et al. [31] propose the learning shutter function for PEs to expose each pixel
independently with an end-to-end training strategy. They obtain an exposure function
by training a U-Net neural network and compiling these trained functions on the
sensor for inference. Later, So et al. [71] presented in-pixel control for snapshot
HDR imaging with irradiance encoding.
1.3.2.2 Contour and Skeleton Extraction
Contours are important features for objects within an image, which can help to
identify different entities. Contour extraction algorithms were proposed based on a
pixel-level snake with very low latency [49]. In 2007, Alonso-Montes et al. proposed
the in-sensor automatic retinal vessel tree extraction based on the Cellular Neural
Networks [50]. The shared key concept for these works [48–50] is to extract contour
iteratively based on the active contour model and Cellular Neural Networks. In 2008,
[72] proposed an image pre-processing method based on the cellular automata for a
robotic scenario. The skeleton within a binary image shows the object size, position,
and simplified shape. Fast image skeletonization [51] is implemented by [52] based
on the wave-trigger propagation/collision. Examples of image contour and skeleton
extraction based on the SCAMP PPA can be seen in Fig.1.8.
Fig. 1.8 Examples of image contour and skeleton extraction using SCAMP PPA. Left: Extracted
Retinal Vascular Tree, Figure from [50]. Right: Extracted skeletons, Figure from [47]](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-27-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 13
Other Feature Extraction Methods: Other image processing methods, such as
background extraction, is exploited by Wang et al. [46, 47]. For higher-level feature
extraction, the edge feature can be obtained by deploying Sobel kernel filters or
Laplacian filters, which are used in the later work for focal plane visual odometry
[32] and neural networks [60]. As for other features, such as corner points extraction,
Chen [33] utilised the DREG for corner points extraction based on the FAST16
algorithm, which is used in later work on visual odometry [65]. Based on the above-
mentioned low- and mid-level image processing methods, researchers are motivated
to exploit more general high-level image processing with up-to-date techniques by
taking advantage of the earlier milestone work and the state-of-art progress, such as
neural networks which would be illustrated in Sect.1.3.2.4.
1.3.2.3 In-Sensor Visual Feature Extraction for Robots
Two major constraints that preclude mobile robots from long-term and diverse appli-
cations are their short battery life and limited load. Emerging sensors may hold the
key to solving this challenge due to their unique low-level hardware design. The
portable SCAMP-5d vision system (171g including the lens) can perform spatial
AI processing in-sensor, reducing data transfer pressure between sensing and the
main processor, hence increasing overall processing efficiency while maintaining
low power consumption [73].
(a) SCAMP PPA on a Quadrocopter
The SCAMP-5d vision system has been integrated into quadrocopter systems for
target tracking, visual odometry and racing. Greatwood et al. perform various exper-
iments by integrating a SCAMP-5d vision system and a quadrotor [28, 29, 74].
Figure1.9 shows a flight control system in terms of hardware integration and control
block diagram, where a pre-set target can be tracked with extracted useful information
on sensor even facing short periods of target tracking loss [28]. In this application, the
direct in-sensor target position extraction releases the pressure of image capturing,
transmission and processing for the whole system. Later, Greatwood et al. proposed
the in-sensor visual odometry using perspective correction on an agile micro air
Fig. 1.9 Left: The integration of a quadrotor and SCAMP-5 vision system for object tracking.
Right: a diagram of system hardware (Figure from [28])](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-28-320.jpg)
![14 Y. Liu et al.
Fig. 1.10 Quadrotor setup for the drone racing with a front-facing SCAMP (Figure from [74])
vehicle based on a similar hardware platform. After that, a drone racing Fig.1.10
within a pre-set environment is demonstrated by taking advantage of the efficient
image processing ability on the PPA [74], where the target position can be estimated
at around 500 FPS. McConville et al. [30] apply the in-sensor visual odometry devel-
oped by Bose et al. [32] on an unmanned aerial system for real-time control purposes.
(b) SCAMP PPA for Mobile Robot Reactive Navigation
In terms of navigation with a SCAMP PPA, Liu et al. [27] proposed reactive agile
navigation on a non-holonomic ground vehicle using PPA by robustly recognis-
ing pre-set patterns out of complex environment background. Although being very
efficient and accurate, using a pre-set fixed pattern for target tracking is difficult to
expand in the generalised environment where there are usually random features. With
this in mind, Chen et al. [60] use in-focal plane feature extraction from the environ-
ment to perform a recurrent neural network on the M4 micro-controller using this
extracted information to estimate the proximity to the ambient objects for obstacle
avoidance purposes. A similar pattern of concentric circles was employed in [27, 28,
30] to effectively extract the dot centre in the circles out of the complex environment
(Fig.1.11).
(c) In-Sensor Computing for Mapping and Localisation
Mapping and localisation are useful techniques for robot navigation. In-sensor map-
ping and localisation are lightweight and low power cost solutions for mobile plat-
forms. Castillo-Elizalde et al. [75] for the first time proposed 1-D mapping and
localisation technique. For this method, features are firstly extracted as the database
from a sequence of images. Then, the incoming image can be localised by comparing
with the database and the prior knowledge of the motion model. In their work, two
methods were utilised to down-sample the original images: direct resizing and local
binary pattern to apply them to different localisation situations.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-29-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 15
a b c
Fig. 1.11 Tracking pattern for the drone and ground vehicle. a Tracking a ground vehicle [28], b
Tracking a moving target while performing a visual odometry [30], c Tracking a fixed pattern with
a mobile ground vehicle [27]
(d) Pose and Depth Estimation
Fordecades,egocentricstateestimationhasbeenstudiedusingconventionalcameras,
emergingDVSdevices,andCPU/GPUs.Inrecentyears,therehavebeensomestudies
utilising SCAMP PPA. For example, Bose et al. [32] for the first time, proposed in-
sensor4Degree-of-Freedom(DoF)visualodometrywhollyonthesensorbymapping
the real-time input image with the previous keyframe through image scaling, shifting,
rotation and alignment. They demonstrate the visual odometry estimation at over
1000 Hz with around 2 W power cost. Debrunner et al. [76] use the SCAMP to
estimate its global motion with the tiling method at 60 FPS with a low power cost of
100.2 mW. After that, Murai [65] proposed 6 DoF visual odometry based on edge
and corner points extracted on sensor and post-processing on a computer with a
frame rate of 300 FPS. They take advantage of feature edge, and corner extraction
methods [33] and calculate the visual odometry off sensor using a similar strategy
with the standard Visual Odometry (VO) systems [77]. Although they combine in-
sensor feature extraction and ready-to-use VO computing method off the sensor, it is
promising to be a direction in the future to combine the efficient image pre-processing
in-sensor and high-volume post-processing with a powerful CPU/GPU, especially
when facing storage shortage and general calculation resources for the large-scale
computing.
In addition, the SCAMP vision system can also work with other accessories to
share the computation burden for more applications. For example, Martel et al. [62–
64] mounted a controllable liquid lens to generate a semi-dense map in real-time,
which is the first work on depth estimation to take advantage of external physical
accessories. With this focus-tunable lens, a vast amount of computation pressure on
the sensor is relieved. This in-sensor feature extraction and post-image processing
on controller scheme are also widely used in many different applications [60, 65],
where the task requirement of storage and computing resources is out of the capacity
of the PPA.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-30-320.jpg)
![16 Y. Liu et al.
1.3.2.4 Research Progress on Neural Networks with a SCAMP PPA
The algorithms of SCAMP PPA proposed earlier mainly focus on low-level image
processing and/or machine vision methods to enhance image quality and extract
basic textures with combinations of inherent built-in functions based on SCAMP-3
and SCAMP-5 with a PE resolution of 128.×128 and 256.×256, respectively. It
should be noticed that these developed image processing methods are deeply related
to the hardware design of the SCAMP vision system. For example, common methods
used in this period are cellular-based algorithms, including cellular neural networks,
because the SCAMP PPA itself is a cellular processor array.
Research on neural network inference with the SCAMP PPA has been active in
recent years. Table1.4 lists the main research work in the area of neural networks,
which covers fully convolutional neural networks and binary convolutional neural
networks using DREG or AREG with various datasets and applications. High-level
image processing, such as object classification, localisation and segmentation in
sensor, is achieved with the neural network. The deployment of neural networks
onto the PPA is a breakthrough since it enables the PPA open to more possibilities
with universal methods, which is unlike the conventional development methods with
some combinations of low-level image processing methods for specific tasks. With
the use of CNN, several types of tasks, such as classification, regression, localisation,
Table 1.4 Different convolutional neural network implementation with SCAMP and performance
comparison.
Network Filter
number
Layers
(Conv+FC)
Dataset Accuracy Frame rate
(fps)
In-sensor/
Near-sensor
Bose [42] 16 5.×5 1 + 1 MNIST .≈94.2% 210 1Conv/1FC
Bose [58] 16/64 4.×4 1 + 1 MNIST .≈93% .>3000 In sensor
– 16+16 4.×4 2 + 1 MNIST 92–94% 224 In sensor
Liu [40] 64 4.×4 1 + 1 8 Plankton 80.5 % 4016 In sensor
– 16 4.×4 1 + 2 8 Hand
gestures
.<98.7% 2092 1Conv+1FC/1FC
– 16 4.×4 1 + 1 Roshambo .<97.73% 8264 In sensor
– 64 4.×4 1 + 1 0,1 in
MNIST
.<99.1% 17543 In sensor
Liu FCN
[59]
16+64 4.×4
64 1.×1
3 + 0 Simulation – 283 In sensor
Liu [78] 16+64 4.×4 2 + 2 EMNIST .<86.74% 178 2Conv+1FC/
Binarized
CNN
1FC
Chen [60] – 0 + N Collected
indoor
– – Near sensor
AnalogNet
[61]
3 3.×3 1 + 3 MNIST 96.9% 2260 1Conv/2FC](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-31-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 17
and segmentation, can be feasible, hence enabling more applications. Table1.4 shows
the neural network-related work based on the SCAMP PPA vision system.
The research on CNN implementation and inference within PPA is pioneered by
Bose et al. [79] where a CNN with a single convolutional layer performed upon
the PPA array and a fully-connected layer upon its controller chip (M0). They per-
formed 16-bit image convolution operations using 4.×4 DREG “Super Pixel” blocks
and demonstrated live digit classification based on MNIST dataset at around 200 FPS.
In their work, the ternary {.−1, 0, 1} kernel filters are stored on the flash (M4) of
the PPA system, and are effectively encoded in the instructions/operation sent to the
PPA array, performing convolutions sequentially. Furthermore, a mobile car locali-
sation task is then explored using synthetic datasets, where the pre-processed edge
information is mainly the clues for network inference. Notice that the localisation is
realised by classifying the car’s position along the.x and.y axis, respectively. To fully
take advantage of PPA’s parallel computing characteristics and to further improve
the CNN inference efficiency, Bose et al. [58], for the first time, proposed the idea of
in-pixel weight storage, where the network’s weights are directly stored within the
registers of the PPA’s PEs. This method enabled both parallel computations of multi-
ple convolutions, and implementation of a fully connected layer upon the PPA array,
resulting in a.×22 faster CNN inference (4464 FPS) on the same digit recognition
task. Based on these two works, [40] further proposes a high-speed lightweight neural
network using BinaryConnect [80] with a new method for computing convolutions
upon the PPA, allowing for varying convolutional strides. This work demonstrated
four different classification tasks with frame rates ranging from 2,000 to 17,500 per
second with different stride setups. Later, based on this network, a direct servo con-
trol using CNN results [81] and a simulated robot tracking from a drone [82] with
in-sensor CNN computing results are exploited. In addition, the AnalogNet2 [61, 83]
extends the earlier work in [84], implementing a CNN which reaches 96.9% accuracy
on the MNIST dataset at a speed of 2260 fps. However, their method requires all
fully connected layers to be performed externally to the PPA array with only 3 con-
volutional kernel filters implemented in sequence on the PPA as the first layer. More
kernel filters would significantly slow down the inference process. Notice that, in our
work [60], a recurrent neural network is implemented on the micro-controller with
features extracted on a sensor. In this manner, the fully-connected layer of a neural
network can be deployed similarly with conventional embedded devices. It is notable
that Martel et al. trained a neural network of exposure time for each individual pixel
off the sensor for HDR imaging and video compressive sensing [31].
Furthermore, work [78] binarized CNN with batch norm both for classification
and coarse segmentation. To deal with the classifications application with more labels
and more segmentation tasks, they propose the idea of dynamic model swapping by
uploading weights of trained models in sequence or according to the last inference
result, targeting multiple sub-tasks decomposed from a more sophisticated task. They
then demonstrate a servo control directly using the CNN inference results [81], which
potentially indicates that motion control platforms, such as a ground vehicle or drones
can have a light-weight servo control system without using external control units in
the future.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-32-320.jpg)
![18 Y. Liu et al.
Notice that the preceding neural network-related work mainly focuses on clas-
sification or classification-based localisation, both of which require fully connected
layers. However, the parameters in fully-connected layers are typically substantially
larger than those in convolutional layers due to the dense connections of each indi-
vidual neuron. Thus, work developed a fully-convolutional neural network (FCN)
[59], not only presenting in-sensor image segmentation and localisation but also
eliminating dense layers for a smaller memory footprint [85].
1.3.2.5 In-Sensor Cellular Automata
The PPA itself is a cellular neural network architecture where each ‘cell’ is closely
connected with its four neighbours, hence information can be shared efficiently.
With this in mind, the author is inspired to explore the possibility to perform cellular
behaviour, such as Conway’s game of life (demonstration shown from2
) and ele-
mentary cellular automata (demonstration Rule 90 seen from3
) based on the theory
of cellular automata [86]. With the rule of the game of life, all ‘cells’ can update
their states (alive or dead) in-sensor as fast as 53.µs for each iteration based on the
bit-operation with DREG. As can be seen from Fig.1.12, a Sierpiński triangle is effi-
ciently generated based on bit operation on the sensor with 730.µs of 255 iterations
to fill the whole chip.
(a) Elementary CA
One-dimensional CA is one of the simple CA algorithms. Some classical updating
rules, such as Rule 30, Rule 90, and Rule 110, can be implemented by logic bit
Fig. 1.12 Our demonstration of elementary cellular automata with Rule 90 on the SCAMP PPA.
This pattern is generated from top to bottom. We have made this project available from https://
github.com/yananliusdu/1D_CellularAutomata
2 https://youtu.be/X_t4c3f-T4s.
3 https://youtu.be/HgPvoK5EJ_s.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-33-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 21
processing-related work can be potentially explored as long as proper update rules
and associated steps are trained with neural network methods [37, 87].
1.4 Eye-RIS
1.4.1 Introduction
Eye-RIS [22, 88] commercial vision system on Chip (VSoC) extends CMOS pixel
functionality with image storage (7 gray-scale images and 4 binary images) and dig-
ital/analogue signal processing ability. Specifically, a 32-bit RISC (Reduced Instruc-
tion Set Computer) is integrated with a vision sensor for image post-processing after
the parallel in-sensor pre-processing (Fig.1.15). The resolution of the Eye-RIS vision
sensor is 176.×144. Notice that Eye-RIS’s overall functional diagram is similar to
that in the SCAMP vision system, where the counterpart of RISC is the M0 micro-
controller in the SCAMP PPA [89]. The most significant difference, though, is that
the Eye-RIS has a DICop part, which is a digital image co-processor that handles
geometric transformations and can send the results back to the pixel level for more
processing.
The Eye-RIS Vision System on Chip (VSoC) is an autonomous device combining
a parallel CMOS image sensor processor with 32-bit RISC microprocessor perform-
ing post-processing and system control tasks, several I/O and high-speed communi-
cation ports that allow the system to communicate and/or to control external systems,
and on chip memory. The combination of massive parallel image pre-processing in
the sensor with complex image post-processing in the microprocessor results in ultra
compact implementation of a vision system able to perform complex machine vision
algorithms at speeds of several thousands of images per second. The Eye-RIS VSoC
Image acquisition
In-pixel image
processing
A/D
In-pixel image
memory
D/A
On-chip image
memory
Image post-
processing
On-chip data and
program memory
Serial flash
memory
External image
memory
External program
and data memory
I/O and
communications
Eye-RIS v2.1 VSoC
SIS Q-Eye
Sensor-processor
Nios II RISC
Microprocessor
Eye-RIS v2.1
Vision system
Fig. 1.15 Eye-RIS v2.1 VSoC architecture](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-36-320.jpg)
![22 Y. Liu et al.
features a complete application development software environment allowing easy
control of the device and is optimized for industrial applications requiring image
sensing, image processing, and decision making at extreme frame rates.
1.4.2 Applications
The applications with Eye-RIS consist of automotive, machine vision, security,
games and battery powered products. Caballero-Garcia and Jimenez-Marrufo [90]
proposed a series of techniques to deploy image processing algorithms on the Eye-
RIS Vision System on Chip (VSoC) for various applications. One unique character-
istic using the Eye-RIS vision platform compared to conventional visual sensors is
the simultaneous image acquisition and early processing in the analogue domain.
Paper [91] aims to describe how the AnaFocus’ Eye-RIS family of vision sys-
tems has been successfully embedded within the roving robots developed under the
framework of SPARK and SPARK II European projects to solve the action-oriented
perception problem in real time. With the ability of low power cost and efficient par-
allel image processing, Eye-RIS has been equipped to many different mobile robot
platforms, such as Rover II wheeled robot and Gregor III hexapod robot. Visual
homing, object tracking, and navigation using landmarks are demonstrated based on
the robot platforms and in-sensor real-time image processing algorithms (Fig.1.16).
Optical flow based on the Lucas and Kanade by Guzman et al. [92] (Fig.1.17) is
implemented on the Eye-RIS platform taking advantage of both analogue and digital
signals processing using Q-Eye and Nios II RISC respectively (Fig.1.15). In the
experiment, the optical flow estimation reaches over 25 fps which can be used in the
area of robotics in real time. Specifically, the optical flow constraint equation:
.uIx + vIy + It = 0 (1.1)
Fig. 1.16 Eye-RIS VSoC equipped onto robot platforms. a Rover II wheeled robot, b Gregor III
hexapod robot (Figure from [91])](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-37-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 23
Fig. 1.17 Optical flow estimation in traffic sequences using Eye-RIS VSoC architecture (Figure
from [92])
where the partial derivatives of.I are denoted by subscripts, which can be obtained
from the image..u and.v are the.x and.y components of the optical flow vector, which
are the optical flow vectors that need to be found out.
Lucas–Kanade method [93] is a classical method to deal with the optical flow
constraint problem. According to the assumption of Lucas–Kanade method, given a
small pixel block, 3.×3 for example, the optical flow remains identical within this
small block. Then, we can have following equation group:
.
uIx1 + vIy1 = −It1
uIx2 + vIy2 = −It2
. . .
uIxn + vIyn = −Itn
(1.2)
Equation1.2 can then be represented as
.
⎡
⎢
⎢
⎣
Ix1 Iy1
Ix2 Iy2
. . .
Ixn Iyn
⎤
⎥
⎥
⎦
[
u
v
]
=
⎡
⎢
⎢
⎣
−It1
−It2
. . .
−Itn
⎤
⎥
⎥
⎦ (1.3)
we assign Eq.1.3 as . A
−
→
V = −b, here the least squares method can be used to get
optical flow vector.
−
→
V , then. AT
A
−
→
V = AT
(−b), hence, the optical flow vector can
be obtained through:
.
−
→
V = (AT
A)−1
AT
(−b) (1.4)
In detail, Eq.1.4 can be shown as:
.
[
u
v
]
=
[ ∑n
i=1 I2
xi
∑n
i=1 Ixi Iyi
∑n
i=1 Ixi Iyi
∑n
i=1 I2
yi
]−1 [
−
∑n
i=1 Ixi Iti
−
∑n
i=1 Iyi Iti
]
(1.5)](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-38-320.jpg)
![24 Y. Liu et al.
Shift along x axis
A A_x Ix = A-A_x
Shift along y axis
A A_y Iy = A-A_y
(a)
A
(b)
A_1 It = A-A_1
(c)
Fig. 1.18 The parallel calculation of.Ix , Iy, It in the focal plane using analogue signals
Through Eq.1.5, the optical flow vector can be estimated through a sequence of
images. These intensity derivatives including.Ix , Iy, It can be efficiently obtained by
shifting and subtracting between frame sequences as illustrated from Fig.1.18. After
that, the optical vector can then be calculated using conventional computing units.
As for the implementation of optical flow using the above-mentioned formulations,
work [22] takes advantage of both in-sensor pre-processing and post-processing with
computing units achieving up to 28.9 fps.
In the study of [94], authors explored different methods of point tracking on the
platform of Eye-RIS, which is able to equip Unmanned Arial Vehicles (UAVs) with
the ability of on-board sensing and computing with low load and power consumption.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-39-320.jpg)
![1 In-Sensor Visual Devices for Perception and Inference 25
Nicolosi et al. [95] applied the Eye-RIS vision platform to control welding process
byusingthecellular neural networkbasedvisual algorithms. [96] extendedtheir work
onto vision based closed loop control for partial penetration welding of overlap joints.
Work [97] proposed algorithms for the cellular neural network to detect rapidly
approaching object which is bio-inspired by mammalian retina that consists of loom-
ing sensitive circuit based on local interaction of cells.
1.5 Kovilta’s KOVA1
1.5.1 Introduction
The KOvilta Vision Array (KOVA1) (depicted in [11]) employs a meticulously
designed pixel-level processing circuitry that utilizes an efficient combination of
analogue and digital (mixed-mode) computation techniques to execute a diverse
range of pre-programmed operations. These operations include automated sensor
adaption, grayscale filtering, segmentation, and complex object-level visual analy-
sis. The selection of operations to be performed at the pixel-level hardware can be
customized based on the specific requirements of the application, thereby enhancing
the overall implementation efficiency.
The KOVA1 is Kovilta’s inaugural silicon rendition of the KOvilta Vision Array
structure, and it comprises a 96.×96 pixel focal-plane processor array fabricated
using 180nm CMOS technology. The sensor-processor chip is integrated into a
miniature smart camera system equipped with FPGA-based control and Ethernet
I/O. In this focal plane processor architecture, each pixel-cell of the sensor array
includes a reconfigurable processing element that operates directly on the output of
the analog photodiode. This allows real-time data compression for capturing images
with high dynamic ranges without compromising quality. Additionally, processing
in parallel on the pixel plane enables rapid low-level feature analysis and eliminates
the need for time and energy-consuming long-distance data transfers from the sensor
to an external processor. The sensor output may include only the essential feature
data, such as the presence of an object or a set of object coordinates or features,
thereby reducing the amount of hardware needed for further external image content
analysis.
The KOVA1 camera system employs pixel cells that are connected within their
immediate neighbourhood, allowing for direct information exchange during image
analysis activities at the sensor level. Moreover, local memories integrated at the
pixel level facilitate the storage of multiple full images or intermediate processing
outcomes on the sensor plane. An FPGA chip is utilized to manage the program
execution and I/O of the sensor-processor chip in the KOVA1 embedded camera
system. As the control and I/O structures consume only a small fraction of the
FPGA’s resources, supplementary visual analysis operations can be implemented
on the FPGA to enhance the on-chip sensor-level processing. The KEDE software](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-40-320.jpg)
![26 Y. Liu et al.
environment from Kovilta is used to program and manage the camera, with program-
ming and data I/O facilitated by an Ethernet connection, while direct CMOS-signal
outputs bypass the network interface. Additionally, the camera’s internal memory
is capable of storing program code. It can operate autonomously without the need
for a PC connection and transmit output data to an Ethernet-connected or directly
controlled device.
1.5.2 Applications
Line detection: Santti et al. [98] proposed a line detection method by combining
KOVA1 and FPGA for low-level and mid-level feature extraction respectively. This
line detection method can be applied for industrial inspection and control applications
with its performance of high-speed processing and low power consumption.
Seam and Spatter Tracking: In Lahdenoja et al. [24, 99] and Santti et al.’s work
[100], KOVA1 is utilised for seam tracking for real-time robot path optimisation
during a high power laser welding process. The reason to use this type of pixel-level
sensor lies in the ability to control pixel-wise exposure periods facing significant
intensity differences in a laser welding task. Hence, the effective binary feature
points can be extracted on the focal plane and then this compressed information is
sent to the FPGA for laser beam location extraction using Hough transform. With
a frame rate over 1000 fps, the combination of in-sensor image pre-processing and
FPGA-based Hough operation enables a real-time optical seam tracking for robot
control. In addition, spatter can also be tracked in laser and manual arc welding [101]
in extreme radiated light intensity conditions (Fig.1.19).
Fig. 1.19 Straight line extraction process using KOVA1. Left to right: figure captured with KOVA1,
binary feature extraction using pixel-level process, estimated beam line with binary features using
FPGA. Figure from [100]](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-41-320.jpg)
![28 Y. Liu et al.
sification, smart home, wearable devices (glasses, headsets), household appliances,
gaming devices, automotive, and voice processes as claimed in their application
website https://aistorm.ai/applications/.
1.7 Other In-Sensor Computing Devices
MIPA4k: MIPA4k [25] is an earlier sensory-level image processing sensor with
similar architecture to KOVA1. A number of algorithms can be carried out based
on the MIPA4k with the use of mixed signals. These algorithms mainly cover edge
detection [102], local binary pattern [103], locally adaptive image sensing [104],
space-dependent binary image processing [105], and object segmentation and track-
ing [106]. Similar early cellular vision chips also include ACE16K [107].
Memristor-Based Devices: Yao et al. [108] Memristor-based hardware is a plat-
form to deploy the neural network using the programmable resistance within the
integrated circuits mimicking the synaptic connections in a human brain [17, 108–
111]. However, it integrates only storage and processing functions, which can be
regarded as in-memory computing. Hence, signals should be input from sensors or
other storage devices. They are thus usually integrated with other sensory systems for
information processing. Lee et al. [19] take advantage of photo-diode and memristor
crossbar for primary visual information process aiming to extract useful information
from the input images. In-Sensor visual computing with memristor or new materials
are currently not mature enough to support various practical applications for machine
vision tasks. By far, the is few practical applications using memristor in-sensor com-
puting devices.
Dynamic Vision Sensor (DVS): inivation [26] DVS produces data in the form
of sparse contrast-change events that facilitate low-latency visual processing using
external computational hardware [112–114]. These binary events are generated from
in-sensor processing according to the brightness changes. Although the pixels in a
DVS have a primitive in-sensor processing ability by binarising brightness changes,
it achieves an ultra-high-speed response to the environment when working with
external hardware computing units, enabling an enormous potential for robotics and
computer vision in a challenging environment [115].
Other Emerging Sensor Devices: Mennel et al. [14] use a 2D semiconductor
(.W Se2) photodiode array as the vision sensor, the photoresponsivity matrix to store
the connecting weights of the neural network, where both supervised and unsuper-
vised learning for classification are present. However, laser light and a set of optical
systems are needed to project images onto the chip, which prevents it from having
practical usage. Song et al. [116] proposed a CMOS-based PIP (Processing-in-Pixel)
architecture where image convolution (8-bit weight configuration) can be performed
as the image preprocessing before image data is read out. In addition, Datta et al.
proposed the Processing-in-Pixel-in-memory paradigm where the first few convolu-
tional layers of a CNN can be processed and the compressed data then sent to other
near-sensor processors [117].](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-43-320.jpg)
![38 Y. Qian and M. Yang
Fig. 2.1 Typical normal and fish-eye images taken in the same location
provides a larger field of view (FoV), up to 180.
◦
, than ordinary cameras. Therefore,
fewer fish-eye cameras are needed to fully monitor their surroundings. Moreover,
fish-eye cameras are indispensable sensors in autonomous driving and have a very
large role in some scenarios.
Figure2.1 shows a normal image and fisheye image; both were taken at the same
location. Intuitively, the images appear quite different. The fish-eye image in Fig.2.1b
contains more scenery. In addition, serious distortion occurs in the fish-eye image,
e.g., the straight street lamp in Fig.2.1a becomes crooked in the fish-eye image. The
same object in a normal image contains more pixels than in a fish-eye image, that is,
fish-eye images lose information as the distance increases.
In short, fish-eye cameras have three advantages. Similar to fish-eye cameras, lidar
sensors also have large FoVs [1–4]. However, cameras provide much richer semantic
information. In addition, cameras have a higher cost performance [5], which is crucial
for the industry. Moreover, these benefits apply to all camera sensors, not just fish-eye
cameras.
Fish-eye cameras offer a wide FoV, usually up to 180.
◦
, which is crucial in
autonomous driving as covering more blind spots helps improve safety.
The use of fish-eye cameras avoids the need for calibration among multiple cam-
eras. To cover more space, ordinary cameras should be grouped. Therefore, for such
a system, calibration is necessary, and calibration between two different sensors
is complex. Inaccurate calibration will introduce unexpected noise in the system,
leading to the failure of subsequent algorithms.
There are three challenges in applying fish-eye cameras. Deep learning methods
have substantially become more popular in the field of computer vision, including
environmental perception, where datasets have become particularly important. The
use of sample-rich datasets contributes to the better performances of most algorithms.
In recent years, many datasets for autonomous driving environment awareness, such
as KITTI [6] and Cityscapes [7], have been published. However, these datasets con-
tain normal images with a normal FoV. As fish-eye images and normal images appear
completely different, normal images have difficulty benefitting from fish-eye-based](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-52-320.jpg)
![2 Environmental Perception Using Fish-Eye Cameras for Autonomous Driving 39
Fig. 2.2 False results caused by the distortion of the fish-eye image. The red bounding box rep-
resents false negative detection results, and the area in the red circle represents false positive seg-
mentation results
algorithms. In conclusion, few significant fish-eye datasets have been formed, which
severely limits the development of fish-eye-based algorithms.
This is the most significant problem of fish-eye cameras. In Fig.2.1, street lights,
cars and buildings appear in a completely different curved position from the real
shape. Moreover, these distorted objects cause issues for most detectors as most
detectors are trained on normal images. Figure2.2a shows the object detection results
obtained by using the faster region-based convolutional neural network (Faster R-
CNN) [8]. Compared to other normal vehicles, the white vehicles with edges were
so deformed that they were not accurately detected. Figure2.2b shows the semantic
segmentation results generated by the efficient residual-factorized CNN (ERFNet)
[9, 10]. The areas in the red circles are segmentation errors caused by distortion.
As shown in Fig.2.1, it is difficult for fish-eye cameras to clearly observe distant
targets, which limits their application scenarios, that is fish-eye cameras can only be
used for low-speed and short-range scenes. In these cases, the perception of distant
objects is not urgently needed.
This chapter is organized as follows. The typical fish-eye image datasets are
presentedinSect.2.2.Differentfish-eyeprojectionmodelsareintroducedinSect.2.3.
Various applications of fish-eye cameras in autonomous driving are organized in
Sects.2.4 and 2.5. Finally, the development tendency of fish-eye camera applications
is discussed in Sect.2.6.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-53-320.jpg)
![40 Y. Qian and M. Yang
2.2 Fish-Eye Image Datasets
High-quality, fish-eye datasets provide a comparison platform for various algorithms
and can be utilized in the training process to improve the performance of the detector.
Currently, three main methods are used to obtain fish-eye image data. Most works
use real fish-eye cameras to manually collect real fish-eye datasets. Some works use
simulators to generate virtual fish-eye images. Other works focus on generating fish-
eye lens images from real perspective images. These real and virtual images promote
research in this field.
2.2.1 Real Fish-Eye Datasets
We sorted the real fish-eye datasets in the field of automatic driving. Details of these
datasets are shown in Table2.1. The author uses different fish-eye camera settings to
customize their fish-eye lens datasets, including different numbers of fish-eye lenses,
image resolutions, numbers of datasets, and supported tasks.
Levi et al. [11, 12] collected a fish-eye dataset using a vehicle rearview camera to
evaluate the application of rear-view pedestrian detection. This dataset is named the
GM-ATCI rearview pedestrian dataset. This dataset contains 15 video sequences, and
each session is obtained in different scenes on different dates. The dataset contains a
total of 250 fragments, with a total duration of 76min, and more than 200K annotated
pedestrian bounding boxes.
Oxford RobotCar [13], which was proposed by Maddern et al., is a large-scale
dataset that focuses on the long-term autonomy of autonomous vehicles, mainly
supporting positioning and mapping tasks. The dataset collects data under various
weather conditions, including heavy rain, nighttime, direct sunlight, and snowfall,
and has accumulated more than 1000 km of road data in one year. Due to the long data
collection time, the appearances of many roads and buildings has changed, which is
also a feature of this dataset.
Yogamani et al. [14] proposed the first fish-eye image dataset dedicated to intel-
ligent vehicles. This dataset, which is named WoodScape, consists of four cameras
covering 360.
◦
and a high-definition laser scanner, inertial measurement unit (IMU),
and global navigation satellite system (GNSS). Annotations can be used for nine
tasks, especially 3D object detection, depth estimation (superimposed on the front
camera) and semantic segmentation. The semantic annotation of 40 classes at the
instance level exceeds 10000 images, and the annotation of other tasks exceeds
100000 images. Rashed et al. further expanded the dataset in [15].
Liao et al. proposed the suburban dataset KITTI-360 [16]. Compared with the
classic KITTI [6], KITTI-360 has two new fish-eye cameras and a larger amount
of data. Compared with WoodScape [14], KITTI-360 provides temporal, coherent,
semantic instance annotation and 3D annotation for 3D reasoning.](https://image.slidesharecdn.com/26405093-250520104930-83ee0877/85/Autonomous-Driving-Perception-Fundamentals-And-Applications-Rui-Fan-54-320.jpg)
Autonomous Driving Perception Fundamentals And Applications Rui Fan
- 1. Autonomous Driving Perception Fundamentals And Applications Rui Fan download https://ebookbell.com/product/autonomous-driving-perception- fundamentals-and-applications-rui-fan-52810186 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Autonomous Driving Perception Fundamentals And Applications Rui Fan https://ebookbell.com/product/autonomous-driving-perception- fundamentals-and-applications-rui-fan-52858356 Belief State Planning For Autonomous Driving Planning With Interaction Uncertain Prediction And Uncertain Perception Constantin Hubmann https://ebookbell.com/product/belief-state-planning-for-autonomous- driving-planning-with-interaction-uncertain-prediction-and-uncertain- perception-constantin-hubmann-230260504 Autonomous Driving Changes The Future 1st Ed Zhanxiang Chai https://ebookbell.com/product/autonomous-driving-changes-the- future-1st-ed-zhanxiang-chai-32710750 Autonomous Driving And Advanced Driverassistance Systems Adas Applications Development Legal Issues And Testing Lentin Joseph https://ebookbell.com/product/autonomous-driving-and-advanced- driverassistance-systems-adas-applications-development-legal-issues- and-testing-lentin-joseph-36337984
- 3. Autonomous Driving Technical Legal And Social Aspects 1st Edition Markus Maurer https://ebookbell.com/product/autonomous-driving-technical-legal-and- social-aspects-1st-edition-markus-maurer-5485550 Autonomous Driving How The Driverless Revolution Will Change The World Andreas Herrmann https://ebookbell.com/product/autonomous-driving-how-the-driverless- revolution-will-change-the-world-andreas-herrmann-10628760 Autonomous Driving Technical Legal And Social Aspects 1st Edition Gerdes https://ebookbell.com/product/autonomous-driving-technical-legal-and- social-aspects-1st-edition-gerdes-11400820 Autonomous Driving Markus Maurer J Christian Gerdes Barbara Lenz Hermann Winner https://ebookbell.com/product/autonomous-driving-markus-maurer-j- christian-gerdes-barbara-lenz-hermann-winner-59042330 Driving Without A Driver Autonomous Driving As A Legal Challenge Proceedings Of The 38th Congress Of The Society Of Comparative Law In Tubingen September 29 To October 1 2022 Uwe Kischel Editor https://ebookbell.com/product/driving-without-a-driver-autonomous- driving-as-a-legal-challenge-proceedings-of-the-38th-congress-of-the- society-of-comparative-law-in-tubingen-september-29-to- october-1-2022-uwe-kischel-editor-51616670
- 5. Advances in ComputerVision and Pattern Recognition Rui Fan Sicen Guo Mohammud Junaid Bocus Editors Autonomous Driving Perception Fundamentals and Applications
- 6. Advances in Computer Vision and Pattern Recognition Founding Editor Sameer Singh Series Editor Sing Bing Kang, Zillow, Inc., Seattle, WA, USA Advisory Editors Horst Bischof, Graz University of Technology, Graz, Austria Richard Bowden, University of Surrey, Guildford, Surrey, UK Sven Dickinson, University of Toronto, Toronto, ON, Canada Jiaya Jia, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong Kyoung Mu Lee, Seoul National University, Seoul, Korea (Republic of) Zhouchen Lin , Peking University, Beijing, Beijing, China Yoichi Sato, University of Tokyo, Tokyo, Japan Bernt Schiele, Max Planck Institute for Informatics, Saarbrücken, Saarland, Germany Stan Sclaroff, Boston University, Boston, MA, USA
- 7. Titles in this series now included in the Thomson Reuters Book Citation Index! Advances in Computer Vision and Pattern Recognition is a series of books which brings together current developments in this multi-disciplinary area. It covers both theoretical and applied aspects of computer vision, and provides texts for students and senior researchers in topics including, but not limited to: • Deep learning for vision applications • Computational photography • Biological vision • Image and video processing • Document analysis and character recognition • Biometrics • Multimedia • Virtual and augmented reality • Vision for graphics • Vision and language • Robotics
- 8. Rui Fan · Sicen Guo · Mohammud Junaid Bocus Editors Autonomous Driving Perception Fundamentals and Applications
- 9. Editors Rui Fan Control Science & Engineering, Shanghai Research Institute for Intelligent Autonomous Systems Tongji University Shanghai, P. R. China Mohammud Junaid Bocus Electrical and Electronic Engineering University of Bristol Bristol, UK Sicen Guo Control Science & Engineering, Shanghai Research Institute for Intelligent Autonomous Systems Tongji University Shanghai, P. R. China ISSN 2191-6586 ISSN 2191-6594 (electronic) Advances in Computer Vision and Pattern Recognition ISBN 978-981-99-4286-2 ISBN 978-981-99-4287-9 (eBook) https://doi.org/10.1007/978-981-99-4287-9 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore Paper in this product is recyclable.
- 10. Preface With the recent advancements in artificial intelligence, there is a growing expectation that fully autonomous driving vehicles will soon become a reality, leading to signif- icant societal changes. The core competencies of an autonomous vehicle system can be broadly categorized into four main categories: perception, prediction, plan- ning, and control. The environmental perception system serves as the foundation of autonomous vehicles, utilizing cutting-edge computer vision and machine learning algorithms to analyze raw sensor data and create a comprehensive understanding of the surrounding environment. Similar to the visual cognition and understanding of humans, this process allows for a deep and nuanced perception of the world. Conventional autonomous driving perception systems are often hindered by sepa- rate sensing, memory, and processing architectures, which may not meet the demand for ultra-high raw sensor data processing rates and ultra-low power consumption. In contrast, in-sensor computing technology performs signal processing at the pixel level by utilizing the collected analog signals directly, without requiring data to be sent to other processors. This enables highly efficient and low-power consumption visual signal processing by integrating sensing, storage, and computation onto focal planes with innovative circuit designs or new materials. Therefore, the in-sensor computing paradigm holds significant potential for autonomous driving. Further- more, fish-eye cameras have emerged as an essential sensor in the field of autonomous driving. Thanks to the unique projection principle of fish-eye cameras, they offer a significantly larger field of view (FoV) compared to conventional cameras. This distinct characteristic makes fish-eye cameras highly versatile and suitable for a wide range of autonomous driving perception applications. In addition, computer stereo vision is a cost-effective and efficient method for depth information acquisition, and it has found widespread use in 3D environmental perception. Despite the impressive results obtainedbystate-of-the-art (SoTA) stereovisionalgorithms that utilizeconvo- lutional neural networks, their training typically necessitates a substantial amount of accurately labeled disparity ground truth data. Consequently, self-supervised or v
- 11. vi Preface unsupervised deep stereo networks have emerged as the dominant approach in this research area. Research on semantic segmentation has been ongoing for over a decade. However, conventional single-modal networks are unable to fully utilize the spatial informa- tion provided by range sensors, making them less effective in diverse weather and illumination conditions. To address this challenge, data-fusion semantic segmenta- tion networks have been developed, which employ multiple encoders to extract deep features from different visual information sources. These deep features are subse- quently fused to provide a more comprehensive understanding of the surrounding environment. 3D object detection is also a crucial component of autonomous driving systems that has made remarkable progress in recent years. Nonetheless, the various perceptual sensors used for object detection present their unique challenges. Cameras are vulnerable to issues such as foreshortening and flickering effects, over-exposure problems, as well as environmental variations like lighting and weather conditions. Similarly, LiDARs and RADARs suffer from low-resolution and sparse data repre- sentations. Furthermore, occlusion presents a significant challenge to object detec- tion, leading to the partial or complete invisibility of obstructed objects. To address these challenges, collaborative 3D object detection has been proposed as an alterna- tive to conventional approaches. Collaborative object detection facilitates informa- tion sharing between agents, enabling them to perceive the environments beyond line- of-sight and FoV. This approach holds great promise in overcoming the limitations of individual sensors and achieving more robust and accurate 3D object detection in autonomous driving systems. The application of the simultaneous localization and mapping (SLAM) tech- nique to autonomous driving also presents several challenges. Over the past three decades, researchers have made significant progress in addressing the probabilistic SLAM problem by developing a range of theoretical frameworks, efficient solvers, and complete systems. Visual SLAM for texture-less environments is an especially challenging task, as multi-view images cannot be effectively linked using reliable keypoints. However, researchers continue to develop new techniques and algorithms to overcome this limitation. Moreover, the enhancement of SLAM systems is also being driven by the emergence of new sensors or sensor combinations, such as cameras, LiDARs, IMUs, and other similar technologies. As these sensors become more advanced and sophisticated, they offer new opportunities to improve the accuracy and reliability of SLAM systems for autonomous driving applications. Multi-task learning has become a popular paradigm for simultaneously tack- ling multiple tasks while using fewer computational resources and reducing the inference time. Recently, several self-supervised pre-training methods have been proposed, demonstrating impressive performance across a range of computer vision tasks. However, the extent to which these methods can generalize to multi-task situ- ations remains largely unexplored. Additionally, the majority of multi-task algo- rithms are tailored to specific tasks that are usually unrelated to autonomous driving,
- 12. Preface vii posing difficulties when attempting to compare state-of-the-art multi-task learning approaches in the domain of autonomous driving. Bird’s eye view (BEV) perception involves transforming a perspective view into a bird’s eye view and performing various perception tasks, such as 3D detection, map segmentation, tracking, and motion planning. Thanks to its inherent advantages in 3D space representation, multimodal fusion, decision-making, and planning, the topic of BEV perception has attracted significant interest among both academic and industrial researchers. Road environment perception, which includes 3D geometry reconstruction of road surfaces and the intelligent detection of road damages, is also critical for ensuring safe andcomfortabledriving.Roadsurfacedefectscanbeextremelyhazardous,especially when hit at high speeds, as these can not only damage the vehicle’s suspension but also cause the driver to lose control of the vehicle. When one of the vehicle’s tyres enters a pothole, the weight distribution across all tyres becomes unbalanced, causing the vehicle to tilt and shift more towards the tyres that are lower relative to the pothole. This uneven weight distribution can produce a considerable and focused force on the tyre when it hits the edge of the pothole, resulting in deformation, breakage, or even bending of the rim. The damage inflicted on the tyre impacts the driving experience, making it challenging to maintain a straight driving trajectory. This book provides an in-depth, comprehensive, and SoTA review on a range of autonomous driving perception topics, such as stereo matching, semantic segmen- tation, 3D object detection, simultaneous localization and mapping, and BEV perception. The book’s webpage can be accessed at mias.group/ADP2023. The intended readership for this book primarily comprises of tertiary students who seek a comprehensive and yet an introductory understanding of the fundamental concepts and practical applications of machine vision and deep learning techniques. It is also directed at professionals and researchers in autonomous driving who are seeking an assessment of the current state-of-the-art methods available in existing literature, and who aspire to identify potential areas of research for further explo- ration. The extensive range of topics covered in this book makes it an invaluable resource for a variety of university programs that include courses related to machine vision, deep learning, and robotics. InChapter1,thebookdiscussestheuseofin-sensorvisualdevicesforautonomous driving perception. Chapter 2 provides a thorough and up-to-date review of SoTA environmental perception algorithms that are specifically designed for fish-eye cameras. In Chapter 3, the theoretical foundations and algorithms of computer stereo vision are discussed. Chapter 4 presents a review of SoTA single-modal and data-fusion semantic segmentation networks. Chapter 5 reviews 3D object detection methods for autonomous driving. Chapter 6 provides an assessment of the current SoTA collaborative 3D object detection systems and algorithms. In Chapter 7, sensor- fusion robust SLAM techniques for mobile robots are introduced. Chapter 8 discusses visual SLAM in texture-less environments. Chapter 9 presents a comprehensive
- 13. viii Preface survey on multi-task perception frameworks. Chapter 10 specifically covers state-of- the-art BEV perception algorithms. Finally, Chapter 11 discusses road environment perception techniques for safe and comfortable driving. Shanghai, P. R. China Shanghai, P. R. China Bristol, UK Rui Fan Sicen Guo Mohammud Junaid Bocus Acknowledgements This book was supported by the National Key R&D Program of China under Grant2020AAA0108100,theNationalNaturalScienceFoundationofChinaunderGrant62233013, the Science and Technology Commission of Shanghai Municipal under Grant 22511104500, and the Fundamental Research Funds for the Central Universities.
- 14. Contents 1 In-Sensor Visual Devices for Perception and Inference . . . . . . . . . . . . 1 Yanan Liu, Hepeng Ni, Chao Yuwen, Xinyu Yang, Yuhang Ming, Huixin Zhong, Yao Lu, and Liang Ran 2 Environmental Perception Using Fish-Eye Cameras for Autonomous Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Yeqiang Qian, Ming Yang, and John M. Dolan 3 Stereo Matching: Fundamentals, State-of-the-Art, and Existing Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Chuang-Wei Liu, Hengli Wang, Sicen Guo, Mohammud Junaid Bocus, Qijun Chen, and Rui Fan 4 Semantic Segmentation for Autonomous Driving . . . . . . . . . . . . . . . . . 101 Jingwei Yang, Sicen Guo, Mohammud Junaid Bocus, Qijun Chen, and Rui Fan 5 3D Object Detection in Autonomous Driving . . . . . . . . . . . . . . . . . . . . . 139 Peng Yun, Yuxuan Liu, Xiaoyang Yan, Jiahang Li, Jiachen Wang, Lei Tai, Na Jin, Rui Fan, and Ming Liu 6 Collaborative 3D Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Siheng Chen and Yue Hu 7 Enabling Robust SLAM for Mobile Robots with Sensor Fusion . . . . 205 Jianhao Jiao, Xiangcheng Hu, Xupeng Xie, Jin Wu, Hexiang Wei, Lu Fan, and Ming Liu 8 Visual SLAM for Texture-Less Environment . . . . . . . . . . . . . . . . . . . . . 241 Yanchao Dong, Yuhao Liu, and Sixiong Xu 9 Multi-task Perception for Autonomous Driving . . . . . . . . . . . . . . . . . . 281 Xiaodan Liang, Xiwen Liang, and Hang Xu ix
- 15. x Contents 10 Bird’s Eye View Perception for Autonomous Driving . . . . . . . . . . . . . 323 Jiayuan Du, Shuai Su, Rui Fan, and Qijun Chen 11 Road Environment Perception for Safe and Comfortable Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Sicen Guo, Yu Jiang, Jiahang Li, Dacheng Zhou, Shuai Su, Mohammud Junaid Bocus, Xingyi Zhu, Qijun Chen, and Rui Fan
- 16. Chapter 1 In-Sensor Visual Devices for Perception and Inference Yanan Liu, Hepeng Ni, Chao Yuwen, Xinyu Yang, Yuhang Ming, Huixin Zhong, Yao Lu, and Liang Ran Abstract The traditional machine vision systems use separate architectures for per- ception, memory, and processing. This approach may hinder the growing demand for high image processing rates and low power consumption. On the other hand, in-sensor computing performs signal processing at the pixel level, directly utilizing collected analogue signals without sending them to other processors. This means that in-sensor computing may offer a solution for achieving highly efficient and low- power consumption visual signal processing. This can be achieved by integrating Y. Liu School of Microelectronics, Shanghai University, Shanghai, China e-mail: yanan.liu@ieee.org H. Ni (B) School of Mechanical and Electrical Engineering, Shandong Jianzhu University, Jinan, China e-mail: nihepeng20@sdjzu.edu.cn C. Yuwen (B) GAC R&D Center, Guangzhou, China e-mail: yuwenchao@gacrnd.com X. Yang Lancaster University, Bailrigg, Lancaster, UK e-mail: xinyu.yang@ieee.org Y. Ming School of Computer Science, Hangzhou Dianzi University, Hangzhou, China e-mail: yuhang.ming@ieee.org H. Zhong Department of Computer Science, University of Bath, Bath, UK e-mail: hz877@bath.ac.uk Y. Lu School of Computer and Information Security, Guilin University of Electronic Technology, Guilin, China e-mail: sunnyluyao@vip.qq.com L. Ran Pixelcore.ai, Beijing, China e-mail: light.ran@pixelcore.ai © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 R. Fan et al. (eds.), Autonomous Driving Perception, Advances in Computer Vision and Pattern Recognition, https://doi.org/10.1007/978-981-99-4287-9_1 1
- 17. 2 Y. Liu et al. sensing, storage, and computation onto focal planes with novel circuit designs or new materials. This chapter aims to describe the proposed image processing algorithms and neural networks of in-sensor computing, as well as their applications in machine vision and robotics. The goal of this chapter is to help developers, researchers, and users of unconventional visual sensors understand their functioning and applications, especially in the context of autonomous driving. 1.1 Introduction The importance of vision as a means of perception cannot be overstated, as it enables efficient collection and interpretation of information [1]. To apply this capability to fields like machine vision, robotics, Internet of Things (IoT), and artificial intelli- gence (AI), there is a pressing need to develop visual information processing methods and technologies that operate at ultra-high speeds while consuming minimal energy [2, 3]. The conventional machine vision systems and their associated technologies face major constraints in terms of system latency, power consumption, and privacy issues [2, 3]. Unlike the mammalian retina mechanism that rapidly processes raw signals through several layers of cells, the visual signal digitization, storage, and transmission processes involved in conventional machine vision systems can intro- duce significant time latency, which hinders quick responses to dynamic changes and results in inefficiencies due to irrelevant data processing [2, 3]. Additionally, exter- nal image processors like CPU/GPU/VPU/DSPs consume high amounts of power, which is unfavorable for portable tasks [2, 3]. The overwhelming amount of data gen- erated by ubiquitous sensors may obscure the useful information, thus necessitating the extraction of critical information by terminal sensors to reduce data movement from the sensing chip to processing units [4, 5]. Moreover, privacy-sensitive scenar- ios may require the extraction of crucial information from raw analog signals rather than collected images. To address these challenges, a paradigm shift towards in-sensor computing is pro- posed [6]. This approach is inspired by the mammalian retina (Fig.1.1a) and involves the vision sensor not only acquiring visual information but also processing it to pro- duce highly compressed information instead of video frames (Fig.1.1c). In-sensor computing offers image-free visual signal processing, which ensures data confiden- tiality. This interdisciplinary field encompasses various technologies, including sen- sors, analogue signal processing, near-sensor computing, and in-memory computing (Fig.1.3). In-sensor computing devices are sensors that integrate perception, tem- porary storage, and data processing and analysis with raw analogue signals within the sensing chip. While near-sensor computing can reduce the physical distance between sensing and computing, data movement from sensors to processors is still necessary. In-memory computing uses memristors for both memory and computing [7], utilizing tunable resistance as the synaptic weights.
- 18. 1 In-Sensor Visual Devices for Perception and Inference 3 Retina Cone and rod Pre-processed signals Photoreceptors Biopolar cells Ganglion cells CCD/CMOS ADC Memory units Processing units Load Photoreceptors In-sensor computing chip Photoreceptors Multi-layer register arrays Pre-processed/ processed signals a b c Fig. 1.1 The origin of in-sensor visual computing. a The concept of in-sensor computing is bio-inspired by the retina mechanism where visual signals can be generated and pre-processed by different types of cells [8]. b Conventional machine vision system: light density needs to be read out first and converted to digital data which being loaded into memory and then processing units for meaningful information extraction. c Visual data can be generated, stored, and processed in sensor through the bio-inspired hardware design This chapter firstly illustrates the common in-sensor visual computing hardware architecture. Then various emerging in-sensor computing visual sensors are intro- duced in terms of hardware, software, algorithms, and applications within the cate- gory of in-sensor computing architecture. Finally, a summary and future prospective of in-sensor visual computing technology are made in the conclusion.
- 19. 4 Y. Liu et al. 1.2 In-Sensor Computing Devices 1.2.1 Architecture In recent years, progress has been made in the development of in-sensor computing devices. By far, there are mainly two types of in-sensor computing architectures: (1)In-Sensorarchitecturebyintegratingsensing,memory,andcomputingunits: A Focal-Plane Sensor Processor (FPSP) [9] integrates visual sensing, storage and computing units on the focal plane under the architecture of cellular neural networks (Fig.1.2a). As for each Processing Element (PE), the generated analogue signals from the pixel can be transferred to the temporal memory units through the bus and processed using ALU units and registers. Each PE plays a role as a cell interacting with its neighbours for signal exchange and processing. Hence, the in-sensor visual inference is realised by the hardware cellular neural network and its synaptic weights in memory. The representative devices under the FPFS architecture mainly include the SCAMP Pixel Processor Array (PPA), Q-Eye [10] MIPA4k [9], Asynchronous- Synchronous Focal-Plane Sensor-Processor Chip (ASAP) [9], KOVA1 [11], and Ais- torm Mantis2 [12], where the SCAMP PPA is comparatively mature with continuous research and application outputs. ... ... a b Photoreceptors Processing elements Registor Memory ALU ... ... Ni1 Multi-layer register network y1 y2 y3 yn ... ... N11 N1j Ni1 Nij N11 N1i Wi Wj Si SiO2 Detect-and-memorize materials Electrode Light P1 PN Wij W1 ij Wn ij ... ... P1 PN Pixel Multi-layer neural network y1 y2 y3 yn ... Fig. 1.2 In-sensor perception and computing architectures and their associated artificial net- works. a An in-sensor cellular network can be built with an array of PEs which integrates sensing, memory, and computing units. b A neural network with detect-and-memorise materials
- 20. 1 In-Sensor Visual Devices for Perception and Inference 5 Table 1.1 In-sensor computing architecture comparison between FPFS and DAM scheme Scheme Maturity level Speed Application General- purpose Programmable Efficiency FPFS [15, 16] Mature High-speed Many Yes Yes High DAM [5, 6, 17–19] Immature Ultra-fast Few No Partial Ultra-high Fig. 1.3 The position of in-sensor computing in the existing knowledge S e n s i n g M e m o r y Computing Near-sensor computing In-sensor computing I n - m e m o r y c o m p u t i n g M o r p h o l o g i c a l c o m p u t i n g (2) Detect-and-memorise materials for in-sensor computing architectures: Material-based detect-and-memorise (DAM) devices (Fig.1.2b) have recently been proposed to mimic the functional mechanism of the photonic synapses to implement artificial neural networks [5, 13]. Among these emerging materials and devices, the memristor is representative as it facilitates sensing, temporal memory, and comput- ing capability when combined with other photo-sensitive devices [14]. Specifically, visual signals generated from photoreceptors such as photodiodes can be further pro- cessed within the artificial networks composed of memristors with tunable resistance as the weights. Table1.1 shows the difference between the two rising in-sensor computing archi- tectures. As can be seen from Table1.1, the DAM-based in-sensor computing sensors are new and immature compared to the scheme by sensor, memory, and computing integration. Hence, this chapter mainly reviews devices and algorithms based on the first architecture scheme (Fig.1.3). 1.2.2 Focal-Plane Sensor Processor (FPFS) Conventional sensors mainly play the role of information collectors. In recent years, with the development of techniques on integrated circuit design and the growing need for low-power and lower-latency edge-computing, a sensor has gradually been
- 21. 6 Y. Liu et al. Table 1.2 List of in-sensor and in-memory processing vision systems System names Resolution In-sensor storage Speed (FPS) Aistorm Mantis2 96.×96 Gray-scale image 50K [12] Eye-RIS 176.×144 7 gray + 4 binary 10K [22] Photodiode array – – 20M bins [14] SCAMP PPA 256.×256 7 gray + 13 binary 100K [23] Kovilta’s KOVA1 96.×96 – .>1K [24] MIPA4k 64.×64 – .>1K [25] DVS In-senor pre-processing Binary events .>10K [26] integrated with the ability of signal processing independent from general-purpose computers. The goal of near-sensor processing is to use a dedicated machine learning accelerator chip on the same printed circuit board [20], or perhaps 3D-stacked with the CMOS image chip [21]. Despite the fact that this allows CMOS image chip data to be processed closer to the sensor rather than in the cloud, data transport expenses between the sensing chip and the processing chip still exist. In contrast, the in-sensor computing paradigm aims to embed processing capability for each individual pixel. This section introduces classic in-sensor visual computing devices. Table1.2 lists the differences among these above-mentioned in-sensor computing devices. 1.3 SCAMP-5d Vision System and Pixel Processor Array 1.3.1 Introduction SCAMP vision system is one of the emerging in-sensor visual computing devices. Currently, the most up-to-date version of SCAMP series system is the SCAMP-5d (Figs.1.4 and 1.5) which consists of 256.×256 processing elements (PEs) weighted 171g with a normal lens. SCAMP-5d vision system is a general-purpose pro- grammable massively parallel vision system [23] that was invented, designed, and developed by University of Manchester. By far, SCAMP-5d enjoys many applica- tions in the field of robotics [27–30] and computer vision [31–33]. As for the PPA shown in Figs.1.3 and 1.4, the photo-detector converts light into an analogue sig- nal which can be directly parallelly processed on AREG. Different from the current hardware design structure of computer vision systems, the PPA gets rid of the Ana- logue/Digital Conversion (ADC) after sensing and directly operates on analogue electric current using an arithmetic unit, hence accelerating the signal processing speed and, in the meantime, avoiding the bottleneck of ADC and data transmission process. However, errors can be introduced when performing arithmetic operations or temporal information storage on AREG [6].
- 22. 1 In-Sensor Visual Devices for Perception and Inference 7 Fig. 1.4 SCAMP-5d vision system and the pixel processor array (PPA). SCAMP-5d consists of PPA with 256.×256 Processing Elements (PE) and ARM Micro-controller where parallel image processing is conducted on PPA by directly operating on analogue signal (electric current from PIX, which is proportional to the light intensity) within AREG and bit operation within in DREG. Hence, there is no need for time-consuming and energy-inefficient analogue-to-digital conversion. Bio-inspired by the efficient information processing of neurons connected by synapses, PPA is designed to have highly interconnected PE and registers where information can be shared and accessed adjacently enabling efficient parallel machine vision computing. ARM micro-controller is in charge of sending instructions to the PPA, receiving the processed information from the PPA, and more fully-connected layers for deeper CNN In terms of hardware techniques, the PPA integrates information storage on reg- isters, image processing and analogue information operation (arithmetic operation, shifting, etc.), digital/bit operation, and logical operations. As can be seen from Fig.1.4, for each Processing element (PE), there are seven read/write AREG (A to F) which can be used for signed value storage and computation with basic arith- metic operations, such as addition, subtraction, division, etc. In addition, thirteen 1-bit DREG (R0 to R12) in each PE (256.×256 in total) can execute the Boolean logical operations, such as AND, OR, XNOR, and NOT [15] with information after binary thresholding on AREG. Each register in PE executes identical instructions synchronously under SIMD instructions, hence enabling parallel image processing. In addition, the FLAG register can activate different areas of registers given corre- sponding patterns for more flexible operation. With the neighbour access function where each pixel is able to communicate with its four neighbours (north, west, east, south), an efficient parallel image shifting can be implemented easily. Instructions for thePPAaredispatchedbytheARM-basedmicro-controllerwithaCortexM0running at 204MHz. The analogue operations is executed at 5MHz and digital at 10MHz.
- 23. 8 Y. Liu et al. Fig. 1.5 The development process of SCAMP series vision system from the University of Manch- ester.ThisreviewmainlyfocusesontheSCAMP-3andSCAMP-5visionsystembecausetheirhigher resolution and performance would enable more research and applications. Source Piotr Dudek’s talk in Second International Workshop on Event-based Vision and Smart Cameras (CVPRW) [39] Other I/O functions, such as USB2.0, GPIO, SPI, and UART, are performed on Cortex M4 Core [15]. Notice that other names for a similar type of focal-plane sensor processor can be seen from [9] with names, e.g. ASPA (Asynchronous-synchronous Focal Plane Sensor Processor Array), FPSP (Focal-Plane Sensor-Processor). The PPA is a hardware implementation of Cellular Neural Network (CeNN) with the new optimisation on a mixture of both analogue and digital computing using AREGs and DREGs, respectively. The studies based on the PPA reviewed in this work utilise the parallel nature of the CeNN architecture for efficient and high- performance computing, where each “cell” is intricately connected with its four neighboursandinformationcanbesharedefficiently.Hence,thePPAcanbemodelled as a CeNN architecture for visual information computing. The CeNN processing circuit architecture was first proposed by Leon Chua and Lin Yang [34], followed by the CeNN universal machine [35] as a prototype. After that, as an invention of new circuit architecture and a parallel computing paradigm, it enjoys widespread popularity in academia with a substantial number of research outputs and applications in pattern recognition [36], image processing [37], and biological vision modelling [38]. With above-mentioned hardware features, the SCAMP PPA mainly consists of the following advantages over conventional machine systems. Efficiency and Low Latency: It is obvious to see from Fig.1.1c that in-sensor computing gets rid of signal digitisation, transmission, and storage processes onto external devices, hence enabling high-speed image processing [23] and CNN infer- ence [40] which can be integrated with agile mobile robot platforms [27–30]. In addition, the PE distribution and simultaneous instruction execution on PEs allow
- 24. 1 In-Sensor Visual Devices for Perception and Inference 9 efficient parallel signal processing. Carey et al. [23] demonstrate an object detection with a frame rate of 100,000 per second using the SCAMP vision system and Liu et al. [40] propose binary shallow neural network on the PPA with binary classification problem at up to 17,000 FPS. These works show the efficiency of image processing in a sensor once the parallelism mechanism of the PPA is fully taken advantage of. Low power consumption: According to Fig.1.1c, there are no external process- ing units or data processing needed, hence the power consumption can be saved to a large degree. The maximum power cost of the SCAMP-3 vision system for a complex object tracking and counting system is 29 mW [41]. And the overall power consump- tion on image processing and CNN inference tasks within a SCAMP-5 vision system is lower than 2 W [42]. This feature makes the SCAMP vision system suitable for mobile platforms, usually with short battery life. In addition, according to the power consumption test from work [43], given 8 popular kernel filters, the SCAMP PPA generates the same convolution results with much lower power consumption (.>20 times) at a higher speed (.>100 times) compared to common CPUs and GPUs. Data Security and Privacy Protection: An unique but non-negligible feature of in-sensor analogue computing with the PPA is its inherent feature of data security and privacy protection. Data security is feasible because of the focal-plane analogue information processing without ADC, extra data recording, storage, or intermediate transmission procedures. Usually, the only output after analogue computing is the extracted useful target information without redundant information, which is hardly reversible to get the original data for sensitive information or user re-identification [44]. Data security and privacy protection have become prominent challenges with the emergence of the internet of things. Smart devices such as autonomous vehicles, domestic robots, and smart household appliances are usually equipped with percep- tual sensors and collect data pervasively in public and private spaces, threatening users’ privacy and data security. Conventional sensors usually directly upload raw data to the cloud for data processing [45], which can be a fault line of data security. When data is processed manually or the network is attacked, crucially sensitive data can be directly obtained. The acquired data can then be applied to determine individ- uals’ habits (e.g., motion sensors) or to conduct surveillance (e.g., facial recognition systems), which can cause significant violations of EU GENERAL DATA PROTEC- TION REGULATION Article 25.1 In sensor computing first enables only valuable information to be extracted as it’s output, without redundant information. Moreover, the minimised raw data are further mocked by the analog signals, which leads to re-identification almost impossible. Hence, users’ privacy can be strictly protected with in-sensor processing mechanisms. 1 https://gdpr-info.eu/art-25-gdpr/.
- 25. 10 Y. Liu et al. 1.3.2 Algorithms and Applications This section mainly illustrates algorithms and their potential applications (Fig.1.7) based on the SCAMP-5 vision systems (Table1.3 and Fig.1.6). 1.3.2.1 Image Enhancement Image enhancement comes along with the imaging process on the PPA compared to the conventional image enhancement which only happens after the image data is captured. Later, other methods are exploited on different image processing tasks. For example, Wang et al. [67] proposed a simple coarse grain mapping method to process bigger images than the PPA resolution itself by temporarily storing sub-images into different registers. Table 1.3 List of main studies with the SCAMP PPA Image processing methods Applications References Background extraction Segmentation [46, 47] Contour extraction Object detection [48–50] Skeleton extraction Shape simplification [51–53] HDR Image enhancement [31, 48, 54, 55] Feature corner/edge extraction Edge/feature-based VO [33] Target detection/localisation High-speed object tracking [23, 56, 57] Neural network High-level inference [40, 42, 58–61] Depth estimation/visual odometry Robot navigation [29, 30, 32, 60, 62–65] Automatic code generation Neural network inference, face detection [43, 66] Fig. 1.6 Examples of two images with(left)/without(right) HDR algorithms towards the same scene in an outdoor environment
- 26. 1 In-Sensor Visual Devices for Perception and Inference 11 2006 Dudek et.al. Real-time image processing (SCAMP-3) 2008 Lopez Vilarino et. al. Moving object segmentation (SCAMP-3) 2009 Lopich et. al. Skeletonization Algorithm (SCAMP-3) 2013 Carey et. al. 100,000 fps vision sensor (SCAMP-5) 2016 Martel et. al. HDR (SCAMP-5) 2017 Chen et. al. feature extraction (SCAMP-5) Greatwood et. al. Tracking (SCAMP-5) Bose et. al. Visual Odometry (SCAMP-5) 2018 Chen et. al. SCAMP5d development framework (SCAMP-5) 2019 Bose et. al. CNN (Dreg) (SCAMP-5) Greatwood et. al. Drone racing (SCAMP-5) 2020 Martel et. al. Learning Pixel Exposures (SCAMP-5) Bose et. al.CNN (Areg) (SCAMP-5) Liu et. al. High-speed CNN (SCAMP-5) Liu et. al. Agile Navigation (SCAMP-5) 2021 Castillo et. al. Mapping & Localisation Martel et. al. Depth estimation (SCAMP-5) Murai et. al. BIT-VO (SCAMP-5) Stow et. al. Cain (Automatic Code Generation) (SCAMP-5) Liu et. al. sensory-motor (SCAMP-5) Debrunner et al. (SCAMP-5) AUKE (Automatic Kernel Code Generation) 2022 Bose et. al. Gaze estimation So et.al. HDR imaging with in-pixel irradiance encoding Liu et.al. Model swapping Fig. 1.7 Milestones SCAMP PPA-based work and key SCAMP PPA studies and applications during last 16 years
- 27. 12 Y. Liu et al. HDR (Fig.1.6) is a basic low-level image pre-processing method to obtain rich image information even facing extreme lighting conditions, such as the mixture of dim and strong light intensity. However, conventional image sensors rely on either a global or rolling shutter to form an image, which limits the efficiency of HDR imaging [31, 68]. Back in 2006, Dudek [48] proposed sensor-level adaptive sensing and image processing with SCAMP-3 [41, 69], where different exposure settings are combined for an image with a high dynamic range. Martel et al. [55] make significant contributions in this area using the PPA. The first HDR image generation in-sensor is from [54] where pixel-wise exposure can be controlled to generate HDR images, followed by automotive applications [70]. Furthermore, Bose et al. [32] take advantageoftheHDRimagetoextractedgesastherobustinputinformationforvisual odometry estimation. However, the usage of iterative exposure for different regions of the image slows down the image pre-processing. To speed up the HDR imaging, Martel et al. [31] propose the learning shutter function for PEs to expose each pixel independently with an end-to-end training strategy. They obtain an exposure function by training a U-Net neural network and compiling these trained functions on the sensor for inference. Later, So et al. [71] presented in-pixel control for snapshot HDR imaging with irradiance encoding. 1.3.2.2 Contour and Skeleton Extraction Contours are important features for objects within an image, which can help to identify different entities. Contour extraction algorithms were proposed based on a pixel-level snake with very low latency [49]. In 2007, Alonso-Montes et al. proposed the in-sensor automatic retinal vessel tree extraction based on the Cellular Neural Networks [50]. The shared key concept for these works [48–50] is to extract contour iteratively based on the active contour model and Cellular Neural Networks. In 2008, [72] proposed an image pre-processing method based on the cellular automata for a robotic scenario. The skeleton within a binary image shows the object size, position, and simplified shape. Fast image skeletonization [51] is implemented by [52] based on the wave-trigger propagation/collision. Examples of image contour and skeleton extraction based on the SCAMP PPA can be seen in Fig.1.8. Fig. 1.8 Examples of image contour and skeleton extraction using SCAMP PPA. Left: Extracted Retinal Vascular Tree, Figure from [50]. Right: Extracted skeletons, Figure from [47]
- 28. 1 In-Sensor Visual Devices for Perception and Inference 13 Other Feature Extraction Methods: Other image processing methods, such as background extraction, is exploited by Wang et al. [46, 47]. For higher-level feature extraction, the edge feature can be obtained by deploying Sobel kernel filters or Laplacian filters, which are used in the later work for focal plane visual odometry [32] and neural networks [60]. As for other features, such as corner points extraction, Chen [33] utilised the DREG for corner points extraction based on the FAST16 algorithm, which is used in later work on visual odometry [65]. Based on the above- mentioned low- and mid-level image processing methods, researchers are motivated to exploit more general high-level image processing with up-to-date techniques by taking advantage of the earlier milestone work and the state-of-art progress, such as neural networks which would be illustrated in Sect.1.3.2.4. 1.3.2.3 In-Sensor Visual Feature Extraction for Robots Two major constraints that preclude mobile robots from long-term and diverse appli- cations are their short battery life and limited load. Emerging sensors may hold the key to solving this challenge due to their unique low-level hardware design. The portable SCAMP-5d vision system (171g including the lens) can perform spatial AI processing in-sensor, reducing data transfer pressure between sensing and the main processor, hence increasing overall processing efficiency while maintaining low power consumption [73]. (a) SCAMP PPA on a Quadrocopter The SCAMP-5d vision system has been integrated into quadrocopter systems for target tracking, visual odometry and racing. Greatwood et al. perform various exper- iments by integrating a SCAMP-5d vision system and a quadrotor [28, 29, 74]. Figure1.9 shows a flight control system in terms of hardware integration and control block diagram, where a pre-set target can be tracked with extracted useful information on sensor even facing short periods of target tracking loss [28]. In this application, the direct in-sensor target position extraction releases the pressure of image capturing, transmission and processing for the whole system. Later, Greatwood et al. proposed the in-sensor visual odometry using perspective correction on an agile micro air Fig. 1.9 Left: The integration of a quadrotor and SCAMP-5 vision system for object tracking. Right: a diagram of system hardware (Figure from [28])
- 29. 14 Y. Liu et al. Fig. 1.10 Quadrotor setup for the drone racing with a front-facing SCAMP (Figure from [74]) vehicle based on a similar hardware platform. After that, a drone racing Fig.1.10 within a pre-set environment is demonstrated by taking advantage of the efficient image processing ability on the PPA [74], where the target position can be estimated at around 500 FPS. McConville et al. [30] apply the in-sensor visual odometry devel- oped by Bose et al. [32] on an unmanned aerial system for real-time control purposes. (b) SCAMP PPA for Mobile Robot Reactive Navigation In terms of navigation with a SCAMP PPA, Liu et al. [27] proposed reactive agile navigation on a non-holonomic ground vehicle using PPA by robustly recognis- ing pre-set patterns out of complex environment background. Although being very efficient and accurate, using a pre-set fixed pattern for target tracking is difficult to expand in the generalised environment where there are usually random features. With this in mind, Chen et al. [60] use in-focal plane feature extraction from the environ- ment to perform a recurrent neural network on the M4 micro-controller using this extracted information to estimate the proximity to the ambient objects for obstacle avoidance purposes. A similar pattern of concentric circles was employed in [27, 28, 30] to effectively extract the dot centre in the circles out of the complex environment (Fig.1.11). (c) In-Sensor Computing for Mapping and Localisation Mapping and localisation are useful techniques for robot navigation. In-sensor map- ping and localisation are lightweight and low power cost solutions for mobile plat- forms. Castillo-Elizalde et al. [75] for the first time proposed 1-D mapping and localisation technique. For this method, features are firstly extracted as the database from a sequence of images. Then, the incoming image can be localised by comparing with the database and the prior knowledge of the motion model. In their work, two methods were utilised to down-sample the original images: direct resizing and local binary pattern to apply them to different localisation situations.
- 30. 1 In-Sensor Visual Devices for Perception and Inference 15 a b c Fig. 1.11 Tracking pattern for the drone and ground vehicle. a Tracking a ground vehicle [28], b Tracking a moving target while performing a visual odometry [30], c Tracking a fixed pattern with a mobile ground vehicle [27] (d) Pose and Depth Estimation Fordecades,egocentricstateestimationhasbeenstudiedusingconventionalcameras, emergingDVSdevices,andCPU/GPUs.Inrecentyears,therehavebeensomestudies utilising SCAMP PPA. For example, Bose et al. [32] for the first time, proposed in- sensor4Degree-of-Freedom(DoF)visualodometrywhollyonthesensorbymapping the real-time input image with the previous keyframe through image scaling, shifting, rotation and alignment. They demonstrate the visual odometry estimation at over 1000 Hz with around 2 W power cost. Debrunner et al. [76] use the SCAMP to estimate its global motion with the tiling method at 60 FPS with a low power cost of 100.2 mW. After that, Murai [65] proposed 6 DoF visual odometry based on edge and corner points extracted on sensor and post-processing on a computer with a frame rate of 300 FPS. They take advantage of feature edge, and corner extraction methods [33] and calculate the visual odometry off sensor using a similar strategy with the standard Visual Odometry (VO) systems [77]. Although they combine in- sensor feature extraction and ready-to-use VO computing method off the sensor, it is promising to be a direction in the future to combine the efficient image pre-processing in-sensor and high-volume post-processing with a powerful CPU/GPU, especially when facing storage shortage and general calculation resources for the large-scale computing. In addition, the SCAMP vision system can also work with other accessories to share the computation burden for more applications. For example, Martel et al. [62– 64] mounted a controllable liquid lens to generate a semi-dense map in real-time, which is the first work on depth estimation to take advantage of external physical accessories. With this focus-tunable lens, a vast amount of computation pressure on the sensor is relieved. This in-sensor feature extraction and post-image processing on controller scheme are also widely used in many different applications [60, 65], where the task requirement of storage and computing resources is out of the capacity of the PPA.
- 31. 16 Y. Liu et al. 1.3.2.4 Research Progress on Neural Networks with a SCAMP PPA The algorithms of SCAMP PPA proposed earlier mainly focus on low-level image processing and/or machine vision methods to enhance image quality and extract basic textures with combinations of inherent built-in functions based on SCAMP-3 and SCAMP-5 with a PE resolution of 128.×128 and 256.×256, respectively. It should be noticed that these developed image processing methods are deeply related to the hardware design of the SCAMP vision system. For example, common methods used in this period are cellular-based algorithms, including cellular neural networks, because the SCAMP PPA itself is a cellular processor array. Research on neural network inference with the SCAMP PPA has been active in recent years. Table1.4 lists the main research work in the area of neural networks, which covers fully convolutional neural networks and binary convolutional neural networks using DREG or AREG with various datasets and applications. High-level image processing, such as object classification, localisation and segmentation in sensor, is achieved with the neural network. The deployment of neural networks onto the PPA is a breakthrough since it enables the PPA open to more possibilities with universal methods, which is unlike the conventional development methods with some combinations of low-level image processing methods for specific tasks. With the use of CNN, several types of tasks, such as classification, regression, localisation, Table 1.4 Different convolutional neural network implementation with SCAMP and performance comparison. Network Filter number Layers (Conv+FC) Dataset Accuracy Frame rate (fps) In-sensor/ Near-sensor Bose [42] 16 5.×5 1 + 1 MNIST .≈94.2% 210 1Conv/1FC Bose [58] 16/64 4.×4 1 + 1 MNIST .≈93% .>3000 In sensor – 16+16 4.×4 2 + 1 MNIST 92–94% 224 In sensor Liu [40] 64 4.×4 1 + 1 8 Plankton 80.5 % 4016 In sensor – 16 4.×4 1 + 2 8 Hand gestures .<98.7% 2092 1Conv+1FC/1FC – 16 4.×4 1 + 1 Roshambo .<97.73% 8264 In sensor – 64 4.×4 1 + 1 0,1 in MNIST .<99.1% 17543 In sensor Liu FCN [59] 16+64 4.×4 64 1.×1 3 + 0 Simulation – 283 In sensor Liu [78] 16+64 4.×4 2 + 2 EMNIST .<86.74% 178 2Conv+1FC/ Binarized CNN 1FC Chen [60] – 0 + N Collected indoor – – Near sensor AnalogNet [61] 3 3.×3 1 + 3 MNIST 96.9% 2260 1Conv/2FC
- 32. 1 In-Sensor Visual Devices for Perception and Inference 17 and segmentation, can be feasible, hence enabling more applications. Table1.4 shows the neural network-related work based on the SCAMP PPA vision system. The research on CNN implementation and inference within PPA is pioneered by Bose et al. [79] where a CNN with a single convolutional layer performed upon the PPA array and a fully-connected layer upon its controller chip (M0). They per- formed 16-bit image convolution operations using 4.×4 DREG “Super Pixel” blocks and demonstrated live digit classification based on MNIST dataset at around 200 FPS. In their work, the ternary {.−1, 0, 1} kernel filters are stored on the flash (M4) of the PPA system, and are effectively encoded in the instructions/operation sent to the PPA array, performing convolutions sequentially. Furthermore, a mobile car locali- sation task is then explored using synthetic datasets, where the pre-processed edge information is mainly the clues for network inference. Notice that the localisation is realised by classifying the car’s position along the.x and.y axis, respectively. To fully take advantage of PPA’s parallel computing characteristics and to further improve the CNN inference efficiency, Bose et al. [58], for the first time, proposed the idea of in-pixel weight storage, where the network’s weights are directly stored within the registers of the PPA’s PEs. This method enabled both parallel computations of multi- ple convolutions, and implementation of a fully connected layer upon the PPA array, resulting in a.×22 faster CNN inference (4464 FPS) on the same digit recognition task. Based on these two works, [40] further proposes a high-speed lightweight neural network using BinaryConnect [80] with a new method for computing convolutions upon the PPA, allowing for varying convolutional strides. This work demonstrated four different classification tasks with frame rates ranging from 2,000 to 17,500 per second with different stride setups. Later, based on this network, a direct servo con- trol using CNN results [81] and a simulated robot tracking from a drone [82] with in-sensor CNN computing results are exploited. In addition, the AnalogNet2 [61, 83] extends the earlier work in [84], implementing a CNN which reaches 96.9% accuracy on the MNIST dataset at a speed of 2260 fps. However, their method requires all fully connected layers to be performed externally to the PPA array with only 3 con- volutional kernel filters implemented in sequence on the PPA as the first layer. More kernel filters would significantly slow down the inference process. Notice that, in our work [60], a recurrent neural network is implemented on the micro-controller with features extracted on a sensor. In this manner, the fully-connected layer of a neural network can be deployed similarly with conventional embedded devices. It is notable that Martel et al. trained a neural network of exposure time for each individual pixel off the sensor for HDR imaging and video compressive sensing [31]. Furthermore, work [78] binarized CNN with batch norm both for classification and coarse segmentation. To deal with the classifications application with more labels and more segmentation tasks, they propose the idea of dynamic model swapping by uploading weights of trained models in sequence or according to the last inference result, targeting multiple sub-tasks decomposed from a more sophisticated task. They then demonstrate a servo control directly using the CNN inference results [81], which potentially indicates that motion control platforms, such as a ground vehicle or drones can have a light-weight servo control system without using external control units in the future.
- 33. 18 Y. Liu et al. Notice that the preceding neural network-related work mainly focuses on clas- sification or classification-based localisation, both of which require fully connected layers. However, the parameters in fully-connected layers are typically substantially larger than those in convolutional layers due to the dense connections of each indi- vidual neuron. Thus, work developed a fully-convolutional neural network (FCN) [59], not only presenting in-sensor image segmentation and localisation but also eliminating dense layers for a smaller memory footprint [85]. 1.3.2.5 In-Sensor Cellular Automata The PPA itself is a cellular neural network architecture where each ‘cell’ is closely connected with its four neighbours, hence information can be shared efficiently. With this in mind, the author is inspired to explore the possibility to perform cellular behaviour, such as Conway’s game of life (demonstration shown from2 ) and ele- mentary cellular automata (demonstration Rule 90 seen from3 ) based on the theory of cellular automata [86]. With the rule of the game of life, all ‘cells’ can update their states (alive or dead) in-sensor as fast as 53.µs for each iteration based on the bit-operation with DREG. As can be seen from Fig.1.12, a Sierpiński triangle is effi- ciently generated based on bit operation on the sensor with 730.µs of 255 iterations to fill the whole chip. (a) Elementary CA One-dimensional CA is one of the simple CA algorithms. Some classical updating rules, such as Rule 30, Rule 90, and Rule 110, can be implemented by logic bit Fig. 1.12 Our demonstration of elementary cellular automata with Rule 90 on the SCAMP PPA. This pattern is generated from top to bottom. We have made this project available from https:// github.com/yananliusdu/1D_CellularAutomata 2 https://youtu.be/X_t4c3f-T4s. 3 https://youtu.be/HgPvoK5EJ_s.
- 34. 1 In-Sensor Visual Devices for Perception and Inference 19 operations. Let use L,C,R as the three continuous grids and C1 is the grid that should be updated, then rules can be represented as follows: R30: C1 = L XOR (C OR R ) R90: C1 = XOR(L, R) R110: C1 = XOR(OR(R, C), AND(R, C, L)) The pseudo codes for R90 can be shown as follows: begin{code}{ElementaryCellularAutomataR90} for(int i = 1; i <= iterations; i++) { scamp5_kernel_begin(); MOV(R5,R6); //R5 = R6 DNEWS(R0,R6,west); //R0 = shift R6 to right for one step DNEWS(R6,R0,west); XOR(R7,R5,R6); // R7 = XOR(R5,R6) DNEWS(R0,R7,north); DNEWS(R6,R0,east); OR(R6,R5); R6 = OR(R6,R5) scamp5_kernel_end(); } end{code} Implementation for R30 and R110 can also be done similarly with R90 using corresponding unit logic operations. (b) In-Sensor Conway’s Game of Life based on the CA The rule of Game of Life is a new independent non-linear computing scheme. Figure1.13 shows an updating 3.×3 block to illustrate the rule and implementa- tion using a cellular neural network.The pseudo code for the Game of Life can be represented as follows according to its definition: if B0 = 1 & SUM(B1,..,B8) <= 2 B0 = 0 If else B0 = 0 & SUM(B1,..,B8) == 3 B0 = 1 else B0 = 0 where.Bi ∈ {0, 1}, i = {0, 1, 2, . . . , 8}. This chapter tries to implement and run the Game of Life using the DREG of SCAMP-5 vision systems and its parallel nature. We use three DREGs 3-bit
- 35. 20 Y. Liu et al. Fig. 1.13 The update block for the Game of Life B1 B2 B3 B4 B5 B6 B7 B8 B0 .R6, R7, R8 to count the number of ‘1’s around the pivot.B0..R5 is the source binary image in this illustration..R1, R2, R3, R4 and.R9, R10 are DREGs to temporarily store intermediate states. Neighbour Number Counting As the first step of the 2D cellular automata, the number of live neighbours should be counted. As shown in Fig.1.14a,.R5 is the source binary image..R6, R7, R8 are 3-bits to represent the number of live neighbours of R5 in the corresponding posi- tion. For example, if .R5(B0) has 2 live neighbours, then .R6(B0) = 0, R7(B0) = 1, R8(B0) = 0. In conclusion, binary digits.R6, R7, R8 are used to record the num- ber of live neighbours of corresponding cell. Figure1.14b shows for each counting step (8 in total around a pivot,.R6, R7, R8 are updated according to the states of each cell. State Update for Cells With live neighbour information stored in three DREGs, cells’ state can be updated according to the rule of the Game of Life as described in the aforementioned pseudo code. We make the codes of Game of Life on the SCAMP-5 vision system avail- able from https://github.com/yananliusdu/GameofLife. In the future, more image Fig. 1.14 Neighbour number counting using DREGs R6 B0 R8 B0 R7 B0 R5 B0 B0 (a) (b)
- 36. 1 In-Sensor Visual Devices for Perception and Inference 21 processing-related work can be potentially explored as long as proper update rules and associated steps are trained with neural network methods [37, 87]. 1.4 Eye-RIS 1.4.1 Introduction Eye-RIS [22, 88] commercial vision system on Chip (VSoC) extends CMOS pixel functionality with image storage (7 gray-scale images and 4 binary images) and dig- ital/analogue signal processing ability. Specifically, a 32-bit RISC (Reduced Instruc- tion Set Computer) is integrated with a vision sensor for image post-processing after the parallel in-sensor pre-processing (Fig.1.15). The resolution of the Eye-RIS vision sensor is 176.×144. Notice that Eye-RIS’s overall functional diagram is similar to that in the SCAMP vision system, where the counterpart of RISC is the M0 micro- controller in the SCAMP PPA [89]. The most significant difference, though, is that the Eye-RIS has a DICop part, which is a digital image co-processor that handles geometric transformations and can send the results back to the pixel level for more processing. The Eye-RIS Vision System on Chip (VSoC) is an autonomous device combining a parallel CMOS image sensor processor with 32-bit RISC microprocessor perform- ing post-processing and system control tasks, several I/O and high-speed communi- cation ports that allow the system to communicate and/or to control external systems, and on chip memory. The combination of massive parallel image pre-processing in the sensor with complex image post-processing in the microprocessor results in ultra compact implementation of a vision system able to perform complex machine vision algorithms at speeds of several thousands of images per second. The Eye-RIS VSoC Image acquisition In-pixel image processing A/D In-pixel image memory D/A On-chip image memory Image post- processing On-chip data and program memory Serial flash memory External image memory External program and data memory I/O and communications Eye-RIS v2.1 VSoC SIS Q-Eye Sensor-processor Nios II RISC Microprocessor Eye-RIS v2.1 Vision system Fig. 1.15 Eye-RIS v2.1 VSoC architecture
- 37. 22 Y. Liu et al. features a complete application development software environment allowing easy control of the device and is optimized for industrial applications requiring image sensing, image processing, and decision making at extreme frame rates. 1.4.2 Applications The applications with Eye-RIS consist of automotive, machine vision, security, games and battery powered products. Caballero-Garcia and Jimenez-Marrufo [90] proposed a series of techniques to deploy image processing algorithms on the Eye- RIS Vision System on Chip (VSoC) for various applications. One unique character- istic using the Eye-RIS vision platform compared to conventional visual sensors is the simultaneous image acquisition and early processing in the analogue domain. Paper [91] aims to describe how the AnaFocus’ Eye-RIS family of vision sys- tems has been successfully embedded within the roving robots developed under the framework of SPARK and SPARK II European projects to solve the action-oriented perception problem in real time. With the ability of low power cost and efficient par- allel image processing, Eye-RIS has been equipped to many different mobile robot platforms, such as Rover II wheeled robot and Gregor III hexapod robot. Visual homing, object tracking, and navigation using landmarks are demonstrated based on the robot platforms and in-sensor real-time image processing algorithms (Fig.1.16). Optical flow based on the Lucas and Kanade by Guzman et al. [92] (Fig.1.17) is implemented on the Eye-RIS platform taking advantage of both analogue and digital signals processing using Q-Eye and Nios II RISC respectively (Fig.1.15). In the experiment, the optical flow estimation reaches over 25 fps which can be used in the area of robotics in real time. Specifically, the optical flow constraint equation: .uIx + vIy + It = 0 (1.1) Fig. 1.16 Eye-RIS VSoC equipped onto robot platforms. a Rover II wheeled robot, b Gregor III hexapod robot (Figure from [91])
- 38. 1 In-Sensor Visual Devices for Perception and Inference 23 Fig. 1.17 Optical flow estimation in traffic sequences using Eye-RIS VSoC architecture (Figure from [92]) where the partial derivatives of.I are denoted by subscripts, which can be obtained from the image..u and.v are the.x and.y components of the optical flow vector, which are the optical flow vectors that need to be found out. Lucas–Kanade method [93] is a classical method to deal with the optical flow constraint problem. According to the assumption of Lucas–Kanade method, given a small pixel block, 3.×3 for example, the optical flow remains identical within this small block. Then, we can have following equation group: . uIx1 + vIy1 = −It1 uIx2 + vIy2 = −It2 . . . uIxn + vIyn = −Itn (1.2) Equation1.2 can then be represented as . ⎡ ⎢ ⎢ ⎣ Ix1 Iy1 Ix2 Iy2 . . . Ixn Iyn ⎤ ⎥ ⎥ ⎦ [ u v ] = ⎡ ⎢ ⎢ ⎣ −It1 −It2 . . . −Itn ⎤ ⎥ ⎥ ⎦ (1.3) we assign Eq.1.3 as . A − → V = −b, here the least squares method can be used to get optical flow vector. − → V , then. AT A − → V = AT (−b), hence, the optical flow vector can be obtained through: . − → V = (AT A)−1 AT (−b) (1.4) In detail, Eq.1.4 can be shown as: . [ u v ] = [ ∑n i=1 I2 xi ∑n i=1 Ixi Iyi ∑n i=1 Ixi Iyi ∑n i=1 I2 yi ]−1 [ − ∑n i=1 Ixi Iti − ∑n i=1 Iyi Iti ] (1.5)
- 39. 24 Y. Liu et al. Shift along x axis A A_x Ix = A-A_x Shift along y axis A A_y Iy = A-A_y (a) A (b) A_1 It = A-A_1 (c) Fig. 1.18 The parallel calculation of.Ix , Iy, It in the focal plane using analogue signals Through Eq.1.5, the optical flow vector can be estimated through a sequence of images. These intensity derivatives including.Ix , Iy, It can be efficiently obtained by shifting and subtracting between frame sequences as illustrated from Fig.1.18. After that, the optical vector can then be calculated using conventional computing units. As for the implementation of optical flow using the above-mentioned formulations, work [22] takes advantage of both in-sensor pre-processing and post-processing with computing units achieving up to 28.9 fps. In the study of [94], authors explored different methods of point tracking on the platform of Eye-RIS, which is able to equip Unmanned Arial Vehicles (UAVs) with the ability of on-board sensing and computing with low load and power consumption.
- 40. 1 In-Sensor Visual Devices for Perception and Inference 25 Nicolosi et al. [95] applied the Eye-RIS vision platform to control welding process byusingthecellular neural networkbasedvisual algorithms. [96] extendedtheir work onto vision based closed loop control for partial penetration welding of overlap joints. Work [97] proposed algorithms for the cellular neural network to detect rapidly approaching object which is bio-inspired by mammalian retina that consists of loom- ing sensitive circuit based on local interaction of cells. 1.5 Kovilta’s KOVA1 1.5.1 Introduction The KOvilta Vision Array (KOVA1) (depicted in [11]) employs a meticulously designed pixel-level processing circuitry that utilizes an efficient combination of analogue and digital (mixed-mode) computation techniques to execute a diverse range of pre-programmed operations. These operations include automated sensor adaption, grayscale filtering, segmentation, and complex object-level visual analy- sis. The selection of operations to be performed at the pixel-level hardware can be customized based on the specific requirements of the application, thereby enhancing the overall implementation efficiency. The KOVA1 is Kovilta’s inaugural silicon rendition of the KOvilta Vision Array structure, and it comprises a 96.×96 pixel focal-plane processor array fabricated using 180nm CMOS technology. The sensor-processor chip is integrated into a miniature smart camera system equipped with FPGA-based control and Ethernet I/O. In this focal plane processor architecture, each pixel-cell of the sensor array includes a reconfigurable processing element that operates directly on the output of the analog photodiode. This allows real-time data compression for capturing images with high dynamic ranges without compromising quality. Additionally, processing in parallel on the pixel plane enables rapid low-level feature analysis and eliminates the need for time and energy-consuming long-distance data transfers from the sensor to an external processor. The sensor output may include only the essential feature data, such as the presence of an object or a set of object coordinates or features, thereby reducing the amount of hardware needed for further external image content analysis. The KOVA1 camera system employs pixel cells that are connected within their immediate neighbourhood, allowing for direct information exchange during image analysis activities at the sensor level. Moreover, local memories integrated at the pixel level facilitate the storage of multiple full images or intermediate processing outcomes on the sensor plane. An FPGA chip is utilized to manage the program execution and I/O of the sensor-processor chip in the KOVA1 embedded camera system. As the control and I/O structures consume only a small fraction of the FPGA’s resources, supplementary visual analysis operations can be implemented on the FPGA to enhance the on-chip sensor-level processing. The KEDE software
- 41. 26 Y. Liu et al. environment from Kovilta is used to program and manage the camera, with program- ming and data I/O facilitated by an Ethernet connection, while direct CMOS-signal outputs bypass the network interface. Additionally, the camera’s internal memory is capable of storing program code. It can operate autonomously without the need for a PC connection and transmit output data to an Ethernet-connected or directly controlled device. 1.5.2 Applications Line detection: Santti et al. [98] proposed a line detection method by combining KOVA1 and FPGA for low-level and mid-level feature extraction respectively. This line detection method can be applied for industrial inspection and control applications with its performance of high-speed processing and low power consumption. Seam and Spatter Tracking: In Lahdenoja et al. [24, 99] and Santti et al.’s work [100], KOVA1 is utilised for seam tracking for real-time robot path optimisation during a high power laser welding process. The reason to use this type of pixel-level sensor lies in the ability to control pixel-wise exposure periods facing significant intensity differences in a laser welding task. Hence, the effective binary feature points can be extracted on the focal plane and then this compressed information is sent to the FPGA for laser beam location extraction using Hough transform. With a frame rate over 1000 fps, the combination of in-sensor image pre-processing and FPGA-based Hough operation enables a real-time optical seam tracking for robot control. In addition, spatter can also be tracked in laser and manual arc welding [101] in extreme radiated light intensity conditions (Fig.1.19). Fig. 1.19 Straight line extraction process using KOVA1. Left to right: figure captured with KOVA1, binary feature extraction using pixel-level process, estimated beam line with binary features using FPGA. Figure from [100]
- 42. 1 In-Sensor Visual Devices for Perception and Inference 27 1.6 Aistorm Mantis2 Mantis system is based on the event-driven charge domain for analogue signal pro- cessing without digitisation and provides an “always on” solution for analogue sig- nal processing. One of the key features claimed by Aistorm is the noise cancelling techniques associated with the analogue signals. In addition, artificial intelligence can also be integrated into a chip for various applications. A 96.×96 AI-in-Sensor machine learning SoC that is particularly well suited for classification jobs is the AIS-C100A Mants. It has a robust post wake-up ML capabilities as well as a con- figurable “always on” wake-up CNN engine that can be used to activate external micro-controllers when an object of interest is recognised. On a single monolithic device with the fewest possible external components, all necessary supporting cir- cuitry is provided, including power management, timing, artificial intelligence, and communications in addition to a (up to) 40mA LED driver enabling both linear and PWM control. An SPI port is used for communication. The on-board camera’s photos and videos as well as those transferred via the SPI connection can both be processed by Mantis. To provide the best contrast for AI calculations, the exposure time can be either internally or externally regulated. The AIS-C100A is housed in an OLGA package of 6.4.×6.4mm. The AI-in-Imager solutions that can directly take pixel data in its native charge form are AIStorm’s Mantis Family of AI-in-Imager processors. The end result is the only method in the world that can wake up a person, face, object, or action based on an image (Fig.1.20). There are several businesses that provide analogue AI solutions, but fundamental physical noise and bandwidth constraints prevent these products from being success- ful. The approach used by AIStorm is charge domain processing. This technology uses charge to create AI-in-Sensor processing opportunities for picture or audio improvement that simply cannot be realised through any other means. Other ana- logue methods’ noise and bandwidth restrictions are solved through charge domain processing, a revolution in processing. Over both analogue and digital solutions, charge domain solutions are preferable because they can immediately take IoT sen- sor data without the expense, power, or delay of digitization. The major applications proposed using the Mantis AI-in-Sensor chip cover motion tracking, gesture clas- Fig. 1.20 AIS technology SKIPS digitization and moves directly to processing Sensor Shared analog to digital converters Machine learning
- 43. 28 Y. Liu et al. sification, smart home, wearable devices (glasses, headsets), household appliances, gaming devices, automotive, and voice processes as claimed in their application website https://aistorm.ai/applications/. 1.7 Other In-Sensor Computing Devices MIPA4k: MIPA4k [25] is an earlier sensory-level image processing sensor with similar architecture to KOVA1. A number of algorithms can be carried out based on the MIPA4k with the use of mixed signals. These algorithms mainly cover edge detection [102], local binary pattern [103], locally adaptive image sensing [104], space-dependent binary image processing [105], and object segmentation and track- ing [106]. Similar early cellular vision chips also include ACE16K [107]. Memristor-Based Devices: Yao et al. [108] Memristor-based hardware is a plat- form to deploy the neural network using the programmable resistance within the integrated circuits mimicking the synaptic connections in a human brain [17, 108– 111]. However, it integrates only storage and processing functions, which can be regarded as in-memory computing. Hence, signals should be input from sensors or other storage devices. They are thus usually integrated with other sensory systems for information processing. Lee et al. [19] take advantage of photo-diode and memristor crossbar for primary visual information process aiming to extract useful information from the input images. In-Sensor visual computing with memristor or new materials are currently not mature enough to support various practical applications for machine vision tasks. By far, the is few practical applications using memristor in-sensor com- puting devices. Dynamic Vision Sensor (DVS): inivation [26] DVS produces data in the form of sparse contrast-change events that facilitate low-latency visual processing using external computational hardware [112–114]. These binary events are generated from in-sensor processing according to the brightness changes. Although the pixels in a DVS have a primitive in-sensor processing ability by binarising brightness changes, it achieves an ultra-high-speed response to the environment when working with external hardware computing units, enabling an enormous potential for robotics and computer vision in a challenging environment [115]. Other Emerging Sensor Devices: Mennel et al. [14] use a 2D semiconductor (.W Se2) photodiode array as the vision sensor, the photoresponsivity matrix to store the connecting weights of the neural network, where both supervised and unsuper- vised learning for classification are present. However, laser light and a set of optical systems are needed to project images onto the chip, which prevents it from having practical usage. Song et al. [116] proposed a CMOS-based PIP (Processing-in-Pixel) architecture where image convolution (8-bit weight configuration) can be performed as the image preprocessing before image data is read out. In addition, Datta et al. proposed the Processing-in-Pixel-in-memory paradigm where the first few convolu- tional layers of a CNN can be processed and the compressed data then sent to other near-sensor processors [117].
- 44. 1 In-Sensor Visual Devices for Perception and Inference 29 1.8 Conclusion In-sensor visual computing is an emerging technology that enables efficient extrac- tion of information solely on the sensing chip of the sensory system, facilitating signal collection, storage, and processing. Vision is the primary sense for human beings, and it accounts for a vast majority of information acquisition worldwide. Despite significant advances in computer vision using conventional camera vision systems and associated methodologies, various limitations, such as system latency and power consumption, persist. Image digitisation, storage, and transmission intro- duce latency, which is a bottleneck that impedes conventional machine vision sys- tems from promptly responding to environmental changes. Additionally, traditional machine vision systems with loosely integrated sensors and processors have high energy costs, weight, and size, making them unsuitable for portable tasks. To address the limitations of conventional machine vision systems, this chapter explores a new visual information processing scheme using a focal plane sensor processor (FPSP) that directly processes signals where they are collected, thereby avoiding issues with latency, power consumption, and size. The chapter focuses on mobile robotic control systems that utilize in-sensor computed results for multiple navigation research, and investigates novel parallel visual inference approaches, par- ticularly machine learning-based algorithms, to extract higher-level information from analogue signals. To achieve this, a lightweight and high-speed binary convolutional neural network is proposed to categorize objects using efficient addition/subtraction and bit shifting operations. However, implementing neural networks on the focal plane is challenging due to hardware resource constraints and analogue noises. There- fore, this work proposes purely binarised convolutional neural networks with both binary weights and activations, trained with batch normalisation and adaptive thresh- oldtobinariseactivationsandalleviatenoise.Withthisapproach,onlyasmallamount of extracted information is obtained, allowing for more efficient data transmission with less bandwidth, and enabling the establishment of an edge computing platform based on the neural network and PPA. Furthermore, prior research has explored visual sensors for information collection but not for signal processing or motion control; however, this chapter investigates the direction of image processing on the sensing chip and servo motor control using the sensor’s digested data directly. Thus, by merging in-sensor neural network inference and direct servo motor control, a sensory-motor system is presented. Moreover, with our proposed dynamic model swapping scheme, more sophisticated classification tasks than earlier work can be achieved. Lastly, a new in-sensor neural network architecture, fully convolutional neural networks, is presented for localisation and coarse segmentation tasks without using the fully connected layers. To deploy this new architecture of a three-layer neural network on the sensor, group convolution is introduced and implemented, with both binary weights and activations, making the fully convolutional neural network compact enough to be embedded on the sensor.
- 45. 30 Y. Liu et al. Acknowledgements This work is funded by National Natural Science Foundation of China (grant 62202285), EPSRC FARSCOPE CDT for future robotics and autonomous system, and China Schol- arship Council (grant 201700260083). References 1. Frisby JP, Stone JV (2010) Seeing: the computational approach to biological vision. MIT Press, Cambridge, MA 2. Lao J, Yan M, Tian B, Jiang C, Luo C, Xie Z, Zhu Q, Bao Z, Zhong N, Tang X et al (2022) Ultralow-power machine vision with self-powered sensor reservoir. Adv Sci 2106092 3. Wan T, Shao B, Ma S, Zhou Y, Li Q, Chai Y (2022) In-sensor computing: materials, devices, and integration technologies. Adv Mater 2203830 4. Interim report for the decadal plan for semiconductors (2020) Semiconductor research cor- poration and semiconductor industry association, pp 1–21 5. Pan W, Zheng J, Wang L, Luo Y (2022) A future perspective on in-sensor computing. Engi- neering 6. Zhou F, Chai Y (2020) Near-sensor and in-sensor computing. Nat Electr 3(11):664–671 7. Ielmini D, Wong H-SP (2018) In-memory computing with resistive switching devices. Nat Electr 1(6):333–343 8. Kaneda M (2013) Signal processing in the mammalian retina. J Nippon Med School 80(1):16– 24 9. Zarándy Á (2011) Focal-plane sensor-processor chips. Springer Science & Business Media, New York City 10. Rodriguez-Vázquez Á, Dominguez-Castro R, Jiménez-Garrido F, Morillas S, Listán J, Alba L, Utrera C, Espejo S, Romay R (2008) The eye-ris cmos vision system. Analog circuit design. Springer, New York City, pp 15–32 11. https://kovilta.fi/technology/ 12. Aistorm, Ai in sensor. Accessed 22 Nov 2021. https://aistorm.ai/ 13. Zhang J, Dai S, Zhao Y, Zhang J, Huang J (2020) Recent progress in photonic synapses for neuromorphic systems. Adv Intell Syst 2(3):1900136 14. Mennel L, Symonowicz J, Wachter S, Polyushkin DK, Molina-Mendoza AJ, Mueller T (2020) Ultrafast machine vision with 2d material neural network image sensors. Nature 579(7797):62–66 15. Chen J, Carey SJ, Dudek P (2018) Scamp5d vision system and development framework. In: Proceedings of the 12th international conference on distributed smart cameras, pp 1–2 16. Dudek P, Hicks PJ (1999) An simd array of analogue microprocessors for early vision. In: Proceedings of the conference on postgraduate research in electronics, photonics and related fields (PREP’99), pp 359–362 17. Wang Y, Gong Y, Yang L, Xiong Z, Lv Z, Xing X, Zhou Y, Zhang B, Su C, Liao Q et al (2021) Mxene-zno memristor for multimodal in-sensor computing. Adv Funct Mater 31(21):2100144 18. Lyapunov N, Zheng XD, Yang K, Liu HM, Zhou K, Cai SH, Ho TL, Suen CH, Yang M, Zhao J et al (2022) A bifunctional memristor enables multiple neuromorphic computing applications. Adv Electr Mater 8(7):2101235 19. Lee D, Park M, Baek Y, Bae B, Heo J, Lee K (2022) In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction towards visual cognitive processing 20. Pinkham R, Berkovich A, Zhang Z (2021) Near-sensor distributed dnn processing for aug- mented and virtual reality. IEEE J Emerg Selected Topics Circuits Syst 11(4):663–676 21. Sony, Intelligent vision sensors with ai processing functionality. Accessed 14 May 2020. https://www.sony.com/en/SonyInfo/News/Press/202005/20-037E/
- 46. 1 In-Sensor Visual Devices for Perception and Inference 31 22. Rodríguez-Vázquez A, Domínguez-Castro R, Jiménez-Garrido F, Morillas S (2010) A cmos vision system on-chip with multicore sensory processing architecture for image analysis above 1,000 f/s. In: Sensors, cameras, and systems for industrial/scientific applications XI, vol 7536. Spie, pp 213–223 23. Carey SJ, Lopich A, Barr DR, Wang B, Dudek P (2013) A 100,000 fps vision sensor with embedded 535gops, w 256.× 256 simd processor array. In: 2013 symposium on VLSI circuits. IEEE, pp C182–C183 24. Lahdenoja O, Säntti T, Laiho M, Paasio A, Poikonen JK (2014) Seam tracking with adaptive image capture for fine-tuning of a high power laser welding process. In: Seventh international conference on machine vision (ICMV 2014), vol 9445. SPIE, pp 374–380 25. Poikonen J, Laiho M, Paasio A (2009) Mipa4k: a 64.× 64 cell mixed-mode image processor array. In: 2009 IEEE international symposium on circuits and systems. IEEE, pp 1927–1930 26. inivation (2021) Dynamic vision platform. https://inivation.com/dvp/ 27. Liu Y, Bose L, Greatwood C, Chen J, Fan R, Richardson T, Carey SJ, Dudek P, Mayol-Cuevas W (2021) Agile reactive navigation for a non-holonomic mobile robot using a pixel processor array. IET Image Process 1–10 28. Greatwood C, Bose L, Richardson T, Mayol-Cuevas W, Chen J, Carey SJ, Dudek P (2017) Tracking control of a uav with a parallel visual processor. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 4248–4254 29. Greatwood C, Bose L, Richardson T, Mayol-Cuevas W, Chen J, Carey SJ, Dudek P (2018) Perspective correcting visual odometry for agile mavs using a pixel processor array. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 987– 994 30. McConville A, Bose L, Clarke R, Mayol-Cuevas W, Chen J, Greatwood C, Carey S, Dudek P, Richardson T (2020) Visual odometry using pixel processor arrays for unmanned aerial systems in gps denied environments. Front Robot AI 7 31. Martel JN, Mueller LK, Carey SJ, Dudek P, Wetzstein G (2020) Neural sensors: learning pixel exposures for hdr imaging and video compressive sensing with programmable sensors. IEEE Trans Pattern Anal Mach Intell 42(7):1642–1653 32. Bose L, Chen J, Carey SJ, Dudek P, Mayol-Cuevas W (2017) Visual odometry for pixel processor arrays. In: Proceedings of the IEEE international conference on computer vision, pp 4604–4612 33. Chen J, Carey SJ, Dudek P (2017) Feature extraction using a portable vision system. In: IEEE/RSJ international conference intelligent and robotic systems, Workshop vision-based Agile Auton. Navigation UAVs 34. Chua LO, Yang L (1988) Cellular neural networks: theory. IEEE Trans Circuits Syst 35(10):1257–1272 35. Roska T, Chua LO (1993) The cnn universal machine: an analogic array computer. IEEE Trans Circuits Syst II: Analog Digit Signal Process 40(3):163–173 36. Orovas C (2000) Cellular associative neural networks for pattern recognition. PhD disserta- tion, Citeseer 37. Rosin PL (2006) Training cellular automata for image processing. IEEE Trans Image Process 15(7):2076–2087 38. Torralba AB (1999) Analogue architectures for vision cellular neural networks and neuro- morphic circuits. Doctorat thesis, Institute national Polytechnique Grenoble, Laboratory of Images and Signals 39. Dudek P (2019) Scamp-5: vision sensor with pixel parallel simd processor array. https://youtu. be/D3VcmkQiPR4 40. Liu Y, Bose L, Chen J, Carey SJ, Dudek P, Mayol-Cuevas W (2020) High-speed light-weight cnn inference via strided convolutions on a pixel processor array. In: The 31st British machine vision conference (BMVC 2020) 41. Carey SJ, Barr DR, Dudek P (2013) Low power high-performance smart camera system based on scamp vision sensor. J Syst Archit 59(10):889–899
- 47. 32 Y. Liu et al. 42. Bose L, Chen J, Carey SJ, Dudek P, Mayol-Cuevas W (2019) A camera that cnns: towards embedded neural networks on pixel processor arrays. In: Proceedings of the IEEE/CVF inter- national conference on computer vision, pp 1335–1344 43. Debrunner T, Saeedi S, Kelly PH (2019) Auke: automatic kernel code generation for an analogue simd focal-plane sensor-processor array. ACM Trans Archit Code Optim (TACO) 15(4):1–26 44. Malekzadeh M, Clegg RG, Cavallaro A, Haddadi H (2020) Privacy and utility pre- serving sensor-data transformations. Pervasive Mobile Comput 63:101132. https://www. sciencedirect.com/science/article/pii/S1574119220300201 45. Lin W, Liang C, Wang JZ, Buyya R (2014) Bandwidth-aware divisible task scheduling for cloud computing. Softw: Pract Exp 44(2):163–174 46. Wang B, Dudek P (2013) Amber: adapting multi-resolution background extractor. In: 2013 IEEE international conference on image processing. IEEE, pp 3417–3421 47. Wang B, Dudek P (2014) A fast self-tuning background subtraction algorithm. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 395–398 48. Dudek P (2006) Adaptive sensing and image processing with a general-purpose pixel-parallel sensor, processor array integrated circuit. In: 2006 International workshop on computer archi- tecture for machine perception and sensing. IEEE, pp 1–6 49. Dudek P, Vilariño DL (2006) A cellular active contours algorithm based on region evolution. In: 2006 10th international workshop on cellular neural networks and their applications. IEEE, pp 1–6 50. Alonso-Montes C, Vilarino D, Penedo M (2005) Cnn-based automatic retinal vascular tree extraction. In: 2005 9th international workshop on cellular neural networks and their appli- cations. IEEE, pp 61–64 51. Wang B, Mroszczyk P, Dudek P (2014) A new method for fast skeletonization of binary images on cellular processor arrays. In: 2014 14th international workshop on cellular nanoscale networks and their applications (CNNA). IEEE, pp 1–2 52. Mroszczyk P, Dudek P (2012) Trigger-wave collision detecting asynchronous cellular logic array for fast image skeletonization. In: 2012 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 2653–2656 53. Razmjooei S, Dudek P (2010) Approximating euclidean distance transform with simple oper- ations in cellular processor arrays. In: 2010 12th international workshop on cellular nanoscale networks and their applications (CNNA 2010). IEEE, pp 1–5 54. Martel JN, Müller LK, Carey SJ, Dudek P (2016) Parallel hdr tone mapping and auto-focus on a cellular processor array vision chip. In: 2016 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 1430–1433 55. Martel JN (2019) Unconventional processing with unconventional visual sensing: parallel, distributed and event based vision algorithms & systems. PhD dissertation, ETH Zurich 56. Carey SJ, Barr DR, Wang B, Lopich A, Dudek P (2012) Locating high speed multiple objects using a scamp-5 vision-chip. In: 2012 13th international workshop on cellular nanoscale networks and their applications. IEEE, pp 1–2 57. Barr DR, Carey SJ, Dudek P (2012) Low power multiple object tracking and counting using a scamp cellular processor array. In: 2012 13th international workshop on cellular nanoscale networks and their applications. IEEE, pp 1–2 58. Bose L, Dudek P, Chen J, Carey SJ, Mayol-Cuevas WW (2020) Fully embedding fast con- volutional networks on pixel processor arrays. In: European conference on computer vision. Springer, pp 488–503 59. Liu Y, Bose L, Lu Y, Dudek P, Mayol-Cuevas W (2022) On-sensor binarized fully convolu- tional neural network with a pixel processor array. arXiv:2202.00836 60. Chen J, Liu Y, Carey SJ, Dudek P (2020) Proximity estimation using vision features computed on sensor. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE, pp 2689–2695 61. Wong MZ, Guillard B, Murai R, Saeedi S, Kelly PH (2020) Analognet: convolutional neural network inference on analog focal plane sensor processors. arXiv:2006.01765
- 48. 1 In-Sensor Visual Devices for Perception and Inference 33 62. Martel JN, Müller LK, Carey SJ, Müller J, Sandamirskaya Y, Dudek P (2017) Real-time depth from focus on a programmable focal plane processor. IEEE Trans Circuits Syst I: Regul Papers 65(3):925–934 63. Martel JN, Müller LK, Carey SJ, Dudek P (2017) High-speed depth from focus on a pro- grammable vision chip using a focus tunable lens. In: 2017 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 1–4 64. Martel JN, Müller LK, Carey SJ, Müller J, Sandamirskaya Y, Dudek P (2017) Live demon- stration: depth from focus on a focal plane processor using a focus tunable liquid lens. In: 2017 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 1–1 65. Murai R, Saeedi S, Kelly PH (2020) Bit-vo: visual odometry at 300 fps using binary features from the focal plane. arXiv:2004.11186 66. Stow E, Murai R, Saeedi S, Kelly PH (2021) Cain: automatic code generation for simultaneous convolutional kernels on focal-plane sensor-processors. arXiv:2101.08715 67. Wang B, Dudek P (2012) Coarse grain mapping method for image processing on fine grain cellular processor arrays. In: 2012 13th international workshop on cellular nanoscale networks and their applications. IEEE, pp 1–6 68. Martel JN, Sandamirskaya Y, Dudek P (2016) A demonstration of tracking using dynamic neural fields on a programmable vision chip. In: Proceedings of the 10th international con- ference on distributed smart camera, pp 212–213 69. Dudek P, Carey S (2006) General-purpose 128/spl times/128 simd processor array with inte- grated image sensor. Electr Lett 42(12):678–679 70. Martel JN, Müller LK, Carey SJ, Dudek P (2016) A real-time high dynamic range vision system with tone mapping for automotive applications. In: CNNA 2016; 15th international workshop on cellular nanoscale networks and their applications. VDE, pp 1–2 71. So HM, Martel JN, Wetzstein G, Dudek P (2022) Mantissacam: learning snapshot high- dynamic-range imaging with perceptually-based in-pixel irradiance encoding. In: 2022 IEEE international conference on computational photography (ICCP). IEEE, pp 1–12 72. Dudek P, Hülse M, Barr DR (2008) Cellular automata and non-static image processing for embodied robot systems on a massively parallel processor array. In: Automata-2008: theory and applications of cellular automata. Luniver Press, pp 504–510 73. Davison AJ (2018) Futuremapping: the computational structure of spatial ai systems. arXiv:1803.11288 74. Greatwood C, Bose L, Richardson T, Mayol-Cuevas W, Clarke R, Chen J, Carey SJ, Dudek P (2019) Towards drone racing with a pixel processor array. In: 11th international micro air vehicles, conferences and competitions, pp 73–79 75. Castillo-Elizalde H, Liu Y, Bose L, Mayol-Cuevas W (2021) Weighted node mapping and localisation on a pixel processor array. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE 76. Debrunner T, Saeedi S, Bose L, Davison AJ, Kelly PH (2019) Camera tracking on focal-plane sensor-processor arrays. In: Proceedings of the workshop on programmability and architec- tures for heterogeneous multicores (MULTIPROG), Vancouver, BC, Canada, vol 15 77. Klein G, Murray D (2007) Parallel tracking and mapping for small ar workspaces. In: 2007 6th IEEE and ACM international symposium on mixed and augmented reality. IEEE, pp 225–234 78. Liu Y, Bose L, Chen J, Fan R, Dudek P, Mayol-Cuevas W (2021) On-sensor cnn parallel computing with a pixel processor array. IEEE Trans Comput Imaging, manuscript 79. Bose L, Chen J, Carey SJ, Dudek P, Mayol-Cuevas W (2019) A camera that cnns: towards embedded neural networks on pixel processor arrays. In: Proceedings of the IEEE/CVF inter- national conference on computer vision (ICCV) 80. Courbariaux M, Bengio Y, David J-P (2015) Binaryconnect: training deep neural networks with binary weights during propagations. In: Advances in neural information processing systems, pp 3123–3131 81. Liu Y, Chen J, Bose L, Dudek P, Mayol-Cuevas W (2021) Direct servo control from in-sensor cnn inference with a pixel processor array. In: 2021 IEEE international conference on robotics and automation (ICRA) Workshop: on and near-sensor vision processing, from photons to applications. IEEE
- 49. 34 Y. Liu et al. 82. Liu Y, Chen J, Bose L, Dudek P, Mayol-Cuevas W (2021) Bringing a robot simulator to the scamp vision system. In: 2021 IEEE international conference on robotics and automation (ICRA) workshop: on and near-sensor vision processing, from photons to applications. IEEE 83. Guillard B (2019) Optimising convolutional neural networks for super fast inference on focal- plane sensor-processor arrays. PhD dissertation, Imperial College London 84. Wong M, Saeedi S, Kelly PH (2018) Analog vision-neural network inference acceleration using analog simd computation in the focal plane. PhD dissertation, Master’s thesis, Imperial College London-Department of Computing 85. Liu Y (2022) On-sensor visual inference with a pixel processor array. PhD dissertation, University of Bristol 86. Wolfram S (1984) Cellular automata as models of complexity. Nature 311(5985):419–424 87. Mordvintsev A, Randazzo E, Niklasson E, Levin M (2020) Growing neural cellular automata. Distill 5(2):e23 88. Eye-RIS, Eye-ris v1.3 hardware description. Accessed 16 Sept 2022. https://imaging. teledyne-e2v.com/products/2d-cmos-image-sensors/eye-ris/ 89. Sotoa LA, Morillasa S, Listána J, Jiméneza A, Arenab P, Patanéb L, De Fioreb S, Embedding the anafocus’ eye-ris vision system in roving robots to enhance the action-oriented perception 90. Caballero-Garcia D, Jimenez-Marrufo A (2014) Visual routines for cognitive systems on the eye-ris platform. In: Spatial temporal patterns for action-oriented perception in roving robots II. Springer, pp 249–316 91. Soto LA, Morillas S, Listán J, Jiménez A, Arena P, Patané L, De Fiore S (2009) Embedding the anafocus’ eye-ris vision system in roving robots to enhance the action-oriented perception. In: Bioengineered and bioinspired systems IV, vol 7365. SPIE, pp 80–90 92. Guzmán P, Díaz J, Agís R, Ros E (2010) Optical flow in a smart sensor based on hybrid analog-digital architecture. Sensors 10(4):2975–2994 93. Lucas BD, Kanade T et al (1981) An iterative image registration technique with an application to stereo vision. Vancouver, vol 81 94. Zarándy Á, Pencz B, Németh M, Zsedrovits T (2014) Implementation of visual naviga- tion algorithms on the eye-ris 1.3 system. In: 2014 14th international workshop on cellular nanoscale networks and their applications (CNNA). IEEE, pp 1–2 95. Nicolosi L, Abt F, Tetzlaff R, Hofler H, Blug A, Carl D (2009) New cnn based algorithms for the full penetration hole extraction in laser welding processes. In: 2009 IEEE international symposium on circuits and systems. IEEE, pp 2713–2716 96. Abt F, Heider A, Weber R, Graf T, Blug A, Carl D, Höfler H, Nicolosi L, Tetzlaff R (2011) Camera based closed loop control for partial penetration welding of overlap joints. Phys Procedia 12:730–738 97. Fülöp T, Zarándy Á (2010) Bio-inspired looming object detector algorithm on the eye-ris focal plane-processor system. In: 2010 12th international workshop on cellular nanoscale networks and their applications (CNNA). IEEE, pp 1–5 98. Säntti T, Lahdenoja O, Paasio A, Laiho M, Poikonen J (2014) Line detection on fpga with parallel sensor-level segmentation. In: 2014 14th international workshop on cellular nanoscale networks and their applications (CNNA). IEEE, pp 1–2 99. Lahdenoja O, Säntti T, Poikonen J, Laiho M, Paasio A (2013) Characterizing spatters in laser welding of thick steel using motion flow analysis. In: Scandinavian conference on image analysis. Springer, pp 675–686 100. Säntti T, Poikonen JK, Lahdenoja O, Laiho M, Paasio A (2015) Online seam tracking for laser welding with a vision chip and fpga enabled camera system. In: 2015 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 1985–1988 101. Lahdenoja O, Säntti T, Laiho M, Poikonen J (2014) Spatter tracking in laser-and manual arc welding with sensor-level pre-processing 102. Poikonen J, Laiho M, Paasio A (2010) Anisotropie filtering with a resistive fuse network on the mipa4k processor array. In: 2010 12th international workshop on cellular nanoscale networks and their applications (CNNA). IEEE, pp 1–5
- 50. 1 In-Sensor Visual Devices for Perception and Inference 35 103. Lahdenoja O, Poikonen J, Laiho M (2010) Extracting local binary patterns with mipa4k vision processor. In: 2010 12th international workshop on cellular nanoscale networks and their applications (CNNA). IEEE, pp 1–5 104. Poikonen J, Laiho M, Paasio A (2009) Locally adaptive image sensing with the 64x64 cell mipa4k mixed-mode image processor array. In: 2009 European conference on circuit theory and design. IEEE, pp 93–96 105. Laiho M, Poikonen J, Paasio A (2009) Space-dependent binary image processing within a 64x64 mixed-mode array processor. In: 2009 European conference on circuit theory and design. IEEE, pp 189–192 106. Laiho M, Poikonen J, Paasio A (2010) Object segmentation and tracking with asynchronous grayscale and binary wave operations on the mipa4k. In: 2010 12th international workshop on cellular nanoscale Networks and their applications (CNNA). IEEE, pp 1–4 107. Linan G, Espejo S, Dominguez-Castro R, Rodriguez-Vázquez A (2002) Architectural and basic circuit considerations for a flexible 128.× 128 mixed-signal simd vision chip. Analog Integr Signal Process 33(2):179–190 108. Yao P, Wu H, Gao B, Tang J, Zhang Q, Zhang W, Yang JJ, Qian H (2020) Fully hardware- implemented memristor convolutional neural network. Nature 577(7792):641–646 109. Wang T-Y, Meng J-L, Li Q-X, He Z-Y, Zhu H, Ji L, Sun Q-Q, Chen L, Zhang DW (2021) Reconfigurable optoelectronic memristor for in-sensor computing applications. Nano Energy 89:106291 110. Sun L, Wang Z, Jiang J, Kim Y, Joo B, Zheng S, Lee S, Yu WJ, Kong B-S, Yang H (2021) In-sensor reservoir computing for language learning via two-dimensional memristors. Sci Adv 7(20):eabg1455 111. Thomas A (2013) Memristor-based neural networks. J Phys D: Appl Phys 46(9):093001 112. Lungu IA, Liu S-C, Delbruck T (2019) Fast event-driven incremental learning of hand sym- bols. In: 2019 IEEE international conference on artificial intelligence circuits and systems (AICAS). IEEE, pp 25–28 113. Lungu I-A, Corradi F, Delbrück T (2017) Live demonstration: convolutional neural network driven by dynamic vision sensor playing roshambo. In: 2017 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 1–1 114. Linares-Barranco A, Rios-Navarro A, Tapiador-Morales R, Delbruck T (2019) Dynamic vision sensor integration on fpga-based cnn accelerators for high-speed visual classification. arXiv:1905.07419 115. Gallego G, Delbruck T, Orchard G, Bartolozzi C, Taba B, Censi A, Leutenegger S, Davison A, Conradt J, Daniilidis K et al (2019) Event-based vision: a survey. arXiv:1904.08405 116. Song R, Huang K, Wang Z, Shen H (2021) An ultra fast low power convolutional neural network image sensor with pixel-level computing. arXiv:2101.03308 117. Datta G, Kundu S, Yin Z, Lakkireddy RT, Beerel PA, Jacob AP, Jaiswal A (2022) P2m: a processing-in-pixel-in-memory paradigm for resource-constrained tinyml applications. arXiv:abs/2203.04737
- 51. Chapter 2 Environmental Perception Using Fish-Eye Cameras for Autonomous Driving Yeqiang Qian, Ming Yang, and John M. Dolan Abstract The fish-eye camera has become an indispensable sensor in autonomous driving. Due to the unique projection principle of the fish-eye camera, it provides a large field of view (FoV). Because of this special feature, the fish-eye camera has abundant applications in environmental perception and autonomous driving. How- ever, many challenges still exist in the practical application of fish-eye cameras. In this chapter, typical fish-eye datasets, including real data and generated virtual data, are introduced. Then, the projection principle of the fish-eye camera and four clas- sical fish-eye image representation models are given. By sorting and summarizing relevant research, we show various applications of fish-eye cameras in environmen- tal perception, including semantic understanding and target detection. These works have designed various strategies to take advantage of fish-eye cameras and prevent image distortions, showing the broad application prospects of fish-eye cameras. The development trend of fish-eye cameras in autonomous driving is discussed. 2.1 Introduction In the past decade, autonomous driving has rapidly developed. Both academia and industry are paying attention to this field. Self-driving cars are equipped with a variety of sensors, such as cameras, lidar, and radar. Cameras provide more semantic information than other sensors. The fish-eye camera is a unique type of camera that Y. Qian University of Michigan-Shanghai Jiao Tong University Joint Institute, Shanghai Jiao Tong University, Shanghai 200240, China e-mail: qianyeqiang@sjtu.edu.cn M. Yang (B) Key Laboratory of System Control and Information Processing, Department of Automation, Shanghai Jiao Tong University, Ministry of Education of China, Shanghai 200240, China e-mail: mingyang@sjtu.edu.cn J. M. Dolan Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA e-mail: jdolan@andrew.cmu.edu © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 R. Fan et al. (eds.), Autonomous Driving Perception, Advances in Computer Vision and Pattern Recognition, https://doi.org/10.1007/978-981-99-4287-9_2 37
- 52. 38 Y. Qian and M. Yang Fig. 2.1 Typical normal and fish-eye images taken in the same location provides a larger field of view (FoV), up to 180. ◦ , than ordinary cameras. Therefore, fewer fish-eye cameras are needed to fully monitor their surroundings. Moreover, fish-eye cameras are indispensable sensors in autonomous driving and have a very large role in some scenarios. Figure2.1 shows a normal image and fisheye image; both were taken at the same location. Intuitively, the images appear quite different. The fish-eye image in Fig.2.1b contains more scenery. In addition, serious distortion occurs in the fish-eye image, e.g., the straight street lamp in Fig.2.1a becomes crooked in the fish-eye image. The same object in a normal image contains more pixels than in a fish-eye image, that is, fish-eye images lose information as the distance increases. In short, fish-eye cameras have three advantages. Similar to fish-eye cameras, lidar sensors also have large FoVs [1–4]. However, cameras provide much richer semantic information. In addition, cameras have a higher cost performance [5], which is crucial for the industry. Moreover, these benefits apply to all camera sensors, not just fish-eye cameras. Fish-eye cameras offer a wide FoV, usually up to 180. ◦ , which is crucial in autonomous driving as covering more blind spots helps improve safety. The use of fish-eye cameras avoids the need for calibration among multiple cam- eras. To cover more space, ordinary cameras should be grouped. Therefore, for such a system, calibration is necessary, and calibration between two different sensors is complex. Inaccurate calibration will introduce unexpected noise in the system, leading to the failure of subsequent algorithms. There are three challenges in applying fish-eye cameras. Deep learning methods have substantially become more popular in the field of computer vision, including environmental perception, where datasets have become particularly important. The use of sample-rich datasets contributes to the better performances of most algorithms. In recent years, many datasets for autonomous driving environment awareness, such as KITTI [6] and Cityscapes [7], have been published. However, these datasets con- tain normal images with a normal FoV. As fish-eye images and normal images appear completely different, normal images have difficulty benefitting from fish-eye-based
- 53. 2 Environmental Perception Using Fish-Eye Cameras for Autonomous Driving 39 Fig. 2.2 False results caused by the distortion of the fish-eye image. The red bounding box rep- resents false negative detection results, and the area in the red circle represents false positive seg- mentation results algorithms. In conclusion, few significant fish-eye datasets have been formed, which severely limits the development of fish-eye-based algorithms. This is the most significant problem of fish-eye cameras. In Fig.2.1, street lights, cars and buildings appear in a completely different curved position from the real shape. Moreover, these distorted objects cause issues for most detectors as most detectors are trained on normal images. Figure2.2a shows the object detection results obtained by using the faster region-based convolutional neural network (Faster R- CNN) [8]. Compared to other normal vehicles, the white vehicles with edges were so deformed that they were not accurately detected. Figure2.2b shows the semantic segmentation results generated by the efficient residual-factorized CNN (ERFNet) [9, 10]. The areas in the red circles are segmentation errors caused by distortion. As shown in Fig.2.1, it is difficult for fish-eye cameras to clearly observe distant targets, which limits their application scenarios, that is fish-eye cameras can only be used for low-speed and short-range scenes. In these cases, the perception of distant objects is not urgently needed. This chapter is organized as follows. The typical fish-eye image datasets are presentedinSect.2.2.Differentfish-eyeprojectionmodelsareintroducedinSect.2.3. Various applications of fish-eye cameras in autonomous driving are organized in Sects.2.4 and 2.5. Finally, the development tendency of fish-eye camera applications is discussed in Sect.2.6.
- 54. 40 Y. Qian and M. Yang 2.2 Fish-Eye Image Datasets High-quality, fish-eye datasets provide a comparison platform for various algorithms and can be utilized in the training process to improve the performance of the detector. Currently, three main methods are used to obtain fish-eye image data. Most works use real fish-eye cameras to manually collect real fish-eye datasets. Some works use simulators to generate virtual fish-eye images. Other works focus on generating fish- eye lens images from real perspective images. These real and virtual images promote research in this field. 2.2.1 Real Fish-Eye Datasets We sorted the real fish-eye datasets in the field of automatic driving. Details of these datasets are shown in Table2.1. The author uses different fish-eye camera settings to customize their fish-eye lens datasets, including different numbers of fish-eye lenses, image resolutions, numbers of datasets, and supported tasks. Levi et al. [11, 12] collected a fish-eye dataset using a vehicle rearview camera to evaluate the application of rear-view pedestrian detection. This dataset is named the GM-ATCI rearview pedestrian dataset. This dataset contains 15 video sequences, and each session is obtained in different scenes on different dates. The dataset contains a total of 250 fragments, with a total duration of 76min, and more than 200K annotated pedestrian bounding boxes. Oxford RobotCar [13], which was proposed by Maddern et al., is a large-scale dataset that focuses on the long-term autonomy of autonomous vehicles, mainly supporting positioning and mapping tasks. The dataset collects data under various weather conditions, including heavy rain, nighttime, direct sunlight, and snowfall, and has accumulated more than 1000 km of road data in one year. Due to the long data collection time, the appearances of many roads and buildings has changed, which is also a feature of this dataset. Yogamani et al. [14] proposed the first fish-eye image dataset dedicated to intel- ligent vehicles. This dataset, which is named WoodScape, consists of four cameras covering 360. ◦ and a high-definition laser scanner, inertial measurement unit (IMU), and global navigation satellite system (GNSS). Annotations can be used for nine tasks, especially 3D object detection, depth estimation (superimposed on the front camera) and semantic segmentation. The semantic annotation of 40 classes at the instance level exceeds 10000 images, and the annotation of other tasks exceeds 100000 images. Rashed et al. further expanded the dataset in [15]. Liao et al. proposed the suburban dataset KITTI-360 [16]. Compared with the classic KITTI [6], KITTI-360 has two new fish-eye cameras and a larger amount of data. Compared with WoodScape [14], KITTI-360 provides temporal, coherent, semantic instance annotation and 3D annotation for 3D reasoning.
- 55. Random documents with unrelated content Scribd suggests to you:
- 56. geeignet sind, Lüsternheit zu erwecken. Diese Eigenschaft tritt so stark hervor, daß nach einer Behauptung des Vorworts des Uebersetzers und nach Notizen der öffentlichen Blätter die Lektüre des Romans in Rußland zu geschlechtlichen Ausschweifungen, namentlich bei jugendlichen Lesern Anlaß gegeben hat. Hiernach ist der Roman in der deutschen Uebersetzung nach seinem Gesamtcharakter und nach einzelnen Stellen geeignet, das normale im deutschen Volk herrschende Scham- und Sittlichkeitsgefühl in geschlechtlicher Beziehung gröblich zu verletzen. Der Inhalt des Romans ist also unzüchtig. Daran ändert die künstlerische, wissenschaftliche oder geschichtliche Bedeutung, die dem Roman von manchen zugesprochen wird, nichts, sie ist nicht so erheblich, daß durch sie der unzüchtige Charakter in den Hintergrund gedrängt würde. An dieser Beurteilung des Romans ändern auch die teils vom Beschwerdeführer, teils von anderer Seite vorgelegten öffentlichen Kritiken nichts; sie sind trotz vielfacher Abweichungen im wesentlichen darüber einig, daß die literarische Bedeutung des Romanes keine außergewöhnliche ist, daß die dort vertretene Auffassung geschlechtlicher Sittlichkeit mit der in Deutschland herrschenden, sittlichen Auffassung in grobem Widerspruch steht; ein Teil dieser Kritiken spricht sich überdies mehr oder weniger offen auch über die sittlichen Eigenschaften des Romans verurteilend aus; wenn einige der Kritiken bestreiten, daß der Roman unsittlich oder pornographisch sei, so mag dies auf einer Verkennung der Begriffe oder auf anderen besonderen Gründen beruhen, ist aber jedenfalls für die allgemeine Beurteilung nicht entscheidend. Der Roman Ssanin stellt daher in seiner Villard-Bugowschen deutschen Uebersetzung eine unzüchtige Schrift im Sinne des § 184 Ziffer 1 St.-G.-B. dar. Das Amtsgericht hat also nach § 40 ff. R.-St.-G.- B., 94 ff. R.-St.-P.-O. mit Recht die Beschlagnahme angeordnet. Die Beschwerde des von der Beschlagnahme betroffenen Verlegers ist daher unbegründet und zurückzuweisen. Die Kosten treffen nach § 505 St.-P.-O. den Beschwerdeführer. (L. S.)
- 57. gez. Dobeneck, Dr. Heuser, Maier. 4. Freigabebeschluß des Landgerichts. Ziffer des Anz.-Verz. VII, 610/08. Beglaubigte Abschrift. Die 4. Strafkammer des Kgl. Landgerichts München I hat am 16. März 1909, vormittags 10 Uhr, versammelt in geheimer Sitzung, wobei zugegen waren: der Vorsitzende kgl. Ldgr.-Direktor Hezner die Landgerichtsräte Dr. Heuser Cl. und Graf in der Untersuchungssache gegen Müller Georg, Verleger in München, wegen Vergehens wider die Sittlichkeit nach § 184 I R.-St.-G.-B. nach Einsicht und Verlesung der wichtigeren Aktenstücke des bisherigen Verfahrens nach Ansicht des vom kgl. Staatsanwalt unterm 12. Februar 1909 gestellten Antrags den Beschuldigten außer Verfolgung zu setzen, beschlossen: 1. Der Angeschuldigte Georg Müller wird wegen eines Vergehens wider die Sittlichkeit nach § 184 Abs. 1 Z. 1 D. R.-St.-G.-B. außer Verfolgung gesetzt. 2. Der Beschlagnahmebeschluß des kgl. Amtsgerichts München I Abteilung für Strafsachen vom 23. November 1908 wird aufgehoben. 3. Die Kosten des Verfahrens fallen der Staatskasse zur Last. Gründe: Der gesetzliche Tatbestand des § 184 Abs. 1 Z. 1 D. R.-St.-G.-B. erfordert nach der objektiven Seite, daß die Schrift in ihrer gegenständlichen Erscheinung und dem daraus sich ergebenden geistigen Inhalte geeignet ist, das allgemeine Scham- und Sittlichkeitsgefühl — nicht das abgestumpfte gewisser Personenkreise — in geschlechtlicher Beziehung zu verletzen. Trifft dies zu, dann ist die Schrift als unzüchtig zu erachten. Ob sie
- 58. geeignet oder darauf berechnet ist, im Leser wollüstige Empfindungen zu erregen, ist belanglos, ebenso auch, zu welchen Zwecken die Schrift nach dem bloßen inneren Wollen des Verfassers dienen soll. Vgl. E. d. R. G. i. St. S. Einziehung von Anekdoten. Urteil vom 9. Dezember 1907, dann Bd. 31 S. 260, 4 S. 87, 24 S. 365, 27 S. 114. Maßgebend ist also auch nicht die Anschauung und individuelle Empfindung einer einzelnen Person, insbesondere auch nicht die Besorgnis für die sittliche Integrität der Jugend. Vor Eröffnung der Voruntersuchung hat sich nur ein Sachverständiger, Professor Dr. Voll, über den Roman Ssanin gutachtlich geäußert. Sein Gutachten läuft darauf hinaus, daß wohl das Werk als unzüchtig zu erachten ist, daß aber die bona fides des Verlegers, er sei sich des unzüchtigen Charakters der Schrift nicht bewußt geworden, nicht bezweifelt werden könne. Mit Durchführung der Voruntersuchung haben sich sechs weitere Sachverständige über den Charakter des Buches geäußert. Darunter haben mehrere namhafte Kenner der Literatur sich dafür ausgesprochen, daß es sich nicht um eine unzüchtige Schrift, dagegen um ein dichterisches Werk von hohem kulturgeschichtlichem und auch literarischem Wert handelt. Hierzu äußert sich Professor Dr. Munker wörtlich: „Was der Verfasser an solchen Stellen (den beanstandeten) erzählt, das scheint mir meistens zur Charakteristik der Menschen und Zustände, um die es sich handelt, künstlerisch und psychologisch geradezu notwendig; wie er es aber erzählt, das beweist durchaus den vornehmen Schriftsteller, der rein sachlich, objektiv episch darstellt und von jeder Lüsternheit weit entfernt ist.“ Von den übrigen Sachverständigen haben sich Wilhelm Weigand, Prof. Dr. Schneegans und Ludwig Ganghofer ohne Einschränkung dahin ausgesprochen, daß der Roman Ssanin nicht als unzüchtig zu erachten sei. Prof. Dr. Brunner erachtet ihn nur mit dem Vorworte, nicht aber für sich unzüchtig, während Oberstudienrat Dr. Nicklas den Roman, in der Hauptsache wohl von der durchaus zu billigenden Ansicht geleitet, daß „Ssanin“ für die heranwachsende Jugend eine keineswegs geeignete, in unreifen Köpfen nur Verwirrung erzeugende Lektüre ist, für unzüchtig hält.
- 59. Angesichts der den unzüchtigen Charakter der Schrift verneinenden Gutachten der Sachverständigen, denen sich das Gericht nach sorgfältiger Prüfung des Werkes auf seinen gesamten Inhalt und den sich daraus ergebenden Gesamtcharakter, den seines literarischen und kulturgeschichtlichen Wertes angeschlossen hat, kann nicht davon gesprochen werden, daß der Roman Ssanin eine unzüchtige Schrift im Sinne des § 184 des St.-G.-B. nach der eingangs gegebenen Begriffserklärung ist. Fehlt es aber schon an einem strafbaren Tatbestande in objektiver Hinsicht, so entfällt ohne weiteres die Prüfung nach der subjektiven Seite, und es bedarf also der Einwand des Angeschuldigten, er sei sich des unzüchtigen Charakters des Werkes nicht bewußt gewesen, keinerlei Erörterung. Hiernach erübrigte nur dem Antrage des Staatsanwalts, den Angeschuldigten wegen eines Vergehens nach § 184 Abs. I Ziffer 1 des R.-St.-G.-B. außer Verfolgung zu setzen, stattzugeben, wie geschehen. Mangels strafbaren Tatbestandes aus § 184 war auch der Beschlagnahmebeschluß des k. Amtsgericht München I Abt. f. St.-S. vom 23. November 1908 aufzuheben. Die Kosten des Strafverfahrens fallen der k. Staatskasse zur Last. R.-St.-P.-O. §§ 202, 99, 496, 499. L. S. gez. Hezner, Dr. Heuser, Graf. München, den 23. März 1909. 5. Gutachten von Professor Dr. Karl Brunner in Pforzheim. Erster Teil. Das Vorwort des einen Uebersetzers (André Villard) ist in seinen starken Uebertreibungen und in seiner einseitigen Tendenz, die dem Buch selber gar nicht in dem Maße eigen ist, geradezu irreführend
- 60. und zwar, wie mir scheint, in der Absicht, dem Roman einen sensationellen Empfehlungsbrief mit auf den Weg zu geben. Ich halte diese Vorrede für bedenklich vom literarischen wie vom ethischen Standpunkt aus, und zwar aus dem Grunde, weil sie ein vorurteilsloses Herantreten an das Buch erschwert, ja für viele unmöglich macht und diesem eine Tendenz vindiziert, die, so an die Reklameglocke gehängt, den Verdacht erweckt, als wäre die Uebertragung des Romans auf den deutschen Büchermarkt in dem Bestreben erfolgt, durch Hervorkehren der erotischen Reize als den Aeußerungen einer neuen, fast möchte man sagen, beglückenden Weltanschauung auf das Lesepublikum bestechend zu wirken. Kann dies nicht ohne starke Uebertreibungen, ja Entstellungen des Inhalts des Romans geschehen, so ist der Verfasser des Vorworts auch um solche Uebertreibungen nicht verlegen, wenn es sich darum handelt, diese meines Erachtens unter § 184 Ziffer 1 fallenden Versuche einer mit Sensationsmitteln arbeitenden Reklame mit hochtrabenden, den oberflächlichen Leser bestrickenden Redewendungen von politischen, kulturellen und wissenschaftlichen Beziehungen und Tendenzen des Buches so zu verschleiern, daß sie nicht ohne weiteres faßbar erscheinen. Aber ein dünn verschleiertes Objekt erotischer Neigungen ist erfahrungsgemäß viel geheimnis- und reizvoller, als ein unverhülltes. Und wenn beispielsweise S. 8 auf den wilden sexuellen Rausch, der auf den Ssanin zurückgeht, in ziemlicher Breite hingewiesen wird. Anmerkung. Die Stelle lautet: „Der wilde sexuelle Rausch, der auf den Ssanin zurückgeht, hat auch schon genug von sich hören lassen. Die Organisation der Ssaninisti, die Propaganda-Vereine der freien Liebe, die Verbindungen zum ungehinderten Geschlechtsgenuß unter Gymnasiasten und Gymnasiastinnen, die orgiastischen Klubs, die fälschlicherweise behaupteten, die Weltanschauung des Ssanin zu vertreten und es jedenfalls mit Verve taten, haben nur das Recht der Geschmacklosigkeit für sich.“ — So muß das den oben ausgesprochenen Verdacht um so mehr bekräftigen, als derartige „orgiastische Klubs“ in Rußland längst vor dem Erscheinen des Ssanin bestanden haben, hier also gewissermaßen künstlich herbeigezogen werden, als eine Folge des Romans. Daran ändert
- 61. nichts die scheinbare Verurteilung solcher Exzesse durch den Vorredner. Wenn aber tatsächlich solche „Propaganda-Vereine der freien Liebe“ u. a. in Schüler- und Schülerinnenkreisen, wie ausdrücklich hervorgehoben wird, ihre Daseinsberechtigung auf Ssanin zurückführen, dann möchte man wahrlich noch vor Beginn der Lektüre selbst das Buch weit, weit wegwünschen aus dem Bannkreis deutschen Lebens — zurück in den tiefen moralischen Sumpf, aus dem es erwachsen ist. Doch damit habe ich noch nicht das Buch selbst charakterisieren wollen. Es sind dies nur Gedanken, die die Vorrede nahelegt. Es erwächst mir nur die Pflicht, durch eine eingehende Kritik des Vorwortes diese meine Auffassung zu begründen, um dann im zweiten Teil meines Gutachtens das Buch selbst zu behandeln. Wenn ich auf die kaum acht Seiten umfassenden Vorbemerkungen so ausführlich eingehe, so dürfte das in der starken Wirkung dieser Darlegungen seine Berechtigung finden. Mir scheint, als hätte selten ein Verleger einen so geschickten Verkünder der Sensation, die zudem der Roman selbst gar nicht hat, gefunden. Offenbar haben die Ausführungen Villards über das Buch viel mehr als dieses selbst die Auffassung weiterer Kreise über Ssanin beeinflußt und die Lust es zu lesen hervorgerufen und gesteigert. Ich konnte das aus dem Mund verschiedener Leute erfahren, die den Ssanin vom Hörensagen kennen, wie ich es auch aus den Kritiken über das Buch ersehen konnte, deren Abhängigkeit vom Vorwort unverkennbar ist. Auf Seite 8 heißt es: „Selbst wenn er (Ssanin) nicht durch seine künstlerischen Qualitäten zu einer der wichtigsten Erscheinungen in der modernen Literatur Rußlands geworden wäre, hätten ihm doch kulturhistorische Gründe bleibende Bedeutung gegeben. Man wird die gegenwärtige Epoche, also die, welche die revolutionäre ablöste, psychologisch und soziologisch nicht beurteilen können, ohne den „Ssanin“ als ihren charakteristischen Niederschlag in den Mittelpunkt der Betrachtung zu ziehen.“
- 62. Welche starke Uebertreibung in diesem Lobpreis der künstlerischen Qualitäten des Buches liegt, wird im zweiten Teil zu beweisen sein. „Die kulturhistorischen Gründe“, die Ssanin als den charakteristischen Niederschlag der ganzen jetzigen Epoche in den Mittelpunkt der Betrachtung stellen sollen, sind keineswegs so besonderer, eigentümlicher Art. Vielmehr lassen sich gerade die Prinzipien der freien Liebe und ihre Betätigung längst vor der Revolution, so gut wie in der Revolution und nach ihr, als in breiten Kreisen der russischen Intelligenz vorhanden, nachweisen (s. u.). Durch diese Einreihung an hervorragender, kulturhistorischer Stelle sollte nur dem Buch, das eine realistische Schilderung tatsächlicher, keineswegs neuer Zustände bietet, eine interessante Folie gegeben werden. Auf Seite 9 heißt es: „Interessanter ist die Feststellung, wie es überhaupt dazu kam, daß ein ganzes Volk für seine Gesamtäußerungen mit einem Mal nur noch erotische Beziehungen finden konnte. Und daß ein einziges Werk — eben der Roman Ssanin — genügt, um sie hervorzurufen, und sie mit seinem Namen zu decken. — — — Die einzige Antwort ist: Ein russisches Volk existiert gar nicht — wohl aber eine russische Gesellschaft, die den Charakter des nationalen Lebens ausprägt ... Einst beschränkte sie sich auf den Adel, — heute umfaßt sie die Schichten der akademisch gebildeten Berufe — die Intelligenz.“ „Ein ganzes Volk“ — bedeutet doch etwas anderes — auch für Rußland —, als wie wenige Zeilen weiter unten gesagt ist mit der Bezeichnung „russische Gesellschaft“. Denn abgesehen davon, daß selbst diese Kreise der Intelligenz keineswegs ein irgendwie einheitliches Gepräge tragen, kommen doch für jedwede Beobachtung des russischen Lebens in seiner Gesamtheit die Millionen von Angehörigen anderer sozialer Schichten in Betracht, die selbst in ihrer Stagnation und Hemmung eine Macht bilden, wie die Bauern, wenn sie nicht, wie die sozialistischen Kreise der Städtebevölkerung eine aktive Rolle spielen.
- 63. Doch selbst diese Beschränkung des Begriffs „russisches Volk“ zugegeben, — heißt es nicht die Tatsachen vergewaltigen, wenn man 1. die ganze russische Gesellschaft „für ihre Gesamtäußerungen nur noch erotische Beziehungen finden läßt“, das lehrt uns ja der Roman mit seinen zahlreichen Personen ganz und gar nicht — und 2. „mit einem Mal“ und zwar „durch ein einziges Werk“ so etwas hervorgerufen sehen will? Die erste kühne Bemerkung muß jeden auf die Lektüre des Buches äußerst gespannt machen. Denn selbst die Rolle der Erotik im Siecle de Louis XIV. oder im Zeitalter der französischen Revolution, die dem Geschichtskenner denkbar wichtig und tiefgehend erscheint, muß weit zurücktreten, gegenüber einer so absolut allgemein gewordenen Lebens- und Weltanschauung, wie sie uns hier verkündigt wird. Wohl folgen auf Perioden ungeheuren Aufschwungs des Gemeinsinns und der Opferwilligkeit, kurz der praktisch betätigten Begeisterung für hohe Ideale, wie sie der russischen Revolutionsbewegung ohne Zweifel zugrunde lagen, solche des Gegenteils, sei es stumpfer Apathie oder krassen Egoismus. Und gerade der slavische Rassencharakter neigt mehr als ein anderer besonders stark zu solchen Widersprüchen. Aber daß eine große Volksbewegung die im Zug der Zeit liegt und sich wohl vorübergehend — aus inneren sozialen, kulturellen und nicht zum wenigsten religiösen Gründen — unterdrücken, niemals aber mehr ganz ausrotten läßt, eine Bewegung, die bei allem Terrorismus so gut wie einstens die französische Revolution vom Idealismus getragen war, sich ganz und gar auf den Standpunkt des Zynismus zurückwerfen läßt — durch ein einziges „Buch der Contre- Revolution“, das zu behaupten ist eine Ungeheuerlichkeit. Aber es klingt äußerst pikant, wenn gesagt wird: „Nichts hat in Rußland die sozialrevolutionäre Bewegung, nachdem sie zum Stillstand gekommen war, so endgültig der Zersetzung zugeführt, wie Ssanin mit seiner erotischen Suggestion.“ Der Uebersetzer Villard, der solche Ungeheuerlichkeiten als Kenner der russischen Verhältnisse, der er sein will, im Vorwort geschrieben hat, kann nicht als so naiv erscheinen, daß er sich dessen nicht bewußt gewesen wäre. Ich sehe in seinem Vorwort, das von Widersprüchen mit feststehenden Tatsachen und mit dem Inhalt des
- 64. Romans strotzt, einen plumpen Versuch „einer erotischen Suggestion“ auf den Leser um seine eigenen Worte zu gebrauchen, d. h. einer Spekulation auf die besondere Empfänglichkeit des Lesers für Erotika. Und dieser Suggestion kann sich angesichts solch hoher Studienzwecke, wie sie die Vorrede vorspiegelt, mit Beziehung auf den Ausgang der russischen Revolution, für die sich doch alle Welt lebhaft interessiert hat, gar mancher nicht entziehen. Ich behaupte, — ich habe an einzelnen, sogar graß erscheinenden Stellen mit urteilsfähigen Lesern, die die Vorrede nicht kannten, die Probe darauf gemacht — daß man, ohne die Vorrede gelesen zu haben, an dem Buch als Ganzem keinen oder wenigstens keinen erheblicheren Anstoß nehmen kann, als an zahlreichen anderen Büchern auch, die ungehindert im Verkehr sind und daß die meisten der inkriminierten Stellen geradezu besonders aufgesucht werden müssen, um an ihnen etwas Schlimmes zu finden. Ich behaupte aber zugleich, daß das Vorwort geradezu diese Stellen heraushebt, indem es der Erotik in dem Buch einen solchen Einfluß zuschreibt mit der Aeußerung: „Wohl noch niemals wurden durch ein Buch in so kurzer Zeit die gesamten Anschauungen einer Gesellschaft von Grund aus verändert zum Ausdruck gebracht.“ Mit wenigen Ausnahmen werden die meisten Leser nach Einblick in die Vorrede auch in den Kreisen, denen allein das Buch für 6,50 Mark zugänglich ist, — Anmerkung: Ich betone das, weil mir im Kampf gegen die schlechte Literatur öfters entgegen gehalten wurde, daß Bücher mit verhältnismäßig hohem Preise, schon ihres beschränkteren Leserkreises wegen nicht zur Schundliteratur gerechnet werden könnten, — von vornherein ihr besonderes Augenmerk auf die erotischen Stellen richten und darüber die künstlerische Seite des Werkes und die tatsächlich interessante Darstellung des sozialen und kulturellen Lebens, kurz die wissenschaftliche Seite, vernachlässigen oder die oft langatmigen, nicht eben tiefen Philosopheme einfach überschlagen, bis sie wieder den von dem Vorwort angedeuteten Spuren begegnen. Dadurch bekommt das Werk einen Charakter, den ihm der Verfasser ganz und gar nicht gegeben hat.
- 65. Statt daß der Leser mit einem auf das Große gerichteten Interesse eine realistische, meines Erachtens überhaupt kaum tendenziös gefärbte Schilderung der Verhältnisse und Zustände in der russischen Gesellschaft hinnimmt, muß er sich im Vorwort suggerieren lassen, daß es sich hier um „das Neue“ handelt, das man sucht, er muß sich sagen lassen, daß es sich hier — ausgerechnet in diesem Buch — um die Veränderung einer gesamten Anschauung von Grund aus handle, er hört — und das wird vielen aus gesellschaftlichen Gründen verkappten Anhängern der „freien Liebe“ willkommen sein, — daß die Ssaninisten, „endlich die leidige Konspirativität, die traditionelle Geheimniskrämerei beiseite werfen können“, daß die Erotomanie — frei von gesellschaftlichen Vorurteilen eben wegen jener fundamentalen Umwälzung der Gesellschaftsanschauungen, nun stolz das Haupt erheben darf — nur soll sie es nicht so laut und lärmend machen, wie die Ssaninisten, sondern mit „behutsamem Stolz.“ Diese Ausführungen des Vorworts stehen, wie bereits angedeutet, in völligem Widerspruch 1. mit den Tatsachen der Wirklichkeit wie 2. mit dem Inhalt des Romans und involvieren eine Tendenz, die meines Erachtens mit aller Entschiedenheit zu bekämpfen ist, denn sie machen erst den Roman zu einem sittlich minderwertigen Literaturprodukt, das er, an sich betrachtet, nicht ist. Fürs erste (Widerspruch mit den Tatsachen) mögen einige wenige Hinweise genügen. Aus eigener Kenntnis und auf speziell eingezogene Erkundigungen weiß ich, daß sich bei russischen und polnischen Studenten, die bei uns auf deutschen Hochschulen weilen, das weitaus größte Interesse in der Betätigung erotischer Neigungen erschöpft — ein besonders grasser Fall ist erst unlängst in Karlsruhe vorgekommen, der mit Mord und Selbstmord endete. Und beim russischen Militär, insbesondere beim Offizierskorps, ist eine ähnliche oder noch niedrigere Stellung des Weibes, wie sie ihm in diesem Roman der Offizier Sarudin zuweist, wahrlich schon längst traditionell geworden. Ich kann dafür Belege aus unserem hiesigen industriellen Leben beibringen, die ich mir für diesen Zweck aus authentischen Quellen verschafft habe.
- 66. Eine einzige hiesige Firma hat zusammen mit ihrem Pariser Hause während des russisch-japanischen Krieges Bijouterie für Damen nach Ostasien, speziell an ihr eigenes dafür errichtetes Geschäftshaus in Charbin im Betrag von 4-5 Mill. Frcs. geliefert — für die mit der russischen Armee ausgerückten Scharen von Halbweltdamen. Nach Beendigung des Krieges mußte das Importhaus dieser Firma in Charbin wieder ganz aufhören, und der Vertreter einer anderen hiesigen Bijouteriefirma, der persönlich monatelang auf dem Kriegsschauplatz war, kann grauenhafte Dinge von der Erotik im russischen Lager erzählen, der man ja wohl mit Recht einen großen Teil der Schuld gibt an der Niederlage der Russen. Daß solche Anschauungen bei russischen Studenten, wie bei russischen Offizieren herrschten, wußte man im Grunde genommen bei uns und überall; man stieß sich nicht einmal daran, man nahm das eben als „russisch“ hin, als Ausdruck der Korruption gewisser Kreise, mit denen wir sonst keine Gemeinschaft suchen. Wenn aber, wie das im Vorwort geschieht, diese speziell „slavische Schweinerei“ — so hört man sie wohl gelegentlich bei uns nennen — auf eine allgemein menschliche Basis gestellt wird, wenn damit eine Passivität gegenüber aller Auflehnung gegen die staatliche Ordnung, namentlich aber die Befriedigung eines persönlichen Freiheitsdranges („Ich lebe für mich“ Seite 12), verknüpft wird, so liegt darin meines Erachtens eine schwere Gefahr, zugleich aber auch eine ungeheuere Anmaßung der Träger jener zersetzenden Weltanschauung, die die Kraft der slavischen Rasse so verhängnisvoll untergraben hat, die auch erschreckend am Mark der romanischen Völker nagt und auch in unserem Volk, besonders durch die herrschenden Richtungen in der Literatur, immer mehr Boden gewinnt. Es ist wohl kein Zufall, daß sich ein Franzose — als solchen darf ich wohl Herrn A. Villard, den Verfasser des Vorworts, vermuten — so lebhaft zum Propheten des neuen Evangeliums der freien Liebe aufwirft. Wenn in der Tat das Buch als Ausdruck einer längst vorhandenen Stimmung „dessen was in der Jugend schon seit Jahren gärte —“ so korrigiert Kurt Aram in der „Frankfurter Zeitung“ das Vorwort — („Frkf. Ztg.“ 1908, 16. September, Abendblatt), nicht als Ausgang
- 67. einer neuen Bewegung — auch nur annähernd die Wirkung in Rußland hervorrief, die die Vorrede andeutet — ich kann das mit den mir zur Verfügung stehenden Mitteln nicht genügend beurteilen, habe aber Nachforschungen in Rußland selbst angestellt, über die literarischen Wirkungen des Buches, deren Ergebnis ich nachzutragen hoffe —, wenn diese Wirkungen tatsächlich hervorgerufen wurden, dann ist bei der Sensationssucht, mit der unsere Presse sowohl wie nicht selten auch aus rein geschäftlichen Gründen unser Buchhandel spekuliert (vgl. den Fall Ganter), nicht ausgeschlossen, daß auch bei uns das Buch eine Mission erfüllt — von erschreckender Wirkung. Denn die Faktoren, die einem Buch — ohne Rücksicht auf seinen Inhalt und wirklichen Wert — einen Riesenerfolg bereiten, sind vollkommen unberechenbar, wie die Schicksale mancher literarischer Produkte der jüngsten Zeit beweisen. Die reklamehaften Behauptungen des Vorworts stehen aber zweitens auch im Widerspruch mit dem Roman und seinem Inhalt selbst. Aus der überschwenglichen Art, wie hier im Vorwort gerade die erotische Seite des „Ssanin“ dargestellt wird, möchte ich fast den Schluß ziehen, daß die erotischen Momente, die im Roman selbst für das ins Auge gefaßte Publikum zu nüchtern behandelt sind, im Vorwort zum Zweck größeren Erfolgs aufgebauscht, ja direkt in falsches Licht der Beurteilung gerückt werden. Sonst könnte Villard nicht von „Veränderungen der gesamten Anschauungen einer Gesellschaft von Grund aus“ sprechen, er könnte auch nicht sprechen vom Beiseitewerfen der traditionellen Geheimniskrämerei in solchen Dingen. Die beiden praktischen Vertreter der freien Liebe im „Ssanin“ sind genau so befangen in den konventionellen Schranken und genau so ehrbar nach außen, wie solche Leute bei uns zu sein pflegen. Ja, wenn sie noch die Schranken durchbrächen und in ihrer „neuen Weltanschauung“ sich vor die breite Oeffentlichkeit stellten, das denkt und wünscht sich offenbar der Uebersetzer, der Autor ist ganz anderer Meinung. Zunächst konnte der im Vorwort besprochene Zusammenhang zwischen dem Sexualleben und der russischen Revolution einem, der von sexuellen Erregungen, von besonderen seelischen
- 68. Schwingungen bei derartigen sozialen Erschütterungen gehört hat, einleuchtend erscheinen, zumal, wenn es sich um solche Erscheinungen während der Revolution handelt, von denen Villard (S. 11) andeutungsweise spricht. Der Hinweis auf die Anarchisten mit ihrem Terrorismus, die Villard als die Pfadfinder Ssanins und der Ssaninisten bezeichnet, erinnert daran, daß in den Kreisen der Anarchisten in der Tat auch „ein sexueller Rausch“ herrschte, freilich ganz anderer Art, als der von Villard angedeutete, — Anmerkung: Die von Villard geradezu als vorbildlich für die russische Intelligenz hingestellte Betätigung der freien Liebe, seitens der terroristischen Anarchisten, ist bekanntlich eine propagandistische Forderung des Anarchismus überhaupt, — nämlich eine pathologische Ausprägung des Sexuallebens, der Sadismus. Ueber diesen sagt Forel (Die sexuelle Frage, S. 239) in Uebereinstimmung mit Lombroso, daß „durch Kampf und Schlacht und die vom Krieg entfesselte Mordlust aufgeregte Soldaten zu viehischen sexuellen, wollüstigen Exzessen geführt werden.“ Ein erschütternder, sinnverwirrender Beleg dafür findet sich speziell mit Bezug auf die russische Revolution bei Bloch (Das Sexualleben unserer Zeit, 2. und 3. Aufl., S. 646 ff.) mitgeteilt von Magnus Hirschfeld: „Ein Beitrag zur Psychologie der russischen Revolution“ (Entwicklungsgeschichte eines algotagnistischen Revolutionärs, von diesem selbst verfaßt). Von sadistischen Regungen ist aber in vorliegendem Roman nur ganz selten andeutungsweise die Rede, solche Regungen kommen nach dem Urteil hervorragender Sexualpsychologen häufig bei exzessiver Betätigung eines normalen Geschlechtstriebes vor, sie geben darum auch dem „Ssanin“ keineswegs ein abnormes Gepräge. Zweiter Teil. (Disposition nach den im Schreiben des Untersuchungsrichters vom 31. Dezember 1908 gestellten Fragen:) 1. Ueber den literarischen und kulturhistorischen Wert des Romans.
- 69. Der Gesamteindruck des Werkes in literarischer Hinsicht ist kein günstiger. Es fehlt der einheitliche Aufbau und die das Ganze mehr oder minder beherrschende Haupthandlung. Es sind vielmehr aneinandergereihte, innerlich wenig oder gar nicht verknüpfte Bilder, meist Gruppenbilder von Personen, die der Verfasser uns eigens vorführt, um interessante Typen aus dem russischen Gesellschaftsleben zu zeigen. So ist meines Erachtens die Bezeichnung „Roman“ nicht recht angebracht. Die Bilder sind mit starkem Realismus gezeichnet und gewähren in der Tat tiefe Einblicke in die russischen Verhältnisse, da sie sowohl das landschaftliche, das soziale und das geistige Milieu, überhaupt das Zuständliche, wie auch das individuelle Seelenleben anschaulich und eindrucksvoll schildern. Insofern kommt dem „Ssanin“ ohne Zweifel ein kulturhistorischer Wert zu. Was aber den literarischen Genuß empfindlich beeinträchtigt, ist die breite Wiedergabe zahlreicher, meist ziemlich oberflächlicher philosophischer Erörterungen und Monologe, die — im Gegensatz zum Vorwort sei dies betont — häufig nichts von Erotik enthalten. Direkt unkünstlerisch, ja unästhetisch wirkt nach meinem Geschmack die Behandlung geschlechtlicher Vorgänge — ich lasse dabei zunächst die moralische Seite außer acht. Wenn das Weib sonst in der schönen Literatur auftritt, selbst da, wo ihm eine im ganzen erniedrigende Rolle zugewiesen wird — ich erinnere nur an Sudermanns neuesten Roman „Das hohe Lied“ — so trägt seine Erscheinung doch wenigstens stellenweise das Gepräge echter Schönheit und Weiblichkeit im edlen Sinne. Das fehlt hier bei Artzibaschew ganz. Stets ist nur vom Weib in rein physischer Beziehung, ich möchte sagen, in zynischer Nacktheit, die Rede, von seinen Schenkeln, Brüsten, dem Rücken usw. — sei es in der Phantasie des Mannes oder in Wirklichkeit. Und selbst der Bruder tritt der Schwester mit solchen Empfindungen gegenüber, diese einseitige Auffassung und Vorstellung des Verfassers hat literarisch einen schweren, meines Erachtens seine ganze künstlerische Qualität in Frage stellenden Mangel zur Folge, nämlich die Unfähigkeit zum Differenzieren, was gerade bei einem so schwierigen Problem, wie dem des Weibes und des Sexuallebens, besonders schwer ins Gewicht fällt. Darunter
- 70. leidet namentlich auch seine Charakterschilderung der Männer in ihrem Verhältnis zum weiblichen Geschlecht. Man hat das Gefühl, wie wenn alle diese Männer das Weib durch genau die gleiche Brille besehen, gleich begehrlich, gleichweit entfernt von jeder noch so bescheidenen Würdigung des weiblichen Charakters als etwas nicht rein Sinnlichen, Animalischen. Nun mag vielleicht seine Grundauffassung vom Weib seitens des Mannes in Rußland typisch sein — ich kann mir das zwar nicht recht vorstellen, daß in der Heimat der Frauenemanzipation und des weiblichen Studententums die Frau keine andere Stellung gegenüber dem Manne sich zu erringen vermochte und stehe diesem „Typus“ des russischen Gesellschaftslebens skeptisch gegenüber — künstlerisch ist sie gewiß nicht. Die Verwickelungen, die das Buch enthält, stehen zum Teil auf sehr schwachen Füßen, oft spielen Zufälligkeiten, die keinerlei logischen und psychologischen Zusammenhang mit dem sonstigen Verlauf der Dinge haben (Karssawina und Ssanin im intimen Verkehr) eine ausschlaggebende Rolle, und Lösungen (wie der Selbstmord Juriis) der Konflikte treten mitunter ganz plötzlich und unmotiviert ein. So darf denn meines Erachtens der literarische Wert des Buches trotz mancher anerkennenswerter Lichtseiten im ganzen nicht hoch angeschlagen werden. Schwerwiegende Mängel treten zu sehr in den Vordergrund, als daß der Gesamteindruck ein erfreulicher genannt werden könnte. 2. Ueber die wissenschaftliche Bedeutung der Behandlung erotischer Fragen. Wissenschaftlich, d. h. in der Absicht, unser Wissen über das menschliche Liebesleben irgendwie zu fördern, hat der Verfasser erotische Fragen sicher nicht behandelt. Nicht ein höherer Zweck, wie ihn das wissenschaftliche Streben im Auge hat, sondern einfach die Absicht, Zustände und Anschauungen wie sie sind, darzustellen, hat dem Verfasser die Feder geführt. Dann kann ich — unbeschadet meiner späteren Ausführungen unter Nr. 4 — gleich hier beifügen, daß dem Verfasser nach meiner Meinung auch kein verwerflicher Nebenzweck, etwa absichtlichen Sinnenreizes bei Darstellung geschlechtlicher Vorgänge, vorgeschwebt hat.
- 71. 3. Ueber die Güte der Uebersetzung. Darüber vermag ich mich nicht zu äußern in Ermangelung russischer Sprachkenntnisse. Aber der Eindruck, den die sprachliche Form in der vorliegenden Uebersetzung macht, ist fast durchweg günstig. Ich habe nur an einigen Stellen Anstoß an Wortformen genommen, die anscheinend auf kleine Mängel in der Sprachkenntnis zurückgehen, u. a. an dem öfters wiederkehrenden Adjektiv „bange“, das — entgegen unserem Sprachgebrauch — zu sachlichen Qualifikationen verwendet wird, so S. 263, 359, 364 und 373. 4. Wird die Darstellung geschlechtlicher Vorgänge durch die vorherrschenden wissenschaftlichen Zwecke dermaßen in den Hintergrund gedrängt, daß das Scham- und Sittlichkeitsgefühl des normal empfindenden Lesers nicht verletzt wird? Die so gestellte Frage muß ich bejahen. Wer ohne Voreingenommenheit an die Lektüre des Ganzen herantritt, kann wohl an der oft bis zum Krassen, Frivolen, ja Zynischen gesteigerten Realistik (bes. S. 196-197, 215, 231-233, 248, Kap. 26, S. 318, 428- 429, 440-441 u. a.) Anstoß nehmen. Er wird aber aus dieser Realistik nicht dem Autor einen sittlichen Vorwurf machen, weil meines Erachtens auch das Ausmalen widerlicher Vorgänge, die Darstellung abstoßender, niedriger Denkweise nirgends in der Absicht geboten ist, Freude an pikanten, den Sinnenreiz erregenden Szenen zu bezeigen und diese Freude etwa auf andere zu übertragen. Die ganze Eigenart des Buches, das den Eindruck starker Wahrheitsliebe und rückhaltloser Ehrlichkeit macht, ist nicht auf den gemeinen Grundton pornographischer Schreibweise gestimmt. Der Verfasser, der nun einmal die Zustände der russischen Gesellschaft ohne Schminke schildern wollte, mußte die Farbe so auftragen, wenn aus einem Gemälde überhaupt etwas Echtes, Brauchbares werden, wenn dabei ein „kulturhistorischer“ rein „wissenschaftlicher“ Zweck im Auge behalten werden sollte. Wenn ich krasse Vorkommnisse zu erzählen und dekadente Stimmungen zu schildern habe, dann kann ich jene nicht formlos und diese nicht ideal hinstellen — will ich nicht der höchsten sittlichen Pflicht als Autor mich entledigen. So sind nun einmal die
- 72. Dinge in Rußland. Würde sie Artzibaschew nicht schildern, wie sie sind, sondern, wie er sie haben möchte, dann läge in Anbetracht zahlreicher Stellen seines „Ssanin“ ein Bestreben vor, das bei solch rauhem Realismus, wie er da zum Ausdruck kommt, als äußerst bedenklich bezeichnet werden müßte. Das schließt aber ohne Zweifel die Behauptung in sich, daß wir es hier mit einem ausgesprochenen Tendenzroman zu tun haben. Und dieser Behauptung widerspreche ich mit Entschiedenheit. Wer sie verteidigt, der verurteilt indirekt den Ssanin als eine pornographische Schrift. Denn wenn das geschlechtliche Leben und alles, was damit zusammenhängt, nicht wirklich so wäre, wie es uns hier vor Augen geführt wird, wenn es vom Autor mit seiner angeblichen Tendenz nur so erdacht wäre, um die Unterlage einer neuen, erotischen Weltanschauung zu bilden, — das Vorwort Villards will das glauben machen — dann müßte sich allerdings das Scham- und Sittlichkeitsgefühl des normal empfindenden Lesers schwer verletzt fühlen. Zustände und Anschauungen, noch dazu in einem fremden Lande, die aus tausend Ursachen so geworden sind, die können wir aber, ob wir nun Kenntnis davon haben oder nicht, ob sie sittlich gut oder verwerflich sind, nicht ändern, nicht ablehnen, nicht durch literarische Konfiskationen usw. beseitigen, Zumutungen aber, Aufdringlichkeiten unsittlicher Art, kurzum Tendenzen, die im Gehirn eines Schriftstellers ihren Ursprung haben, können wir ablehnen, von uns fernhalten, ebenso gut wie die Zudringlichkeiten von Leuten, die uns am Leib und Leben, an Hab und Gut schädigen wollen, nötigenfalls unter Anrufung des gesetzlichen Schutzes. Wer dem Ssanin des Artzibaschew anstelle der harten und herben Realistik der Tatsachen eine solche Tendenz vindiziert, der tut ihm meines Erachtens unrecht. Indem nun aber das oben im ersten Teil meines Gutachtens ausführlich erörterte Vorwort in denkbar anspruchsvollster Weise dem Buch eine geradezu das Ganze beherrschende erotische Tendenz zuschreibt, und indem es die Schilderung der ungeheuerlichen Wirkung solcher Tendenz dem, der das Buch zu lesen beabsichtigt, als die beste Empfehlung, die
- 73. wirksamste Reklame zur Weiterempfehlung aufdrängt, stempelt es, wie ich oben bereits ausgeführt habe, das ganze Buch zu einem erotisch-tendenziösen. Es zerstört seine nüchterne Realistik und nimmt ihm seinen unverkennbaren kulturhistorischen, belehrenden, wissenschaftlichen Charakter. Ich stehe nicht an, zum Schluß meines Gutachtens meine Meinung dahin zusammenzufassen: Artzibaschews Ssanin an sich halte ich vom Standpunkt des § 183 Ziffer 1 des St.-G.- B. für einwandfrei. Die vorliegende deutsche Ausgabe jedoch mit Villards Vorwort fällt meines Erachtens unter den Begriff einer unzüchtigen Schrift im Sinne des erwähnten Paragraphen. Pforzheim, 2. Februar 1909. gez.: Prof. Dr. Karl Brunner. 6. Gutachten von Ludwig Ganghofer in München. Der mir zur Beurteilung vorgelegte Roman ist als eine dichterisch hochstehende, aus künstlerischem Geist entsprungene Schöpfung zu bezeichnen, die in der Geschichte der russischen Literatur neben den Meisterwerken von Gogol, Turgenjew, Dostojewski und Gontscharow ihren verdienten Ehrenplatz finden wird. Die Komposition des Romans ist von schöner und strenger Geschlossenheit; die Handlung ist ein starkes und überzeugendes Bild des Lebens, klar geschaut, großzügig erfaßt und mit künstlerischer Kraft aus der Wirklichkeit emporgehoben zu dichterischer Wahrheit; alle Gestalten des Buches sind mit Meisterhand gezeichnet, ohne Beschönigung, ohne Uebertreibung, ohne Absichtlichkeit, im plumperen Sinne dieses Wortes, ohne tendenziöse Verschiebung der Linien, die das Leben dem Dichter zeigte; von fesselnder Wirkung sind die mit den Lebensvorgängen künstlerisch verwobenen Naturschilderungen, die in ihrem malerischen Reiz an die feinsten landschaftlichen Stimmungen bei
- 74. Turgenjew erinnern; und als glühende Seele dieses Buches redet zu uns die leidenschaftliche Vaterlandsliebe des Dichters, seine schwermutsvolle Trauer über das Schicksal seiner Heimat und ihrer Jugend, deren wertvolle Lebenskräfte zwischen politischem Unglück und sozialer Entartung nutzlos verbraucht und zerrieben werden. Die Lektüre dieses Buches brachte mir jenen hohen Genuß, wie ihn nur das Werk eines echten Dichters dem Leser zu bieten vermag — gleichviel, ob es nun licht und erhebend oder erschütternd und bedrückend wirkt. Ich würde nicht nur als Schriftsteller das lebhafteste Bedauern darüber empfinden, wenn dieses Werk aus Gründen, die nur außerhalb seines dichterischen Wertes liegen könnten, bei uns in Bayern eine zensurelle Maßregelung erführe. Denn die an mich gestellte Frage, ob die im „Ssanin“ geschilderten sexuellen Vorgänge geeignet wären, das Scham- und Sittlichkeitsgefühl eines „normal empfindenden Lesers“ zu verletzen, muß ich mit Nein beantworten. Dabei verstehe ich allerdings unter einem „normal empfindenden Leser“ auch einen gesunden, natürlich fühlenden Menschen von relativer Bildung und ernsten Kulturinteressen. Auf entartete und krankhaft gereizte Menschheitsexemplare kann ja auch ein viel harmloseres Kunsterzeugnis, als es der Ssanin ist, eine zu sexueller Nervosität umschlagende Wirkung üben. Aber in einem halbwegs gesunden, natürlich fühlenden und verständigen Menschen wird die Lektüre dieses Buches niemals ein Gefühl der Lüsternheit erwecken, also meines Erachtens auch nie ein Gefühl der sittlichen Empörung über die Form hervorrufen, in welcher die von der Handlung des Romans untrennbaren sexuellen Vorgänge hier geschildert sind. Von einer „wissenschaftlichen Behandlung der erotischen Fragen“ oder von einem „wissenschaftlichen Zwecke“ der hier gegebenen Darstellung geschlechtlicher Vorgänge zu sprechen, erscheint mir diesem Werke gegenüber als nicht ganz zutreffend. Der Schöpfer dieses Werkes wollte nicht als Arzt oder als Gelehrter sprechen, sondern als Dichter. Er wollte kein wissenschaftliches Compendium der modernen Erotik seiner Heimat verfassen, sondern mit Wahrheit und künstlerischer Kraft das versinkende Leben einer bedrückten und irregeleiteten Jugend schildern, die sich — von höheren
- 75. kulturellen Lebenszielen auf politischem und sozialem Gebiete gewaltsam abgedrängt — auf die tierischen Reservatrechte der Menschheit beschränkt sieht und sich in verschärfter Intensität mit allem beschäftigt, was ihr als erotisches Problem erscheint. Diese Jugend sagt: Untergang und Niederbruch auf allen Seiten, suchen wir also wenigstens einen Fortschritt auf diesem einen Gebiete zu erzielen, auf dem uns die Arbeit nicht unterbunden werden kann. Die latente Nähe des Galgens erzeugt sexuelle Subtilitäten. Ich halte es für einen der Grundgedanken des vorliegenden Romans, daß der Dichter aussprechen wollte: Hindert eine blutvoll und stürmisch heranwachsende Jugend an der redlichen Betätigung ihrer besten Lebenskräfte, nehmt ihr die fliegenden Hoffnungen, die sie emportragen über die Dunkelheiten des ewig Menschlichen, so werden die Feinfühligsten dieser Jugend zu Unglücklichen wie Ssoloveitschik, die gedankenvollen Schwächlinge zu ratlosen Selbstmördern wie Jurii, die geistig minderwertigen Menschtiere zu perversen Schweinen wie Woloschin und Sarudin, die seelisch und körperlich Starken zu einsamen Spöttern und zu rücksichtslosen Lebenstrinkern wie Wladimir Ssanin, der aus allem schwülen Sturm dieses Buches höhnisch und lachend hinausschreitet ins Ziellose, ein Zyniker und doch ein Held, der keinen Menschen haßt, aber auch um keines Menschen willen leidet. Man mag erschrecken vor aller Wahrheit, die hier geschildert wird; aber man darf für diese Wahrheit nicht den Dichter verantwortlich machen, der solche Wahrheit sah und zeigte. Aus dem ernsten Lebensklang seines Buches, aus dem Gang der Handlung, aus dem Kontrast der geschilderten Figuren, aus der Schärfe des Gegensatzes, mit dem der Dichter das Helle gegen das Dunkle stellt, das Kraftvolle gegen die Schwäche, gesunde Natur gegen das Verkrüppelte und Entartete, die Reinheit gegen die Vertierung, das Ersehnenswerte gegen das wirklich Bestehende — aus diesen kontrastierenden Farben und Bildern des Werkes wäre ohne mühsame Kombination zu erweisen, daß dieses erschütternde Werk geschaffen wurde, um an giftige Lebenswunden das brennende Eisen zu legen. Aber mit Worten ist eine solche Absicht im „Ssanin“ nirgends ausgesprochen. Der echte Künstler hat es nicht
- 76. nötig, seinem großgefaßten Werke jenes Moralfähnchen anzuhängen, wie es in der kleinen Fabel des Kinderbuches üblich ist. Im Zusammenhange mit allem mutigen Ernste dieses Buches vermag die Darstellung der für die Handlung unumgänglichen erotischen Szenen einen normalen und gesunden Leser weder sinnlich zu erregen, noch sein Scham- und Sittlichkeitsgefühl zu verletzen. Aber auch losgelöst aus dem künstlerischen Zusammenhange dieses Buches, jede der inkriminierten Stellen für sich allein betrachtet — so, wie sie in der an mich gerichteten Zuschrift von der Kgl. Staatsanwaltschaft nach Seitenzahlen aufgeführt wurden — können diese Darstellungen nicht als unsittlich oder unzüchtig bezeichnet werden. Diese Schilderungen gehen nie von der Absicht aus, die Lüsternheit des Lesers zu erwecken oder eine Wirkung durch die Spekulation auf seine tierischen Instinkte zu erzielen. Hinter all diesen Szenen stehen ernste, psychologische, kulturelle und nationale Werte; die Form der Darstellung ist immer ruhig, abgeklärt, reinlich und vornehm, sie vermeidet mit Geschmack jede Linie und jedes Wort, das über die Grenzen des künstlerisch Notwendigen und Zulässigen hinausginge. Mit einer einzigen Ausnahme. In der Szene zwischen Jurii und Karssawina — Seite 440 — störte mich die textliche Brutalität der drei Worte auf Zeile 25. Diese Wendung ist geschmacklos, eine stilistische Entgleisung, von der ich nicht entscheiden kann, ob sie dem Dichter oder dem Uebersetzer anzukreiden ist. Ich möchte das letztere vermuten. Im übrigen wird die Uebersetzung, von kleinen Nachlässigkeiten der Sprache abgesehen, dem ernsten Charakter des Buches wohl gerecht, so daß sie als literarische Arbeit zu bezeichnen ist. Nicht völlig einverstanden bin ich mit einem Abschnitt der Vorrede. Die Behauptung, daß „der wilde sexuelle Rausch“, der einen Teil der russischen Jugend erfaßte, „auf den „Ssanin“ zurückgeht“, scheint mir historisch nicht richtig. Ich erwähne das, weil ich mich zu der Bemerkung verpflichtet fühle, daß jene Stelle der Vorrede — Seite VIII — für mich den unbehaglichen Beigeschmack einer nicht sehr delikaten Anpreisung des Buches bekam. Es ist möglich, daß ein solcher Eindruck in mir vorbereitet war durch den geschmacklosen und marktschreierischen Aufdruck der Buchhändlerschleife, mit
- 77. welcher der Band verschlossen war. Es ist richtig, daß der „Ssanin“ in Rußland verboten wurde. Aber mit dieser Tatsache buchhändlerischer Reklame zu machen, erscheint mir als unanständig. Und Reklame der gleichen Gattung ist der Aufdruck: „Ursprung der sexuellen Revolution“. Dieser Reklameschrei, der sich übel an eine literarisch und kulturhistorische Sache anhängt, ist überdies eine Unwahrheit, denn der „Ssanin“ ist weder der „Weltanschauungsroman des heutigen Rußland“, noch weniger „der Ursprung der sexuellen Revolution“. Durch solche Reklame wird das Anstandsgefühl eines normalen Menschen verletzt, nicht aber durch dieses künstlerisch wertvolle Buch, das meines Wissens in Rußland nicht aus Sittlichkeitsgründen, sondern aus politischen Motiven verboten wurde. Denn dieses Buch — dessen sittlicher Wert allein schon durch das grauenvolle Schicksal dokumentiert wird, dem der Dichter die Gestalt des Masochisten Sarudin überantwortet — dieses Buch mit seinem flammende Geiste und seiner peitschenden Ironie war geeignet, die russische Jugend aus ihrer seit Jahrzehnten entwickelten, schon vor dem „Raskolnikow“ und „Oblomow“ angebahnten Verirrung und Versumpfung aufzurütteln und zu neuem Widerstande gegen die in Rußland herrschenden politischen Mißstände zu beseelen. Daß der „Ssanin“ nach seinem Erscheinen in unreifen Gehirnen und krankhaften Organismen der russischen Jugend mancherlei Mißverständnisse und Verwirrungen anrichtete, das ist dem Dichter und seinem Werke ebensowenig zur Last zu legen, wie die nationale und kulturelle Wirkung Goethes durch die Tatsache zu belasten wäre, daß sich nach dem Erscheinen der „Leiden des jungen Werthers“ ein paar sensible Schwächlinge aus törichter Eitelkeit erschossen. Geniale dichterische Werke pflegen nach einigen Erschütterungen, die sie bei Unverständigen anrichten, reinigend zu wirken und gesunde Erneuerungen des Lebens vorzubereiten. München, den 29. Januar 1909. gez. Ludwig Ganghofer.
- 78. 7. Gutachten von Dr. Franz Muncker, Professor an der Universität München. München, 7. Januar 1909. Ein Sachverständigengutachten über den Roman „Ssanin“ von Artzibaschew in dem ganzen Umfang, wie es von mir gefordert wurde, kann ich nicht abgeben: Zunächst kann ich über den Wert der Uebersetzung nur mit Einschränkung urteilen. Das russische Original liegt mir nicht vor, und auch wenn dies der Fall wäre, würde meine — ziemlich dürftige — Kenntnis der russischen Sprache nicht ausreichen, daß ich wirklich über die Treue und Güte der Uebersetzung sprechen dürfte. Von meinen näheren Kollegen an der Universität wäre dazu meines Wissens am ersten Professor Dr. Krumbacher befähigt. Ich kann nur beurteilen, ob das Deutsch, das der Uebersetzer schreibt, gut und künstlerisch ist. Darin sind mir hie und da kleine grammatikalische Sorglosigkeiten, bisweilen auch eine allzu russisch klingende Wendung aufgefallen; im ganzen aber ist die sprachliche Darstellung ungezwungen, frisch und gewandt: die Uebersetzung liest sich wie ein gutes deutsches Originalwerk. Auch über den kulturhistorischen Wert des Romans habe ich kein eigentliches Sachverständigenurteil. Von Berufs wegen gehen mich die kulturellen Verhältnisse des modernen Rußland nichts an; was ich von ihnen weiß, stammt in der Hauptsache aus den Quellen, aus denen sich jeder andere Gebildete ebensogut wie ich über solche Dinge unterrichten kann, aus Zeitungen oder aus Gesprächen mit Leuten, die mehr davon zu wissen scheinen. So vermag ich nicht mit Sicherheit darüber zu urteilen, ob Ssanin wirklich, wie es in der Vorrede der deutschen Ausgabe heißt, die sexuelle Revolution in Rußland hervorgerufen hat, oder ob er nur ein künstlerisches Abbild von dieser Revolution gibt. Daß jedoch die in ihm gekennzeichneten philosophisch sittlichen Anschauungen und Freiheiten des geschlechtlichen Lebens
- 79. tatsächlich der Wahrheit entsprechen, steht nach den Berichten der Zeitungen außer Frage. Auch könnte ich mich dafür auf bestätigende Aeußerungen berufen, die eine der ersten Persönlichkeiten der Petersburger Universität, Staatsrat Th. v. Zielinski, hiesigen Freunden — namentlich auch dem Geheimrat Professor Dr. Crusius hier, dem ich den Roman zu rascher Lektüre gab, eben weil ich wußte, daß Zielinski gerade mit ihm über diese russischen Verhältnisse ausführlich gesprochen hatte; Crusius, einer der größten Kenner alter und neuerer Literatur, stimmt übrigens in allem Wesentlichen meinem Urteil über „Ssanin“ bei — gegenüber getan hat. Durch diese Wahrheit des Inhalts gewinnt der Roman Ssanin, gleichviel wie seine Bedeutung in Rußland selbst geschätzt wird, für uns deutsche Leser allerdings einen hohen kulturgeschichtlichen Wert; und insofern verdient er zweifellos ins Deutsche übersetzt zu werden. Daß er in Rußland verboten worden ist, kann dabei nicht in Betracht kommen. In Rußland wird manches von der Zensur unterdrückt, was bei uns als vortrefflich gilt. Den Roman Ssanin verbot man dort, weil man fürchtete, sein Inhalt möchte der dortigen Jugend gefährlich werden, weil man sah, daß diese Jugend die in dem Roman geschilderte freie Liebe und überhaupt die Lebensanschauung des Titelhelden in wildem Rausche praktisch zum Gesetz erheben wolle. Diese Gefahr besteht bei uns durchaus nicht, weil bei uns die ganze revolutionäre Gärung, überhaupt die politisch sozialen Voraussetzungen fehlen, die in Rußland solchen Bestrebungen die Wege bahnen. Unbestreitbar aber verdiente „Ssanin“ auch um seines literarischen Wertes willen die Uebersetzung ins Deutsche. Mit großer Kraft und Kunst zeichnet der Verfasser eine Reihe von Personen lebenswahr und psychologisch sorgfältig in allen ihren Gedanken, Empfindungen, Reden und Handlungen individuelle Charaktere, die zugleich bedeutsame Typen der verschiedenen Arten von Menschen sind, mit nicht geringerer Kunst erzählt er eine Reihe von Vorgängen, die sich zu einem lebensvollen Gesamtbilde zusammenschließen, und trotz der Breite, mit der er das Meiste in ihnen ausmalt, trotz mancher ermüdenden Einförmigkeit der einzelnen Geschehnisse weiß er sehr wohl den Leser dichterisch anzuziehen, zu spannen und zu fesseln.
- 80. Ohne falsche Ueberladung, aber anschaulich und wirksam schildert er bald die Natur, bald Einzelheiten aus dem sozialen Treiben. Ausführliche Gespräche über Religion, Christentum, philosophische Weltanschauung flicht er ein, um die wechselnden Gedanken und Bestrebungen der russischen Jugend genau zu beleuchten. Diese Gespräche erstrecken sich oft über Dutzende von Seiten; ihnen sind unter anderem die Kapitel 23-25 (S. 276-311), 31-33 (S. 378-409) usw. gewidmet. Mehr als alles übrige beweisen diese umfangreichen und nicht immer gerade kurzweiligen Abschnitte, wie ernste Absichten der Verfasser mit seinem Roman verfolgte. Wer mit unreinen sinnlichen Begierden zu dem Buche greifen würde, den müßten diese Abschnitte unbedingt abschrecken. Er käme aber auch sonst wohl nicht auf seine Rechnung, obwohl von geschlechtlichen Regungen und Handlungen mehrfach in dem Roman die Rede ist. Was der Verfasser an solchen Stellen erzählt, das scheint mir meistens zur Charakteristik der Menschen und der Zustände, um die es sich handelt, künstlerisch und psychologisch geradezu notwendig; wie er es aber erzählt, beweist durchaus den vornehmen Schriftsteller, der rein sachlich, objektiv episch darstellt und von jeder Lüsternheit weit entfernt ist. Ich gehe sogleich zu den einzelnen Stellen über, die in dem gerichtlichen Schreiben an mich vom 28. Dezember 1908 besonders hervorgehoben sind. S. 88-90. Die Hingabe Lydas an Sarudin ist eines der Grundmotive des Romans, als solches daher unentbehrlich. Die Darstellung dieser Hingabe ist ganz sachlich, fast nüchtern, von jeder Beschönigung durch den Erzähler, von jeder lüstern schlüpfrigen Ausmalung frei; streng genommen wird nur das fieberhafte Verlangen Lydas vor dem Akt der Hingabe selbst charakterisiert und zwar durch kurze Andeutungen. Für kleine Mädchen und unreife Jüngelchen sind diese Andeutungen freilich nicht, der ausgewachsene, normal empfindende Leser aber kann in ihnen nichts Unsittliches entdecken. S. 94 u. 96. Wo hier überhaupt etwas Unsittliches liegen soll, kann ich nicht herausbringen. Ebenso geht es mir bei S. 211-213 und 465- 466.
- 81. S. 196-197. Es handelt sich um die Charakteristik eines gemeinen Lüstlings, deren Berechtigung in einem Roman kein literarisch verständiger Mensch leugnen wird. Dieser Charakteristik dient die rohe Rede. Aber der Verfasser streicht selbst das roheste Wort und deutet es nur unbestimmt an, so daß es der Leser nicht einmal mit Sicherheit ergänzen kann. Er weicht hier also geradezu dem aus, was das Schamgefühl des Lesers verletzen könnte. S. 231-233 und 236. Eine sittlich verwerfliche Anschauung wird von dem Helden des Romans ausgesprochen, den der Dichter aber keineswegs als Ideal gezeichnet hat, dessen Gesinnungen er in keiner Weise billigt. Dabei werden verschiedene rücksichtslose Ausdrücke (z. B. das Wort „schwanger“) gebraucht, allein noch besonders hervorgehoben, daß diese unverblümte Rede die schuldige Hörerin aufs tiefste beschämte. Wie diese Stellen aber das Sittlichkeitsgefühl des normal empfindenden Lesers verletzen sollen, ist mir unfaßbar. S. 246-248. Ssanins Worte sind roh, aber ohne jeden lüsternen, geschlechtlich erregenden Sinn. Die künstlerische Wahrheit erforderte übrigens gerade hier unbedingt die Roheit des Ausdrucks, und die gröbste Stelle in Ssanins Rede (S. 248) ist vom dichterischen Standpunkt aus ebenso notwendig wie etwa die Schimpfwörter, die der sterbende Valentin in Goethes „Faust“ ausstößt. S. 316-318. Wieder handelt es sich um die Charakteristik zweier elender Gesellen, die der Verfasser überdies wiederholt als schamlos bezeichnet. Ganz objektiv nüchtern berichtet er über ihre gemeinen Reden, die er verurteilt, deutet aber von diesen Reden nur das Nötigste an, und läßt ihre unzüchtig-witzige Pointe nicht einmal ahnen. Die Stelle ist geradezu ein Beweis dafür, daß er nichts weniger als lüstern wirken will, sonst hätte er von dem Gespräch der erbärmlichen Patrone, sogar mit einem Schein von künstlerischer Berechtigung, viel mehr mitteilen können. Was er sagt, ist kaum unsittlicher als was bei einer Verhandlung über seinen Roman im Gerichtssaal Ankläger und Verteidiger auch sagen müßten. Denn auch aus seinen Worten klingt überall der sittliche Ernst heraus; sein sittliches Urteil schwankt nicht einen Augenblick, hier so wenig wie an anderer Stelle des Romans.
- 82. S. 419-421. Das Gespräch der zwei Männer, ob man eine nackte Frau betrachten dürfe oder nicht, ist rein theoretisch, von jeder Roheit oder Niedrigkeit frei, die folgende Szene aber, wie beide die badenden Mädchen beobachten, ist so einfach, fast naiv, jedenfalls dichterisch so hübsch, daß sie einen gebildeten, rein empfindenden Leser ebenso wenig verletzen kann, wie etwa ein schönes Gemälde, das eine nackte Frau zeigt. Jede unsittliche Wirkung ist hier ausgeschlossen, wenn die Phantasie des Lesers nicht an sich schon verdorben ist. S. 430 u. 435. Rein sachlich, ohne Lüsternheit von seiten des Schriftstellers, wird hier ausgesprochen, daß sich in die Liebesgedanken Juriis auch sinnlich begehrliche Vorstellungen einmischen. Solange nicht bewiesen wird, daß so etwas bei einem jungen Mann, der von wirklicher Liebe erfüllt ist, niemals vorkommt, kann ich das Anstößige oder Verwerfliche dieser Darstellung nicht verstehen. S. 439-443 und 470-473. Sinnlich geschlechtliche Vorgänge werden hier allerdings dargestellt, aber in sachlicher, nüchtern objektiver Weise ohne lüsterne Zutat. Die Vorgänge selbst sind im Gefüge des Romans unentbehrlich; die Form der Darstellung aber kann nicht unzüchtig wirken, weil sie einen rein geschichtlichen Charakter trägt. Wollte man um dieser Szenen willen das Buch verurteilen, so müßte man vorher zahllose Werke alter und neuer Literatur verbieten, so z. B. allerlei griechische, lateinische, italienische, französische, englische, ältere wie moderne deutsche Dichtungen berühmter Autoren, besonders auch mehrere Erzählungen Wielands und Heinses, die viel reicher an ähnlichen, nur zwanzigmal sinnlicheren Stellen sind als der Roman „Ssanin“. S. 494. Auch hier fehlt jede Lüsternheit in dem geschichtlich nüchternen Bericht, von Unsittlichkeit kann keine Rede sein. Ueberblicke ich alle diese Stellen auf einmal und fasse zugleich den Sinn und Inhalt des ganzen Romans zusammen, so kann ich nirgends etwas wahrnehmen, was als unzüchtig gelten könnte. Regungen einer starken Sinnlichkeit werden in den Personen des Romans geschildert, entsprechend den kulturgeschichtlichen und sozialen Absichten, die der Verfasser als künstlerischer Darsteller der
- 83. modernen russischen Jugend verfolgt. Das Buch ist somit keine Lektüre für unreife Leser, für Kinder oder für Ungebildete. Solche könnten sich allerdings an einzelne unverstandene Szenen halten und dann allerlei Anstoß daran nehmen; die Schuld daran trüge aber nur ihr eigener literarisch und moralisch nicht genügend ausgebildeter Geist. Normal empfindende Leser, die auch die nötige künstlerische Bildung besitzen, können meines Erachtens unmöglich in ihrem Scham- und Sittlichkeitsgefühl durch diesen Roman verletzt werden; solche Leser werden vielmehr die Anklage und eine etwaige Verurteilung des Romans, wenn diese aus mir unbekannten juristischen Gründen möglich sein sollte, nicht verstehen können. gez. Dr. Franz Muncker. 8. Gutachten des kgl. Oberstudienrats J. Nicklas in München. Der Roman „Ssanin“ von Artzibaschew ist nach meiner Meinung eine Publikation ohne künstlerischen bezw. literarischen Wert. Das Werk, das die jetzige Jugend Rußlands schildern will, wie sie von revolutionären Ideen und Handlungen zur Erotomanie überging, entbehrt vor allem des Rückgrates eines jeden literarischen Kunstwerks, der Handlung und der Charaktere. Der Verfasser läßt seine „Helden“, die mit Ausnahme des Egoisten Ssanin blasierte, abgelebte, lüsterne, erbärmliche junge Leute sind, lediglich Zigaretten rauchen, in breiter, selbstgefälliger und geschwätzigster Weise ohne tiefere Kenntnis der Welt über alles mögliche, namentlich über ihre weltschmerzlichen Gefühle Raisonnements anstellen; da sie sich in ihrem überreizten Empfinden in der Welt nicht zurechtfinden und keinen Begriff von der Bedeutung der Pflicht und der Arbeit haben, gefallen sie sich fortgesetzt in nichtigen Gefühlsentladungen und suchen den Wert des Lebens in der Befriedigung sexuellen Genusses. Nur die Darstellung der
- 84. psychologischen Vorgänge in der Brust der gefallenen Mädchen Lyda und Karssawina erhebt sich zu einer gewissen literarischen Bedeutsamkeit. Die Uebersetzung ist in fließender und gewandter Sprache gegeben, wenn sie auch nicht immer frei ist von Inkorrektheiten. Eine kulturhistorische Bedeutung, wie sie etwa Goethes „Werthers Leiden“ hat, oder gar einen wissenschaftlichen Wert kann ich dem Buche nicht beimessen; denn auch ohne diesen Roman hat die Welt Kenntnis von den gegenwärtigen soziologisch wichtigen Verhältnissen Rußlands und von seiner Jugend. Diese Bedeutung kann das Werk schon deshalb nicht haben, weil das Geschlechtsproblem nicht in ernster, zurückhaltender und taktvoller Weise behandelt ist, sondern weil die Absicht des Verfassers immer wieder allzu deutlich hervortritt, unter dem Deckmantel künstlerischer Offenbarung auf den Kitzel niedriger Sinnlichkeit und auf gemeine und teilweise perverse Instinkte zu spekulieren. Geradezu abstoßend, ekelerregend und schamlos sind die unflätigen Szenen, in denen ausführlich, eingehend und mit breiter Behaglichkeit dargestellt wird, wie Sarudin gegenüber Lyda, sowie Jurii und besonders Ssanin dem Mädchen Karssawina gegenüber sich benehmen. (S. 211 ff., 430 ff., 439 ff., 470 ff.) Diese Darstellungen haben mit Kunst gar nichts zu schaffen, da sie nicht die mindeste ästhetische Befriedigung hervorrufen und himmelweit entfernt sind von einer Erhebung zu höherer sittlicher oder ästhetischer Auffassung; sie gehen nur darauf aus, die Lüsternheit zu erwecken. Schon diese Stellen allein würden das Urteil rechtfertigen, daß das Buch geeignet ist, eine Verwirrung in die Vorstellungen von Sittlichkeit zu bringen; aber auch noch viele andere Partien sind geeignet, die normalen sittlichen Empfindungen der Leser zu verletzen. (S. 233, 248 ff., 311, 316 ff., 338, 419 f., 494.) Auch das Vorwort, besonders S. VIII, wo von der Organisation der Ssaninisti und von Verbindungen zum freien Geschlechtsgenuß unter Gymnasiasten und Gymnasiastinnen die Rede ist, ist angetan, die Jugend sittlich zu gefährden; es ist dies umsomehr zu befürchten,
- 85. als anzunehmen ist, daß das Buch, falls es frei gegeben würde, besonders von der Jugend gelesen werden würde. Die Rücksicht auf die körperliche und seelische Gesundheit unserer Jugend verlangt gebieterisch, die deutsche Jugend vor der Lektüre solcher literarischer Erzeugnisse zu schützen, und zwar umsomehr, als die Welt nichts verliert, wenn das in Rußland beschlagnahmte Buch auch in Deutschland verboten wird. München, 15. Januar 1909. gez. J. Nicklas, K. Oberstudienrat. 9. Gutachten des Dr. H. Schneegans, Univ.-Professor, Würzburg. Dem mir im Schreiben des kgl. Untersuchungsrichters E. A.-V.-Z. VII 610-08 Tab. Nr. 73/08 E. vom 28. Dezember 1908 auferlegten Auftrage, ein Gutachten abzugeben „über den literarischen und kulturhistorischen Wert des Romans „Ssanin“ von Artzibaschew, über die wissenschaftliche Bedeutung der Behandlung erotischer Fragen in ihm und die Güte der Uebersetzung, sowie darüber, ob die Darstellung geschlechtlicher Vorgänge (s. insbesondere S. 88-90, 94, 96, 196-197, 211-213, 231-233, 236, 246-248, 316-318, 419-421, 430, 435, 439-443, 445-446, 470-473, 494) durch die vorherrschenden wissenschaftlichen Zwecke dermaßen in den Hintergrund gedrängt wird, daß das Scham- und Sittlichkeitsgefühl des normal empfindenden Lesers nicht verletzt wird“, erlaube ich mir im folgenden nachzukommen. Zunächst glaube ich feststellen zu müssen, daß nach meiner Ansicht im Roman von einer wissenschaftlichen Bedeutung oder Tendenz keine Rede sein kann. Ich wüßte nicht, welche „wissenschaftlichen Zwecke“ hier vorherrschen sollten. Eine wissenschaftliche „Belehrung“ über erotische Fragen will der Roman nicht geben. Er ist nicht wissenschaftlicher als irgend ein anderer
- 86. Roman. Es wäre ein ungerechtfertigter Mißbrauch, wenn er diesen Namen beanspruchen wollte. Eine andere Sache ist es natürlich, ob man vielleicht in späterer Zeit aus der Darstellung erotischer Vorgänge, resp. der Erörterung erotischer Fragen in diesem Roman für eine Kulturgeschichte Rußlands im zwanzigsten Jahrhundert Nutzen wird ziehen können. Das glaube ich allerdings, doch gilt das mutatis mutandis von jedem kulturgeschichtlich interessanten Roman. Was die literarische Bedeutung des Romans betrifft, so ist er an und für sich als dichterische Komposition nach meinem Dafürhalten keine hervorragende Leistung. Die ziemlich lose aneinander gereihten Bilder der Liebesverhältnisse mäßiger russischer Kleinstädter vermögen kein sonderliches ästhetisches Interesse zu erwecken. Es fehlt dem Roman an Geschlossenheit der Handlung und an der Erzählung einer spannenden Begebenheit, die Personen sind nicht alle scharf gezeichnet, einige Nebenfiguren heben sich nicht von den andern ab. Im schönsten gelungen ist die Schilderung der Naturvorgänge und am tiefsten die Darstellung der psychischen Zustände der einzelnen Personen. Die wirkliche Bedeutung des Romans ist aber gewiß nach der kulturgeschichtlichen Seite zu suchen. Darüber sagt treffend das Vorwort S. VIII: „Man wird die gegenwärtige Epoche, also die, welche die revolutionäre ablöste, psychologisch und soziologisch nicht beurteilen können, ohne den Ssanin als ihren charakteristischen Niederschlag in den Mittelpunkt der Betrachtung zu ziehen.“ Nach dem Scheitern der Revolution zog sich die „Intelligenz“ in Rußland, wie aus S. X, XI hervorgeht, von der Politik zurück. „Man suchte nach dem Neuen.“ Dieses Neue fand man, wie es scheint, in der praktischen Ausübung der freien Liebe. „Man sah, daß es Gebiete des täglichen Lebens gab, die, trotzdem sie polizeilich nicht strafbar, doch ganz annehmlich waren. Aber niemals hätte man diesem Beispiel zu folgen gewagt, wenn nicht in diesem Zeitpunkt das erlösende „Wort“ für die unbewußten Empfindungen gesprochen worden wäre.“ Dieses Wort sprach Ssanin aus. Deshalb gilt nach dem Vorwort Artzibaschew als der charakteristische Vertreter des heutigen Rußland. Der Roman scheint ungeheuren
- 87. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com