SlideShare a Scribd company logo

Camera 2.0 in Android 4.2

Download as PPTX, PDF
Balwinder Kaur, profile picture
Balwinder Kaur

Camera 2.0 provides a new camera hardware interface in Android 4.2 that addresses limitations in the previous Camera API and enables new camera capabilities. It focuses on per-frame control of the camera, improved access to frame metadata, and support for features like burst mode photography and panoramic shots. The new interface involves changes to the Camera HAL, camera service, and framework to support a more stream-based and configurable camera pipeline.

Technology
1 of 55
Downloaded 642 times
1
2
3
4
5
6
Most read
7
Most read
8
9
10
Most read
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
Camera 2.0 The New Camera Hardware Interface in Android 4.2 Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging Ashutosh Gupta, Principal Software Engineer, Aptina Imaging Android Builder’s Summit, San Francisco, CA 2.18.2013 © 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners. 1 | © 2013 Aptina Imaging Corporation
Agenda • Camera Use Cases ‣ Prominent Use Cases ‣ Limitations of existing APIs ‣ Overview of android.hardware.Camera • Android Framework : Camera Service ‣ Native Camera service ‣ New - Under the hood – Camera 2.0 • It’s all about the Camera hardware ! ‣ Camera Hardware Abstraction Layer ‣ Camera Device Driver • Challenges in Camera HAL development • Emerging Trends • Q&A 2 | © 2013 Aptina Imaging Corporation
Section I Android Camera APIs 3 | © 2013 Aptina Imaging Corporation
Prominent Camera Use Cases • Main Use Cases ‣ Live Preview of Camera Stream • Live Preview + copy of the Frame returned to the application ‣ Capture a frame ‣ Video Recording of a Camera Stream • Secondary Use Cases ‣ Configuring the Camera ‣ Snapshot during video recording ‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved • Information Related Use cases • Receiving more than an image back . e.g. face detection data • Meta Data of an Image 4 | © 2013 Aptina Imaging Corporation
Limitations in Camera API • No support for Burst Mode Photography • No support for Panoramic Shots • Very limited support for frame metadata • No per-frame control of the camera or the image processing pipeline • No access to control subsystems within the camera e.g. Focus, Flash, Image Sensor 5 | © 2013 Aptina Imaging Corporation
Overview of android.hardware.Camera 6 Classes 8 Callback Interfaces • Camera.AutoFocusCallback • Camera • Camera.ErrorCallback • Camera.CameraInfo • Camera.FaceDetectionListener • Camera.Parameters • Camera.OnZoomChangeListener • Camera.Size • Camera.PictureCallback • Camera.Face • Camera.PreviewCallback • Camera.Area • Camera.ShutterCallback • Camera.AutoFocusMoveCallback 6 | © 2013 Aptina Imaging Corporation
Camera class Contains all the methods for the Camera Lifecycle • Open & Release • Access to the Camera Controls • Preview ‣ Direct Live Preview to the display or a texture ‣ Get Preview Frame in a Callback • Capture ‣ Callbacks: Shutter, JPEG, RAW, “Postview” • Lock & Unlock • Actions: startAutoFocus, startSmoothZoom & startFaceDetection 7 | © 2013 Aptina Imaging Corporation
Camera.Parameters Class for Camera Controls ① Mandatory Feature Set ‣ getSupportedPreviewSizes + set/get ② Optional Feature Set ‣ isVideoStabilizationSupported + set/get ③ Custom Feature Set ‣ Camera.Parameters class provides a “dumb” pipe to the hardware for custom controls ‣ set (“your_param_string”, value); get(“your_param_string”); Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes, Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding 8 | © 2013 Aptina Imaging Corporation
… the rest of the Camera Classes • Camera.CameraInfo ‣ For each camera, front or back facing, orientation of the camera image • Camera.Size ‣ width and height of the image • Camera.Face ‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of the face • Camera.Area ‣ Rectangular bounds with a weight ‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto Exposure 9 | © 2013 Aptina Imaging Corporation
Camera 2 Internals Released as part of Android 4.2 Camera 1 => android.hardware.Camera Camera 2 => android.hardware.ProCamera New Camera HAL implementation based on Camera 2.0 – Samsung exynos5 based No application that actually uses Camera 2.0 Disclaimer: All reference to Camera2 in this slide deck is based on reverse engineering AOSP code ! 10 | © 2013 Aptina Imaging Corporation
Section II Android Framework - Camera 11 | © 2013 Aptina Imaging Corporation
High Level Architecture 12 | © 2013 Aptina Imaging Corporation
Android High Level Architecture Source: Android Anatomy and Physiology, Google IO 2008 13 | © 2013 Aptina Imaging Corporation
Hardware Abstraction Layer Camera Source: Android Anatomy and Physiology, Google IO 2008 14 | © 2013 Aptina Imaging Corporation
Camera Subsystem Application Application framework Hardware Independent Camera Service Camera HAL Implementation Camera Device Driver Hardware Dependent Camera Hardware HAL = Hardware Abstraction Layer 15 | © 2013 Aptina Imaging Corporation
Process View App App App Binder IPC ICamera Camera Service Back Facing Front Facing Binder IPC Surfaceflinger ISurface Camera Hardware Camera Hardware Object Object Media server System Call System Call Kernel Driver Kernel Driver 16 | © 2013 Aptina Imaging Corporation
Inside the Camera App Application Code android framework code Camera.java JNI android_hardware_Camera.cpp IBinder Interfaces Camera Service libcameraservice.so Media server 17 | © 2013 Aptina Imaging Corporation
JNI Layer 18 | © 2013 Aptina Imaging Corporation
android_hardware_Camera • Allocates Memory from the Java memory heap for JPEG images. • If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here. • Creates a persistent context for callbacks from native code to Java (JNICameraContext) • Holds references to the Java Camera, Face and Area objects. 19 | © 2013 Aptina Imaging Corporation
Camera Service - with Camera 2.0 20 | © 2013 Aptina Imaging Corporation
Camera Service Application Code android framework code Camera.java JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera Service libcameraservice.so camera.h Camera HAL implementation Media server 21 | © 2013 Aptina Imaging Corporation
Camera Service with Camera 2 Application Code android framework code Camera.java JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera2Client Camera Service libcameraservice.so Camera2Device camera.h camera2.h Camera HAL Camera2 HAL implementation implementation Media server 22 | © 2013 Aptina Imaging Corporation
Camera Service • Runs in the media server process • It is a shared library libcameraservice.so • Queries System for Camera HAL Api version - accordingly creates a CameraClient or Camera2Client for the application • For Camera 1, libcameraservice does: Permission check android.permission.CAMERA Ensures only one Client connects to a Camera Hardware Object Ensures each Process connects to a single Camera Hardware Object Redirects callbacks back to the app layer Accessed over IBinder Interface Number of Cameras Available CameraInfo Details 23 | © 2013 Aptina Imaging Corporation
Camera2Client • Implements android.hardware.camera API on top of camera device HAL version 2 • Has processors that run in their own threads Recording Manages Preview & Recording Streams Streaming Processor Preview Stream Stream Still Image capture output processing JPEG Processor Capture Stream Processes Callback Callback Processor Callback Stream ZSL Queue Processing ZSL Processor ZSL Stream Metadata processing for output frames Frame Processor Manages Capture Sequences for ZSL Capture Sequencer regular and Burst modes 24 | © 2013 Aptina Imaging Corporation
Camera2Device • Manages Input and Output Streams between the Camera HAL and the Service • Functions ‣ Retrieves Static Metadata Information of the Camera ‣ Sends requests for Capture and Streaming ‣ Manages input and output streams including re-processing streams and metadata buffers ‣ StreamAdaptor between ANativeWindow Interface and the Camera HAL stream ops • ANativeWindows is Android Native Window interface ‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and AWB) 25 | © 2013 Aptina Imaging Corporation
Metadata • Camera2 has a lot of emphasis on retrieving metadata from the camera and making it available to the application • Two kinds of Metadata ‣ Static & Frame-Based ‣ Static does not change and is available without opening the camera device (Camera Info Class) ‣ *_INFO is the static metadata ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO, ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO, ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO, ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC, ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP , ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER, ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS, ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO, ANDROID_QUIRKS_INFO etc. • Extensible - there is provision for Vendor Specific Tags 26 | © 2013 Aptina Imaging Corporation
Android Open Source Project (AOSP) Structure • Android Framework ‣ Java: frameworks/base/core/java/android/hardware ‣ JNI: frameworks/base/core/jni • Camera Service ‣ frameworks/av/services/camera/libcameraservice/ • IBinder Interfaces ‣ frameworks/av/include/camera/ICamera.h etc. • IBinder Implementation ‣ frameworks/av/camera/ICamera.cpp etc. • Camera HAL Interface ‣ hardware/libhardware/include/hardware/camera2.h etc. • Camera HAL Implementation ‣ hardware/<vendor>/camera (typically) • Camera Metadata ‣ system/media/camera/include/system/camera_metadata_tags.h 27 | © 2013 Aptina Imaging Corporation
Section III Its all about the Camera Hardware ! 28 | © 2013 Aptina Imaging Corporation
Camera Hardware Abstraction Layer Review of a Typical Implementation 29 | © 2013 Aptina Imaging Corporation
Camera Stack – Camera HAL Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer 30 | © 2013 Aptina Imaging Corporation
Android CameraHAL Library • The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform ‣ Written by hardware vendors (Qualcomm, Samsung, TI, others) • CameraHAL maps Android Camera Service calls to driver functions ‣ Ice Cream Sandwich (ICS) uses camera.h ‣ Jelly Bean (and above) use camera2.h • CameraHAL low level interface communicates with the kernel level driver ‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX) ‣ Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc) 31 | © 2013 Aptina Imaging Corporation
Sample ICS CameraHAL Functional Diagram Camera Service I/F Memory Manager Display Surface Manager CameraHAL Event Notification Manager Camera Manager User Camera Driver Kernel Source: TI OMAP4 /dev/videoX git.omapzoom.org 32 | © 2013 Aptina Imaging Corporation
Sample JB CameraHAL Functional Diagram Camera Service I/F Stream Manager CameraHAL Operations Handler Metadata Handler Reprocess Stream Management User Camera Driver Kernel Source: Samsung Exynos 5 /dev/videoX 33 | © 2013 Aptina Imaging Corporation
JB CameraHAL Block Diagram Discussion • Block Functions ‣ CameraHAL • Initialization, thread creation, function dispatch ‣ Stream Manager • Handle stream events, coordinate buffer and stream usage, manage state machine ‣ Metadata Handler • Acquire per shot metadata, convert to Android format ‣ Reprocess Stream Management • Setup and manage reprocess streams 34 | © 2013 Aptina Imaging Corporation
JB CameraHAL Changes • For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is moved to libcameraservice • CameraHAL 2.0 targeted at HW specific functionality • New CameraHAL functionality currently not passed up to applications • Image metadata has more importance • Reprocessing introduced – Process an already captured image stream • Stream based rather than stream function based ‣ CameraHAL 1.0 has methods for specific stream types (e.g., start_preview(), take_picture()) ‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream()) 35 | © 2013 Aptina Imaging Corporation
Camera Device Driver 36 | © 2013 Aptina Imaging Corporation
Camera Stack – Camera Driver Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer 37 | © 2013 Aptina Imaging Corporation
Android Kernel Camera Driver • The kernel driver presents a standard interface for different types of camera hardware ‣ Camera hardware specific attributes are handled by the low level kernel driver ‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor - differences are handled at the driver level • For Android, Video for Linux 2 (V4L2) is used in many implementations ‣ V4L2 has been in existence for many years ‣ OpenMax (OMX) is also used for a low level driver interface by some vendors. 38 | © 2013 Aptina Imaging Corporation
V4L2 Kernel Driver Block Diagram V4L2 Driver Interface - IOCTL support/dispatch - V4L2 driver infrastructure Controlling Interface - Support for different device configurations - Control device flow Buffer/Memory Management Camera HW Management - Memory allocation (if needed) - One of these blocks for each camera - Buffer management type - Buffer queue/de-queue - Device discovery - Device initialization - Power management - Set/get device specific parameter - Enable/disable image streaming 39 | © 2013 Aptina Imaging Corporation
V4L2 Kernel Driver Resources • Memory ‣ Memory can be either driver-allocated or user-provided ‣ The image transfers from the camera to memory through hardware Direct Memory Access (DMA) ‣ Hardware memory management may be used to avoid contiguous memory requirement • Interrupts ‣ Camera ports support for interrupts on events such as frame start, finish, focus events, etc. • Camera Control: I2C/SPI ‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers ‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C • Control Signals/GPIO ‣ All controlled by the low level driver • Power ‣ Sensor power management is critical to embedded device operation ‣ Sensors support standby mode where settings are maintained while power usage is reduced 40 | © 2013 Aptina Imaging Corporation
V4L2 Driver Buffer Management • One or more buffers are supported • User buffers or kernel-allocated buffers are supported Buff_0 • Buffers are treated the same for preview, capture, video (output resolution does not Buff_x Buff_1 matter) • Buffers are queued to a circular list Buff_3 Buff_2 • Buffer filling starts when the V4L2 Stream_On command is executed • Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re- queues the buffer • The Stream_Off command causes all buffer to be released 41 | © 2013 Aptina Imaging Corporation
Typical V4L2 Preview Sequence (1) • V4L2 preview start up sequence is given below V4L2 Call Driver Events Hardware Events VIDIOC_S_FMT – set format Set image format and size Set both resolution and output pixel format VIDIOC_G_PARM – get Get a camera driver or hardware Read camera parameter parameter parameter VIDIOC_S_PARM – set Set a camera driver or hardware Write camera parameter parameter parameter VIDIOC_CROPCAP – get Return camera cropping None cropping capabilities capabilities VIDIOC_S_CROP – set Set cropping rectangle Set camera cropping rectangle cropping VIDIOC_REQBUFS – request Request buffer support from the None camera buffer driver (user vs. kernel) Loop: VIDIOC_QUERYBUF For kernel allocated buffers, return None – query buffer caps buffer characteristics V4L2_MMAP – map buffers For kernel allocated buffers, None to user space memory map to user space 42 | © 2013 Aptina Imaging Corporation
Typical V4L2 Preview Sequence (2) • V4L2 preview start up sequence (cont) V4L2 Call Driver Events Hardware Events Loop: VIDIOC_QBUF – Queue buffers in the circular queue none queue buffers VIDIOC_STREAM_ON – start Start image capture state Enable image output streaming • V4L preview shut-down sequence V4L2 Call Driver Events Hardware Events VIDIOC_STREAM_OFF – stop Stop streaming, deallocate Disable image output streaming buffer 43 | © 2013 Aptina Imaging Corporation
V4L2 Media Controller Architecture • Designed to support dynamically re-connectable hardware blocks. • Allows for greater programmer control • Introduces the notion of entities, pads and links 44 | © 2013 Aptina Imaging Corporation
V4L2 Media Controller Architecture(Cont.) 45 | © 2013 Aptina Imaging Corporation
Section IV Challenges in Camera HAL Development 46 | © 2013 Aptina Imaging Corporation
Challenges in Camera HAL Development • Memory Management • Various Implementation of Camera Driver • Color Format Conversion • Buffer Synchronization between Camera & Display • Support for Advanced Features 47 | © 2013 Aptina Imaging Corporation
Are all Camera HALs equal ? No • Supported Features depend on hardware capabilities • Supported Features depend on the implementation in the Camera HAL • Performance • Reliability • Extensions to the standard Android Feature Set 48 | © 2013 Aptina Imaging Corporation
A Peek into the Future 49 | © 2013 Aptina Imaging Corporation
Emerging Trends • Computer Vision Applications go mainstream ‣ APIs on Object Tracking, Gesture Recognition become more common place • Computation Photography application ‣ Developers get fine grained control of flash and camera • High Dynamic Range ‣ Ability to capture larger exposure range • 3D Imaging ‣ Use of 2 cameras to generate a 3D image 50 | © 2013 Aptina Imaging Corporation
What’s coming ? 51 | © 2013 Aptina Imaging Corporation
Q&A 52 | © 2013 Aptina Imaging Corporation
References • http://developer.android.com/ • http://www.codeaurora.org • http://omappedia.org • http://source.android.com • http://stackoverflow.com/questions/10775942/android -sdk-get-raw-preview-camera-image-without- displaying-it • http://www.cjontechnology.com/blog/?p=14 • http://graphics.stanford.edu/projects/camera-2.0/ 53 | © 2013 Aptina Imaging Corporation
About Aptina enables imaging everywhere. A leading innovator of CMOS imaging technology, Aptina delivers excellent pixel performance, sensor functionality and camera system capability to a world going visual. Aptina Camera Software Stack (Access) is Aptina’s implementation of the Android Camera HAL with some custom extensions. Contact Information Balwinder Kaur bkaur@aptina.com Ashutosh Gupta ashutosh@aptina.com 54 | © 2013 Aptina Imaging Corporation
Camera 2.0 in Android 4.2

More Related Content

PPTX
Hidden Camera 3 APIs in Android 4.4 (KitKat)
Balwinder Kaur
 
ODP
Android Camera Architecture
Picker Weng
 
PDF
Camera2 API, SHIM, and HAL 3.2 in Android 5.1
Cheng Hsien Chen
 
PDF
Android's HIDL: Treble in the HAL
Opersys inc.
 
PDF
The Android graphics path, in depth
Chris Simmonds
 
PDF
Embedded Android : System Development - Part IV
Emertxe Information Technologies Pvt Ltd
 
PDF
Android Internals
Opersys inc.
 
PPTX
Android 5.0 Camera2 APIs
Balwinder Kaur
 
Hidden Camera 3 APIs in Android 4.4 (KitKat)
Balwinder Kaur
 
Android Camera Architecture
Picker Weng
 
Camera2 API, SHIM, and HAL 3.2 in Android 5.1
Cheng Hsien Chen
 
Android's HIDL: Treble in the HAL
Opersys inc.
 
The Android graphics path, in depth
Chris Simmonds
 
Embedded Android : System Development - Part IV
Emertxe Information Technologies Pvt Ltd
 
Android Internals
Opersys inc.
 
Android 5.0 Camera2 APIs
Balwinder Kaur
 

What's hot (20)

ODP
Embedded Android : System Development - Part III
Emertxe Information Technologies Pvt Ltd
 
PDF
Android IPC Mechanism
National Cheng Kung University
 
PDF
Understaing Android EGL
Suhan Lee
 
PPTX
How to create a camera2
Booch Lin
 
PDF
Android's Multimedia Framework
Opersys inc.
 
PDF
Android Internals at Linaro Connect Asia 2013
Opersys inc.
 
PDF
Embedded Android : System Development - Part IV (Android System Services)
Emertxe Information Technologies Pvt Ltd
 
PPTX
Android audio system(audioplicy_service)
fefe7270
 
PDF
Embedded Android : System Development - Part III (Audio / Video HAL)
Emertxe Information Technologies Pvt Ltd
 
PPT
Learning AOSP - Android Linux Device Driver
Nanik Tolaram
 
PDF
Embedded Android : System Development - Part II (HAL)
Emertxe Information Technologies Pvt Ltd
 
PDF
Embedded Android : System Development - Part I
Emertxe Information Technologies Pvt Ltd
 
PDF
Android Binder IPC for Linux
Yu-Hsin Hung
 
PPTX
U-boot and Android Verified Boot 2.0
GlobalLogic Ukraine
 
PDF
Hardware Accelerated 2D Rendering for Android
National Cheng Kung University
 
PDF
Android ipm 20110409
Tetsuyuki Kobayashi
 
ODP
Q4.11: Porting Android to new Platforms
Linaro
 
PPTX
Binder: Android IPC
Shaul Rosenzwieg
 
PDF
AndroidとSELinux
android sola
 
PPT
Android Audio System
Yi-Hsiang Huang
 
Embedded Android : System Development - Part III
Emertxe Information Technologies Pvt Ltd
 
Android IPC Mechanism
National Cheng Kung University
 
Understaing Android EGL
Suhan Lee
 
How to create a camera2
Booch Lin
 
Android's Multimedia Framework
Opersys inc.
 
Android Internals at Linaro Connect Asia 2013
Opersys inc.
 
Embedded Android : System Development - Part IV (Android System Services)
Emertxe Information Technologies Pvt Ltd
 
Android audio system(audioplicy_service)
fefe7270
 
Embedded Android : System Development - Part III (Audio / Video HAL)
Emertxe Information Technologies Pvt Ltd
 
Learning AOSP - Android Linux Device Driver
Nanik Tolaram
 
Embedded Android : System Development - Part II (HAL)
Emertxe Information Technologies Pvt Ltd
 
Embedded Android : System Development - Part I
Emertxe Information Technologies Pvt Ltd
 
Android Binder IPC for Linux
Yu-Hsin Hung
 
U-boot and Android Verified Boot 2.0
GlobalLogic Ukraine
 
Hardware Accelerated 2D Rendering for Android
National Cheng Kung University
 
Android ipm 20110409
Tetsuyuki Kobayashi
 
Q4.11: Porting Android to new Platforms
Linaro
 
Binder: Android IPC
Shaul Rosenzwieg
 
AndroidとSELinux
android sola
 
Android Audio System
Yi-Hsiang Huang
 
Ad

Viewers also liked (20)

PPTX
Camera2 introdction
Booch Lin
 
PDF
Camera2
Hiroshi Hashimoto
 
PPTX
Introduction of Android Camera1
Booch Lin
 
PPTX
IoT in the Enterprise
Balwinder Kaur
 
PPTX
Going Beyond the Device Heart Beat
Balwinder Kaur
 
PDF
Understanding the Android System Server
Opersys inc.
 
PDF
Video Drivers
Anil Kumar Pugalia
 
PDF
Android Multimedia Framework
Picker Weng
 
PDF
BKK16-403 Android HAL Consolidation Status
Linaro
 
PDF
CS-ISP Overview
chipnmedia-bd
 
PDF
Using Hardware Security with Android Apps
Abhijeet Rane
 
PDF
Imaging on embedded GPUs
Mikael Bourges-Sevenier
 
PPTX
GreenDao Introduction
Booch Lin
 
PDF
Java Svet - Communication Between Android App Components
Aleksandar Ilić
 
PDF
Kernel Recipes 2013 - Overview display in the Linux kernel
Anne Nicolas
 
ODP
Embedded Linux Multimedia
Caglar Dursun
 
PDF
Multimedia con GNU/Linux
Stefano Sabatini
 
PDF
Kernel Recipes 2014 - Testing Video4Linux Applications and Drivers
Anne Nicolas
 
PDF
안드로이드 MediaPlayer & VideoView
Eunjoo Im
 
PDF
Core Android
Dominik Helleberg
 
Camera2 introdction
Booch Lin
 
Camera2
Hiroshi Hashimoto
 
Introduction of Android Camera1
Booch Lin
 
IoT in the Enterprise
Balwinder Kaur
 
Going Beyond the Device Heart Beat
Balwinder Kaur
 
Understanding the Android System Server
Opersys inc.
 
Video Drivers
Anil Kumar Pugalia
 
Android Multimedia Framework
Picker Weng
 
BKK16-403 Android HAL Consolidation Status
Linaro
 
CS-ISP Overview
chipnmedia-bd
 
Using Hardware Security with Android Apps
Abhijeet Rane
 
Imaging on embedded GPUs
Mikael Bourges-Sevenier
 
GreenDao Introduction
Booch Lin
 
Java Svet - Communication Between Android App Components
Aleksandar Ilić
 
Kernel Recipes 2013 - Overview display in the Linux kernel
Anne Nicolas
 
Embedded Linux Multimedia
Caglar Dursun
 
Multimedia con GNU/Linux
Stefano Sabatini
 
Kernel Recipes 2014 - Testing Video4Linux Applications and Drivers
Anne Nicolas
 
안드로이드 MediaPlayer & VideoView
Eunjoo Im
 
Core Android
Dominik Helleberg
 
Ad

Similar to Camera 2.0 in Android 4.2 (20)

PDF
Implementing Computational Camera
Qualcomm Developer Network
 
PDF
UplinQ - implementing computational camera
Satya Harish
 
PPT
Cloud Computing
Goa App
 
PPTX
COSCUP 2017 FACE OFF
PRADA Hsiung
 
PDF
"Accessing Advanced Image Processing Feature Sets with Alvium Cameras Using a...
Edge AI and Vision Alliance
 
PDF
Mopcon2017 - AppDevKit x CameraKit
anistar sung
 
PDF
AnDevCon 2014: Building a Custom Camera Application
Huyen Tue Dao
 
PDF
Building a Custom Camera Application in Android
Huyen Dao
 
PDF
Avigilon acc6.10.0.24-release-notes
Danilo Moreira
 
PDF
Droidcon NYC 2014: Building Custom Camera Applications
Huyen Tue Dao
 
PDF
Droidcon NYC 2014: Building Custom Camera Applications
Huyen Dao
 
PDF
IP Surveillance Rotakin White Paper
Steven Kenny
 
PPT
Omron and InduSoft Web Studio Vision Systems
AVEVA
 
PDF
CameraX: Make photography easier on Android!
Bapusaheb Patil
 
PPT
070515 iseephotovista
bgb1234
 
PPTX
Device APIs at TakeOff Conference
dianacheng
 
PDF
Are Your Applications Delivering What Your End-Users Expect?
Compuware APM
 
PDF
App Engine overview (Android meetup 06-10)
jasonacooper
 
PPTX
Camera camcorder framework overview(ginger bread)
fefe7270
 
DOC
Video Surveillance Report
Mihika Shah
 
Implementing Computational Camera
Qualcomm Developer Network
 
UplinQ - implementing computational camera
Satya Harish
 
Cloud Computing
Goa App
 
COSCUP 2017 FACE OFF
PRADA Hsiung
 
"Accessing Advanced Image Processing Feature Sets with Alvium Cameras Using a...
Edge AI and Vision Alliance
 
Mopcon2017 - AppDevKit x CameraKit
anistar sung
 
AnDevCon 2014: Building a Custom Camera Application
Huyen Tue Dao
 
Building a Custom Camera Application in Android
Huyen Dao
 
Avigilon acc6.10.0.24-release-notes
Danilo Moreira
 
Droidcon NYC 2014: Building Custom Camera Applications
Huyen Tue Dao
 
Droidcon NYC 2014: Building Custom Camera Applications
Huyen Dao
 
IP Surveillance Rotakin White Paper
Steven Kenny
 
Omron and InduSoft Web Studio Vision Systems
AVEVA
 
CameraX: Make photography easier on Android!
Bapusaheb Patil
 
070515 iseephotovista
bgb1234
 
Device APIs at TakeOff Conference
dianacheng
 
Are Your Applications Delivering What Your End-Users Expect?
Compuware APM
 
App Engine overview (Android meetup 06-10)
jasonacooper
 
Camera camcorder framework overview(ginger bread)
fefe7270
 
Video Surveillance Report
Mihika Shah
 

Recently uploaded (20)

PDF
A comparative study of natural language inference in Swahili using monolingua...
IAESIJAI
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PPTX
Chapter 5: Probability Theory and Statistics
RandyFin
 
PDF
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
PDF
project resource management chapter-09.pdf
ssuser00e6e21
 
PPTX
O2C Customer Invoices to Receipt V15A.pptx
ssuserbfa915
 
PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
PDF
Transform Your ITIL® 4 & ITSM Strategy with AI in 2025.pdf
PDCA Consulting
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PPTX
The various Industrial Revolutions .pptx
bandileshozi3
 
PPT
Module 1.ppt Iot fundamentals and Architecture
nanditha7766
 
PDF
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
PPTX
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
PDF
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
PDF
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
PDF
Hybrid model detection and classification of lung cancer
IAESIJAI
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
2021 HotChips TSMC Packaging Technologies for Chiplets and 3D_0819 publish_pu...
KeXue5
 
A comparative study of natural language inference in Swahili using monolingua...
IAESIJAI
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
Chapter 5: Probability Theory and Statistics
RandyFin
 
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
project resource management chapter-09.pdf
ssuser00e6e21
 
O2C Customer Invoices to Receipt V15A.pptx
ssuserbfa915
 
Getting Started with Data Integration: FME Form 101
Safe Software
 
Transform Your ITIL® 4 & ITSM Strategy with AI in 2025.pdf
PDCA Consulting
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
The various Industrial Revolutions .pptx
bandileshozi3
 
Module 1.ppt Iot fundamentals and Architecture
nanditha7766
 
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
Hybrid model detection and classification of lung cancer
IAESIJAI
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
2021 HotChips TSMC Packaging Technologies for Chiplets and 3D_0819 publish_pu...
KeXue5
 

Camera 2.0 in Android 4.2

  • 1. Camera 2.0 The New Camera Hardware Interface in Android 4.2 Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging Ashutosh Gupta, Principal Software Engineer, Aptina Imaging Android Builder’s Summit, San Francisco, CA 2.18.2013 © 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners. 1 | © 2013 Aptina Imaging Corporation
  • 2. Agenda • Camera Use Cases ‣ Prominent Use Cases ‣ Limitations of existing APIs ‣ Overview of android.hardware.Camera • Android Framework : Camera Service ‣ Native Camera service ‣ New - Under the hood – Camera 2.0 • It’s all about the Camera hardware ! ‣ Camera Hardware Abstraction Layer ‣ Camera Device Driver • Challenges in Camera HAL development • Emerging Trends • Q&A 2 | © 2013 Aptina Imaging Corporation
  • 3. Section I Android Camera APIs 3 | © 2013 Aptina Imaging Corporation
  • 4. Prominent Camera Use Cases • Main Use Cases ‣ Live Preview of Camera Stream • Live Preview + copy of the Frame returned to the application ‣ Capture a frame ‣ Video Recording of a Camera Stream • Secondary Use Cases ‣ Configuring the Camera ‣ Snapshot during video recording ‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved • Information Related Use cases • Receiving more than an image back . e.g. face detection data • Meta Data of an Image 4 | © 2013 Aptina Imaging Corporation
  • 5. Limitations in Camera API • No support for Burst Mode Photography • No support for Panoramic Shots • Very limited support for frame metadata • No per-frame control of the camera or the image processing pipeline • No access to control subsystems within the camera e.g. Focus, Flash, Image Sensor 5 | © 2013 Aptina Imaging Corporation
  • 6. Overview of android.hardware.Camera 6 Classes 8 Callback Interfaces • Camera.AutoFocusCallback • Camera • Camera.ErrorCallback • Camera.CameraInfo • Camera.FaceDetectionListener • Camera.Parameters • Camera.OnZoomChangeListener • Camera.Size • Camera.PictureCallback • Camera.Face • Camera.PreviewCallback • Camera.Area • Camera.ShutterCallback • Camera.AutoFocusMoveCallback 6 | © 2013 Aptina Imaging Corporation
  • 7. Camera class Contains all the methods for the Camera Lifecycle • Open & Release • Access to the Camera Controls • Preview ‣ Direct Live Preview to the display or a texture ‣ Get Preview Frame in a Callback • Capture ‣ Callbacks: Shutter, JPEG, RAW, “Postview” • Lock & Unlock • Actions: startAutoFocus, startSmoothZoom & startFaceDetection 7 | © 2013 Aptina Imaging Corporation
  • 8. Camera.Parameters Class for Camera Controls ① Mandatory Feature Set ‣ getSupportedPreviewSizes + set/get ② Optional Feature Set ‣ isVideoStabilizationSupported + set/get ③ Custom Feature Set ‣ Camera.Parameters class provides a “dumb” pipe to the hardware for custom controls ‣ set (“your_param_string”, value); get(“your_param_string”); Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes, Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding 8 | © 2013 Aptina Imaging Corporation
  • 9. … the rest of the Camera Classes • Camera.CameraInfo ‣ For each camera, front or back facing, orientation of the camera image • Camera.Size ‣ width and height of the image • Camera.Face ‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of the face • Camera.Area ‣ Rectangular bounds with a weight ‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto Exposure 9 | © 2013 Aptina Imaging Corporation
  • 10. Camera 2 Internals Released as part of Android 4.2 Camera 1 => android.hardware.Camera Camera 2 => android.hardware.ProCamera New Camera HAL implementation based on Camera 2.0 – Samsung exynos5 based No application that actually uses Camera 2.0 Disclaimer: All reference to Camera2 in this slide deck is based on reverse engineering AOSP code ! 10 | © 2013 Aptina Imaging Corporation
  • 11. Section II Android Framework - Camera 11 | © 2013 Aptina Imaging Corporation
  • 12. High Level Architecture 12 | © 2013 Aptina Imaging Corporation
  • 13. Android High Level Architecture Source: Android Anatomy and Physiology, Google IO 2008 13 | © 2013 Aptina Imaging Corporation
  • 14. Hardware Abstraction Layer Camera Source: Android Anatomy and Physiology, Google IO 2008 14 | © 2013 Aptina Imaging Corporation
  • 15. Camera Subsystem Application Application framework Hardware Independent Camera Service Camera HAL Implementation Camera Device Driver Hardware Dependent Camera Hardware HAL = Hardware Abstraction Layer 15 | © 2013 Aptina Imaging Corporation
  • 16. Process View App App App Binder IPC ICamera Camera Service Back Facing Front Facing Binder IPC Surfaceflinger ISurface Camera Hardware Camera Hardware Object Object Media server System Call System Call Kernel Driver Kernel Driver 16 | © 2013 Aptina Imaging Corporation
  • 17. Inside the Camera App Application Code android framework code Camera.java JNI android_hardware_Camera.cpp IBinder Interfaces Camera Service libcameraservice.so Media server 17 | © 2013 Aptina Imaging Corporation
  • 18. JNI Layer 18 | © 2013 Aptina Imaging Corporation
  • 19. android_hardware_Camera • Allocates Memory from the Java memory heap for JPEG images. • If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here. • Creates a persistent context for callbacks from native code to Java (JNICameraContext) • Holds references to the Java Camera, Face and Area objects. 19 | © 2013 Aptina Imaging Corporation
  • 20. Camera Service - with Camera 2.0 20 | © 2013 Aptina Imaging Corporation
  • 21. Camera Service Application Code android framework code Camera.java JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera Service libcameraservice.so camera.h Camera HAL implementation Media server 21 | © 2013 Aptina Imaging Corporation
  • 22. Camera Service with Camera 2 Application Code android framework code Camera.java JNI android_hardware_Camera.cpp ICameraClient | ICameraService | ICamera Camera2Client Camera Service libcameraservice.so Camera2Device camera.h camera2.h Camera HAL Camera2 HAL implementation implementation Media server 22 | © 2013 Aptina Imaging Corporation
  • 23. Camera Service • Runs in the media server process • It is a shared library libcameraservice.so • Queries System for Camera HAL Api version - accordingly creates a CameraClient or Camera2Client for the application • For Camera 1, libcameraservice does: Permission check android.permission.CAMERA Ensures only one Client connects to a Camera Hardware Object Ensures each Process connects to a single Camera Hardware Object Redirects callbacks back to the app layer Accessed over IBinder Interface Number of Cameras Available CameraInfo Details 23 | © 2013 Aptina Imaging Corporation
  • 24. Camera2Client • Implements android.hardware.camera API on top of camera device HAL version 2 • Has processors that run in their own threads Recording Manages Preview & Recording Streams Streaming Processor Preview Stream Stream Still Image capture output processing JPEG Processor Capture Stream Processes Callback Callback Processor Callback Stream ZSL Queue Processing ZSL Processor ZSL Stream Metadata processing for output frames Frame Processor Manages Capture Sequences for ZSL Capture Sequencer regular and Burst modes 24 | © 2013 Aptina Imaging Corporation
  • 25. Camera2Device • Manages Input and Output Streams between the Camera HAL and the Service • Functions ‣ Retrieves Static Metadata Information of the Camera ‣ Sends requests for Capture and Streaming ‣ Manages input and output streams including re-processing streams and metadata buffers ‣ StreamAdaptor between ANativeWindow Interface and the Camera HAL stream ops • ANativeWindows is Android Native Window interface ‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and AWB) 25 | © 2013 Aptina Imaging Corporation
  • 26. Metadata • Camera2 has a lot of emphasis on retrieving metadata from the camera and making it available to the application • Two kinds of Metadata ‣ Static & Frame-Based ‣ Static does not change and is available without opening the camera device (Camera Info Class) ‣ *_INFO is the static metadata ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO, ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO, ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO, ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC, ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP , ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER, ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS, ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO, ANDROID_QUIRKS_INFO etc. • Extensible - there is provision for Vendor Specific Tags 26 | © 2013 Aptina Imaging Corporation
  • 27. Android Open Source Project (AOSP) Structure • Android Framework ‣ Java: frameworks/base/core/java/android/hardware ‣ JNI: frameworks/base/core/jni • Camera Service ‣ frameworks/av/services/camera/libcameraservice/ • IBinder Interfaces ‣ frameworks/av/include/camera/ICamera.h etc. • IBinder Implementation ‣ frameworks/av/camera/ICamera.cpp etc. • Camera HAL Interface ‣ hardware/libhardware/include/hardware/camera2.h etc. • Camera HAL Implementation ‣ hardware/<vendor>/camera (typically) • Camera Metadata ‣ system/media/camera/include/system/camera_metadata_tags.h 27 | © 2013 Aptina Imaging Corporation
  • 28. Section III Its all about the Camera Hardware ! 28 | © 2013 Aptina Imaging Corporation
  • 29. Camera Hardware Abstraction Layer Review of a Typical Implementation 29 | © 2013 Aptina Imaging Corporation
  • 30. Camera Stack – Camera HAL Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer 30 | © 2013 Aptina Imaging Corporation
  • 31. Android CameraHAL Library • The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform ‣ Written by hardware vendors (Qualcomm, Samsung, TI, others) • CameraHAL maps Android Camera Service calls to driver functions ‣ Ice Cream Sandwich (ICS) uses camera.h ‣ Jelly Bean (and above) use camera2.h • CameraHAL low level interface communicates with the kernel level driver ‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX) ‣ Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc) 31 | © 2013 Aptina Imaging Corporation
  • 32. Sample ICS CameraHAL Functional Diagram Camera Service I/F Memory Manager Display Surface Manager CameraHAL Event Notification Manager Camera Manager User Camera Driver Kernel Source: TI OMAP4 /dev/videoX git.omapzoom.org 32 | © 2013 Aptina Imaging Corporation
  • 33. Sample JB CameraHAL Functional Diagram Camera Service I/F Stream Manager CameraHAL Operations Handler Metadata Handler Reprocess Stream Management User Camera Driver Kernel Source: Samsung Exynos 5 /dev/videoX 33 | © 2013 Aptina Imaging Corporation
  • 34. JB CameraHAL Block Diagram Discussion • Block Functions ‣ CameraHAL • Initialization, thread creation, function dispatch ‣ Stream Manager • Handle stream events, coordinate buffer and stream usage, manage state machine ‣ Metadata Handler • Acquire per shot metadata, convert to Android format ‣ Reprocess Stream Management • Setup and manage reprocess streams 34 | © 2013 Aptina Imaging Corporation
  • 35. JB CameraHAL Changes • For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is moved to libcameraservice • CameraHAL 2.0 targeted at HW specific functionality • New CameraHAL functionality currently not passed up to applications • Image metadata has more importance • Reprocessing introduced – Process an already captured image stream • Stream based rather than stream function based ‣ CameraHAL 1.0 has methods for specific stream types (e.g., start_preview(), take_picture()) ‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream()) 35 | © 2013 Aptina Imaging Corporation
  • 36. Camera Device Driver 36 | © 2013 Aptina Imaging Corporation
  • 37. Camera Stack – Camera Driver Upper Camera Stack … Camera Hardware Abstraction Layer SurfaceFlinger (HAL) /Overlay Buffers User Vendor Specific HAL Implementation Camera Driver Kernel Image Sensor Image Sensor Hardware Processer 37 | © 2013 Aptina Imaging Corporation
  • 38. Android Kernel Camera Driver • The kernel driver presents a standard interface for different types of camera hardware ‣ Camera hardware specific attributes are handled by the low level kernel driver ‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor - differences are handled at the driver level • For Android, Video for Linux 2 (V4L2) is used in many implementations ‣ V4L2 has been in existence for many years ‣ OpenMax (OMX) is also used for a low level driver interface by some vendors. 38 | © 2013 Aptina Imaging Corporation
  • 39. V4L2 Kernel Driver Block Diagram V4L2 Driver Interface - IOCTL support/dispatch - V4L2 driver infrastructure Controlling Interface - Support for different device configurations - Control device flow Buffer/Memory Management Camera HW Management - Memory allocation (if needed) - One of these blocks for each camera - Buffer management type - Buffer queue/de-queue - Device discovery - Device initialization - Power management - Set/get device specific parameter - Enable/disable image streaming 39 | © 2013 Aptina Imaging Corporation
  • 40. V4L2 Kernel Driver Resources • Memory ‣ Memory can be either driver-allocated or user-provided ‣ The image transfers from the camera to memory through hardware Direct Memory Access (DMA) ‣ Hardware memory management may be used to avoid contiguous memory requirement • Interrupts ‣ Camera ports support for interrupts on events such as frame start, finish, focus events, etc. • Camera Control: I2C/SPI ‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers ‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C • Control Signals/GPIO ‣ All controlled by the low level driver • Power ‣ Sensor power management is critical to embedded device operation ‣ Sensors support standby mode where settings are maintained while power usage is reduced 40 | © 2013 Aptina Imaging Corporation
  • 41. V4L2 Driver Buffer Management • One or more buffers are supported • User buffers or kernel-allocated buffers are supported Buff_0 • Buffers are treated the same for preview, capture, video (output resolution does not Buff_x Buff_1 matter) • Buffers are queued to a circular list Buff_3 Buff_2 • Buffer filling starts when the V4L2 Stream_On command is executed • Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re- queues the buffer • The Stream_Off command causes all buffer to be released 41 | © 2013 Aptina Imaging Corporation
  • 42. Typical V4L2 Preview Sequence (1) • V4L2 preview start up sequence is given below V4L2 Call Driver Events Hardware Events VIDIOC_S_FMT – set format Set image format and size Set both resolution and output pixel format VIDIOC_G_PARM – get Get a camera driver or hardware Read camera parameter parameter parameter VIDIOC_S_PARM – set Set a camera driver or hardware Write camera parameter parameter parameter VIDIOC_CROPCAP – get Return camera cropping None cropping capabilities capabilities VIDIOC_S_CROP – set Set cropping rectangle Set camera cropping rectangle cropping VIDIOC_REQBUFS – request Request buffer support from the None camera buffer driver (user vs. kernel) Loop: VIDIOC_QUERYBUF For kernel allocated buffers, return None – query buffer caps buffer characteristics V4L2_MMAP – map buffers For kernel allocated buffers, None to user space memory map to user space 42 | © 2013 Aptina Imaging Corporation
  • 43. Typical V4L2 Preview Sequence (2) • V4L2 preview start up sequence (cont) V4L2 Call Driver Events Hardware Events Loop: VIDIOC_QBUF – Queue buffers in the circular queue none queue buffers VIDIOC_STREAM_ON – start Start image capture state Enable image output streaming • V4L preview shut-down sequence V4L2 Call Driver Events Hardware Events VIDIOC_STREAM_OFF – stop Stop streaming, deallocate Disable image output streaming buffer 43 | © 2013 Aptina Imaging Corporation
  • 44. V4L2 Media Controller Architecture • Designed to support dynamically re-connectable hardware blocks. • Allows for greater programmer control • Introduces the notion of entities, pads and links 44 | © 2013 Aptina Imaging Corporation
  • 45. V4L2 Media Controller Architecture(Cont.) 45 | © 2013 Aptina Imaging Corporation
  • 46. Section IV Challenges in Camera HAL Development 46 | © 2013 Aptina Imaging Corporation
  • 47. Challenges in Camera HAL Development • Memory Management • Various Implementation of Camera Driver • Color Format Conversion • Buffer Synchronization between Camera & Display • Support for Advanced Features 47 | © 2013 Aptina Imaging Corporation
  • 48. Are all Camera HALs equal ? No • Supported Features depend on hardware capabilities • Supported Features depend on the implementation in the Camera HAL • Performance • Reliability • Extensions to the standard Android Feature Set 48 | © 2013 Aptina Imaging Corporation
  • 49. A Peek into the Future 49 | © 2013 Aptina Imaging Corporation
  • 50. Emerging Trends • Computer Vision Applications go mainstream ‣ APIs on Object Tracking, Gesture Recognition become more common place • Computation Photography application ‣ Developers get fine grained control of flash and camera • High Dynamic Range ‣ Ability to capture larger exposure range • 3D Imaging ‣ Use of 2 cameras to generate a 3D image 50 | © 2013 Aptina Imaging Corporation
  • 51. What’s coming ? 51 | © 2013 Aptina Imaging Corporation
  • 52. Q&A 52 | © 2013 Aptina Imaging Corporation
  • 53. References • http://developer.android.com/ • http://www.codeaurora.org • http://omappedia.org • http://source.android.com • http://stackoverflow.com/questions/10775942/android -sdk-get-raw-preview-camera-image-without- displaying-it • http://www.cjontechnology.com/blog/?p=14 • http://graphics.stanford.edu/projects/camera-2.0/ 53 | © 2013 Aptina Imaging Corporation
  • 54. About Aptina enables imaging everywhere. A leading innovator of CMOS imaging technology, Aptina delivers excellent pixel performance, sensor functionality and camera system capability to a world going visual. Aptina Camera Software Stack (Access) is Aptina’s implementation of the Android Camera HAL with some custom extensions. Contact Information Balwinder Kaur bkaur@aptina.com Ashutosh Gupta ashutosh@aptina.com 54 | © 2013 Aptina Imaging Corporation

Editor's Notes

  • #10: In photography, the metering mode refers to the way in which a camera determines the exposure.Autobracketing is a feature of some more advanced cameras, whether film or digital cameras, particularly single-lens reflex cameras, where the camera will take several successive shots (often three) with slightly different settings. Two modes are used: The images are automatically combined, for example Automatic Exposure Bracketing into oneHigh dynamic range image, or the best-looking separately stored pictures can be picked later from the batch.White balance is a camera setting that adjusts for lighting in order to make white objects appear white in photos. This is more difficult than it might seem due to the fact that light cast from different sources is different in color (technically called temperature). That is to say, light is rarely truly white in nature. The light from an incandescent or halogen bulb, for example, is red/orange in color, while that from the sun is relatively blue. A proper white balance setting in a camera will prevent, for example, a white bed sheet in a photo from appearing orange in color when it is being illuminated by a candle.
  • #24: Provides a common shutter sound and recording sound for all camera clients (using the MediaPlayer) &quot;/system/media/audio/ui/camera_click.ogg&quot;);mSoundPlayer[SOUND_RECORDING] = newMediaPlayer(&quot;/system/media/audio/ui/VideoRecord.ogg&quot;
  • #26: Uses a MetadataQueue class