MIPI DevCon 2016: Accelerating Software Development for MIPI CSI-2 Cameras
2 likes1,944 views
The document discusses accelerating software development for MIPI CSI-2 cameras by leveraging Linux resources. It describes using the V4L2 API and modular design approach with media controller and device tree to model hardware in a way that reduces effort and allows software to adapt to changing hardware configurations. Key aspects covered include defining relationships between devices and sub-devices to represent the hardware pipeline, and using the device tree to describe hardware details that drivers can access to configure themselves. This approach allows high-level interaction with different hardware through a standardized interface and modular design.
MIPI DevCon 2016: Accelerating Software Development for MIPI CSI-2 Cameras
- 1. Accelerating Software Development for MIPI CSI-2℠ Cameras Licinio Sousa Synopsys, Inc.
- 2. The Challenge Developing SW for Multiple HW Configs • Nowadays HW platforms are always evolving • It is hard for SW to keep up with the changes • Using a monolithic approach might be simple to begin, but harder to mantain • To support a new device, the designer then needs to either split the code or add support for another device in the initial code • After a while the code will become “spaghetti code” • Each device has specific configuration requirements and characteristics, which complicates the tasks of providing support and integration, due to limited code reusability • Steep learning curve 2
- 3. The Challenge Developing SW for Multiple HW Configs 3 Complete MIPI Camera and Display Solu6on
- 4. The Solution Leverage Linux Resources to Reduce Design Efforts • Systems with embedded Linux can leverage existing libraries, APIs and driver frameworks to reduce effort and accelerate SW development • V4L2 API and modular design approach • Media controller • Device tree
- 5. Agenda • Introduce a few Linux concepts and resources • Share our experience • Conclusion
- 6. Linux # facts - Linux is a free and open source software ~18 000 000 lines of code - 5.64 patches accepted in mainline kernel / h - More than 1000 engineers work in each release # who is working in Linux - Only 17.9 % make kernel hacking for free - The other 82.1% is made by top tech players
- 8. When is Linux’s Birthday? 8 From:torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds) Newsgroup: comp.os.minix Subject: GCC-1.40 and a posix ques6on Message-ID: 1991Jul13, 100050.9886@klaava.Helsinki.FI Date: 3 Jul 91 10:00:50 GMT Hello netlanders, Due a project I'm working on (in minix), I'm interested in the posix standard defini6on. Could somebody please point me to a (preferably) machine- readable format of the latest posix rules? Ftp-sites would be nice. Linus Torvalds torvalds@kruuna.helsinki.fi From:torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds) Newsgroup: comp.os.minix Subject: What would you like to see most in minix? Summary: small poll for my new operaIng system Message-ID: 1991Aug25, 20578.9541@klaava.Helsinki.FI Date: 25 Aug 91 20:57:08 GMT OrganizaIon: University of Helsinki. Hello everybody out there using minix- I'm doing a (free) operaIng system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. (…) Any suggesIons are welcome, but I won't promise I'll implement them :-) Linus Torvalds torvalds@kruuna.helsinki.fi From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds) Newsgroups: comp.os.minix Subject: Free minix-like kernel sources for 386-AT Message-ID: <1991Oct5.054106.4647@klaava.Helsinki.FI> Date: 5 Oct 91 05:41:06 GMT Organiza6on: University of Helsinki Do you pine for the nice days of minix-1.1, when men were men and wrote their own device drivers? Are you without a nice project and just dying to cut your teeth on a OS you can try to modify for your needs? Are you finding it frustra6ng when everything works on minix? No more all- nighters to get a ni`y program working? Then this post might be just for you :-) As I menIoned a month(?) ago, I'm working on a free version of a minix-lookalike for AT-386 computers. It has finally reached the stage where it's even usable (though may not be depending on what you want), and I am willing to put out the sources for wider distribuIon. It is just version 0.02 (+1 (very small) patch already), but I've successfully run bash/gcc/gnu-make/gnu- sed/compress etc under it. There are actually three "birthdays" one can idenIfy for Linux!!
- 9. Inside Linux – Architecture
- 10. User Space vs. Kernel Space • User Space • If SW is running in userspace it’s usually called an Application • Memory area where application software will run • Kernel Space • Strictly reserved for running privileged OS kernel • Runs kernel extensions and device drivers • A driver has direct access to the HW
- 11. User-Kernel Interface • User-kernel interaction is made via Linux APIs • Linux provides APIs in order to allow programmers to write applications consistent with the operating environment • Some APIs can be divided in 2 parts, which run in two different areas • Kernel space • User space
- 12. Video4Linux2 API API for Video capture • One of the Linux APIs is V4L (Video4Linux) a.k.a. V4L2 • V4L2 is an API for supporting real-time video capture in Linux • V4L2 is used to build drivers for many USB webcams, TV tuners, smartphone cameras, and related devices • With V4L2, applications can easily add video support in a standard way
- 13. V4L2 API Kernel Space ApplicaIon Space V4L2 Linux Drivers V4L2 IOCTLs mplayer ffmpeg vlc V4L2 IOCTLs Hardware Hardware
- 14. V4L2 Devices and Subdevices Modular Approach to Model the HW • Hardware is often made of several integrated circuits that need to interact with each other • This can lead to complex drivers. • The drivers must reflect the hardware model in software • Modelling the different hardware components as software blocks called sub-devices. V4L2 device V4L2 sub-device V4L2 sub-device V4L2 sub-device V4L2 sub-device
- 15. Media Controller API to Define Relation Between Device/Sub-Devices • Allows to discover a platform configuration and configure it • Devices are described as building blocks • Connected through pads • Pads can be either sinks or sources • No limit for pads and sinks on each entity
- 16. Device/Sub-device and Media Entity Source & Sink Pads to Represent Data Flow
- 17. Media Device - HW Pipeline V4L2 Device/Sub-Devices Structured Using MediaController API Media Device Camera CSI-2 Host Colorspace converter Video Device /dev/video
- 18. Device Driver Interaction Only High-Level Interaction is Needed • The main driver will call the sub-device drivers to • power on, • start streaming, • change video mode, • and many more. • All this without needing to know anything about the HW it is talking to Video Device /dev/video mplayer Kernel Space ApplicaIon Space
- 19. Device Tree Describes the HW Configuration • The format is expressive and able to describe most board design aspects including • the number and type of CPUs • base addresses and size of RAM • busses and bridges • peripheral device connections • interrupt controllers and IRQ line connections • and others
- 20. Device Tree Hardware Details Listed • The HW specific details will be available in the Device Tree • HW addresses, IRQs and more are always available in the Device Tree • In the case of V4L2 cameras, information like number of lanes, frequency and others is also available
- 21. V4L2, Media Controller and Device Tree Putting it All Together 21
- 22. V4L2, Media Controller and Device Tree Putting it All Together • Device driver is designed using V4L2 API – API for video capture • It is modular and captures the HW elements – Sub- devices • Media controller API is used to define the relationship between the device and sub-devices, forming a HW pipeline that represents the actual HW platform • Device tree describes the HW platform and individual block specific details (address, interrupts, version, number of lanes) 22
- 23. V4L2, Media Controller and Device Tree Putting it All Together • When kernel starts, it will look at the device tree and configure it self • The sub-device drivers will be loaded if present in the device tree and will get the necessary details from it • The device driver HW pipeline will be formed • Application interacts with video device using high level commands • The video device is accessible under /dev • The application will open /dev/videoX and interact using the high level commands (such as change resolution, shutdown, power on, start capture, etc.) • Commands will be translated into low-level commands, specific to each individual sub-device (CSI-2 controller, camera, etc.) 23
- 24. V4L2, Media Controller and Device Tree Putting it All Together • If a change in the platform is required (replace camera or controller, etc.) • The new individual sub-devices need to be added to the kernel • The device tree needs to be updated and recompiled to reflect the HW changes • By re-starting the Kernel, the Device driver HW pipeline will be updated automatically 24
- 25. OUR EXPERIENCE
- 26. CSI-2 SW Development The Media Device HW Pipeline 26
- 27. CSI-2 SW Development The Device Tree Camera PHY Video DMA CSI Controller
- 28. CSI-2 SW Development Simple to Change CSI-2 Camera Camera
- 29. MIPI CSI-2 Host IP Prototyping Kit • Synopsys offers an integrated HW/SW solution that takes advantage of all these concepts Visit Synopsys booth to see demo!
- 30. Leverage Kernel Tools to Accelerate CSI-2 SW Development • Application interacts with driver only using high level Commands • No need to use specific commands for each HW element • The same kernel can serve for several HW platforms by simple changing the Device Tree • No effort in recompiling the kernel • Modular code is easily split among team members • Each driver just needs to provide standard interaction commands • Easy to maintain and add support for new HW All this accelerates driver development by reducing the code complexity and turnaround time! 30