Linaro and Android Kernel
5 likes2,671 views
The document discusses issues around merging Android kernel patches into the mainline Linux kernel, including initial rejection of the patches by kernel developers, limited later acceptance, and Linaro's efforts to upstream individual fixes and improvements. It also covers Linus Torvalds' frustrations with the increasing volume and apparent duplication of work in the ARM architecture area of the kernel.
Linaro and Android Kernel
- 1. Linaro and Android Kernel Thanks to Matt Hsu and John Lee at 0xlabs, AzureWave and the other organizers for being so kind as to invite me
- 2. Linaro and Android Kernel 1. What is Linaro 2. Android kernel patches and Linaro 3. Angry Birds Andy Green, TI Landing Team Lead, Linaro With thanks to John Stultz at Linaro
- 3. Linaro and Android 1. What is Linaro 2. Android kernel patches and Linaro 3. ARM Wrestling with Angry Birds Andy Green, TI Landing Team Lead, Linaro With thanks to John Stultz at Linaro
- 4. Andy Green Hardware, software and SoC design for 25 years; FPGA early adopter in 1989; wrote HTML5 libwebsockets Worked as kernel maintainer at Openmoko with many people now at 0xlabs; architected txtr e-book reader Texas Instruments Landing Team Leader at Linaro Moving to Taiwan in August 2011 with my family My wife is progressing learning 中國語言 I am stuck at 好, know a little にほんご though
- 5. Part 1: What is Linaro?
- 6. Part 1: What is Linaro?
- 7. Structure Paid for by its members “Not for profit” - money is spent on engineers and engineering “Core” members get a director on the board and representatives on the “Technical Steering Committee” Three engineering sections: Working groups Platform engineering Landing Teams
- 8. Working Groups Specialist teams focused on doing difficult tasks inside their domain Kernel – eg, pushing Device Tree, Thumb2 kernel Toolchain – eg, advanced gcc work Power Management – eg, idle state latency improvement, dynamic CPU multicore hotplug Graphics / Multimedia – eg, OpenGLES 2 compiz, prepare upcoming ARM GPUs kernel frameworks http://status.linaro.org
- 9. Landing Teams Work closely with vendors Three person team from Linaro works with vendor assignees Cooperate to get stuff upstream and in Nicolas Pitre's linux-linaro tree I lead the TI Landing Team, work with TI folks in Nice, France, and Bangalore, India OMAP4 Panda and OMAP5 stuff shortly
- 10. Manpower Started with engineers from Canonical last year Contract Engineers Employees Member Assignees Over 100 engineers now in total Some big names like Nicolas Pitre
- 11. Linaro Goals Agree common problems with the vendors and try to solve them directly, or be the basis of wider solutions Some problems can't be solved by one vendor Some problems might otherwise not be solved by volunteers Some problems require a funded, stable, trusted organization to lead the way to the solution
- 12. Linaro Goals Agree common problems with the vendors and try to solve them directly, or be the basis of wider solutions Some problems can't be solved by one vendor Some problems might otherwise not be solved by volunteers Some problems require a funded, stable, trusted organization to lead the way to the solution *
- 13. Part 2: Android Kernel patches & Linaro
- 14. Android becoming ubiquitous All the SoC vendors are quite clear Android is very important for them Many of the vendors' customers are quite clear Android is very important to them Excepting a certain Finnish phone company recently for some reason Linaro already engages with Android and has Android experts on board Android is really “a Linux distro”
- 15. The whole Android “distro” Linux Kernel Android Kernel Patchset Special minimal libc – Bionic Other libraries The Dalvik VM Application Frameworks Apps themselves
- 16. Just going to cover Android kernel Linux Kernel Android Kernel Patchset Special minimal libc – Bionic Other libraries The Dalvik VM Application Frameworks Apps themselves
- 17. Android Kernel Patches Overview All features that mainline does not offer, or improvements
- 18. Android Kernel Patches scope 1 / 6 Ashmem Android Shared Memory, allows naming allocated memory and having it visible between multiple processes /dev/ashmem device node tracks usage count on the named memory blocks Under memory pressure, kernel can free blocks with no current users Binder RPC scheme like CORBA
- 19. Android Kernel Patches scope 2 / 6 Pmem Allows userspace to map a given range of contiguous physical address Used for userland talking to DSPs and similar that cannot tolerate scatter-gather Logger Android's kernel logging, supports four logging buffers “main”, “events”, “radio”, and “system” Userland can read from the files down /dev/log/...
- 20. Android Kernel Patches scope 3 / 6 Early Suspend Allows userland to coordinate a staged suspend action of four levels From most awake to suspended, it first allows screen to be blanked, then stop access to the framebuffer, then turn off the framebuffer / crtc, and lastly stop all non-wake input devices Wakelocks Allow userland to veto either low power state (eg, stop CPU Clock) and / or entry to suspend; a few kernel places need to use this too – ie, covering pending timers in input device driver
- 21. Android Kernel Patches scope 4 / 6 Alarm Timer Deal with tracking elapsed time while in suspend Schedule wakeups from RTC alarm while CPU completely OFF; kernel already allows some of this Low Memory Killer More proactive and cooperative version of existing OOM Killer Paranoid Network Restricts certain network operations to processes with magic group IDs
- 22. Android Kernel Patches scope 5 / 6 RAM Console Debugging helper similar to EARLY_PRINTK but buffering very early messages to RAM, good for silently dying boot Apanic Panic handler tries to write the panic dump to flash where it can be recovered the next boot ADB Gadget Driver USB Gadget driver enables debugger operations from host PC – also able to listen on a network interface
- 23. Android Kernel Patches scope 6 / 6 Timed GPIO Adds a period capability to generic GPIO, allowing primitives like “set this high for 50ms” “Other” Small hacks and fixes for ARM, MMC, Bluetooth etc
- 24. Android Kernel Patches ~250 patches for 2.6.38, 3.3MBytes of patches Relatively thin layer of improvements over mainline, adding features that are not otherwise supported “Hey this OS is working great, let's send those patches upstream!” Submitted Jan 2009
- 25. Lkml vs Android Patches threads emails 1000 Use DBUS! Use OOM Killer! 500 Binder LowMem Killer Alarm Ashmem 0
- 26. Lkml vs Android Patches threads emails 2000 1000 wakelock Binder LowMem Killer Alarm Ashmem 0
- 27. Lkml vs Google 2009 experience Proposed patchsets: 13 Accepted: 0
- 28. Not Invented Here Political, cultural and practical problems dealing with a large after-the-fact patchbomb Google just wanted their patches accepted since they were shipping insane volumes already; stopped caring and got on with their job out of tree As a compromise a copy of the Google patches (not proposed by Google itself) allowed in staging Some folks tried to work with kernel process but the patches were removed from staging in 2.6.33
- 29. Lkml vs Google 2010 experience
- 30. Limited re-engagement with lkml After two years of rejection and stalled progress there are real commits from @android.com in mainline Over 100, > 50 from 2011 But these are not the 13 critical patchsets Out-of-tree model works well enough Wakelock impact on drivers makes practical difficulty working on drivers at both mainline and Android though Everyone would be happier if they were all in mainline
- 31. Now Invented Here runtime_pm apis added to kernel to do similar job to wakelock by Rafael Wysocki; design at least influenced by wakelock discussion GregKH, who removed android stuff from staging, writes on his blog 23rd March that android drivers using this native implementation are now welcome http://www.kroah.com/log/linux/android-wakelock-solution.html Competes with existing Google implementation, political as well as technical decision for Google; how to transition? Unclear if runtime_pm is full wakelock replacement
- 32. Summary Google felt their contribution should have been taken in mainline based on awesome Android volume success and high quality results there. lkml felt targeted by a fait accompli that did not consider wider kernel architecture issues outside of phone case If Google had included lkml in the loop to start with they probably get easier ride, but lkml does not follow their schedule
- 33. Linaro and the patchset Controversy about some elements of the patches overshadowed good fixes and improvements “Trivial Tree” established; fixes split out in topic branches and offered upstream by John Stultz from Linaro Difficulties defending the patches as third-party; uplevelling all the topic branches while out-of-tree is significant effort John also working on mainline Posix Alarm Timers that may cover Android Alarm Timer functionality
- 34. Linaro Android Platform Upstream-focused ARM Android Community getting started Linaro toolchain packages for Android Cloud-based build services ARM board farm for automated testing, validation and benchmarking Integration point for member SoC platform code Based on linux-linaro tree from Nicolas Pitre
- 35. Linaro Android Platform Installable images and public code for low-cost Linaro member SoC boards Combines latest public AOSP platform code with latest Linaro kernel and toolchain, plus member SoC platform code Monthly releases #linaro-android on Freenode IRC *
- 36. Part 3: Angry Birds
- 37. Linus, Sep 1998 To Theodore Ts'o's complaints about lost patches... “...Quite frankly, this particular discussion (and others before it) has just made me irritable, and is ADDING pressure. Instead, I'd suggest that if you have a complaint about how I handle patches, you think about what I end up having to deal with for five minutes. Go away, people. Or at least don't Cc me any more. I'm not interested, I'm taking a vacation, and I don't want to hear about it any more. In short, get the hell out of my mailbox.'' http://lkml.indiana.edu/hypermail/linux/kernel/9809.3/0850.html
- 38. Analysis Symptom of overload due to success This was before bitkeeper and later git, the amount of manual work was too high - “think about what I end up having to deal with” With awesome tools like git, Linus was able to do much more with the same effort and “scaled well” Able to keep up with the ecosystem and Linux has grown for another 12 years since then
- 39. Linus, June 2010 I got a bit frustrated with ten different ARM pulls per day at one point. There's something wrong with ARM development. The amount of pure noise in the patches is incredibly annoying. Right now, ARM is already (despite me not reacting to some of the flood) 55% of all arch/ changes since 2.6.34, and it's all pointless churn in arch/arm/configs/ arch/arm/mach-xyz arch/arm/plat-blah and at a certain point in the merge window I simply could not find it in me to care about it any more. http://lwn.net/Articles/392135/
- 40. Linus sees ARM arch overloading him ARM architecture is growing strongly Bit chaotic because outside the core ARM CPUs, it is very diverse indeed Random peripheral unit macrocells get implemented alongside custom DSP, custom interconnect, custom IPC hardware Same peripheral unit may be wired on different buses on different SoC, need “glue layer” like MUSB
- 41. Platform layer RMK brings order to the chaos with “platform layer” A platform is, eg, OMAP2+ for later TI chips, imx for Freescale, etc, combining commonality as much as possible and offering “just works” core code for chips There are more specific mach- files for boards that use some or all the platform features plus extra support for other chips on the board “pointless churn... arch/arm/plat... arch/arm/mach...” per- board config in “arch/arm/configs”
- 42. How the ARM SoC world works ARM SoC vendors license CPU Core from ARM Cheaper, quicker, more certain, better tool / OS ecosystem than proprietary... and patents So how to differentiate? Pile on specialized integrated peripheral units Talk up special process sauce, eg, smartreflex biasing Sell companion chips with special sauce (eg, OMAP4 vs twl6030 PMIC and twl6040 audio) Twl6030 and CPU talk on dedicated serial link for voltage scaling
- 43. Interversion changed files peak 1700 Files changed in arch/arm 1800 1600 1400 1200 1000 files 800 600 400 200 0 V2.6.14 V2.6.18 V2.6.22 V2.6.26 V2.6.30 V2.6.34 V2.6.12 V2.6.16 V2.6.20 V2.6.24 V2.6.28 V2.6.32 V2.6.36 Source: git diff using linus tree
- 44. Interversion diffstat peak 250kloc 300000 250000 200000 150000 Lines - Lines + 100000 50000 0 V2.6.14 V2.6.18 V2.6.22 V2.6.26 V2.6.30 V2.6.34 V2.6.12 V2.6.16 V2.6.20 V2.6.24 V2.6.28 V2.6.32 V2.6.36 Source: git diff using linus tree
- 45. Why Linus is a bit stressed
- 46. 500 pull / version endurance limit Linus pulls 2.6.26 thru 2.6.35 2500 2000 1500 Patches total Pulls / Patches Patches direct Pulls Total 1000 Pulls in MW 500 0 “2.6.27” “2.6.29” “2.6.31” “2.6.33” “2.6.35” “2.6.26” “2.6.28” “2.6.30” “2.6.32” “2.6.34” version Data Source: http://lwn.net/Articles/393694/
- 47. Interversion commit count stable at ~11K 14000 12000 10000 8000 Commits 6000 4000 2000 0 v2.6.26 v2.6.27 v2.6.28 v2.6.29 v2.6.30 v2.6.31 v2.6.32 v2.6.33 v2.6.34 v2.6.35 v2.6.36 v2.6.37 Source: eg, git log v2.6.26..v2.6.27 --oneline | wc -l
- 48. Linus, March 2011 Gaah. Guys, this whole ARM thing is a f*cking pain in the ass. You need to stop stepping on each others toes. There is no way that your changes to those crazy clock-data files should constantly result in those annoying conflicts, just because different people in different ARM trees do some masturbatory renaming of some random device. Seriously. That usb_musb_init() thing in arch/arm/mach-omap2/usb-musb.c also seems to be totally insane. I wonder what kind of insanity I'm missing just because I don't happen to see the merge conflicts, just because people were lucky enough to happen to not touch the same file within a few lines. Somebody needs to get a grip in the ARM community. I do want to do these merges, just to see how screwed up things are, but guys, this is just ridiculous. The pure amount of crazy churn is annoying in itself, but when I then get these "independent" pull requests from four different people, and they touch the same files, that indicates that something is wrong.
- 49. Linus, March 2011 (continued) And stop the crazy renaming already! Just leave it off. Don't rename boards and drivers "just because", at least not when there clearly are clashes. There's no point. I'm not even talking about the file renames (which happened and can also make it "fun" to try to resolve the conflicts when somebody else then makes _other_ changes), but about the stupid "change human-readable names in board files just to annoy whoever needs to merge the crap". Somebody in the ARM community really needs to step up and tell people to stop dicking around. (I'm replying to the omap pull request, because that's the one I did last, but I don't know who to "blame". I don't care. It really doesn't matter. I realize thar ARM vendors do crazy shit and haven't figured out this whole "platform" thing yet, but you guys need to push back on the people sending you crap). https://lkml.org/lkml/2011/3/17/492
- 50. Analysis Overload handling merge conflicts between ARM arch trees Renames and missed common code opportunities are “crap” Asking for more management in ARM world Subsequently made it clear he doesn't think Russell King (ARM Linux maintainer) is the problem Big crisis for ARM world, boss guy is raging but individual maintainers unable to react alone LOT of people dependent on flow to mainline...
- 51. Difficulty with MUSB – 1 / 5 Linus complained about MUSB, what is it, what's the problem? MUSB is Mentor's USB OTG hardware IP MUSB is a good case study of general issue Mentor sell hardware design tools including macrocells Using someone else's tested CPU core has advantages: same goes for someone else's tested USB OTG implementation
- 52. MUSB peripheral unit generic view – 2/5
- 53. Devil in the details – 3 / 5
- 54. Led to Generic driver + SoC glue – 4 / 5 MUSB Macrocell driver down drivers/usb/musb/ Also set of “glue layer” files for SoCs to take care of the messy and different implementation details, eg, blackfin.c, omap4230.c etc For example, if power interrupts are coming from a different chip, must be set up and enabled by that driver, not MUSB driver, despite MUSB controls the interrupt Seems to be “not liked” but hardware realities can mandate this kind of setup
- 55. Limits of discoverability – 5 / 5 AMBA bus used to hook up peripheral units on ARM SoC is more lightweight than PCI or USB Probing memory with no unit leads to death by bus exception, so we have to know what we have already Connectivity width is NOT a property the peripheral unit can declare, it's defined outside the unit itself OMAP has hwmod “registry”, DT wants to claim that turf Maybe Linus will end up improving or at least impacting future SoC composition visibility
- 56. Unable to react: business as usual linux-next churn post-Linus rant From dirstat 80 Posted by 70 RMK Apr 14 60 50 percentage 40 Percentage 30 20 10 0 arch/arm arch/m68k arch/mips arch/x86 ? arch http://lists.infradead.org/pipermail/linux-arm-kernel/ 2011-April/048133.html
- 57. Xilinx blocked from mainline Russell King - ARM Linux linux at arm.linux.org.uk Mon Apr 18 13:18:57 EDT 2011 I'm very sorry for people with new platforms outstanding like John Linn who are on the sharp end of this (who have platform code ready to go) but I see no solution at the moment. It really is the case that either I can say no to it, or I can put it in my tree and Linus will say no to it. So putting it in my tree *doesn't* help. Will we ever be able to put John's code in the kernel? Honestly, I have no idea. http://lists.infradead.org/pipermail/linux-arm-kernel/2011-April/048553.html
- 58. You are going on a diet Linus Torvalds torvalds at linux-foundation.org Mon Apr 18 12:23:51 EDT 2011 Hint for anybody on the arm list: look at the dirstat that rmk posted, and if your "arch/arm/{mach,plat}-xyzzy" shows up a lot, it's quite possible that I won't be pulling your tree unless the reason it shows up a lot is because it has a lot of code removed. People need to realize that the endless amounts of new pointless platform code is a problem, and since my only recourse is to say "if you don't seem to try to make an effort to fix it, I won't pull from you", that is what I'll eventually be doing. Exactly when I reach that point, I don't know. http://lists.infradead.org/pipermail/linux-arm-kernel/2011-April/048543.html
- 59. Linaro working to form a solution Linaro has “good offices” and good people that can help with a solution for all ARM SoC vendors Eg, Thomas Gleixner, Arnd Bergmann, Nicolas Pitre Well placed to help form a solution Linaro in contact with all the main participants these last weeks *
- 60. Thanks for listening http://status.linaro.org