LCU14 209- LLVM Linux
3 likes3,982 views
The LLVMLinux project aims to fully build the Linux kernel across multiple architectures using the Clang/LLVM toolchain, addressing known issues and upstreaming patches. The status update reveals that many LLVM patches are already upstream, while various kernel patches are still pending acceptance. Community contributions are encouraged through testing, reporting bugs, and working on unsupported features.
![Missing “%a” for inline ASM
● The following error is generated with clang for AArch64:
error: invalid operand in inline asm: 'prfm pldl1keep, ${0:a}'
● Per comments by Tim Northover on the LLVM Bug database:
It's rather unclear how it's better than "prfm pstl1keep, [%0]" though. Not
all instructions can make use of any offset, so wouldn't we have to be
conservative and always map it to "[xN]"?
● When %a0 is changed to [%x0] it uncovered a GCC bug: https://bugs.
linaro.org/show_bug.cgi?id=635
● Changing the "p" to "r" resolves the issue for both clang and GCC.
- asm volatile("prfm pldl1keep, %a0n" : : "p" (ptr));
+ asm volatile("prfm pldl1keep, [%x0]n" : : "r" (ptr));](https://image.slidesharecdn.com/lcu14-209-llvmlinux-140916155120-phpapp01/85/LCU14-209-LLVM-Linux-16-320.jpg)
![Variable Length Arrays In Structs
● VLAIS isn't supported by Clang (undocumented gcc extension)
char vla[n]; /* Supported, C99/C11 */
struct {
char flexible_member[]; /* Supported, C99/C11 */
} struct_with_flexible_member;
struct {
char vlais[n]; /* Explicitly not allowed by C99/C11 */
} variable_length_array_in_struct;
● VLAIS is used in the Linux kernel in a number of places, spreading
mostly through reusing patterns from data structures found in
crypto](https://image.slidesharecdn.com/lcu14-209-llvmlinux-140916155120-phpapp01/85/LCU14-209-LLVM-Linux-19-320.jpg)
![VLAIS Removal Example (from crypto/hmac.c)
- struct {
- struct shash_desc shash;
- char ctx[crypto_shash_descsize(hash)];
- } desc;
+ char desc[sizeof(struct shash_desc)
+
+ crypto_shash_descsize(hash)] CRYPTO_MINALIGN_ATTR;
+ struct shash_desc *shash = (struct shash_desc *)desc;
unsigned int i;
- desc.shash.tfm = hash;
+ shash->tfm = hash;](https://image.slidesharecdn.com/lcu14-209-llvmlinux-140916155120-phpapp01/85/LCU14-209-LLVM-Linux-20-320.jpg)
![VLAIS Removal Example (the missing pieces)
#define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
#define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))
struct shash_desc {
struct crypto_shash *tfm;
u32 flags;
void *__ctx[] CRYPTO_MINALIGN_ATTR;
};](https://image.slidesharecdn.com/lcu14-209-llvmlinux-140916155120-phpapp01/85/LCU14-209-LLVM-Linux-21-320.jpg)
![Huge use of VLAIS in fs/exofs/ore_raid.c
static int _sp2d_alloc(unsigned pages_in_unit, unsigned group_width,
unsigned parity, struct __stripe_pages_2d **psp2d)
[...]
unsigned data_devs = group_width - parity;
struct _alloc_all_bytes {
struct __alloc_stripe_pages_2d {
struct __stripe_pages_2d sp2d;
struct __1_page_stripe _1p_stripes[pages_in_unit];
} __asp2d;
struct __alloc_1p_arrays {
struct page *pages[group_width];
struct page *scribble[group_width];
char page_is_read[data_devs];
} __a1pa[pages_in_unit];
} *_aab;](https://image.slidesharecdn.com/lcu14-209-llvmlinux-140916155120-phpapp01/85/LCU14-209-LLVM-Linux-23-320.jpg)
LCU14 209- LLVM Linux
- 1. LCU14-209: LLVMLinux Behan Webster, LCU14 LCU14 BURLINGAME
- 2. Linaro Overview ● About the LLVMLinux project ● State of Linux kernel being compiled with clang ● Details of specific known problems/patches still being upstreamed ● How can you help? ● Contact info
- 3. The LLVMLinux Project Goals ● Fully build the Linux kernel for multiple architectures, using the Clang/LLVM toolchain ● Discover LLVM/Kernel issues early and find fixes quickly across both communities ● Upstream patches to the Linux Kernel and LLVM projects ● Bring together like-minded developers ● Enable the kernel community to do more in depth analysis of the kernel code
- 4. LLVMLinux Build/Test System ● Fetches, patches, builds, tests: clang, kernel, qemu, etc ○ git clone http://git.linuxfoundation.org/llvmlinux.git ○ cd llvmlinux/target/vexpress (or vexpress64) ○ make
- 5. Patched Mainline Kernel Tree ● A mainline kernel tree with all LLVMLinux patches applied on top is now available: ○ git://git.linuxfoundation.org/llvmlinux/kernel.git ● Dated llvmlinux branches ○ remotes/origin/llvmlinux-2014.09.16 ● The master branch is rebased regularly
- 6. LLVMLinux Project Status ● LLVM/clang: ○ All LLVMLinux patches for LLVM are Upstream ○ Newer LLVM patches to support the Linux kernel are mostly being added by upstream maintainers ● Linux Kernel: ○ Roughly 49 kernel patches for various arches ○ LLVMLinux branch in linux-next
- 7. Remaining LLVMLinux Kernel Patches ● Patches still working their way upstream Architecture Number of Patches Patches Submitted Patches Accepted all 23 17 2 arm 12 11 1 (+7) aarch64 11 8 7 x86_64 3 1 1 TOTAL 49 37 11
- 8. LLVMLinux Kernel Patches ● Total patches currently required for a single architecture Architecture Number of Patches Patches Submitted Patches Accepted arm (+all) 35 (12+23) 28 (11+17) 3 (1+2) aarch64 (+all) 34 (11+23) 25 (8+17) 9 (7+2) x86_64 (+all) 26 (3+23) 18 (1+17) 3 (1+2)
- 9. Kbuild ● Basic Kbuild support for clang and x86 has been upstreamed ● Specific support for ARM and aarch64 ready to be upstreamed
- 10. Integrated Assembly Status ● Renato Golin, Vicinius Tinti, Saleem Abdulrasool and Stepan Dyatkovskiy are working on fixing IA issues in clang to support the Linux ARM kernel code (and ultimately AARCH64) ● David Woodhouse has added .code16 support for X86 ASM ● For now we disable the IA and use GNU as instead
- 11. Different option passing ● gcc passes -march to GNU as ● clang doesn't... (Bug submitted PR) ● Probably should be fixed in clang ● Work around patch for now -CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto +CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto -Wa,-march=armv8-a+crypto
- 12. extern inline: Different for gnu89 and gnu99 ● GNU89/GNU90 (used by gcc) ○ Function will be inlined where it is used ○ No function definition is emitted ○ A non-inlined function may also be provided ● GNU99/C99 (used by clang) ○ Function will be inlined where it is used ○ An external function is emitted ○ No other function of the same name may be provided. ● Solution? Use “static inline” instead. ● Only still an issue for ARM support for ftrace (submitted)
- 13. Attribute Order ● gcc is less picky about placement of __attribute__(()) ● clang requires it at the end of the type or variable -struct __read_mostly va_alignment va_align = { +struct va_alignment __read_mostly va_align = { ● (This particular patch was just accepted)
- 14. Named Registers ● (Named registers for X86 kernel have been removed from the mainline kernel by Andi Kleen) ● ARM and AARCH64 still like using named registers ● Clang now supports using a globally named register for the stack pointer (Thanks Renato!) ● For ARM/AARCH64 move to using a global in asm/thread_info.h register unsigned long current_stack_pointer asm ("sp"); ● Patches for AARCH64 now accepted, acked for ARM
- 15. ARM percpu patch ● One of the uses of Named Registers in the ARM code is due to a deficiency in gcc ● The new code which works with gcc fails in clang ● Solution, provide routines for both, and choose at compile time ● Gcc: asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) : "Q" (*sp)); ● Clang: asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) : : "memory");
- 16. Missing “%a” for inline ASM ● The following error is generated with clang for AArch64: error: invalid operand in inline asm: 'prfm pldl1keep, ${0:a}' ● Per comments by Tim Northover on the LLVM Bug database: It's rather unclear how it's better than "prfm pstl1keep, [%0]" though. Not all instructions can make use of any offset, so wouldn't we have to be conservative and always map it to "[xN]"? ● When %a0 is changed to [%x0] it uncovered a GCC bug: https://bugs. linaro.org/show_bug.cgi?id=635 ● Changing the "p" to "r" resolves the issue for both clang and GCC. - asm volatile("prfm pldl1keep, %a0n" : : "p" (ptr)); + asm volatile("prfm pldl1keep, [%x0]n" : : "r" (ptr));
- 17. Section Mismatch Issues (MergedGlobals) ● By default clang merges globals with internal linkage into one: MergedGlobals ● Allows globals to be addressed using offsets from a base pointer ● Can reduce the number of registers used ● Modpost script in the Linux kernel uses symbol names to look for section mismatches (e.g. regular code calling init code) ● MergedGlobals breaks modpost (false positive section mismatches) ● Current solution: use -mno-global-merge to stop global merging ● Updates to modpost may allow this optimization to be enabled again in the future
- 18. ARM eabi support ● Clang emits code which uses the “aeabi” ARM calls which are implemented in compiler-rt (equivalient to libgcc) ● Compiler-rt doesn't easily cross compile yet... void __aeabi_memcpy(void *dest, const void *src, size_t n) void __aeabi_memmove(void *dest, const void *src, size_t n) void __aeabi_memset(void *s, size_t n, int c) ● Still needed for ARM ● No longer required for AARCH64
- 19. Variable Length Arrays In Structs ● VLAIS isn't supported by Clang (undocumented gcc extension) char vla[n]; /* Supported, C99/C11 */ struct { char flexible_member[]; /* Supported, C99/C11 */ } struct_with_flexible_member; struct { char vlais[n]; /* Explicitly not allowed by C99/C11 */ } variable_length_array_in_struct; ● VLAIS is used in the Linux kernel in a number of places, spreading mostly through reusing patterns from data structures found in crypto
- 20. VLAIS Removal Example (from crypto/hmac.c) - struct { - struct shash_desc shash; - char ctx[crypto_shash_descsize(hash)]; - } desc; + char desc[sizeof(struct shash_desc) + + crypto_shash_descsize(hash)] CRYPTO_MINALIGN_ATTR; + struct shash_desc *shash = (struct shash_desc *)desc; unsigned int i; - desc.shash.tfm = hash; + shash->tfm = hash;
- 21. VLAIS Removal Example (the missing pieces) #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long) #define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN #define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN))) struct shash_desc { struct crypto_shash *tfm; u32 flags; void *__ctx[] CRYPTO_MINALIGN_ATTR; };
- 22. Status of VLAIS in the Linux Kernel ● USB Gadget patch is in mainline ● Mac80211 patch is in mainline ● Netfilter patch is in mainline ● apparmor patch accepted ● Bluetooth patch accepted ● Only the VLAIS patches for crypto are left: (btrfs, dm-crypt, hmac, libcrc32c, testmgr, etc) ● However, we recently found a few previously unknown instances of VLAIS in raid10 and exofs...
- 23. Huge use of VLAIS in fs/exofs/ore_raid.c static int _sp2d_alloc(unsigned pages_in_unit, unsigned group_width, unsigned parity, struct __stripe_pages_2d **psp2d) [...] unsigned data_devs = group_width - parity; struct _alloc_all_bytes { struct __alloc_stripe_pages_2d { struct __stripe_pages_2d sp2d; struct __1_page_stripe _1p_stripes[pages_in_unit]; } __asp2d; struct __alloc_1p_arrays { struct page *pages[group_width]; struct page *scribble[group_width]; char page_is_read[data_devs]; } __a1pa[pages_in_unit]; } *_aab;
- 24. How Can You Help? ● Don’t use the non-C99 practices we showed in previous slides ● Make it known you want to be able to use Clang to compile the kernel (tell your Linaro representative!) ● Test LLVMLinux patches ● Report bugs to the LLVMLinux mailing list ● Help get LLVMLinux patches upstream ● Work on unsupported features and Bugs ○ http://llvm.linuxfoundation.org/index.php/Broken_kernel_options ● Submit new targets and arch support ● Patches welcome
- 25. Embrace the Dragon. He's cuddly. Thank you http://llvm.linuxfoundation.org
- 26. Contribute to the LLVMLinux Project ● Project wiki page ○ http://llvm.linuxfoundation.org ● Project Mailing List ○ http://lists.linuxfoundation.org/mailman/listinfo/llvmlinux ○ http://lists.linuxfoundation.org/pipermail/llvmlinux/ ● IRC Channel ○ #llvmlinux on OFTC ○ http://buildbot.llvm.linuxfoundation.org/irclogs/OFTC/%23llvmlinux/ ● LLVMLinux Community on Google Plus
- 27. More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/members More about Linaro: www.linaro.org/about/