[BUG] Cannot resolve symbols from binary linked with LLD

[mikaheli]

Description

Cannot resolve symbols (crash location) from crash dump from binary that is linked with -fuse=lld. I'm using ndk-stack to resolve symbols from crash dump. When using current default linker from ndk, things work fine.

Here is small example to reproduce:

Structure:

├── application
│   ├── application.cpp
│   └── CMakeLists.txt
└── build_and_run.sh

application/application.cpp

#include <cstdlib>

int main(void)
{
	abort();
}

application/CMakeLists.txt

cmake_minimum_required(VERSION 3.10)

project(application)
add_executable(application application.cpp)

project(application-lld)
add_executable(application-lld application.cpp)
target_link_libraries(application-lld
	PUBLIC
		-fuse-ld=lld
)

build_and_run.sh

#!/usr/bin/env bash

set -e

HERE=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

ANDROID_NDK=$HOME/Android/Sdk/ndk/21.0.6113669
ANDROID_ABI=arm64-v8a
ANDROID_NATIVE_API_LEVEL=29

function compile()
{
	printf '\nConfigure & Compile test applications ...\n\n'

	mkdir "$HERE/build"

	cd "$HERE/build"

	cmake \
		-G Ninja \
		-DCMAKE_BUILD_TYPE:STRING=Debug \
		-DBUILD_SHARED_LIBS:BOOL=ON \
		-DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
		-DANDROID_ABI=$ANDROID_ABI \
		-DANDROID_NATIVE_API_LEVEL=$ANDROID_NATIVE_API_LEVEL \
		-DANDROID_TOOLCHAIN=clang \
		-DANDROID_STL=c++_static \
		-DCMAKE_RUNTIME_OUTPUT_DIRECTORY=$HERE/bin \
		"$HERE/application"

	cmake --build . -- -j1 -v

	printf '\nConfigure & Compile test applications DONE\n\n'
}

function run()
{
	printf '\nRun with & without LLD\n'

	adb logcat -c

	adb push $HERE/bin/application /data/local/tmp/
	adb push $HERE/bin/application-lld /data/local/tmp/

	set +e

	adb shell /data/local/tmp/application
	adb shell /data/local/tmp/application-lld

	set -e
}

function crash_dump()
{
	printf '\nCrash dumps:\n\n'
	adb logcat -d | ndk-stack -sym $HERE/bin
}

compile
run
crash_dump

Output crash dumps:

Default linker

********** Crash dump: **********
Build fingerprint: 'OPPO/PCCM00/OP46C3:9/PKQ1.190223.001/1553783366:user/release-keys'
#00 0x00000000000223bc /system/lib64/libc.so (abort+116)
#01 0x0000000000000704 /data/local/tmp/application
main
/home/mikaheli/Desktop/LLDTEST/application/application.cpp:5:2
#02 0x00000000000ca024 /system/lib64/libc.so (__libc_init+88)
Crash dump is completed

LLD linker

********** Crash dump: **********
Build fingerprint: 'OPPO/PCCM00/OP46C3:9/PKQ1.190223.001/1553783366:user/release-keys'
#00 0x00000000000223bc /system/lib64/libc.so (abort+116)
#01 0x0000000000000094 /data/local/tmp/application-lld (offset 0x1000)
                                                        __libc_init
                                                        ??:0:0
Crash dump is completed

Environment Details

• NDK Version: 21.0.6113669
• Build system: CMake
• Host OS: MX Linux 18.3 / Windows 10
• ABI: arm64-v8a / armeabi-v7a
• NDK API level: 24/29 tested
• Device API level:

[DanAlbert]

@pirama-arumuga-nainar any ideas? I haven't had a chance to test with the latest LLD yet.

[HaiPhan2002]

Hello,

I have the same issue on armeabi-v7a. However, I don't get this issue on arm64-v8a.

Environment Details

NDK Version: 21.0.6113669
Build system: CMake
Host OS:  Windows 10 64bit
ABI: armeabi-v7a
NDK API level: 19/28 tested
Device API level: 21

I made the sample in case you need to recheck it. Link is here

[DanAlbert]

Thanks. I'll check that against a more up to date toolchain tomorrow. You could try the canary if you want, but I think the master branch is fairly behind what we'll actually be shipping in r22.

[DanAlbert]

Although you could try downloading the LLD from https://android.googlesource.com/platform/prebuilts/clang/host/windows-x86/+/refs/heads/master/clang-r383902/ and replacing the LLD that's in your NDK with that. I think r22 will have something slightly newer, but that's what I'll test tomorrow since I don't have anything newer yet :)

[DanAlbert]

It looks like the toolchain we have is going to need further updates, so waiting for the new new one before trying the above.

[DanAlbert]

Ended up cherry-picking to the toolchain rather than a full update, and this is still broken with that version:

********** Crash dump: **********
Build fingerprint: 'google/marlin/marlin:7.1.2/NJH47B/4021576:user/release-keys'
#00 0x000000000006b2ec /system/lib64/libc.so (tgkill+8)
#01 0x0000000000068788 /system/lib64/libc.so (pthread_kill+64)
#02 0x0000000000023ce8 /system/lib64/libc.so (raise+24)
#03 0x000000000001c76c /system/lib64/libc.so (abort+52)
#04 0x00000000000006bc /data/local/tmp/application
main
/usr/local/google/home/danalbert/src/ndk/ndk/foo/application/application.cpp:5:2
#05 0x000000000001a594 /system/lib64/libc.so (__libc_init+88)
#06 0x0000000000000674 /data/local/tmp/application
_start_main
??:0:0
Crash dump is completed

********** Crash dump: **********
Build fingerprint: 'google/marlin/marlin:7.1.2/NJH47B/4021576:user/release-keys'
#00 0x000000000006b2ec /system/lib64/libc.so (tgkill+8)
#01 0x0000000000068788 /system/lib64/libc.so (pthread_kill+64)
#02 0x0000000000023ce8 /system/lib64/libc.so (raise+24)
#03 0x000000000001c76c /system/lib64/libc.so (abort+52)
#04 0x0000000000000660 /data/local/tmp/application-lld
??
??:0:0
Crash dump is completed

Going to leave this triaged here for the moment. Could be that we need another toolchain update, could be that this is broken upstream, could be a bug in ndk-stack. We need to know more before we can actually ship lld as the default, I think.

[DanAlbert]

https://github.com/DanAlbert/lld-stack-dump-bug is the repro case from the original post, updated slightly to accommodate the changes I'm making to master today and to set the linker from the command line rather than CMakeLists.txt in case the issue was the toolchain file not knowing (it wasn't, but had to check).

[DanAlbert]

Seems this is http://b/109657296 (thanks a bunch to @pirama-arumuga-nainar for all the help debugging!). If I add -Wl,--no-rosegment to the link flags it symoblizes fine. The problem is actually the unwinder older devices not understanding the output from LLD.

The fix for the driver that was needed for this never got sent, so we'll patch the build systems and document the problem for now and get the fix in for r23.

[DanAlbert]

https://android-review.googlesource.com/c/platform/ndk/+/1350672 is the fix for r22. Still need to write the driver patch.

@mikaheli @Seal2002 you should be able to fix this locally for old NDK versions as well. Just make sure -Wl,--no-rosegment gets added to your linker flags.

[DanAlbert]

Okay, we can't actually lift this into the driver until the old linkers are gone. The driver doesn't know what linker its using and neither gold nor bfd accept that argument. LLD doesn't know what version of Android it's targeting, so we can't make the fix there either.

This is just going to be a build system fix until we've removed the old linkers. Filed #1294 to track that.

@jomof fyi you'll want to make this fix in VS as well. See the updated doc in the patch in the previous comment for more info.